CN115330540A - Method and device for processing transaction data - Google Patents
Method and device for processing transaction data Download PDFInfo
- Publication number
- CN115330540A CN115330540A CN202211237667.4A CN202211237667A CN115330540A CN 115330540 A CN115330540 A CN 115330540A CN 202211237667 A CN202211237667 A CN 202211237667A CN 115330540 A CN115330540 A CN 115330540A
- Authority
- CN
- China
- Prior art keywords
- data
- real
- processing
- time
- target
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000012545 processing Methods 0.000 title claims abstract description 202
- 238000000034 method Methods 0.000 title claims abstract description 62
- 238000013507 mapping Methods 0.000 claims abstract description 131
- 230000005477 standard model Effects 0.000 claims abstract description 76
- 230000004044 response Effects 0.000 claims abstract description 9
- 238000013499 data model Methods 0.000 claims description 39
- 238000006243 chemical reaction Methods 0.000 claims description 24
- 238000004590 computer program Methods 0.000 claims description 11
- 238000004806 packaging method and process Methods 0.000 claims description 9
- 230000002159 abnormal effect Effects 0.000 claims description 6
- 238000004140 cleaning Methods 0.000 claims description 4
- 239000000126 substance Substances 0.000 claims description 2
- 238000005516 engineering process Methods 0.000 abstract description 5
- 230000008569 process Effects 0.000 description 15
- 238000010586 diagram Methods 0.000 description 9
- 238000004891 communication Methods 0.000 description 6
- 230000006870 function Effects 0.000 description 6
- 230000003287 optical effect Effects 0.000 description 4
- 230000008901 benefit Effects 0.000 description 3
- 238000007726 management method Methods 0.000 description 3
- 230000004048 modification Effects 0.000 description 3
- 238000012986 modification Methods 0.000 description 3
- 238000012216 screening Methods 0.000 description 3
- 238000011161 development Methods 0.000 description 2
- 239000000835 fiber Substances 0.000 description 2
- 230000000644 propagated effect Effects 0.000 description 2
- 239000004065 semiconductor Substances 0.000 description 2
- 230000009286 beneficial effect Effects 0.000 description 1
- 238000010276 construction Methods 0.000 description 1
- 238000013523 data management Methods 0.000 description 1
- 238000013461 design Methods 0.000 description 1
- 230000000694 effects Effects 0.000 description 1
- 238000000605 extraction Methods 0.000 description 1
- 230000006872 improvement Effects 0.000 description 1
- 239000004973 liquid crystal related substance Substances 0.000 description 1
- 239000013307 optical fiber Substances 0.000 description 1
- 238000003672 processing method Methods 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
- 230000009466 transformation Effects 0.000 description 1
- 238000012384 transportation and delivery Methods 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06Q—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
- G06Q40/00—Finance; Insurance; Tax strategies; Processing of corporate or income taxes
- G06Q40/04—Trading; Exchange, e.g. stocks, commodities, derivatives or currency exchange
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/28—Databases characterised by their database models, e.g. relational or object models
- G06F16/284—Relational databases
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/80—Information retrieval; Database structures therefor; File system structures therefor of semi-structured data, e.g. markup language structured data such as SGML, XML or HTML
- G06F16/84—Mapping; Conversion
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Databases & Information Systems (AREA)
- Physics & Mathematics (AREA)
- Business, Economics & Management (AREA)
- General Physics & Mathematics (AREA)
- Accounting & Taxation (AREA)
- General Engineering & Computer Science (AREA)
- Data Mining & Analysis (AREA)
- Finance (AREA)
- Development Economics (AREA)
- Economics (AREA)
- Marketing (AREA)
- Strategic Management (AREA)
- Technology Law (AREA)
- General Business, Economics & Management (AREA)
- Financial Or Insurance-Related Operations Such As Payment And Settlement (AREA)
Abstract
The invention discloses a method and a device for processing transaction data, and relates to the technical field of financial science and technology. One embodiment of the method comprises: in response to receiving real-time original data related to a transaction, determining a target data processing model corresponding to a service type to which the real-time original data belongs; the target data processing model comprises a mapping relation between an original data structure and a target data structure; processing real-time original data corresponding to the original data structure into a data standard model; and providing the data standard model to the data demand side of the service type. The universality, flexibility and expansibility of processing transaction data are improved through one or more data processing models and data standard models; the efficiency of processing transaction data is improved by real-time processing.
Description
Technical Field
The invention relates to the technical field of financial science and technology, in particular to a method and a device for processing transaction data.
Background
At present, the internet technology is widely applied to the financial field, and the transaction data of financial products can be processed by using the internet technology and the data processing technology.
Generally, transaction data has data complexity and data source diversity, currently, transaction data related to different service types need to be processed respectively aiming at different service types, a general processing means for processing the transaction data for different service types or different data demand parties is lacked, and the problems of poor flexibility, low universality and high complexity in processing the transaction data exist.
Disclosure of Invention
In view of this, embodiments of the present invention provide a method and an apparatus for processing transaction data, which can determine, in response to receiving real-time raw data related to a transaction, a target data processing model corresponding to a service type to which the real-time raw data belongs; the target data processing model comprises a mapping relation between an original data structure and a target data structure; processing real-time original data corresponding to the original data structure into a data standard model; and providing the data standard model to the data demander of the service type. The universality, flexibility and expansibility of processing transaction data are improved through one or more data processing models and data standard models; the efficiency of processing transaction data is improved through real-time processing, and the complexity of processing data is reduced.
To achieve the above object, according to an aspect of an embodiment of the present invention, there is provided a method of processing transaction data, including: in response to receiving one or more kinds of real-time original data related to transactions, determining a target data processing model corresponding to a service type to which the real-time original data belongs from a plurality of preset data processing models; the target data processing model comprises a mapping relation between an original data structure and a target data structure; processing real-time original data corresponding to the original data structure into a data standard model based on the mapping relation; and providing the data standard model to a data demand side of the service type.
Optionally, the method of processing transaction data further comprises:
acquiring transaction historical data corresponding to multiple service types; for the transaction history data of each service type, performing: determining a data dictionary corresponding to the transaction historical data; analyzing an original data structure contained in a data dictionary corresponding to the transaction historical data; and constructing a mapping relation between the original data structure and a target data structure constructed for the service type in advance, and packaging the mapping relation into a data processing model.
Optionally, the method of processing transaction data comprises: the mapping relation comprises a data field mapping and a data processing mapping which are constructed aiming at one or more original fields included by the original data structure; wherein the data processing mapping comprises any one or more of a data type mapping, a processing logic mapping, and a data conversion mapping.
Optionally, the processing, based on the mapping relationship, real-time original data corresponding to the original data structure into a data standard model includes:
for each piece of real-time raw data, performing: matching the real-time raw data with one or more first fields corresponding to the raw data structure; searching a data field mapping and a data processing mapping which are matched with one or more first fields from the mapping relation packaged by the target data processing model; converting the first field and real-time original data corresponding to the first field into a second field and target real-time data corresponding to the second field according to data field mapping and data processing mapping matched with one or more first fields; and processing each target real-time data into the data standard model.
Optionally, the method of processing transaction data further comprises: and detecting whether the received real-time original data is abnormal or not, if so, cleaning the real-time original data according to a set strategy, and executing a step of determining a target data processing model corresponding to the service type of the real-time original data based on the cleaned real-time original data.
Optionally, the converting the first field and its corresponding real-time raw data into a second field and target real-time data corresponding to the second field includes: and in the case that the data processing mapping of the first field comprises a data conversion mapping, converting the data value of the real-time original data into a specific type of data value corresponding to the second field based on a set conversion rule corresponding to the data conversion mapping.
Optionally, the processing each target real-time data into the data standard model includes: the method comprises the steps of obtaining one or more data structures of the service type, obtaining a plurality of field identifications of the target real-time data, adding data values of the target real-time data to the data structures based on the corresponding relation between the field identifications and the data structures, and generating the data standard model based on the data structures.
Optionally, the method for processing transaction data further includes: presetting one or more theme data models for the service types based on the data standard model, wherein one theme data model corresponds to one or more data structures; for each of the subject data models, performing the following operations: acquiring data corresponding to the theme data model from the target real-time data, and adding the data corresponding to the theme data model to a data structure corresponding to the theme data model; forming business data corresponding to a theme by using one or more data structures belonging to the same theme data model; the step of providing the data standard model to the data demander of the service type comprises the following steps: and providing the business data of one or more themes to the data demand side.
To achieve the above object, according to a second aspect of an embodiment of the present invention, there is provided an apparatus for processing transaction data, including: the device comprises a determining model module, a data processing module and a data sending module; wherein, the first and the second end of the pipe are connected with each other,
the determining model module is used for responding to one or more kinds of real-time original data related to the transaction and determining a target data processing model corresponding to the business type to which the real-time original data belongs from a plurality of preset data processing models; the target data processing model comprises a mapping relation between an original data structure and a target data structure;
the data processing module is used for processing the real-time original data corresponding to the original data structure into a data standard model based on the mapping relation;
and the data sending module is used for providing the data standard model to the data demander of the service type.
Optionally, the device for processing transaction data is further configured to obtain transaction history data corresponding to multiple service types; for the transaction history data of each service type, performing: determining a data dictionary corresponding to the transaction historical data; analyzing an original data structure contained in a data dictionary corresponding to the transaction historical data; and constructing a mapping relation between the original data structure and a target data structure constructed for the service type in advance, and packaging the mapping relation into a data processing model.
Optionally, the apparatus for processing transaction data comprises: the mapping relation comprises a data field mapping and a data processing mapping which are constructed aiming at one or more original fields included by the original data structure; wherein the data processing mapping comprises any one or more of a data type mapping, a processing logic mapping, and a data conversion mapping.
Optionally, the device for processing transaction data is configured to process real-time raw data corresponding to the raw data structure into a data standard model based on the mapping relationship, and includes:
for each piece of real-time raw data, performing: matching the real-time raw data with one or more first fields corresponding to the raw data structure; searching a data field mapping and a data processing mapping which are matched with one or more first fields from the mapping relation packaged by the target data processing model; converting the first field and real-time original data corresponding to the first field into a second field and target real-time data corresponding to the second field according to data field mapping and data processing mapping matched with one or more first fields; and processing each target real-time data into the data standard model.
Optionally, the device for processing transaction data is further configured to detect whether the received real-time raw data is abnormal, if so, clean the real-time raw data according to a set policy, and execute a step of determining, based on the cleaned real-time raw data, a target data processing model corresponding to a service type to which the real-time raw data belongs.
Optionally, the apparatus for processing transaction data, configured to convert the first field and its corresponding real-time raw data into a second field and target real-time data corresponding to the second field, includes: and in the case that the data processing mapping of the first field comprises a data conversion mapping, converting the data value of the real-time original data into a specific type of data value corresponding to the second field based on a set conversion rule corresponding to the data conversion mapping.
Optionally, the device for processing transaction data is configured to process each of the target real-time data into the data standard model, and includes: acquiring one or more data structures of the service type, acquiring a plurality of field identifications of the target real-time data, adding the data values of the target real-time data to the data structures based on the corresponding relation between the field identifications and the data structures, and generating the data standard model based on the data structures.
Optionally, the device for processing transaction data is further configured to preset one or more theme data models for the service type based on the data standard model, where one theme data model corresponds to one or more data structures; for each of the subject data models, performing the following operations: acquiring data corresponding to the theme data model from the target real-time data, and adding the data corresponding to the theme data model to a data structure corresponding to the theme data model; forming business data corresponding to a theme by using one or more data structures belonging to the same theme data model; the providing the data standard model to the data demander of the service type includes: and providing the business data of one or more themes to the data demand side.
To achieve the above object, according to a third aspect of the embodiments of the present invention, there is provided an electronic device for processing transaction data, including: one or more processors; a storage device for storing one or more programs which, when executed by the one or more processors, cause the one or more processors to implement a method as in any one of the methods of processing transaction data described above.
To achieve the above object, according to a fourth aspect of embodiments of the present invention, there is provided a computer readable medium having stored thereon a computer program, characterized in that the program, when executed by a processor, implements a method as in any one of the methods of processing transaction data as described above.
One embodiment of the above invention has the following advantages or benefits: in response to receiving real-time original data related to a transaction, determining a target data processing model corresponding to a service type to which the real-time original data belongs; the target data processing model comprises a mapping relation between an original data structure and a target data structure; processing real-time original data corresponding to the original data structure into a data standard model; and providing the data standard model to the data demand side of the service type. The universality, flexibility and expansibility of processing transaction data are improved through one or more data processing models and data standard models; the efficiency of processing transaction data is improved through real-time processing, and the complexity of processing data is reduced.
Further effects of the above-mentioned non-conventional alternatives will be described below in connection with the embodiments.
Drawings
The drawings are included to provide a better understanding of the invention and are not to be construed as unduly limiting the invention. Wherein:
FIG. 1 is a flow diagram illustrating a method for processing transaction data according to one embodiment of the invention;
FIG. 2 is a schematic flow chart of processing transaction data according to an embodiment of the present invention;
FIG. 3 is a schematic diagram of an apparatus for processing transaction data according to an embodiment of the present invention;
FIG. 4 is an exemplary system architecture diagram in which embodiments of the present invention may be employed;
fig. 5 is a schematic block diagram of a computer system suitable for use in implementing a terminal device or server of an embodiment of the invention.
Detailed Description
Exemplary embodiments of the present invention are described below with reference to the accompanying drawings, in which various details of embodiments of the invention are included to assist understanding, and which are to be considered as merely exemplary. Accordingly, those of ordinary skill in the art will recognize that various changes and modifications of the embodiments described herein can be made without departing from the scope and spirit of the invention. Also, descriptions of well-known functions and constructions are omitted in the following description for clarity and conciseness.
As shown in fig. 1, an embodiment of the present invention provides a method for processing transaction data, which may include the following steps:
step S101: in response to receiving one or more kinds of real-time original data related to transactions, determining a target data processing model corresponding to a service type to which the real-time original data belongs from a plurality of preset data processing models; the target data processing model includes a mapping relationship between an original data structure and a target data structure.
Specifically, the real-time raw data related to the transaction may be market data, and the like associated with the transaction; market data is data generated by a transaction, such as: real-time quote, real-time deal, exchange rate, etc., and the market data includes price data, curve data, benchmark information, information data, etc., such as: bond price, commodity price, reference curve, fluctuation rate and market index; it will be appreciated that the transaction data generated according to the type of transaction is different, for example: the transaction data associated in the bond transaction includes transaction amount, settlement date, nominal interest rate, net price, point difference, and the like. The transaction data associated in the futures transaction includes contract type, delivery date, expiration date, volume of transaction, price of transaction, etc. Also, transaction data may come from different sources, with diversity, such as: the quotation data in the market data is curve data or a database file or the like. The source of the real-time raw data may be from one or more of a data cache, a data cluster, a relational database, a data queue, and the like. In one embodiment of the invention, real-time raw data (e.g., real-time market data, etc.) is obtained through kafka middleware. It can be seen that there is complexity and diversity in the real-time raw data associated with a transaction.
In embodiments of the present invention, there are two methods of receiving one or more real-time raw data related to a transaction: the first method comprises the following steps: the method has the advantages that the log information of a plurality of external heterogeneous data sources is monitored, the original data is determined and real-time acquisition is carried out, and therefore the beneficial effect of non-invasive data acquisition is achieved; the second method comprises the following steps: receiving the push of real-time original data of an external data source according to a set strategy; further, after the raw data are collected in real time, the received raw data are preferably checked based on transaction rules to determine the validity and reasonability of the raw data, and the efficiency of subsequently processing the transaction data by using a data processing model is further improved. Computational resources are saved.
Further, processing one or more real-time raw data associated with the transaction using a data processing model; in one embodiment of the invention, a plurality of preset data processing models are set according to different service types; the types of business associated with the transaction data include, for example, a transaction class, a security class, a market class, an accounting class, a risk class, a product class, and the like.
The method for constructing the data processing model comprises the steps of determining the mapping relation between an original data structure (which can be obtained by acquiring a data dictionary of historical data) and a processed target data structure aiming at historical data associated with a certain service type, and packaging the mapping relation into the data processing model corresponding to the service; the data processing model is a logic code and is used for automatically processing real-time original data to form standard data so as to provide the standard data to a data demand side. Namely, transaction history data corresponding to a plurality of service types are obtained; for the transaction history data of each service type, performing: determining a data dictionary corresponding to the transaction historical data; analyzing an original data structure contained in a data dictionary corresponding to the transaction historical data; and constructing a mapping relation between the original data structure and a target data structure constructed for the service type in advance, and packaging the mapping relation into a data processing model. Similarly, corresponding preset data processing models are constructed for a plurality of service types.
Preferably, according to the application scenario, multiple mapping relationships can be constructed for the same service type; it can be understood that the same service type may contain original data of a plurality of original data structures, and thus, mapping relationships with standard target data structures are respectively constructed for the various original data structures; and packaging the constructed multiple mapping relations into a data processing model, thereby determining the matching mapping relations according to the identification of the original data structure.
Further preferably, when a new service type is required according to the application scenario, a new mapping relationship with the standard target data structure is constructed according to the original data structure corresponding to the new service type, and the data processing model corresponding to the new service type is encapsulated for the new service type.
Therefore, by flexibly changing (increasing, updating, deleting and the like) the mapping relation, the universality, the flexibility and the expansibility of processing data of different service types, different data types and different data structures are further improved, and the complexity of processing heterogeneous data is reduced to a greater extent for a data demand party.
Further, the mapping relation comprises a data field mapping and a data processing mapping which are constructed aiming at one or more original fields included in the original data structure; wherein the data processing mapping comprises any one or more of a data type mapping, a processing logic mapping, and a data conversion mapping. Wherein the data type mapping includes mapping of data types between data, for example: conversion mapping between integer and string; the processing logic map includes a variety of maps, such as: mapping between different data accuracies, mapping between data screening conditions, and the like; the data conversion map includes a mapping between data values, and the like.
Specifically, due to the diversity and complexity of real-time original data, field names, data formats, data values and the like of the data need to be processed, the processing method is to process the information set by the mapping relation, and the original data structure and the processed target data structure both contain a plurality of field names; it will be appreciated that the number of fields contained in the original data structure and the processed target data structure may be different.
For example: one data field of the target data structure and the original data structure is mapped as:
VALUE_DATE：trade_date
wherein VALUE _ DATE represents the DATE of interest field in the target data structure and track _ DATE represents the DATE of interest field in the original data structure.
Further, the mapping relationship also includes a data processing mapping; wherein the data processing mapping comprises any one or more of a data type mapping, a processing logic mapping and a data conversion mapping; for example, the schematic mapping relationship is as follows:
VALUE_DATE，DATE：trade_date，VARCHAR2(20)， trade_date>2022.6.6
wherein one exemplary mapping comprises a process logic mapping from VARCHAR2 (20) to DATE as a data type mapping, and track _ DATE >2022.6.6 as VALUE _ DATE; in some application scenarios, data values corresponding to data fields need to be converted and processed into numerical values with a uniform format or the same content, that is, data conversion mapping; for example, data "ABC" corresponding to field ABC is converted to "ABC _ ID" in a uniform format, etc.
It can be understood that the mapping relationship between the original data structure and the target data structure constructed for the service type in advance is constructed, and the mapping relationship includes a plurality of data field mappings according to the service type and the application scenario. The mapping relationships may be stored in a table file, a data table, a database, and the like. Further, packaging the mapping relation into a data processing model; therefore, the corresponding data processing model processes the real-time original data into the corresponding target original data based on the encapsulated mapping relation.
Therefore, a target data processing model corresponding to the service type to which the real-time original data belongs is determined from a plurality of preset data processing models; the target data processing model includes a mapping relationship between an original data structure and a target data structure. The universality, expansibility and flexibility of processing various types of data sources are improved.
Step S102: and processing the real-time original data corresponding to the original data structure into a data standard model based on the mapping relation.
In an embodiment of the present invention, the data standard model may be target real-time data obtained by processing with an embodiment of the present invention, or target real-time data with a structured format (e.g., a structured data table containing the target real-time data, etc.), or data with a JSON format containing the target real-time data, or a data access layer containing the target real-time data, etc.
Further, based on the mapping relationship, processing the real-time raw data corresponding to the raw data structure into a data standard model, including: for each piece of real-time raw data, performing: matching the real-time raw data with one or more first fields corresponding to the raw data structure; searching a data field mapping and a data processing mapping which are matched with one or more first fields from the mapping relation packaged by the target data processing model; converting the first field and real-time original data corresponding to the first field into a second field and target real-time data corresponding to the second field according to data field mapping and data processing mapping matched with one or more first fields; and processing each target real-time data into the data standard model.
For example: if the multiple fields included in the real-time raw data of a bank transaction include a trade _ DATE field, then a data field mapping matching the trade _ DATE (i.e. the first field) is searched from multiple mapping relationships encapsulated by a target data processing model corresponding to the real-time raw data, for example, the search result is VALUE _ DATE, DATE: trade _ date, VARCHAR2 (20), trade _ date >2022.6.6; further, the real-time raw data corresponding to the track _ date is, for example: 20220101 converts to target real-time data corresponding to VALUE _ DATE (and the second field), for example, the converted data is "2022-01-01"; it will be appreciated that the strategy of transformation is according to a data processing map, etc., such as: any one or more of data type conversion, data precision conversion, data screening conditions (i.e., data logic processing), and the like; the data processing mapping also includes obtaining the raw data directly after determining the field mapping.
Further, the converting the first field and its corresponding real-time raw data into a second field and target real-time data corresponding to the second field includes: and in the case that the data processing mapping of the first field comprises a data conversion mapping, converting the data value of the real-time original data into a specific type of data value corresponding to the second field based on a set conversion rule corresponding to the data conversion mapping. Specifically, in some application scenarios, it is necessary to process target real-time data, in which a data value corresponding to a first field in the received real-time raw data is a second field, for example: the data value of trade _ num (first field, trade identification) is "abcd-1234-000"; for another example, the set conversion rule corresponding to the conversion mapping of the first field transaction identification data is converted into: converting the transaction identifier into a standard identifier, for example, generating the standard identifier after removing the "-" in the transaction identifier; then "abcd-1234-000" is converted to "abcd1234000" (i.e., a data value of a particular type) corresponding to the TRADE _ ID (second field).
Further, preferably, whether the received real-time original data is abnormal or not is detected, if yes, the real-time original data is cleaned according to a set strategy, and a step of determining a target data processing model corresponding to a service type to which the real-time original data belongs is executed based on the cleaned real-time original data. Specifically, the method for detecting whether the real-time raw data is abnormal may be determined based on a set threshold, for example, the real-time raw data is a numerical value of transaction data, and if the detected numerical value is greater than a set market numerical value (i.e., the set threshold), it is determined that the real-time raw data is abnormal, and the real-time raw data is cleaned according to a set policy, for example: deleting the transaction data which are larger than the set market value, or setting the value of the transaction data which are larger than the set market value to be less than or equal to the set market value, and the like, wherein it can be understood that corresponding set strategies can be set aiming at a plurality of data fields needing to be detected so as to clean corresponding real-time original data; further, a step of determining a target data processing model corresponding to the service type to which the real-time original data belongs is executed based on the cleaned real-time original data. Another example is: the market data can be from multi-channel quotation suppliers, so the data cleaning process of data comparison and screening needs to be correspondingly carried out; by data cleaning, the real-time performance and the accuracy of data processing are improved; the efficiency of processing transaction data is improved.
Preferably, the data of the set time range is further acquired as the target real-time data based on the set time range based on the target real-time data, for example: acquiring transaction data generated in 5-10 minutes in real time; by the operation, real-time transaction data can be screened by using time, the efficiency of processing the transaction data is improved by reducing the data volume of the transaction data, and the consumption of computing resources is reduced.
Further, processing each of the target real-time data into the data standard model includes: the method comprises the steps of obtaining one or more data structures of the service type, obtaining a plurality of field identifications of the target real-time data, adding data values of the target real-time data to the data structures based on the corresponding relation between the field identifications and the data structures, and generating the data standard model based on the data structures. Specifically, for example, the obtained real-time data of each target is directly processed into a data standard model, or the real-time data of each target is processed into a plurality of data structures; for example: if the data structure is a structured data table, correspondingly adding the data values of the target real-time data into the data table according to each structured field, and generating a data standard model based on the data table; or the data structure is data of a JSON structure, the target real-time data is packaged into data of a JSON format, and a data standard model is generated based on the JSON data; or the data structure is in a key-value format, packaging the target real-time data into data in the key-value format to generate a data standard model; or packaging the target real-time data into a data access layer as a data standard model and the like, thereby further providing the data standard model for the data demand side.
Step S103: providing the data standard model to the data demander of the service type
Specifically, after the business type is processed into the target real-time data according to the mapping relation packaged by the target data processing model, a data standard model is generated based on the target real-time data and is provided for the data demand party of the business type. The data standard model processed by the data processing model of the embodiment of the invention is an output standardized data model, and can provide standardized data for the data demander, so that the data demander does not need to consider the data heterogeneity of original data and the difference of data fields of different data source data, the complexity of using target real-time data by the data demander is reduced to a greater extent, and the development efficiency of the data demander is improved.
The method for providing the data standard model (containing the target real-time data) to the data demander of the service type comprises the following steps:
the first method comprises the following steps: providing one or more data files contained in the data standard model to a data demand side, wherein the data files can contain a structured data table, JSON format data, key-value format data and the like; and the data demander can further use, analyze and process the data provided by the code based on the acquired data standard model based on the application scene.
The second method comprises the following steps: presetting a data structure corresponding to one or more theme data models for the service type based on the data standard model, and adding target real-time data into the data structure of a data demand side according to a theme; the data demander can analyze and process the data according to the theme of the service scene based on the theme data structure added with the target real-time data.
The third method comprises the following steps: providing an application access layer contained in the data standard model to a data demanding party so that the data demanding party calls an interface or data contained in the application access layer according to an application scene to obtain required data; therefore, the embodiment of the invention can support simultaneous access of cache, big data and relational databases, and data on different storage media can be accessed on a unified data access layer by establishing a unified data model extraction technology.
Therefore, by providing a standardized data standard model, a data demander can directly use business data from data sources of various types and various data structures by using the existing codes, and the development efficiency of the data demander and the consumption of time cost and labor cost in processing data are greatly reduced.
As shown in fig. 2, an embodiment of the present invention provides a method for processing transaction data, which may include the following steps:
step S201: in response to receiving one or more kinds of real-time original data related to transactions, determining a target data processing model corresponding to a service type to which the real-time original data belongs from a plurality of preset data processing models; the target data processing model comprises a mapping relation between an original data structure and a target data structure;
specifically, the description about constructing the data processing model and determining the target data processing model is consistent with the description of step S101, and is not repeated here.
Step S202: and processing the real-time original data corresponding to the original data structure into a data standard model based on the mapping relation.
Specifically, the description of processing the real-time original data corresponding to the original data structure into the data standard model based on the mapping relationship is consistent with the description of step S102, and is not repeated here.
Step S203: acquiring one or more data structures of the service type, acquiring field identifications of a plurality of target real-time data, adding data values of the target real-time data to the data structures based on the corresponding relation between the field identifications and the table structures, and generating the data standard model based on the data structures; and providing the data standard model to the data demand side.
Specifically, the description about processing each of the target real-time data into the data standard model and the data standard model is consistent with the description of step S102, and is not repeated here. Providing various corresponding data standard models while providing service data; the expansibility and the efficiency of processing transaction data are further improved, and the user experience of a data demand party is improved.
Further, target real-time data contained in the data standard model can be added into one or more data structures corresponding to the theme data model according to the theme data model corresponding to the service type based on the existing data standard model; by further subdividing the topics on the basis of the data standard model, various data requirements of a data demander can be met, the refinement degree of data processing is improved, and the data use efficiency of the data demander is improved. Taking a transaction theme as an example, a transaction theme data model can be constructed for transaction data based on different transaction types (such as stock transaction, bond transaction, option transaction, and the like), wherein the transaction theme data model can include a transaction main theme and/or a transaction sub-theme, so that common elements of transactions can be added to a data structure corresponding to the transaction main theme, and personalized elements of different types of transactions can be added to a data structure corresponding to the transaction sub-theme; according to different application scenes, the theme data model can be further modeled through management dimensions according to the requirements of a data demand party on the data model and the dimensions of data management, and the theme data model for the application scenes of the data demand party is customized. One or more data structures related to the theme data model are used as business data and are provided for the data demanding party, so that the data demanding party further analyzes and processes the data based on the theme data; the method and the device improve the refinement degree of the transaction data processing and improve the flexibility of the transaction data processing. Acquiring one or more theme data models preset for the service types, wherein one theme data model corresponds to one or more data structures; for each of the subject data models, performing the following operations: acquiring data corresponding to the theme data model from the target real-time data, and adding the data corresponding to the theme data model to a data structure corresponding to the theme data model; forming business data corresponding to a theme by using one or more data structures belonging to the same theme data model; the providing the data standard model to the data demander of the service type includes: and providing the business data of one or more themes to the data demand side.
As shown in fig. 3, an embodiment of the present invention provides an apparatus 300 for processing transaction data, including: a model determining module 301, a data processing module 302 and a data sending module 303; wherein the content of the first and second substances,
the model determining module 301 is configured to, in response to receiving one or more types of real-time raw data related to a transaction, determine, from a plurality of preset data processing models, a target data processing model corresponding to a service type to which the real-time raw data belongs; the target data processing model comprises a mapping relation between an original data structure and a target data structure;
the data processing module 302 is configured to process real-time original data corresponding to the original data structure into a data standard model based on the mapping relationship;
the data sending module 303 is configured to provide the data standard model to the data demander of the service type.
An embodiment of the present invention further provides an electronic device for processing transaction data, including: one or more processors; the storage device is used for storing one or more programs, and when the one or more programs are executed by the one or more processors, the one or more processors are enabled to realize the method provided by any one of the above embodiments.
Embodiments of the present invention further provide a computer-readable medium, on which a computer program is stored, where the computer program is executed by a processor to implement the method provided in any of the above embodiments.
Fig. 4 illustrates an exemplary system architecture 400 of a method of processing transaction data or an apparatus for processing transaction data to which embodiments of the invention may be applied.
As shown in fig. 4, the system architecture 400 may include terminal devices 401, 402, 403, a network 404, and a server 405. The network 404 serves as a medium for providing communication links between the terminal devices 401, 402, 403 and the server 405. Network 404 may include various types of connections, such as wire, wireless communication links, or fiber optic cables, to name a few.
A user may use terminal devices 401, 402, 403 to interact with a server 405 over a network 404 to receive or send messages or the like. The terminal devices 401, 402, 403 may have various client applications installed thereon, such as an e-mall client application, a web browser application, a search-type application, an instant messaging tool, a mailbox client, and the like.
The terminal devices 401, 402, 403 may be various electronic devices having display screens and supporting various client applications, including but not limited to smart phones, tablet computers, laptop portable computers, desktop computers, and the like.
The server 405 may be a server providing various services, such as a background management server providing support for client applications used by users with the terminal devices 401, 402, 403. The background management server can process the received real-time original data related to the transaction and send the processed target original data to the terminal equipment.
It should be noted that the method for processing transaction data provided by the embodiment of the present invention is generally executed by the server 405, and accordingly, the device for processing transaction data is generally disposed in the server 405.
It should be understood that the number of terminal devices, networks, and servers in fig. 4 is merely illustrative. There may be any number of terminal devices, networks, and servers, as desired for implementation.
Referring now to FIG. 5, shown is a block diagram of a computer system 500 suitable for use with a terminal device implementing an embodiment of the present invention. The terminal device shown in fig. 5 is only an example, and should not bring any limitation to the functions and the scope of use of the embodiments of the present invention.
As shown in fig. 5, the computer system 500 includes a Central Processing Unit (CPU) 501 that can perform various appropriate actions and processes according to a program stored in a Read Only Memory (ROM) 502 or a program loaded from a storage section 508 into a Random Access Memory (RAM) 503. In the RAM 503, various programs and data necessary for the operation of the system 500 are also stored. The CPU 501, ROM 502, and RAM 503 are connected to each other via a bus 504. An input/output (I/O) interface 505 is also connected to bus 504.
The following components are connected to the I/O interface 505: an input portion 506 including a keyboard, a mouse, and the like; an output portion 507 including a display such as a Cathode Ray Tube (CRT), a Liquid Crystal Display (LCD), and the like, and a speaker; a storage portion 508 including a hard disk and the like; and a communication section 509 including a network interface card such as a LAN card, a modem, or the like. The communication section 509 performs communication processing via a network such as the internet. A drive 510 is also connected to the I/O interface 505 as needed. A removable medium 511 such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, or the like is mounted on the drive 510 as necessary, so that a computer program read out therefrom is mounted into the storage section 508 as necessary.
In particular, according to the embodiments of the present disclosure, the processes described above with reference to the flowcharts may be implemented as computer software programs. For example, embodiments of the present disclosure include a computer program product comprising a computer program embodied on a computer-readable medium, the computer program comprising program code for performing the method illustrated by the flow chart. In such an embodiment, the computer program may be downloaded and installed from a network through the communication section 509, and/or installed from the removable medium 511. The computer program performs the above-described functions defined in the system of the present invention when executed by the Central Processing Unit (CPU) 501.
It should be noted that the computer readable medium shown in the present invention can be a computer readable signal medium or a computer readable storage medium or any combination of the two. A computer readable storage medium may be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination of the foregoing. More specific examples of the computer readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer diskette, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. In the present invention, a computer readable storage medium may be any tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device. In the present invention, however, a computer readable signal medium may include a propagated data signal with computer readable program code embodied therein, for example, in baseband or as part of a carrier wave. Such a propagated data signal may take many forms, including, but not limited to, electro-magnetic, optical, or any suitable combination thereof. A computer readable signal medium may also be any computer readable medium that is not a computer readable storage medium and that can communicate, propagate, or transport a program for use by or in connection with an instruction execution system, apparatus, or device. Program code embodied on a computer readable medium may be transmitted using any appropriate medium, including but not limited to: wireless, wire, fiber optic cable, RF, etc., or any suitable combination of the foregoing.
The flowchart and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods and computer program products according to various embodiments of the present invention. In this regard, each block in the flowchart or block diagrams may represent a module, segment, or portion of code, which comprises one or more executable instructions for implementing the specified logical function(s). It should also be noted that, in some alternative implementations, the functions noted in the block may occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams or flowchart illustration, and combinations of blocks in the block diagrams or flowchart illustration, can be implemented by special purpose hardware-based systems that perform the specified functions or acts, or combinations of special purpose hardware and computer instructions.
The modules and/or units described in the embodiments of the present invention may be implemented by software, and may also be implemented by hardware. The described modules and/or units may also be provided in a processor, and may be described as: a processor includes a determine model module, a process data module, and a send data module. The names of these modules do not in some cases form a limitation on the modules themselves, for example, the data sending module may also be described as a "module that provides the target real-time data to the data consumers of the service type".
As another aspect, the present invention also provides a computer-readable medium that may be contained in the apparatus described in the above embodiments; or may be separate and not incorporated into the device. The computer readable medium carries one or more programs which, when executed by a device, cause the device to comprise: in response to receiving one or more kinds of real-time original data related to transactions, determining a target data processing model corresponding to a service type to which the real-time original data belongs from a plurality of preset data processing models; the target data processing model comprises a mapping relation between an original data structure and a target data structure; processing real-time original data corresponding to the original data structure into a data standard model based on the mapping relation; and providing the data standard model to a data demand side of the service type.
The embodiment of the invention can respond to the received real-time original data related to the transaction and determine the target data processing model corresponding to the service type of the real-time original data; the target data processing model comprises a mapping relation between an original data structure and a target data structure; processing real-time original data corresponding to the original data structure into a data standard model; and providing the data standard model to the data demander of the service type. The universality, flexibility and expansibility of processing transaction data are improved through one or more data processing models and data standard models; the efficiency of processing transaction data is improved through real-time processing, and the complexity of processing data is reduced.
The above-described embodiments should not be construed as limiting the scope of the invention. Those skilled in the art will appreciate that various modifications, combinations, sub-combinations, and substitutions can occur, depending on design requirements and other factors. Any modification, equivalent replacement, and improvement made within the spirit and principle of the present invention should be included in the protection scope of the present invention.
Claims (11)
1. A method of processing transaction data, comprising:
in response to receiving one or more real-time raw data related to a transaction,
determining a target data processing model corresponding to the service type to which the real-time original data belongs from a plurality of preset data processing models; the target data processing model comprises a mapping relation between an original data structure and a target data structure; the mapping relation comprises a data field mapping and a data processing mapping which are constructed aiming at one or more original fields of the original data structure;
processing real-time original data corresponding to the original data structure into a data standard model based on the mapping relation; the data standard model comprises target real-time data obtained by processing based on the real-time original data, and the target real-time data has one or more data structures;
and providing the data standard model to a data demand side of the service type.
2. The method of claim 1, further comprising:
acquiring transaction historical data corresponding to multiple service types;
for the transaction history data of each service type, executing:
determining a data dictionary corresponding to the transaction historical data;
analyzing an original data structure contained in a data dictionary corresponding to the transaction historical data;
and constructing a mapping relation between the original data structure and a target data structure constructed for the service type in advance, and packaging the mapping relation into a data processing model.
3. The method of claim 2,
the data processing map comprises any one or more of a data type map, a processing logic map, and a data conversion map.
4. The method of claim 3,
processing the real-time original data corresponding to the original data structure into a data standard model based on the mapping relationship, including:
for each piece of real-time raw data, performing:
matching the real-time raw data with one or more first fields corresponding to the raw data structure;
searching a data field mapping and a data processing mapping which are matched with one or more first fields from the mapping relation packaged by the target data processing model;
converting the first field and real-time original data corresponding to the first field into a second field and target real-time data corresponding to the second field according to data field mapping and data processing mapping matched with one or more first fields;
and processing each target real-time data into the data standard model.
5. The method of claim 1, further comprising:
and detecting whether the received real-time original data is abnormal or not, if so, cleaning the real-time original data according to a set strategy, and executing a step of determining a target data processing model corresponding to the service type of the real-time original data based on the cleaned real-time original data.
6. The method of claim 4,
the converting the first field and its corresponding real-time raw data into a second field and target real-time data corresponding to the second field comprises:
and in the case that the data processing mapping of the first field comprises a data conversion mapping, converting the data value of the real-time original data into a specific type of data value corresponding to the second field based on a set conversion rule corresponding to the data conversion mapping.
7. The method of claim 4,
processing each of the target real-time data into the data standard model, including:
acquiring one or more data structures of the service type, acquiring a plurality of field identifications of the target real-time data, adding the data values of the target real-time data to the data structures based on the corresponding relation between the field identifications and the data structures, and generating the data standard model based on the data structures.
8. The method of claim 1, further comprising:
presetting one or more theme data models for the service types based on the data standard model, wherein one theme data model corresponds to one or more data structures;
for each of the subject data models, performing the following operations: acquiring data corresponding to the theme data model from the target real-time data, and adding the data corresponding to the theme data model to a data structure corresponding to the theme data model;
forming business data corresponding to a theme by using one or more data structures belonging to the same theme data model;
the step of providing the data standard model to the data demander of the service type comprises the following steps: and providing the business data of one or more themes to the data demand side.
9. An apparatus for processing transaction data, comprising: the device comprises a determining model module, a data processing module and a data sending module; wherein the content of the first and second substances,
the determining model module is used for responding to one or more kinds of real-time original data related to the transaction and determining a target data processing model corresponding to the business type to which the real-time original data belongs from a plurality of preset data processing models; the target data processing model comprises a mapping relation between an original data structure and a target data structure; the mapping relation comprises a data field mapping and a data processing mapping which are constructed aiming at one or more original fields included in the original data structure;
the data processing module is used for processing the real-time original data corresponding to the original data structure into a data standard model based on the mapping relation; the data standard model comprises target real-time data obtained by processing based on the real-time original data, and the target real-time data has one or more data structures;
and the data sending module is used for providing the data standard model to the data demander of the service type.
10. An electronic device, comprising:
one or more processors;
a storage device for storing one or more programs,
the one or more programs, when executed by the one or more processors, cause the one or more processors to implement the method recited in any of claims 1-8.
11. A computer-readable medium, on which a computer program is stored, which, when being executed by a processor, carries out the method according to any one of claims 1-8.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202211237667.4A CN115330540A (en) | 2022-10-11 | 2022-10-11 | Method and device for processing transaction data |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202211237667.4A CN115330540A (en) | 2022-10-11 | 2022-10-11 | Method and device for processing transaction data |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN115330540A true CN115330540A (en) | 2022-11-11 |
Family
ID=83914885
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202211237667.4A Pending CN115330540A (en) | 2022-10-11 | 2022-10-11 | Method and device for processing transaction data |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN115330540A (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN115952770A (en) * | 2023-03-15 | 2023-04-11 | 广州汇通国信科技有限公司 | Data standardization processing method and device, electronic equipment and storage medium |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN107295039A (en) * | 2016-03-31 | 2017-10-24 | 阿里巴巴集团控股有限公司 | Data access treating method and apparatus |
| CN111241107A (en) * | 2020-01-16 | 2020-06-05 | 北京字节跳动网络技术有限公司 | Service processing method, device, medium and electronic equipment |
| CN112395117A (en) * | 2021-01-21 | 2021-02-23 | 武汉中科通达高新技术股份有限公司 | Data processing method, system and storage medium |
| CN112685493A (en) * | 2020-12-30 | 2021-04-20 | 平安普惠企业管理有限公司 | Report processing method and device, electronic equipment and storage medium |
| CN113836131A (en) * | 2021-09-29 | 2021-12-24 | 平安科技（深圳）有限公司 | Big data cleaning method and device, computer equipment and storage medium |
-
2022
- 2022-10-11 CN CN202211237667.4A patent/CN115330540A/en active Pending
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN107295039A (en) * | 2016-03-31 | 2017-10-24 | 阿里巴巴集团控股有限公司 | Data access treating method and apparatus |
| CN111241107A (en) * | 2020-01-16 | 2020-06-05 | 北京字节跳动网络技术有限公司 | Service processing method, device, medium and electronic equipment |
| CN112685493A (en) * | 2020-12-30 | 2021-04-20 | 平安普惠企业管理有限公司 | Report processing method and device, electronic equipment and storage medium |
| CN112395117A (en) * | 2021-01-21 | 2021-02-23 | 武汉中科通达高新技术股份有限公司 | Data processing method, system and storage medium |
| CN113836131A (en) * | 2021-09-29 | 2021-12-24 | 平安科技（深圳）有限公司 | Big data cleaning method and device, computer equipment and storage medium |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN115952770A (en) * | 2023-03-15 | 2023-04-11 | 广州汇通国信科技有限公司 | Data standardization processing method and device, electronic equipment and storage medium |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN109034988B (en) | Accounting entry generation method and device | |
| CN112184154A (en) | Business approval method and device | |
| CN111427971B (en) | Business modeling method, device, system and medium for computer system | |
| CN111429241A (en) | Accounting processing method and device | |
| CN111382279A (en) | Order examination method and device | |
| CN111460129A (en) | Method and device for generating identification, electronic equipment and storage medium | |
| CN111881329A (en) | Account balance management method and system | |
| CN111127181A (en) | Voucher bookkeeping method and device | |
| CN111857888A (en) | Transaction processing method and device | |
| CN111339743B (en) | Account number generation method and device | |
| CN115330540A (en) | Method and device for processing transaction data | |
| CN111104556A (en) | Service processing method and device | |
| CN109740130B (en) | Method and device for generating file | |
| CN115858345A (en) | Application service module verification method and device, electronic equipment and storage medium | |
| CN113485763A (en) | Data processing method and device, electronic equipment and computer readable medium | |
| CN112579673A (en) | Multi-source data processing method and device | |
| CN112256566A (en) | Test case preservation method and device | |
| CN113434754A (en) | Method and device for determining recommended API (application program interface) service, electronic equipment and storage medium | |
| CN113780921B (en) | Price inquiry processing method and device | |
| CN114997977B (en) | Data processing method, device, electronic equipment and computer readable medium | |
| CN115858325B (en) | Project log adjusting method, device, equipment and storage medium | |
| CN116319716A (en) | Information processing method, no-service system, electronic device, and storage medium | |
| CN116228384A (en) | Data processing method, device, electronic equipment and computer readable medium | |
| CN114022296A (en) | Service processing method and device, electronic equipment and storage medium | |
| CN115422589A (en) | Data processing method and device, electronic equipment and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20221111 |