CN110008282A - Transaction data synchronization interconnection method, device, computer equipment and storage medium - Google Patents

Transaction data synchronization interconnection method, device, computer equipment and storage medium Download PDF

Info

Publication number
CN110008282A
CN110008282A CN201910184626.5A CN201910184626A CN110008282A CN 110008282 A CN110008282 A CN 110008282A CN 201910184626 A CN201910184626 A CN 201910184626A CN 110008282 A CN110008282 A CN 110008282A
Authority
CN
China
Prior art keywords
data
transaction data
cleaned
platform
transaction
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Pending
Application number
CN201910184626.5A
Other languages
Chinese (zh)
Inventor
马万里
甘瑞华
叶丽娜
杨明
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Ping An Trust Co Ltd
Original Assignee
Ping An Trust Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Ping An Trust Co Ltd filed Critical Ping An Trust Co Ltd
Priority to CN201910184626.5A priority Critical patent/CN110008282A/en
Publication of CN110008282A publication Critical patent/CN110008282A/en
Pending legal-status Critical Current

Links

Classifications

    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/20Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
    • G06F16/21Design, administration or maintenance of databases
    • G06F16/215Improving data quality; Data cleansing, e.g. de-duplication, removing invalid entries or correcting typographical errors
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/20Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
    • G06F16/27Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor

Landscapes

  • Engineering & Computer Science (AREA)
  • Databases & Information Systems (AREA)
  • Theoretical Computer Science (AREA)
  • Data Mining & Analysis (AREA)
  • Physics & Mathematics (AREA)
  • General Engineering & Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Quality & Reliability (AREA)
  • Computing Systems (AREA)
  • Information Retrieval, Db Structures And Fs Structures Therefor (AREA)

Abstract

This application involves big data technical field, a kind of transaction data synchronization interconnection method, device, computer equipment and storage medium are provided, each platform transaction data of acquisition is passed through;Each platform transaction data is cleaned according to data cleansing algorithm, data have been cleaned in acquisition;The logical value for obtaining input obtains target data corresponding with logical value from having cleaned in data;Read the common ID character for having cleaned data and target data;According to common ID character, processing is synchronized to each platform transaction data, obtains synchrodata.In whole process, by obtaining transaction data from multiple platforms, whitewash transaction data, isochronous transaction data realize the synchronization of transaction data on the basis of accurate with data extensively based on data source, easy to use.

Description

Transaction data synchronization interconnection method, device, computer equipment and storage medium
Technical field
This application involves big data processing technology field, more particularly to a kind of transaction data synchronization interconnection method, device, Computer equipment and storage medium.
Background technique
With the arrival of big data era, the growth rate of data becomes faster, and process of exchange networking mode is gradually risen, and hands over The scale of easy data becomes huge, and the type and structure of transaction data are also more various, at the confluence analysis of transaction data Reason becomes particularly important.
In general, financing corporation is all to search customer data and transaction record etc. by certain channels by full-time staff Information is integrated by manual analysis table, provides more good service for client, or provide to company from whole financial circles Analysis in business.
Using manually to the processing of transaction data, data acquisition channel is limited, data are dispersed and transaction data becomes in real time Change, analytical integration result inaccuracy is inconvenient to use.
Summary of the invention
Based on this, it is necessary to be directed to problem inconvenient for use, provide a kind of transaction data synchronization interconnection method easy to use, Device, computer equipment and storage medium.
A kind of transaction data synchronization interconnection method, comprising:
Acquire each platform transaction data;
Each platform transaction data is cleaned according to data cleansing algorithm, data have been cleaned in acquisition;
The logical value for obtaining input obtains target data corresponding with logical value from having cleaned in data;
Read the common ID character for having cleaned data and target data;
According to common ID character, processing is synchronized to each platform transaction data, obtains synchrodata.
Acquiring each platform transaction data in one of the embodiments, includes: identifying the transaction data class that each platform is sent Type;
If the JSON that transaction data type is internal platform formats transaction data type, call default internal interface into Row acquisition;
It is flat to inside using regular expression if transaction data type is internal platform unformatted transaction data type Platform unformatted transaction data is parsed, and is generated JSON format transaction data and is acquired;
If transaction data type is external platform unformatted transaction data type, grabbed using web crawlers technology outer Portion's platform unformatted transaction data solves the external platform unformatted transaction data grabbed by regular expression Analysis processing generates JSON and formats transaction data, and acquires.
Each platform transaction data is cleaned according to data cleansing algorithm in one of the embodiments, is obtained clear Washing data includes:
The cleaning of repeated data based on Cosine similarity function is carried out, based on calorie enthesis to each platform transaction data Missing data cleaning, be based onThe abnormal data of principle cleans and the noise data cleaning based on branch mailbox method, obtains clear Wash data.
The logical value for obtaining input in one of the embodiments, from having cleaned, acquisition in data is corresponding with logical value Target data includes:
Receive the logical value of input;
According to logical value search it is corresponding with logical value cleaned data, record is corresponding with logical value, which to have cleaned data, has Effect property;
According to effective record, target data is obtained.
The logical value for obtaining input in one of the embodiments, from having cleaned, acquisition in data is corresponding with logical value After target data further include:
When receiving logical value change message, customized label is generated according to the logical value of change;
According to customized label, effectively record, the target data after obtaining change are changed.
In one of the embodiments, according to common ID character, data synchronization processing is carried out, obtains synchronization packets It includes:
Data relationship net is established using association rule algorithm according to common relation character;
It to target data and has cleaned data by data relationship net and has been associated, according to associated data, to target data Processing is synchronized, synchrodata is obtained.
In one of the embodiments, according to common ID character, processing is synchronized to each platform transaction data, is obtained After synchrodata further include:
Synchrodata is inputted to the Data Analysis Model constructed based on normalization method, by Data Analysis Model to synchrodata Output format is normalized, and obtains the unified synchrodata of output format;
Visualization mapping is carried out according to the unified synchrodata of output format, generates visual analyzing chart.
A kind of transaction data synchronization docking facilities, comprising:
Transaction data obtains module, for acquiring each platform transaction data;
Cleaning module, for being cleaned according to data cleansing algorithm to data to be cleaned, data have been cleaned in acquisition;
Target data obtains module, and for obtaining the logical value of input, from having cleaned, acquisition in data is corresponding with logical value Target data;
Character read module, for reading the common ID character for having cleaned data and target data;
Synchronization module, for synchronizing processing to each platform transaction data, obtaining same step number according to common ID character According to.
A kind of computer equipment, including memory and processor, memory are stored with computer program, and processor executes meter It is performed the steps of when calculation machine program
Acquire each platform transaction data;
Each platform transaction data is cleaned according to data cleansing algorithm, data have been cleaned in acquisition;
The logical value for obtaining input obtains target data corresponding with logical value from having cleaned in data;
Read the common ID character for having cleaned data and target data;
According to common ID character, processing is synchronized to each platform transaction data, obtains synchrodata.
A kind of computer readable storage medium is stored thereon with computer program, when computer program is executed by processor It performs the steps of
Acquire each platform transaction data;
According to data cleansing algorithm, each platform transaction data is cleaned, data have been cleaned in acquisition;
The logical value for obtaining input obtains target data corresponding with logical value from having cleaned in data;
Read the common ID character for having cleaned data and target data;
According to common ID character, processing is synchronized to each platform transaction data, obtains synchrodata.
Above-mentioned transaction data synchronization interconnection method, device, computer equipment and storage medium are traded by acquiring each platform Data;Each platform transaction data is cleaned according to data cleansing algorithm, data have been cleaned in acquisition;Obtain the logic of input Value obtains target data corresponding with logical value from having cleaned in data;Read the common mark for having cleaned data and target data Character learning symbol;According to common ID character, processing is synchronized to each platform transaction data, obtains synchrodata.In whole process, By obtaining transaction data from multiple platforms, whitewash transaction data, isochronous transaction data, based on data source extensively and data On the basis of accurately, the synchronization of transaction data is realized, it is easy to use.
Detailed description of the invention
The one of embodiment flow diagram of the above-mentioned transaction data synchronization interconnection method of Fig. 1;
Fig. 2 is another embodiment flow diagram of above-mentioned transaction data synchronization interconnection method;
Fig. 3 is the above-mentioned one of example structure schematic diagram of transaction data synchronization docking facilities;
Fig. 4 is the one of embodiment schematic diagram of internal structure of computer equipment.
Specific embodiment
It is with reference to the accompanying drawings and embodiments, right in order to which the objects, technical solutions and advantages of the application are more clearly understood The application is further elaborated.It should be appreciated that specific embodiment described herein is only used to explain the application, not For limiting the application.
In one of the embodiments, as shown in Figure 1, providing a kind of transaction data synchronization interconnection method, including it is following Step:
S110: each platform transaction data is acquired.
Transaction data in each transaction platform is acquired, wherein transaction platform refers to that third-party transaction security is protected Hinder platform, the transaction discussed under line is moved on the net, traded by third party transaction platform on the net by both parties, and net Upper transaction, which is more client, finds product required for oneself by transaction platform, to trade, therefore and generation Transaction data, the transaction data in the present embodiment can be the products transactions of enterprises operation and the transaction data that generates, It can be and be acquired from external transaction platform, also include the essential information etc. of transacting customer, further, transaction platform can be with It is gold trade platform, stock exchange platform, stock exchange platform, insures transaction platform and real estate trade platform etc., specifically , the transaction data of acquisition can have for client A, gender female, household register H, 2018 September 22 in place K, in B platform, purchase Security C is bought, quantity is transaction data and essential information of friend relatives of 2 and client A etc..
S120: cleaning each platform transaction data according to data cleansing algorithm, and data have been cleaned in acquisition.
Data cleansing, which refers to the process of, to be audited and is verified again to data, it is therefore intended that is deleted duplicate message, is corrected Existing mistake etc..From the transaction data that each transaction platform obtains, the transaction data of magnanimity there is it is a large amount of it is imperfect, It repeats, have phenomena such as abnormal, seriously affect working efficiency.Data cleansing algorithm refer to by similarity function,Principle etc. pair Repetition values in data, abnormal data are cleaned, and the dirty data in data is filtered off, and dirty data refers to the data in the system of source not Given range or to practical business it is beyond all doubt or data format is illegal, and there are nonstandard in the system of source Coding and ambiguous service logic etc., in the present embodiment, by data cleansing algorithm, clean each platform transaction data, The dirty datas such as repeated data, abnormal data are filtered out, data have been cleaned in acquisition.
S130: obtaining the logical value of input, and target data corresponding with logical value is obtained in data from having cleaned.
In face of magnanimity transaction data, financing manager does not need to be clearly seen the specific transaction data of each single item, logical value It can be a certain screening conditions, when the client of financing manager's service is A, it is only necessary to the All Activity data of client A are obtained, this When screening conditions be client A, target data be data information related with client A, therefore manage money matters manager given according to business demand Determine logical value, obtains target data corresponding with logical value.Specifically, server connects when given screening conditions are client A The logical value of receipts is client A, and screening and the related all data informations of client A, obtain number of targets from the data cleaned According to, such as age information, certificate address information, kinsfolk's information, company's information, exchange hour and the purchase product class of client A Type etc..
S140: the common ID character for having cleaned data and target data is read.
Mark character refers to, for identifying the character of some entity, there is different meanings under different application environments, In computer programming language, the name that mark character uses when being user program, for variable, constant, function and sentence The name such as block, to set up the relationship between title and use, in the present embodiment, common ID character is equivalent to one jointly The factor includes the information such as client A and client B in having cleaned data, after logical value is screened for client A, is obtained Target data client's A transaction data, the client A and target data A cleaned in data have common factor, i.e. common ID word Symbol, further, in having cleaned data, client A and client B are there are relatives' relationship or when having common intersection, target data visitor It family A and has cleaned client B in data and can also have common ID character, common ID character has been read out.
S150: according to common ID character, processing is synchronized to each platform transaction data, obtains synchrodata.
According in S140 after having cleaned in data and reading the common ID character with target data, according to co-owning Mark character, establish cleaned data and target data synchronize contact, when the cleaning data with common ID character When information changes, the target data with common ID character detects the update for having cleaned data, automatically to common mark The target data of character learning symbol synchronizes update, such as when target data is client A, having cleaned in data has and client A Common ID character changes client A purchase product quantity, according to the reading to common ID character, recognizes clearly The corresponding product quantity of common ID character washed in data is changed, and is produced in the target data of corresponding common ID character Product quantity also makes consistent modification, carries out data and synchronizes, obtains synchrodata.
Above-mentioned transaction data synchronization interconnection method acquires each platform transaction data;According to data cleansing algorithm, to each platform Transaction data is cleaned, and data have been cleaned in acquisition;The logical value for obtaining input obtains and logical value pair from having cleaned in data The target data answered;Read the common ID character for having cleaned data and target data;According to common ID character, to each platform Transaction data synchronizes processing, obtains synchrodata.In entire transaction data synchronization docking operation, obtains and hand over from multiple platforms Easy data, whitewash transaction data, isochronous transaction data realize on the basis of accurate with data extensively based on data source The synchronization of transaction data, it is easy to use.
In one of the embodiments, as described in Figure 2, acquiring each platform transaction data includes:
S210: the transaction data type that each platform is sent is identified.
S211: it if the JSON that transaction data type is internal platform formats transaction data type, calls default internal Interface is acquired.
S212: if transaction data type is the unformatted transaction data type of internal platform, using regular expression Internal unformatted transaction data is parsed, JSON format transaction data is generated and is acquired.
S213: if transaction data type is external platform unformatted transaction data type, using web crawlers technology External platform unformatted transaction data is grabbed, by regular expression to the external platform unformatted transaction data grabbed Dissection process is carried out, JSON is generated and formats transaction data, and acquire.
JSON (JavaScript Object Notation, JS, object numbered musical notation) is a kind of data exchange lattice of lightweight Formula, it based on European Computer association formulate JS a subset, using the text formatting for being totally independent of programming language come Storage and expression data are easy to machine parsing and generate, and effectively promote network transmission efficiency.Internal platform JSON is formatted Data refer to, the transaction carried out in the transaction platform of enterprises operation, and are stored with JSON formatting;Internal non-lattice Formula data refer to, the transaction carried out in the transaction platform of enterprises operation, with one in addition to JSON format data Kind or the transaction data of a variety of data formats storage;External platform nonformatted data refers to, outside business system, other enterprises The transaction data that the transaction platform of industry or website generates.
The transaction data of each system registration of up-stream system transmission is acquired from each system by ODS, wherein ODS is Subject-oriented retains the data in the current and regular period, provides data normalization and distribution.Further, to transaction Data type is identified, when the JSON for identifying that transaction data type is internal platform formats transaction data type, is called The internal interface of systemic presupposition formats transaction data to the JSON of internal platform and is acquired, wherein and internal interface refers to, To the reference type that agreement is defined, other types realize interface, to guarantee that they support certain operations, in the present embodiment Operation is to realize the acquisition of insider transaction data.When the unformatted transaction data class that identification transaction data type is internal platform When type, processing, the conversion of data encoding etc. using regular expression to spcial character is carried out in transaction data, dissection process Afterwards, unified JSON format data is generated, wherein regular expression refers to a kind of logical formula to string operation, just It is the combination with predefined some specific characters and these specific characters, forms a regular character string, this rule Character string is used to express a kind of filter logic to character string.Regular expression is for the special word in matching treatment transaction data Symbol, change data specific coding, unified transaction data format are JSON format data.When identification transaction data type is outside When platform nonformatted data type, passes through web crawlers technology and carry out data grabber, wherein web crawlers is otherwise known as webpage Spider, network robot, webpage follower etc., be it is a kind of according to certain rules, automatically grab the program of web message Or script.
In the present embodiment, server search engine using web crawlers technology crawl Web page, document even picture, The resources such as audio and screen obtain transaction data by corresponding index technology, by regular expression, to the number of deals of acquisition It is matched according to character string, uniform format processing generates JSON format data.After unified transaction data format, data are carried out Acquisition, it is non-essential, after collecting the data, data can be stored in interface table, facilitate calling.In the present embodiment, by not The transaction data of same type uses different types of acquisition method, ensure that the accuracy of transaction data and from a wealth of sources.
Each platform transaction data is cleaned according to data cleansing algorithm in one of the embodiments, is obtained clear Washing data includes: to carry out repeated data cleaning based on Cosine similarity function to each platform transaction data, filled out based on calorie The missing data cleaning of reinforcing method is based onThe abnormal data of principle cleans and the noise data cleaning based on branch mailbox method, obtains Data are cleaned.Wherein, Cosine similarity function is also known as cosine similarity, is the included angle cosine by calculating two vectors Value assesses their similarity;Calorie enthesis is that the variable comprising missing values is found and its most phase in the database As object, be then filled using the value of this analogical object, different problems may select pair of different standards It is similar to be judged;Principle is also known as Pauta criterion, it is first to assume that one group of detection data contains only random error, to it It carries out calculation processing and obtains standard deviation, by one section of certain determine the probability, it is believed that all errors more than this section, just not Belong to random error but gross error, the data containing the error are rejected;Branch mailbox method be by investigate adjacent data come Determine end value, actually according to the subinterval of attribute value division, if an attribute value is within the scope of some subinterval, In " chest " with regard to claiming the attribute value to put into the representative of this section, data to be processed are put into according to certain rules The data in each " chest " are investigated in " chest ", carry out side using data of the customized interval method of user to each " chest " Boundary's smoothing processing.
In the present embodiment, by two transaction data attribute vectors, A and B, cosine similarity θ is by dot product and vector Length provides:
In formula, Ai, BiRespectively indicate a component of transaction data A and B, similarity ranges indicated from -1 to 1:-1 two to The direction that amount is directed toward just is completely contradicted, and is independent between 1 two vectors of expression, and value between then indicates centre Similitude and diversity delete any one data in transaction data A and B when similarity is 1, carry out data deduplication processing; When the information for there was only mono- people of client A in the essential information of client A, and client's B essential information shows that there are relatives passes with client A System, by the similarity relation of client A and client B, supplements the essential information of client A;Assuming that client A in transaction data, On November 28th, 2018, fund M was had purchased with 500,000 price, and client A in recording, on November 28th, 2018, with 55 Ten thousand price has purchased fund M, then judges that having purchased fund M with 550,000 price is that error information is rejected: according to boundary Value is smooth, determines two boundaries, then successively calculates in addition to boundary value at a distance from two boundaries of other values, minimum with distance Boundary value replace each data, determine smooth boundary value, filter off noise data, when in certain case transaction data be 4,8,9, 15, | 8-4 |=4, | 15-8 |=7, | 9-4 |=5, | 15-9 |=6;Therefore selecting 4 is smooth boundary value.It is non-essential, it can will be clear The data of cleaning after washing are stored in middle table, and middle table is the concept in service logic, and calculated result is exactly stored in one In a interim table, subsequent processing is carried out to data in this table, the complexity of server process data can be reduced.This reality It applies in example, by cleaning to each platform transaction data, guarantees transaction data accurately and consistently.
The logical value for obtaining input in one of the embodiments, from having cleaned, acquisition in data is corresponding with logical value Target data includes: receiving the logical value of input;Data have been cleaned according to logical value lookup is corresponding with logical value, has recorded and patrols Volume value is corresponding has cleaned data validity;According to effective record, target data is obtained.Wherein, financing manager's input is obtained Logical value, logical value can be used as the critical fielies such as time, name, place and products transactions type, and data have been cleaned in lookup In containing critical field, that is, logical value transaction data, when finding the data in transaction data containing logical value, to its into Row record, transaction data of the record containing logical value are valid data, and the transaction data without logical value is invalid data, according to Validity record obtains the target data that financing manager intentionally gets, and non-essential, the lookup of transaction data validity can be in Between carry out in table, validity result is stored in middle table.Specifically, when the logical value of financing manager's input is place value Z It saves, the transaction data for having cleaned that loco generation is saved in Z in data is searched in middle table, at this point, for loco in Z The transaction data record of province be it is effective, be that save data record be invalid data by non-Z for loco, record result be stored in It in middle table, is recorded according to validity, obtains effective target data.Further, when server is stored with the industry of manual entry When business table transaction data, it can be recorded according to the transaction data validity of middle table, be modified to traffic table data, such as when Middle table is recorded as Hunan, and Zhang San, 2018.11.20, N fund, quantity 2 is the invalid friendship in Hubei to record place in traffic table Easy data are deleted, when transaction data in traffic table are as follows: Hunan, Zhang San, 2018.11.20N fund.Then in traffic table not It is Hunan, Zhang San, 2018.11.20, N fund, quantity 2 that partial data, which carries out supplement,.In the present embodiment, by logical value to friendship Easy data carry out validity record, obtain accurate target data, easy to use.
The logical value for obtaining input in one of the embodiments, from having cleaned, acquisition in data is corresponding with logical value After target data further include: when receiving logical value change message, generate customized label according to the logical value of change;According to Customized label changes effectively record, the target data after obtaining change.Wherein, label be to a kind of mark of things, it is customized Label refers to be changed correspondingly according to the change label of logical value, is carried out by logical value to label customized.Further, work as reason When finance and economics reason first time input logic value is that time value is greater than 2018.10.22, transaction data in data is cleaned and has existed 2018.10.22 the transaction data record occurred after is valid data, when financing manager modifies to logical value, service Device receives logical value and changes message, parses to logical value change message, the logical value after obtaining change, for example, after change Logical value be A fund simultaneously the time be greater than 2017.5.23, according to after change logical value generate customized label marked Note, in the data that first time is recorded by logical value, the transaction data that 2018.11.02 has purchased fund A is significant figure According to after logical value change, this transaction record is consistent with customized label, keeps original and is recorded as valid data;When first In the secondary data recorded by logical value, 2017.12.25 has purchased fund A and is recorded as invalid data, logical value change Afterwards, this transaction record is consistent with customized label content, and change invalid record is effectively to record, more according to validity record Change, the target data after obtaining change.Target data can be changed in time in the present embodiment to the change of logical value, guarantees number of targets It is easy to use according to energy synchronized update.
In one of the embodiments, according to common ID character, processing is synchronized to each platform transaction data, is obtained Synchrodata includes: establishing data relationship net using association rule algorithm according to common relation character;Pass through data relationship net It to target data and has cleaned data and has been associated, according to associated data, processing has been synchronized to target data, obtains same step number According to.Wherein, correlation rule is the implication shaped like x → y, wherein X and Y is the referred to as guide of correlation rule and subsequent respectively, Middle correlation rule x, y, there are support and degree of belief, association rule algorithm includes Aprion algorithm, based on planning algorithm and FP- Frequency set algorithm is set, in the present embodiment, using Aprion algorithm, finds out all frequency collection first, frequencies that these frequency collection occur At least as predefined minimum support, then collected by frequency and generate Strong association rule, these rules support minimum support And Minimum support4 generates strictly all rules only comprising set entry, each of them rule then using frequency collection generation expectation rule Most right part then only has one, when rule generation, can just be left greater than the regular of confidence level that user gives.In this implementation In example, according to reading target data and the common ID character in data has been cleaned, when common ID character is M fund, root According to association rule algorithm, when obtaining M fund and N stock greater than user preset confidence level, M fund also has altogether with N stock at this time It establishes the target transaction data with M fund common ID character by common ID character with mark character and has cleaned number There is the network of personal connections between incidence relation and the N stock and transaction data of common ID character in, cobweb can be likened into, Become when having cleaned transaction data and N stock exchange data with M fund common ID character and incidence relation in data When dynamic, processing is synchronized to target transaction data, obtains synchrodata.Real-time update data ensure that the timeliness of data, Financing manager's working efficiency is improved, it is easy to use.
In one of the embodiments, according to common ID character, processing is synchronized to each platform transaction data, is obtained After synchrodata further include: synchrodata is inputted to the Data Analysis Model constructed based on normalization method, analyzes mould by data Synchrodata output format is normalized in type, obtains the unified data of output format;Unified according to output format Data carry out visualization mapping, generate visual analyzing chart.Wherein, normalization method realizes the standardization of data, and different evaluation refers to Mark often has different a dimension and dimensional unit, such situation influence whether data analysis as a result, in order to eliminate each index Between dimension impact, need to carry out data normalization processing, the output form of expression of uniform data, Data Analysis Model is base It is constructed in normalization method, further, Data Analysis Model can be 5W2H basic model, also known as 7 what analysis model, for 5W and 2H keyword carries out the selection of data target, is analyzed according to the data of selection: 5W why, what, who, when, Why where, 2H how, how much can trade according to client in the present embodiment, what traded, who is made that Transaction and exchange hour loco, mode of doing business, transaction cost etc. carry out data analysis as data target, are analyzing It can be standardized in the process with extraction time index by time normalizing, it is the date that exchange hour, which is unified format, or extracts ground Point index, is standardized by place normalizing, cities and counties and junior, cities and counties place is unified for province etc., the normalization condition of normalizing is not only It is limited to when and where, can be adjusted according to the expected results of financing manager, be obtained after Data Analysis Model is handled The standardized data of uniform format.Further, when transaction data unit is different or difference is larger, standard is carried out to data Change, removes the unit limitation of data, be converted into nondimensional pure values, it can be into convenient for the index of not commensurate or magnitude Row relatively and weighting, for example, select a number appropriate by determining base ratio method, can for sample initial value, average, in Digit, standard deviation or other given numerical value, are divisor divided by all samples, in order to eliminate the magnitude difference of dimension, are equipped with and hand over Easy data sample x1,x2,..xn, radix V determines base rate are as follows:
Specifically, visualization refer to data or image or calculate involved in, generate digital information become intuitive Indicated with image information, at any time with the physical phenomenon of spatial variations or be presented in face of manager.Synchrodata is passed through After the processing of Data Analysis Model normalized analysis, visualization mapping is carried out, synchrodata is mapped to diagrammatic form, can be column The chart of generation is pushed to financing manager by shape figure, curve graph and radar map etc., for example, when logical value is customer name, it can The transaction data of this client is mapped as radar map, the clear particularity for intuitively seeing this client trading product data and The relevance of the factors such as product type and time, place, obtains customer priorities product tendency, carries out Products Show to client.This Customer data can be provided in real time to financing manager by Data Analysis Model and Visual Chart in embodiment, be based on customer information Data analysis carries out product-specific investments suggestion to client, easy to use.
In one of the embodiments, as shown in figure 3, providing a kind of transaction data synchronization docking facilities, including it is following Module:
Transaction data obtains module 310, for acquiring each platform transaction data;
Cleaning module 320, for being cleaned according to data cleansing algorithm to data to be cleaned, data have been cleaned in acquisition;
Target data obtains module 330, for obtaining the logical value of input, obtains and logical value pair from having cleaned in data The target data answered;
Character read module 340, for reading the common ID character for having cleaned data and target data;
Synchronization module 350 obtains same for synchronizing processing to each platform transaction data according to common ID character Step data.
Transaction data obtains module 310 and is also used in one of the embodiments, identifies the transaction data that each platform is sent Type;If the JSON that transaction data type is internal platform formats transaction data type, default internal interface is called to carry out Acquisition;If transaction data type is internal platform unformatted transaction data type, using regular expression to internal platform Unformatted transaction data is parsed, and is generated JSON format transaction data and is acquired;If transaction data type is external platform When unformatted transaction data type, external platform unformatted transaction data is grabbed using web crawlers technology, passes through canonical Expression formula carries out dissection process to the external platform unformatted transaction data grabbed, generates JSON and formats transaction data, And it acquires.
Cleaning module 320 is also used in one of the embodiments, carries out each platform transaction data based on Cosine phase Like the repeated data cleaning of degree function, the missing data cleaning based on calorie enthesis, it is based onThe abnormal data of principle cleans And data have been cleaned in the noise data cleaning based on branch mailbox method, acquisition.
Target data obtains module 330 and is also used in one of the embodiments, receives the logical value of input;According to patrolling Volume value search it is corresponding with logical value cleaned data, record is corresponding with logical value to have cleaned data validity;According to effective Record obtains target data.
Synchronization module 350 is also used in one of the embodiments, according to common relation character, is calculated using correlation rule Method establishes data relationship net;It to target data and has cleaned data by data relationship net and has been associated, according to associated data, Processing is synchronized to target data, obtains synchrodata.
Specific restriction about transaction data synchronization docking facilities may refer to dock above for transaction data synchronization The restriction of method, details are not described herein.Modules in above-mentioned transaction data synchronization docking facilities can be fully or partially through Software, hardware and combinations thereof are realized.Above-mentioned each module can be embedded in the form of hardware or independently of the place in computer equipment It manages in device, can also be stored in a software form in the memory in computer equipment, in order to which processor calls execution above each The corresponding operation of a module.
A kind of computer equipment is provided in one of the embodiments, which can be server, in Portion's structure chart can be as shown in Figure 4.The computer equipment includes that the processor, memory, network connected by system bus connects Mouth and database.Wherein, the processor of the computer equipment is for providing calculating and control ability.The storage of the computer equipment Device includes non-volatile memory medium, built-in storage.The non-volatile memory medium be stored with operating system, computer program and Database.The built-in storage provides environment for the operation of operating system and computer program in non-volatile memory medium.It should The database of computer equipment is for depositing transaction data synchronization Interworking Data.The network interface of the computer equipment is used for and outside Terminal by network connection communication.The computer program realizes a kind of transaction data synchronization docking side when being executed by processor Method.
It will be understood by those skilled in the art that structure shown in Fig. 4, only part relevant to application scheme is tied The block diagram of structure does not constitute the restriction for the computer equipment being applied thereon to application scheme, specific computer equipment It may include perhaps combining certain components or with different component layouts than more or fewer components as shown in the figure.
A kind of computer equipment, including memory and processor are provided in one of the embodiments, are deposited in memory Computer program is contained, which performs the steps of each platform transaction data of acquisition when executing computer program;According to number Each platform transaction data is cleaned according to cleaning algorithm, data have been cleaned in acquisition;The logical value for obtaining input, from having cleaned number Target data corresponding with logical value is obtained according to middle;Read the common ID character for having cleaned data and target data;According to altogether With mark character, processing is synchronized to each platform transaction data, obtains synchrodata.
Each platform hair of identification is also performed the steps of when processor executes computer program in one of the embodiments, The transaction data type sent;If the JSON that transaction data type is internal platform formats transaction data type, call default Internal interface is acquired;If transaction data type is internal platform unformatted transaction data type, using regular expressions Formula parses internal platform unformatted transaction data, generates JSON format transaction data and acquires;If transaction data class When type is external platform unformatted transaction data type, external platform unformatted number of deals is grabbed using web crawlers technology According to by regular expression to the external platform unformatted transaction data progress dissection process grabbed, generation JSON format Change transaction data, and acquires.
It also performs the steps of when processor executes computer program in one of the embodiments, and trades to each platform Data carry out the cleaning of the repeated data based on Cosine similarity function, the missing data cleaning based on calorie enthesis, are based onThe abnormal data of principle cleans and data have been cleaned in the noise data cleaning based on branch mailbox method, acquisition.
It is also performed the steps of when processor executes computer program in one of the embodiments, and receives patrolling for input Collect value;According to logical value search it is corresponding with logical value cleaned data, record is corresponding with logical value, and to have cleaned data effective Property;According to effective record, target data is obtained.
It is also performed the steps of when processor executes computer program in one of the embodiments, when reception logical value When changing message, customized label is generated according to the logical value of change;According to customized label, effectively record is changed, is obtained more Target data after changing.
It also performs the steps of when processor executes computer program in one of the embodiments, according to common relation Character establishes data relationship net using association rule algorithm;To target data and data progress has been cleaned by data relationship net Association synchronizes processing to target data, obtains synchrodata according to associated data.
It is also performed the steps of when processor executes computer program in one of the embodiments, synchrodata is defeated Enter the Data Analysis Model based on normalization method building, place is normalized to synchrodata output format by Data Analysis Model Reason obtains the unified synchrodata of output format;Visualization mapping is carried out according to the unified synchrodata of output format, generation can Chart is analyzed depending on changing.
A kind of computer readable storage medium is provided in one of the embodiments, is stored thereon with computer program, Each platform transaction data of acquisition is performed the steps of when computer program execution processed;According to data cleansing algorithm to each flat Platform transaction data is cleaned, and data have been cleaned in acquisition;Obtain input logical value, from cleaned in data obtain and logical value Corresponding target data;Read the common ID character for having cleaned data and target data;According to common ID character, to each flat Platform transaction data synchronizes processing, obtains synchrodata.
Each platform of identification is also performed the steps of when computer program is executed by processor in one of the embodiments, The transaction data type of transmission;If the JSON that transaction data type is internal platform formats transaction data type, call pre- If internal interface is acquired;If transaction data type is internal platform unformatted transaction data type, using canonical table Internal platform unformatted transaction data is parsed up to formula, generate JSON format transaction data and is acquired;If transaction data When type is external platform unformatted transaction data type, using the crawl external platform unformatted transaction of web crawlers technology Data carry out dissection process to the external platform unformatted transaction data grabbed by regular expression, generate JSON lattice Formula transaction data, and acquire.
It is also performed the steps of when computer program is executed by processor in one of the embodiments, and each platform is handed over Easy data carry out the cleaning of the repeated data based on Cosine similarity function, the missing data cleaning based on calorie enthesis, base InThe abnormal data of principle cleans and data have been cleaned in the noise data cleaning based on branch mailbox method, acquisition.
It is also performed the steps of when computer program is executed by processor in one of the embodiments, and receives input Logical value;According to logical value search it is corresponding with logical value cleaned data, record is corresponding with logical value, which to have cleaned data, has Effect property;According to effective record, target data is obtained.
It is also performed the steps of when computer program is executed by processor in one of the embodiments, when reception logic When value change message, customized label is generated according to the logical value of change;According to customized label, effectively record is changed, is obtained Target data after change.
It also performs the steps of when computer program is executed by processor in one of the embodiments, and is closed according to common It is character, using association rule algorithm, establishes data relationship net;By data relationship net to target data and cleaned data into Row association synchronizes processing to target data, obtains synchrodata according to associated data.
It also performs the steps of when computer program is executed by processor in one of the embodiments, by synchrodata The Data Analysis Model constructed based on normalization method is inputted, synchrodata output format is normalized by Data Analysis Model Processing obtains the unified synchrodata of output format;Visualization mapping is carried out according to the unified synchrodata of output format, is generated Visual analyzing chart.
Those of ordinary skill in the art will appreciate that realizing all or part of the process in above-described embodiment method, being can be with Relevant hardware is instructed to complete by computer program, the computer program can be stored in a non-volatile computer In read/write memory medium, the computer program is when being executed, it may include such as the process of the embodiment of above-mentioned each method.Wherein, To any reference of memory, storage, database or other media used in each embodiment provided herein, Including non-volatile and/or volatile memory.Nonvolatile memory may include read-only memory (ROM), programming ROM (PROM), electrically programmable ROM (EPROM), electrically erasable ROM (EEPROM) or flash memory.Volatile memory may include Random access memory (RAM) or external cache.By way of illustration and not limitation, RAM is available in many forms, Such as static state RAM (SRAM), dynamic ram (DRAM), synchronous dram (SDRAM), double data rate sdram (DDRSDRAM), enhancing Type SDRAM (ESDRAM), synchronization link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic ram (DRDRAM) and memory bus dynamic ram (RDRAM) etc..
Each technical characteristic of embodiment described above can be combined arbitrarily, for simplicity of description, not to above-mentioned reality It applies all possible combination of each technical characteristic in example to be all described, as long as however, the combination of these technical characteristics is not deposited In contradiction, all should be considered as described in this specification.
The embodiments described above only express several embodiments of the present invention, and the description thereof is more specific and detailed, but simultaneously It cannot therefore be construed as limiting the scope of the patent.It should be pointed out that coming for those of ordinary skill in the art It says, without departing from the inventive concept of the premise, various modifications and improvements can be made, these belong to protection of the invention Range.Therefore, the scope of protection of the patent of the invention shall be subject to the appended claims.

Claims (10)

1. a kind of transaction data synchronization interconnection method, which is characterized in that the described method includes:
Acquire each platform transaction data;
Each platform transaction data is cleaned according to data cleansing algorithm, data have been cleaned in acquisition;
The logical value for obtaining input obtains target data corresponding with the logical value from described cleaned in data;
Read the common ID character for having cleaned data Yu the target data;
According to the common ID character, processing is synchronized to each platform transaction data, obtains synchrodata.
2. transaction data synchronization interconnection method according to claim 1, which is characterized in that each platform transaction data of acquisition Include:
Identify the transaction data type that each platform is sent;
If the JSON that the transaction data type is internal platform formats transaction data type, call default internal interface into Row acquisition;
If the transaction data type is internal platform unformatted transaction data type, using regular expression to described interior Portion's platform unformatted transaction data is parsed, and is generated JSON format transaction data and is acquired;
If the transaction data type is external platform unformatted transaction data type, institute is grabbed using web crawlers technology External platform unformatted transaction data is stated, by regular expression to the external platform unformatted number of deals grabbed According to dissection process is carried out, generates JSON and format transaction data, and acquire.
3. transaction data synchronization interconnection method according to claim 1, which is characterized in that described according to data cleansing algorithm pair Each platform transaction data is cleaned, and acquisition has cleaned data and included:
The cleaning of repeated data based on Cosine similarity function is carried out, based on calorie enthesis to each platform transaction data Missing data cleaning, be based onThe abnormal data of principle cleans and the noise data cleaning based on branch mailbox method, described in acquisition Data are cleaned.
4. transaction data synchronization interconnection method according to claim 1, which is characterized in that the logical value for obtaining input, It obtains corresponding with logical value target data from described cleaned in data and includes:
Receive the logical value of input;
According to the logical value search it is corresponding with the logical value it is described cleaned data, record is corresponding with the logical value The validity for having cleaned data;
According to effective record, the target data is obtained.
5. transaction data synchronization interconnection method according to claim 4, which is characterized in that the logical value for obtaining input, It has been cleaned in data after acquisition target data corresponding with the logical value from described further include:
When receiving the logical value change message, customized label is generated according to the logical value of change;
According to the customized label, effective record is changed, the target data after obtaining change.
6. transaction data synchronization interconnection method according to claim 1, which is characterized in that described according to the common ID word Symbol carries out data synchronization processing, obtains synchrodata and includes:
Data relationship net is established using association rule algorithm according to the common relation character;
The target data and the data of having cleaned are associated by the data relationship net, it is right according to associated data The target data synchronizes processing, obtains the synchrodata.
7. transaction data synchronization interconnection method according to claim 1, which is characterized in that described according to the common ID word It accords with, progress data synchronization processing, after acquisition synchrodata further include:
The synchrodata is inputted to the Data Analysis Model constructed based on normalization method, by the Data Analysis Model to described Synchrodata output format is normalized, and obtains the unified synchrodata of output format;
Visualization mapping is carried out according to the unified synchrodata of the output format, generates visual analyzing chart.
8. a kind of transaction data synchronization docking facilities, which is characterized in that described device includes:
Transaction data obtains module, for acquiring each platform transaction data;
Cleaning module, for being cleaned according to data cleansing algorithm to the data to be cleaned, data have been cleaned in acquisition;
Target data obtains module, for obtaining the logical value of input, obtains and the logical value from described cleaned in data Corresponding target data;
Character read module: for reading the common ID character for having cleaned data Yu the target data;
Synchronization module obtains same for synchronizing processing to each platform transaction data according to the common ID character Step data.
9. a kind of computer equipment, including memory and processor, the memory are stored with computer program, feature exists In the step of processor realizes any one of claims 1 to 7 the method when executing the computer program.
10. a kind of computer readable storage medium, is stored thereon with computer program, which is characterized in that the computer program The step of method described in any one of claims 1 to 7 is realized when being executed by processor.
CN201910184626.5A 2019-03-12 2019-03-12 Transaction data synchronization interconnection method, device, computer equipment and storage medium Pending CN110008282A (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN201910184626.5A CN110008282A (en) 2019-03-12 2019-03-12 Transaction data synchronization interconnection method, device, computer equipment and storage medium

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN201910184626.5A CN110008282A (en) 2019-03-12 2019-03-12 Transaction data synchronization interconnection method, device, computer equipment and storage medium

Publications (1)

Publication Number Publication Date
CN110008282A true CN110008282A (en) 2019-07-12

Family

ID=67166853

Family Applications (1)

Application Number Title Priority Date Filing Date
CN201910184626.5A Pending CN110008282A (en) 2019-03-12 2019-03-12 Transaction data synchronization interconnection method, device, computer equipment and storage medium

Country Status (1)

Country Link
CN (1) CN110008282A (en)

Cited By (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN111680083A (en) * 2020-04-30 2020-09-18 四川弘智远大科技有限公司 Intelligent multi-stage government financial data acquisition system and data acquisition method
CN112287411A (en) * 2020-11-05 2021-01-29 南京中泾数据系统有限公司 Storage array type data crushing device

Citations (7)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN104112207A (en) * 2014-07-29 2014-10-22 浪潮软件集团有限公司 Electronic commerce transaction monitoring method based on internet data
CN104966172A (en) * 2015-07-21 2015-10-07 上海融甸信息科技有限公司 Large data visualization analysis and processing system for enterprise operation data analysis
CN107633077A (en) * 2017-09-25 2018-01-26 南京安链数据科技有限公司 A kind of system and method for more strategy cleaning social media text datas
CN107784074A (en) * 2017-09-25 2018-03-09 平安科技(深圳)有限公司 The recognition methods of connected transaction, device and, computer equipment and storage medium
US10019650B1 (en) * 2017-11-28 2018-07-10 Bank Of America Corporation Computer architecture for emulating an asynchronous correlithm object processing system
CN108710665A (en) * 2018-05-15 2018-10-26 阿里巴巴集团控股有限公司 Data reflow method, device, system and equipment
CN109388634A (en) * 2018-09-18 2019-02-26 平安科技(深圳)有限公司 Processing method, terminal device and the computer readable storage medium of address information

Patent Citations (7)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN104112207A (en) * 2014-07-29 2014-10-22 浪潮软件集团有限公司 Electronic commerce transaction monitoring method based on internet data
CN104966172A (en) * 2015-07-21 2015-10-07 上海融甸信息科技有限公司 Large data visualization analysis and processing system for enterprise operation data analysis
CN107633077A (en) * 2017-09-25 2018-01-26 南京安链数据科技有限公司 A kind of system and method for more strategy cleaning social media text datas
CN107784074A (en) * 2017-09-25 2018-03-09 平安科技(深圳)有限公司 The recognition methods of connected transaction, device and, computer equipment and storage medium
US10019650B1 (en) * 2017-11-28 2018-07-10 Bank Of America Corporation Computer architecture for emulating an asynchronous correlithm object processing system
CN108710665A (en) * 2018-05-15 2018-10-26 阿里巴巴集团控股有限公司 Data reflow method, device, system and equipment
CN109388634A (en) * 2018-09-18 2019-02-26 平安科技(深圳)有限公司 Processing method, terminal device and the computer readable storage medium of address information

Cited By (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN111680083A (en) * 2020-04-30 2020-09-18 四川弘智远大科技有限公司 Intelligent multi-stage government financial data acquisition system and data acquisition method
CN112287411A (en) * 2020-11-05 2021-01-29 南京中泾数据系统有限公司 Storage array type data crushing device
CN112287411B (en) * 2020-11-05 2023-09-12 南京中泾数据系统有限公司 Storage array type data crushing device

Similar Documents

Publication Publication Date Title
WO2019218699A1 (en) Fraud transaction determining method and apparatus, computer device, and storage medium
CN109345282A (en) A kind of response method and equipment of business consultation
CN109033132B (en) Method and device for calculating text and subject correlation by using knowledge graph
CN112015721A (en) E-commerce platform storage database optimization method based on big data
JP2009151760A (en) Method and system for calculating competitiveness metric between objects
US20210117978A1 (en) Graph decomposition for fraudulent transaction analysis
US20210366055A1 (en) Systems and methods for generating accurate transaction data and manipulation
CN111651552B (en) Structured information determining method and device and electronic equipment
WO2022156525A1 (en) Object matching method and apparatus, and device
CN111199474A (en) Risk prediction method and device based on network diagram data of two parties and electronic equipment
CN112509661B (en) Methods, computing devices, and media for identifying physical examination reports
CN110162681A (en) Text identification, text handling method, device, computer equipment and storage medium
CN110363206B (en) Clustering of data objects, data processing and data identification method
CN105303447A (en) Method and device for carrying out credit rating through network information
CN113434628A (en) Comment text confidence detection method based on feature level and propagation relation network
CN110008282A (en) Transaction data synchronization interconnection method, device, computer equipment and storage medium
CN109902129B (en) Insurance agent classifying method and related equipment based on big data analysis
CN114118816A (en) Risk assessment method, device and equipment and computer storage medium
CN113887214A (en) Artificial intelligence based wish presumption method and related equipment thereof
CN114357184A (en) Item recommendation method and related device, electronic equipment and storage medium
CN112446777B (en) Credit evaluation method, device, equipment and storage medium
Prematilake et al. Evaluation and prediction of polygon approximations of planar contours for shape analysis
CN113343700B (en) Data processing method, device, equipment and storage medium
EP3493082A1 (en) A method of exploring databases of time-stamped data in order to discover dependencies between the data and predict future trends
CN115758271A (en) Data processing method, data processing device, computer equipment and storage medium

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
WD01 Invention patent application deemed withdrawn after publication
WD01 Invention patent application deemed withdrawn after publication

Application publication date: 20190712