CN104933112A - Distributed Internet transaction information storage and processing method - Google Patents
Distributed Internet transaction information storage and processing method Download PDFInfo
- Publication number
- CN104933112A CN104933112A CN201510302559.4A CN201510302559A CN104933112A CN 104933112 A CN104933112 A CN 104933112A CN 201510302559 A CN201510302559 A CN 201510302559A CN 104933112 A CN104933112 A CN 104933112A
- Authority
- CN
- China
- Prior art keywords
- data
- distributed
- warehouse
- transaction information
- fairground
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/28—Databases characterised by their database models, e.g. relational or object models
- G06F16/283—Multi-dimensional databases or data warehouses, e.g. MOLAP or ROLAP
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/25—Integrating or interfacing systems involving database management systems
- G06F16/254—Extract, transform and load [ETL] procedures, e.g. ETL data flows in data warehouses
Abstract
The invention provides a distributed Internet transaction information storage and processing method. The method comprises the following steps that: an operation data storage function module extracts structured transaction information data from an external data layer; a distributed data warehouse and a non-relational database read non-structured network log data from a distributed file system respectively; an Internet transaction information data warehouse extracts, transforms and loads the transaction information data and the network log data from the operation data storage function module and the distributed data warehouse, integrates the transaction information data and the network log data into a data tuple, and stores the data tuple with a star model; a data market extracts, transforms and loads market-specified data from the Internet transaction information data warehouse and/or the distributed data warehouse; and a business intelligence system acquires data required by analysis from the data market and/or the non-relational database according to data analysis requirements. The method is specific to the application scene of Internet transaction data analysis and processing.
Description
Technical field
The present invention relates to a kind of distributed interconnection Transaction Information storage processing method.
Background technology
The model of Data Analysis Services and result only apply to could produce real value in some decision-making application, and to know by carrying out aid decision making based on the back-up system of the fact etc. that some serial Theories and methods are business intelligence (Bussiness Intelligence are called for short BI).Along with information-based high development, business intelligence is more and more taken seriously, especially associating on-line analysis OLAP(is namely based on the online express-analysis of database, large data), be the main methods of business intelligence especially in a way, the core data warehouse architecture design in the BI solution of traditional industries as shown in Figure 1.
But internet business information data is not suitable for adopting BI solution, main cause have following some:
1. the data source of internet electronic business transaction is varied, comprises the data and network log data etc. of different electric business's platforms, various relevant database, the generation of social software; So single ETL instrument cannot handle all data well.
2. the data volume of e-commerce transaction is huge, and traditional centralized relevant database cannot meet the requirement processing large-scale data like this.
3. internet business monitoring is higher to the requirement of real-time of Data Analysis Services, and most monitoring and warning needs to complete the value that process could embody data at short notice, and traditional off-line ETL processing mode cannot satisfy the demands.
4. the number of users of internet business is huge, and traditional BI represents tool design mainly for the user on the middle and senior level of enterprise, is transplanted to e-commerce industry and all there is larger difference from interactive efficiency and Consumer's Experience.The business intelligence system of current traditional mode, can not well be applicable to the extensive of internet business information, heterogeneous data source and the demand to data analysis real-time.
Summary of the invention
The present invention is directed to the application scenarios of internet business Data Analysis Services, in conjunction with on the feature basis of internet electronic business analysis and early warning, for defect of the prior art, the object of this invention is to provide distributed interconnection Transaction Information storage processing method.
According to a kind of distributed interconnection Transaction Information storage processing method provided by the invention, comprising:
Operation data storage function module is from the trading information data of external data layer drawing-out structure;
Distributed Data Warehouse and non-relational database read non-structured network log data respectively from distributed file system;
Internet business information data warehouse extracts respectively after conversion loads described trading information data and network log data and carries out being integrated into data tuple from described operation data storage function module and described Distributed Data Warehouse, and stores this data tuple with Star Model;
Data Mart root from described internet business information data warehouse, and/or extracts conversion loading fairground specific data in described Distributed Data Warehouse;
Business intelligence system needs from described Data Mart according to data analysis, and/or obtains analysis desired data in described non-relational database.
As a kind of prioritization scheme, described Data Mart comprises exchanges, sells fairground and customer service fairground;
Described exchanges, sale fairground are extracted conversion according to service needed from described internet business information data warehouse and are loaded described fairground specific data;
Described customer service fairground is extracted conversion according to service needed and is loaded described fairground specific data from described Distributed Data Warehouse.
As a kind of prioritization scheme, between described customer service fairground and described business intelligence system, also carry out data transmission through MemCache caching system;
Described business intelligence system first checks data needed for asked analysis whether in MemCache caching system when obtaining analyze desired data to described customer service fairground, if have, then directly to obtain from MemCache caching system, if do not exist, then obtain from described customer service fairground analyze desired data and MemCache caching system buffer memory a.
As a kind of prioritization scheme, described business intelligent system is used for data mining, enterprise diagnosis, customer analysis, data file analysis and on-line analysis.
As a kind of prioritization scheme, described operation data storage function module comprises three-decker:
Mapping layer: the field mappings of former for the data of external data layer table in the local data base of operation data storage function module, complete the association of data from operation layer to analysis layer;
Data prediction layer: pre-service is carried out to described trading information data, this pre-service comprises integration, screening and increases contingency table;
Data cleansing layer: data cleansing operation is carried out for defective in quality trading information data.
As a kind of prioritization scheme, the fact table that described Star Model is positioned at star centre comprise some data tuple time address, number of transaction corresponding to domain addresses, store address, product address and this data tuple and dealing money;
The dimension table of described Star Model comprises shop dimension table, time dimension table, product dimension table and region dimension table;
Described shop dimension table comprises electric business's platform information, landing slab block message and shop management information;
Described time dimension table comprises temporal information;
Described product dimension table comprises name of product, product description, product price and product quality;
Described region dimension table comprises geographical location information, and this geographical location information comprises country, province, city.
As a kind of prioritization scheme, described operation data storage function module uses full dose load mode, is specially:
S101, empties the object table of described operation data storage function module,
S102, inserts this object table by the full dose trading information data of external data layer.
As a kind of prioritization scheme, described operation data storage function module uses step increment method mode, is specially:
S201, empties the temporary table of described operation data storage function module,
S202, inserts this temporary table by the increment trading information data of external data layer,
S203, the data that deletion object table and this temporary table repeat,
Data in temporary table are inserted after in object table and are returned step S1 until data all extract end by S204.
As a kind of prioritization scheme, the dimension table in described internet business information data warehouse is step increment method mode, and fact table is full dose load mode;
The dimension table step increment method process in internet business information data warehouse is specially:
S301, calculates sequence to incremental data according to the line number of dimension table historical data in temporary table,
S302, inserts temporary table by the incremental data with described sequence,
S303, empties the dimension table in internet business information data warehouse,
S304, associates the tables of data of described operation data storage function module with described temporary table, the described incremental data with described sequence is inserted the dimension table in internet business information data warehouse;
The step increment method process of fact table is specially:
After emptying temporary table, judge whether that index takes from different fact tables in addition;
If also have index to take from different fact tables, then repeat the process of the different business event of taking out in fact table within the scope of timestamp, terminate until all business event are all removed Posterior circle;
If do not have index to take from different fact tables, then take out the different business event in fact table within the scope of timestamp, the data of the first temporary table are aggregated in the second temporary table according to dimension field, delete the data repeated with this second temporary table in target fact table, the data of described second temporary table are inserted target fact table.
Compared with prior art, the present invention has following beneficial effect:
The present invention is directed to internet electronic business Transaction Information feature, design application oriented, integrated, that there is temporal characteristics, stable data acquisition, for transaction data process, analysis and monitoring decision-making provide support.
Accompanying drawing explanation
In order to be illustrated more clearly in the technical scheme of the embodiment of the present invention, below the accompanying drawing used required in describing embodiment is briefly described, obviously, accompanying drawing in the following describes is only some embodiments of the present invention, for those skilled in the art, under the prerequisite not paying creative work, other accompanying drawing can also be obtained according to these accompanying drawings.In accompanying drawing:
Fig. 1 is the core data warehouse Organization Chart in the BI solution of traditional industries;
Fig. 2 is a kind of distributed interconnection Transaction Information storage architecture schematic diagram in embodiment;
Fig. 3 is the principle framework of data warehouse;
Fig. 4 is a kind of distributed interconnection Transaction Information storage processing method schematic diagram in embodiment;
Fig. 5 is data warehouse Star Model schematic diagram;
Fig. 6 is that operation data storage function module (ODS) full dose loads process flow diagram;
Fig. 7 is operation data storage function module (ODS) step increment method process flow diagram;
Fig. 8 is the dimension table step increment method process flow diagram in internet business information data warehouse (DW);
Fig. 9 is fact table (DM) the step increment method process flow diagram of DW;
Figure 10 is operation data storage function module ODS, relation schematic diagram between internet business information data warehouse DW, fact table DM.
Embodiment
Hereafter in the mode of specific embodiment, the present invention is described in detail by reference to the accompanying drawings.Following examples will contribute to those skilled in the art and understand the present invention further, but not limit the present invention in any form.It should be pointed out that the embodiment that can also use other, or the amendment on 26S Proteasome Structure and Function is carried out to the embodiment enumerated herein, and can not depart from the scope and spirit of the present invention.
The present invention is directed to the application scenarios of internet business Data Analysis Services, in conjunction with on the feature basis of internet electronic business analysis and early warning, mainly comprise leading portion Reports module, ETL module, data warehouse module, database management module, data dispatch module and Web service module composition.The present invention is directed to internet electronic business Transaction Information feature, design application oriented, integrated, that have temporal characteristics, stable data acquisition, for transaction data process, analysis and monitoring decision-making provide support, overall architecture as shown in Figure 2.
In the embodiment of a kind of distributed interconnection Transaction Information storage processing method provided by the invention, as shown in Figure 2 and Figure 4, comprising:
Operation data storage function module is from the trading information data of external data layer drawing-out structure;
Distributed Data Warehouse and non-relational database read non-structured network log data respectively from distributed file system;
Internet business information data warehouse extracts respectively after conversion loads described trading information data and network log data and carries out being integrated into data tuple from described operation data storage function module and described Distributed Data Warehouse, and stores this data tuple with Star Model;
Data Mart root from described internet business information data warehouse, and/or extracts conversion loading fairground specific data in described Distributed Data Warehouse;
Business intelligence system needs from described Data Mart according to data analysis, and/or obtains analysis desired data in described non-relational database.
Described extraction conversion is loaded as ETL, is the abbreviation of English Extract-Transform-Load, is used for describing data are passed through extraction (extract) from source terminal, change (transform), loaded (load) process to destination.
Along with the development of ecommerce, need to carry out on-line analysis to the trading activity of user in real time, such as show all history access and the inquiry record of certain shop on net, simultaneously real-time tracing shows this shop on electric business's platform just in information such as the accessed pages, adopts the relevant database of traditional support off-line analysis and complex query to be difficult to satisfied such demand.Simultaneously, the data mining process of the semi-structured large data processing combination complexity that the increasing web log file of e-commerce industry, user behavior are such, therefore the present invention adopts the large data platform of power stone science and technology or Hadoop to realize mass data volume handling implement.The large data processing core module of power stone science and technology mainly comprises cloud database, cloud storage, search engine and data analysis, structuring, semi-structured, unstructured data can be processed, support standard interface, provide the data of one-stop robotization to dispose, move, back up, recovery, the function such as disaster tolerance.The principle framework of data warehouse as shown in Figure 3.
The data of data warehouse, by the data-interface of standard, are derived from internet electronic business transaction platform and open to applications.Data warehouse is divided into three-decker according to data flow: data Layer, Information Level and analysis layer, as shown in Figure 4.
Data Layer
By the standard data interface of propelling movement type, use the external data obtaining electric business's platform with the mode of the consistent model of electric business's platform database or middle table, then carry out data pick-up data grabber in other words by ODS, the form of extraction comprises XML and TXT etc.
Information Level
Operation data storage function module (ODS:Operation Data Storage) is increased in the middle of data Layer and internet business information data depot layer.Object is as a buffer pool, by the data integration of multiple data source a to extra buffer for data warehouse, effectively alleviate the pressure of data source and ETL.
Wherein, ODS comprises three-decker:
Mapping layer: the field mappings of former for the data of external data layer table in the local data base of operation data storage function module, complete the association of data from operation layer to analysis layer is also manage mapping layer by system to concentrate for external data.
Data prediction layer: pre-service is carried out to described trading information data, this pre-service comprises integration, screening and increases contingency table, and object is the work simplifying and promote ETL;
Data cleansing: data cleansing operation is carried out for defective in quality trading information data.
What ODS stored is all the internet business information data captured from all kinds of electric business's platform.
Analysis layer
By BI system and Hadoop instrument, the process such as data mining, enterprise diagnosis, customer analysis, Data support and on-line analysis are carried out to the trading information data of all kinds of electric business's platform and non-structured web log file.Described business intelligent system is used for data mining, enterprise diagnosis, customer analysis, data file analysis and on-line analysis.
The easy information data warehouse in internet for stores processor structuring trading information data adopts the synthesis of relational database, memory database and distributed data base, adopts applicable traditional BI service to carry out analyzing and processing for relational database; For the non-relational database (Nosql) based on the storage of a large amount of real time datas and the Hadoop distributed file system HDFS of real-time query analysis employing support HBase.
In embodiment as shown in Figure 4, described Data Mart comprises exchanges, sells fairground and customer service fairground;
Described exchanges, sale fairground are extracted conversion according to service needed from described internet business information data warehouse and are loaded described fairground specific data;
Described customer service fairground is extracted conversion according to service needed and is loaded described fairground specific data from described Distributed Data Warehouse.
Described exchanges are used for the transaction related information in stores processor buyer and shop, as exchange hour, number of transaction etc.
Described sale fairground is used for the sale related data in stores processor shop, as shop visit capacity, sales situation etc.
The calling information that described customer service fairground is used for stores processor buyer is mutual with the both sides in transaction.
Distributed Data Warehouse in the present embodiment is power stone cloud database, the distributed relation database all-in-one of high-performance, High Availabitity is provided, OLAP, OLTP and Combination application can be supported, support high-performance (distributed), High Availabitity, support that thermophoresis, Hot Spare, heat are recovered, support stsndard SQL, support main flow development language, support based on x86, Godson, soar, the chip server such as PowerPC, low to hardware requirement.
Also can adopt HBase, i.e. Hadoop Database, be one high
reliablyproperty, high-performance, towards row, telescopic
distributed memory system.
In Fig. 4, Distributed Data Warehouse and non-relational database read non-structured network log data respectively from distributed file system.When customer service needs to transfer transaction record and transaction data corresponding with it, directly extracted by ETL mode from HBASE, and if third-party business intelligence system needs to carry out statistical study to internet business data, without the need to detailed Transaction Information, then the direct network log data obtaining all kinds of electric business's platform from described NOSQL.Which thereby enhance travelling speed, make system storage process more efficient.
As a kind of embodiment, between described customer service fairground and described business intelligence system, also carry out data transmission through MemCache caching system;
Described business intelligence system first checks data needed for asked analysis whether in MemCache caching system when obtaining analyze desired data to described customer service fairground, if have, then directly to obtain from MemCache caching system, if do not exist, then obtain from described customer service fairground analyze desired data and MemCache caching system buffer memory a.
Memcache is a high performance distributed memory object caching system, and by safeguarding the huge hash table of a unification in internal memory, it can be used for storing the data of various form, comprises the result etc. of image, video, file and database retrieval.Be exactly briefly by data call in internal memory, then read from internal memory, thus greatly improve reading speed.
The metadata store pattern in internet business information data warehouse adopts the Star Model being applicable to dimension and being separated with the fact, as shown in Figure 5.Data are through pre-service, and the dimensional information about the fact has detached out and has been based upon in corresponding dimension table from the fact.Treatment scheme is: ODS layer is drawn into about the relevant data of operation flow from operation layer; Data Layer design (mainly comprising the common dimension such as time dimension table, region dimension table) is carried out according to business function at depot layer (DW layer); Fact table (DM layer) is the fact of historical data, does not have and repeats operational, and correspond to business association relation table, prestige table, visit capacity etc., fact table is the core of star structure, the trunk content of record main body.
In the embodiment as shown in fig .5, the fact table that described Star Model is positioned at star centre comprise some data tuple time address, number of transaction corresponding to domain addresses, store address, product address and this data tuple and dealing money;
The dimension table of described Star Model comprises shop dimension table, time dimension table, product dimension table and region dimension table;
Described shop dimension table comprises electric business's platform information, landing slab block message and shop management information;
Described time dimension table comprises temporal information;
Described product dimension table comprises name of product, product description, product price and product quality;
Described region dimension table comprises geographical location information, and this geographical location information comprises country, province, city.
As a kind of embodiment, as shown in Figure 6, described operation data storage function module uses full dose load mode, is specially:
S101, empties the object table of described operation data storage function module,
S102, inserts this object table by the full dose trading information data of external data layer.
As a kind of embodiment, as shown in Figure 7, described operation data storage function module uses step increment method mode, is specially:
S201, empties the temporary table of described operation data storage function module,
S202, inserts this temporary table by the increment trading information data of external data layer,
S203, the data that deletion object table and this temporary table repeat,
Data in temporary table are inserted after in object table and are returned step S1 until data all extract end by S204.
As a kind of embodiment, as shown in Figure 8, the dimension table in described internet business information data warehouse is step increment method mode, and fact table is full dose load mode;
The dimension table step increment method process in internet business information data warehouse is specially:
S301, calculates sequence to incremental data according to the line number of dimension table historical data in temporary table,
S302, inserts temporary table by the incremental data with described sequence,
S303, empties the dimension table in internet business information data warehouse,
S304, associates the tables of data of described operation data storage function module with described temporary table, the described incremental data with described sequence is inserted the dimension table in internet business information data warehouse.
As a kind of embodiment, as shown in Figure 9, the step increment method process of fact table is specially:
After emptying temporary table, judge whether that index takes from different fact tables in addition;
If also have index to take from different fact tables, then repeat the process of the different business event of taking out in fact table within the scope of timestamp, terminate until all business event are all removed Posterior circle;
If do not have index to take from different fact tables, then take out the different business event in fact table within the scope of timestamp, the data of the first temporary table are aggregated in the second temporary table according to dimension field, delete the data repeated with this second temporary table in target fact table, the data of described second temporary table are inserted target fact table.
One is all also comprised by the step of the situation of insertion writing system daily record after all kinds of full dose loading of the present embodiment intelligence or step increment method complete.
Adopt Multi-dimension on-line analytical process (OLAP:Online Analysis Processing) to come according to different business demand, from different demand angle (as visual angles such as sale, customer service, finance, time, region, industries), alternate analysis is carried out to the related data from other data structure.By presenting the analysis of data and the multidimensional of leading portion system, realize internet business information display and real-time trend analysis early warning.First dimension and indication information is extracted; Secondly because dimension between each main body is all separate when defining, in order to consistance and the incidence relation of data, will set up the relation information of each analysis personnel dimension, realization body is expanded and is associated; After establishing data model, utilize ETL that the data in data warehouse are carried out corresponding statistical summaries according to customer demand and obtain multidimensional analysis data, finally form form.
The present invention devises rational system architecture and the design of the ETL adapted, data warehouse and data dimension, makes it be applicable to the stores processor of internet business information.
Consider and the data of the electric business's platform of difference are processed and stored, so the demand of application layer is different; Be not only that unitem can be used to the changes in demand of user simultaneously, so need to take into account the demand of search efficiency and data dynamics and good extensibility simultaneously, adopt star-like Multidimensional Data Model as data warehouse model.
Through pretreated data, the dimensional information about fact table extracted from the fact is based upon in corresponding dimension table.Therefore, processing layer functional module only needs to carry out inquiry to fact table just can obtain Transaction Information, substantially increases the efficiency of access.Operation data storage function module ODS, relation schematic diagram between internet business information data warehouse DW, fact table DM are as shown in Figure 10.
ODS layer correspondence is drawn into the data about internet business main body and behavior from data Layer; DW layer is data warehouse layer, main store to tie up correlation time with internet business show, region ties up and the public information such as to show and tie up and show; DM corresponding fact table, the namely core of hub-and-spoke configuration, the information such as the incidence relation of record transaction.
ETL update mechanism, warehouse upgrades at first, and fairground is in renewal; Dimension first upgrades, the true mechanism upgraded again.
The foregoing is only preferred embodiment of the present invention, those skilled in the art know, without departing from the spirit and scope of the present invention, can carry out various change or equivalent replacement to these characteristic sum embodiments.In addition, under the teachings of the present invention, can modify to adapt to concrete situation and material to these characteristic sum embodiments and can not the spirit and scope of the present invention be departed from.Therefore, the present invention is not by the restriction of specific embodiment disclosed herein, and the embodiment in the right of all the application of falling into all belongs to protection scope of the present invention.
Claims (9)
1. a distributed interconnection Transaction Information storage processing method, is characterized in that, comprising:
Operation data storage function module is from the trading information data of external data layer drawing-out structure;
Distributed Data Warehouse and non-relational database read non-structured network log data respectively from distributed file system;
Internet business information data warehouse extracts respectively after conversion loads described trading information data and network log data and carries out being integrated into data tuple from described operation data storage function module and described Distributed Data Warehouse, and stores this data tuple with Star Model;
Data Mart from described internet business information data warehouse, and/or extracts conversion loading fairground specific data in described Distributed Data Warehouse;
Business intelligence system needs from described Data Mart according to data analysis, and/or obtains analysis desired data in described non-relational database.
2. a kind of distributed interconnection Transaction Information storage processing method according to claim 1, is characterized in that, described Data Mart comprises exchanges, sells fairground and customer service fairground;
Described exchanges, sale fairground are extracted conversion according to service needed from described internet business information data warehouse and are loaded described fairground specific data;
Described customer service fairground is extracted conversion according to service needed and is loaded described fairground specific data from described Distributed Data Warehouse.
3. a kind of distributed interconnection Transaction Information storage processing method according to claim 2, is characterized in that, also carry out data transmission through MemCache caching system between described customer service fairground and described business intelligence system;
Described business intelligence system first checks data needed for asked analysis whether in MemCache caching system when obtaining analyze desired data to described customer service fairground, if have, then directly to obtain from MemCache caching system, if do not exist, then obtain from described customer service fairground analyze desired data and MemCache caching system buffer memory a.
4. a kind of distributed interconnection Transaction Information storage processing method according to claim 1, is characterized in that, described business intelligent system is used for data mining, enterprise diagnosis, customer analysis, data file analysis and on-line analysis.
5. a kind of distributed interconnection Transaction Information storage processing method according to claim 1, it is characterized in that, described operation data storage function module comprises three-decker:
Mapping layer: the field mappings of former for the data of external data layer table in the local data base of operation data storage function module, complete the association of data from operation layer to analysis layer;
Data prediction layer: pre-service is carried out to described trading information data, this pre-service comprises integration, screening and increases contingency table;
Data cleansing layer: data cleansing operation is carried out for defective in quality trading information data.
6. a kind of distributed interconnection Transaction Information storage processing method according to claim 1, it is characterized in that, the fact table that described Star Model is positioned at star centre comprise some data tuple time address, number of transaction corresponding to domain addresses, store address, product address and this data tuple and dealing money;
The dimension table of described Star Model comprises shop dimension table, time dimension table, product dimension table and region dimension table;
Described shop dimension table comprises electric business's platform information, landing slab block message and shop management information;
Described time dimension table comprises temporal information;
Described product dimension table comprises name of product, product description, product price and product quality;
Described region dimension table comprises geographical location information, and this geographical location information comprises country, province, city.
7. a kind of distributed interconnection Transaction Information storage processing method according to claim 1, is characterized in that, described operation data storage function module uses full dose load mode, is specially:
S101, empties the object table of described operation data storage function module,
S102, inserts this object table by the full dose trading information data of external data layer.
8. a kind of distributed interconnection Transaction Information storage processing method according to claim 1, is characterized in that, described operation data storage function module uses step increment method mode, is specially:
S201, empties the temporary table of described operation data storage function module,
S202, inserts this temporary table by the increment trading information data of external data layer,
S203, the data that deletion object table and this temporary table repeat,
Data in temporary table are inserted after in object table and are returned step S1 until data all extract end by S204.
9. a kind of distributed interconnection Transaction Information storage processing method according to claim 1, is characterized in that, the dimension table in described internet business information data warehouse is step increment method mode, and fact table is full dose load mode;
The dimension table step increment method process in internet business information data warehouse is specially:
S301, calculates sequence to incremental data according to the line number of dimension table historical data in temporary table,
S302, inserts temporary table by the incremental data with described sequence,
S303, empties the dimension table in internet business information data warehouse,
S304, associates the tables of data of described operation data storage function module with described temporary table, the described incremental data with described sequence is inserted the dimension table in internet business information data warehouse;
The step increment method process of fact table is specially:
After emptying temporary table, judge whether that index takes from different fact tables in addition;
If also have index to take from different fact tables, then repeat the process of the different business event of taking out in fact table within the scope of timestamp, terminate until all business event are all removed Posterior circle;
If do not have index to take from different fact tables, then take out the different business event in fact table within the scope of timestamp, the data of the first temporary table are aggregated in the second temporary table according to dimension field, delete the data repeated with this second temporary table in target fact table, the data of described second temporary table are inserted target fact table.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201510302559.4A CN104933112B (en) | 2015-06-04 | 2015-06-04 | Distributed interconnection Transaction Information storage processing method |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201510302559.4A CN104933112B (en) | 2015-06-04 | 2015-06-04 | Distributed interconnection Transaction Information storage processing method |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN104933112A true CN104933112A (en) | 2015-09-23 |
| CN104933112B CN104933112B (en) | 2018-12-21 |
Family
ID=54120280
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201510302559.4A Active CN104933112B (en) | 2015-06-04 | 2015-06-04 | Distributed interconnection Transaction Information storage processing method |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN104933112B (en) |
Cited By (28)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN105320757A (en) * | 2015-10-19 | 2016-02-10 | 杭州华量软件有限公司 | Business intelligent analysis method for quickly processing data |
| CN105590259A (en) * | 2015-11-04 | 2016-05-18 | 中国银联股份有限公司 | Device and method for diagnosis of transaction system |
| CN105589940A (en) * | 2015-12-16 | 2016-05-18 | 南京联成科技发展有限公司 | Safety management operation and maintenance service platform based on unstructured real-time database |
| CN105653696A (en) * | 2015-12-29 | 2016-06-08 | 台山核电合营有限公司 | Data processing method and system for nuclear power plant databases |
| CN105787660A (en) * | 2016-02-24 | 2016-07-20 | 国家电网公司 | Information management system for photovoltaic power distribution network |
| JP2016170778A (en) * | 2015-03-10 | 2016-09-23 | 技研商事インターナショナル株式会社 | Market area analysis system |
| CN106227862A (en) * | 2016-07-29 | 2016-12-14 | 浪潮软件集团有限公司 | E-commerce data integration method based on distribution |
| CN106934023A (en) * | 2017-03-13 | 2017-07-07 | 山东浪潮云服务信息科技有限公司 | A kind of data managing method and device |
| CN107832392A (en) * | 2017-10-31 | 2018-03-23 | 链家网（北京）科技有限公司 | A kind of metadata management system |
| CN107944866A (en) * | 2017-10-17 | 2018-04-20 | 厦门市美亚柏科信息股份有限公司 | Transaction record rearrangement and computer-readable recording medium |
| CN107958046A (en) * | 2017-11-24 | 2018-04-24 | 小花互联网金融服务（深圳）有限公司 | Internet finance big data warehouse analysis mining method |
| CN108280084A (en) * | 2017-01-06 | 2018-07-13 | 上海前隆信息科技有限公司 | A kind of construction method of data warehouse, system and server |
| CN108595685A (en) * | 2018-05-04 | 2018-09-28 | 北京顶象技术有限公司 | A kind of data processing method and device |
| CN108733758A (en) * | 2018-04-11 | 2018-11-02 | 北京三快在线科技有限公司 | Hotel's static data method for pushing, device, electronic equipment and readable storage medium storing program for executing |
| CN109189861A (en) * | 2018-06-29 | 2019-01-11 | 深圳市彬讯科技有限公司 | Data stream statistics method, server and storage medium based on index |
| CN109325648A (en) * | 2018-06-29 | 2019-02-12 | 深圳市彬讯科技有限公司 | Multi-dimensional data stream statistics method, server and storage medium based on index |
| CN109656910A (en) * | 2018-12-06 | 2019-04-19 | 哈尔滨工业大学 | Expansible Large Scale Biology medicine sample management and Visualization Platform |
| CN111581254A (en) * | 2020-05-03 | 2020-08-25 | 上海维信荟智金融科技有限公司 | ETL method and system based on internet financial data |
| CN112256523A (en) * | 2020-09-23 | 2021-01-22 | 贝壳技术有限公司 | Service data processing method and device |
| CN112380218A (en) * | 2020-11-18 | 2021-02-19 | 浪潮天元通信信息系统有限公司 | ETL-based automatic triggering method for summarizing data tables of data warehouse layers |
| CN112395345A (en) * | 2020-12-04 | 2021-02-23 | 江苏苏宁云计算有限公司 | HBase full data import method and device, computer equipment and storage medium |
| CN112416630A (en) * | 2020-12-10 | 2021-02-26 | 湖南新云网科技有限公司 | Data flow architecture and data processing method |
| CN112650738A (en) * | 2020-12-31 | 2021-04-13 | 广西中科曙光云计算有限公司 | Construction method of open database |
| CN112947844A (en) * | 2019-12-11 | 2021-06-11 | 北京金山云网络技术有限公司 | Data storage method and device, electronic equipment and medium |
| CN113362018A (en) * | 2021-05-25 | 2021-09-07 | 北京明略软件系统有限公司 | Conference time processing method and system |
| CN113515362A (en) * | 2021-07-12 | 2021-10-19 | 广州云从洪荒智能科技有限公司 | Data processing method, data processing device, computer equipment and storage medium |
| CN113742320A (en) * | 2021-11-05 | 2021-12-03 | 亿景智联(北京)科技有限公司 | Management method and device of OLAP data warehouse |
| WO2022133981A1 (en) * | 2020-12-25 | 2022-06-30 | 京东方科技集团股份有限公司 | Data processing method, platform, computer-readable storage medium, and electronic device |
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102043841A (en) * | 2010-12-10 | 2011-05-04 | 上海市城市建设设计研究院 | Multi-source information supplying method based on Web technology and integrated service system thereof |
| CN102867282A (en) * | 2012-09-13 | 2013-01-09 | 福建富士通信息软件有限公司 | Implementation method for mobile Internet-based customer service quality analysis system |
| CN103678665A (en) * | 2013-12-24 | 2014-03-26 | 焦点科技股份有限公司 | Heterogeneous large data integration method and system based on data warehouses |
| CN104298779A (en) * | 2014-11-04 | 2015-01-21 | 中国银行股份有限公司 | Processing method and system for massive data processing |
-
2015
- 2015-06-04 CN CN201510302559.4A patent/CN104933112B/en active Active
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102043841A (en) * | 2010-12-10 | 2011-05-04 | 上海市城市建设设计研究院 | Multi-source information supplying method based on Web technology and integrated service system thereof |
| CN102867282A (en) * | 2012-09-13 | 2013-01-09 | 福建富士通信息软件有限公司 | Implementation method for mobile Internet-based customer service quality analysis system |
| CN103678665A (en) * | 2013-12-24 | 2014-03-26 | 焦点科技股份有限公司 | Heterogeneous large data integration method and system based on data warehouses |
| CN104298779A (en) * | 2014-11-04 | 2015-01-21 | 中国银行股份有限公司 | Processing method and system for massive data processing |
Cited By (35)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| JP2016170778A (en) * | 2015-03-10 | 2016-09-23 | 技研商事インターナショナル株式会社 | Market area analysis system |
| CN105320757A (en) * | 2015-10-19 | 2016-02-10 | 杭州华量软件有限公司 | Business intelligent analysis method for quickly processing data |
| CN105590259A (en) * | 2015-11-04 | 2016-05-18 | 中国银联股份有限公司 | Device and method for diagnosis of transaction system |
| CN105589940A (en) * | 2015-12-16 | 2016-05-18 | 南京联成科技发展有限公司 | Safety management operation and maintenance service platform based on unstructured real-time database |
| CN105653696A (en) * | 2015-12-29 | 2016-06-08 | 台山核电合营有限公司 | Data processing method and system for nuclear power plant databases |
| CN105787660A (en) * | 2016-02-24 | 2016-07-20 | 国家电网公司 | Information management system for photovoltaic power distribution network |
| CN106227862A (en) * | 2016-07-29 | 2016-12-14 | 浪潮软件集团有限公司 | E-commerce data integration method based on distribution |
| CN108280084A (en) * | 2017-01-06 | 2018-07-13 | 上海前隆信息科技有限公司 | A kind of construction method of data warehouse, system and server |
| CN106934023A (en) * | 2017-03-13 | 2017-07-07 | 山东浪潮云服务信息科技有限公司 | A kind of data managing method and device |
| CN107944866B (en) * | 2017-10-17 | 2021-08-31 | 厦门市美亚柏科信息股份有限公司 | Transaction record duplication elimination method and computer-readable storage medium |
| CN107944866A (en) * | 2017-10-17 | 2018-04-20 | 厦门市美亚柏科信息股份有限公司 | Transaction record rearrangement and computer-readable recording medium |
| CN107832392A (en) * | 2017-10-31 | 2018-03-23 | 链家网（北京）科技有限公司 | A kind of metadata management system |
| CN107958046A (en) * | 2017-11-24 | 2018-04-24 | 小花互联网金融服务（深圳）有限公司 | Internet finance big data warehouse analysis mining method |
| CN108733758A (en) * | 2018-04-11 | 2018-11-02 | 北京三快在线科技有限公司 | Hotel's static data method for pushing, device, electronic equipment and readable storage medium storing program for executing |
| CN108733758B (en) * | 2018-04-11 | 2022-04-05 | 北京三快在线科技有限公司 | Hotel static data pushing method and device, electronic equipment and readable storage medium |
| CN108595685A (en) * | 2018-05-04 | 2018-09-28 | 北京顶象技术有限公司 | A kind of data processing method and device |
| CN109189861A (en) * | 2018-06-29 | 2019-01-11 | 深圳市彬讯科技有限公司 | Data stream statistics method, server and storage medium based on index |
| CN109325648A (en) * | 2018-06-29 | 2019-02-12 | 深圳市彬讯科技有限公司 | Multi-dimensional data stream statistics method, server and storage medium based on index |
| CN109656910A (en) * | 2018-12-06 | 2019-04-19 | 哈尔滨工业大学 | Expansible Large Scale Biology medicine sample management and Visualization Platform |
| CN109656910B (en) * | 2018-12-06 | 2021-04-13 | 哈尔滨工业大学 | Extensible large-scale biomedical sample management and visualization platform |
| CN112947844A (en) * | 2019-12-11 | 2021-06-11 | 北京金山云网络技术有限公司 | Data storage method and device, electronic equipment and medium |
| CN111581254A (en) * | 2020-05-03 | 2020-08-25 | 上海维信荟智金融科技有限公司 | ETL method and system based on internet financial data |
| CN112256523A (en) * | 2020-09-23 | 2021-01-22 | 贝壳技术有限公司 | Service data processing method and device |
| CN112380218A (en) * | 2020-11-18 | 2021-02-19 | 浪潮天元通信信息系统有限公司 | ETL-based automatic triggering method for summarizing data tables of data warehouse layers |
| CN112380218B (en) * | 2020-11-18 | 2023-03-28 | 浪潮通信信息系统有限公司 | ETL-based automatic triggering method for summarizing data tables of data warehouse layers |
| CN112395345A (en) * | 2020-12-04 | 2021-02-23 | 江苏苏宁云计算有限公司 | HBase full data import method and device, computer equipment and storage medium |
| CN112416630A (en) * | 2020-12-10 | 2021-02-26 | 湖南新云网科技有限公司 | Data flow architecture and data processing method |
| WO2022133981A1 (en) * | 2020-12-25 | 2022-06-30 | 京东方科技集团股份有限公司 | Data processing method, platform, computer-readable storage medium, and electronic device |
| CN112650738B (en) * | 2020-12-31 | 2021-09-21 | 广西中科曙光云计算有限公司 | Construction method of open database |
| CN112650738A (en) * | 2020-12-31 | 2021-04-13 | 广西中科曙光云计算有限公司 | Construction method of open database |
| CN113362018A (en) * | 2021-05-25 | 2021-09-07 | 北京明略软件系统有限公司 | Conference time processing method and system |
| CN113515362A (en) * | 2021-07-12 | 2021-10-19 | 广州云从洪荒智能科技有限公司 | Data processing method, data processing device, computer equipment and storage medium |
| CN113515362B (en) * | 2021-07-12 | 2023-10-20 | 广州云从洪荒智能科技有限公司 | Data processing method, device, computer equipment and storage medium |
| CN113742320A (en) * | 2021-11-05 | 2021-12-03 | 亿景智联(北京)科技有限公司 | Management method and device of OLAP data warehouse |
| CN113742320B (en) * | 2021-11-05 | 2022-03-01 | 亿景智联(北京)科技有限公司 | Management method and device of OLAP data warehouse |
Also Published As
| Publication number | Publication date |
|---|---|
| CN104933112B (en) | 2018-12-21 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN104933112A (en) | Distributed Internet transaction information storage and processing method | |
| CN103365929B (en) | The management method of a kind of data base connection and system | |
| CN111459985B (en) | Identification information processing method and device | |
| CN104750681B (en) | A kind of processing method and processing device of mass data | |
| Bao et al. | Managing massive trajectories on the cloud | |
| US9798813B2 (en) | Extensible person container | |
| CN112767058A (en) | AIOT DaaS digital twin cloud platform | |
| CN103268336A (en) | Fast data and big data combined data processing method and system | |
| CN103678665A (en) | Heterogeneous large data integration method and system based on data warehouses | |
| CN103440288A (en) | Big data storage method and device | |
| CN104239377A (en) | Platform-crossing data retrieval method and device | |
| CN110990474A (en) | Regional industry image analysis method and device | |
| CN110674152B (en) | Data synchronization method and device, storage medium and electronic equipment | |
| US20190050435A1 (en) | Object data association index system and methods for the construction and applications thereof | |
| CN112148718A (en) | Big data support management system for city-level data middling station | |
| CN107832392A (en) | A kind of metadata management system | |
| CN110837520A (en) | Data processing method, platform and system | |
| CN105630934A (en) | Data statistic method and system | |
| CN111737364B (en) | Safe multi-party data fusion and federal sharing method, device, equipment and medium | |
| CN107832463A (en) | A kind of finance data service platform | |
| CN113127741B (en) | Cache method for reading and writing data of mass users and posts in part-time post recommendation system | |
| CN102945270B (en) | Parallel distribution type network public opinion data management method and system | |
| CN109446167A (en) | A kind of storage of daily record data, extracting method and device | |
| Singh et al. | Easy designing steps of a local data warehouse for possible analytical data processing | |
| CN111708895A (en) | Method and device for constructing knowledge graph system |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| C10 | Entry into substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant | ||
| PE01 | Entry into force of the registration of the contract for pledge of patent right |
Denomination of invention: Distributed Internet transaction information storage and processing method Effective date of registration: 20200306 Granted publication date: 20181221 Pledgee: Huaxia Bank Co., Ltd. Hangzhou Yuhang sub branch Pledgor: Zhejiang Li Shi Science and Technology Co., Ltd. Registration number: Y2020330000080 |
|
| PE01 | Entry into force of the registration of the contract for pledge of patent right |