CN104933112B - Distributed interconnection Transaction Information storage processing method - Google Patents
Distributed interconnection Transaction Information storage processing method Download PDFInfo
- Publication number
- CN104933112B CN104933112B CN201510302559.4A CN201510302559A CN104933112B CN 104933112 B CN104933112 B CN 104933112B CN 201510302559 A CN201510302559 A CN 201510302559A CN 104933112 B CN104933112 B CN 104933112B
- Authority
- CN
- China
- Prior art keywords
- data
- business
- distributed
- information
- warehouse
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Abstract
The present invention provides a kind of distributed interconnection Transaction Information storage processing methods, comprising: trading information data of the operation data storage function module from external data layer drawing-out structure；Distributed Data Warehouse and non-relational database read non-structured network log data from distributed file system respectively；Internet business information data warehouse carries out being integrated into data tuple after extracting the conversion load trading information data and network log data in the operation data storage function module and the Distributed Data Warehouse respectively, and stores the data tuple with Star Model；Data Mart extracts conversion load fairground from internet business information data warehouse and/or the Distributed Data Warehouse and specifies data；Business intelligence system needs to obtain from the Data Mart and/or the non-relational database according to data analysis analyzes required data.The present invention is directed to the application scenarios of internet business Data Analysis Services.
Description
Technical field
The present invention relates to a kind of distributed interconnection Transaction Information storage processing methods.
Background technique
The model and result of Data Analysis Services only apply in certain decision applications that really value could be generated, and
By based on the fact that support system know etc. that some serial theory and methods are business intelligence come aid decision
(Bussiness Intelligence, abbreviation BI).With information-based high development, business intelligence is increasingly taken seriously, especially
It is the joint on-line analysis OLAP(i.e. online quick analysis based on database, big data), it is even more business intelligence in a way
Can main methods, the core data warehouse architecture design in the BI solution of traditional industries is as shown in Figure 1.
However, internet business information data is not suitable for using BI solution, main cause has the following:
1. the data source of internet electronic business transaction is varied, including different electric business platforms, various relationship type numbers
Data and network log data for being generated according to library, social software etc.；So single ETL tool can not handle well it is all
Data.
2. the data volume of e-commerce transaction is huge, it is so big that traditional centralized relevant database is unable to satisfy processing
The requirement of scale data.
3. internet business monitoring is higher to the requirement of real-time of Data Analysis Services, most monitoring and warning needs
Processing is completed in short time could embody the value of data, and traditional offline ETL processing mode can no longer meet demand.
4. the number of users of internet business is huge, and traditional BI shows tool design primarily directed to the middle height of enterprise
Layer user, is transplanted to no matter e-commerce industry from interactive efficiency and user experience all has biggish difference.Current tradition
The business intelligence system of mode, can not be extensive suitable for internet business information well, heterogeneous data source and right
The demand of data analysis real-time.
Summary of the invention
The present invention is directed to the application scenarios of internet business Data Analysis Services, in conjunction with internet electronic business analysis and early warning
The characteristics of on the basis of, for the defects in the prior art, the object of the present invention is to provide the storages of distributed interconnection Transaction Information
Processing method.
A kind of distributed interconnection Transaction Information storage processing method provided according to the present invention, comprising:
Trading information data of the operation data storage function module from external data layer drawing-out structure；
Distributed Data Warehouse and non-relational database read non-structured net from distributed file system respectively
Network daily record data；
Internet business information data warehouse is from the operation data storage function module and the Distributed Data Warehouse
Middle extracted after conversion loads the trading information data and network log data respectively carries out being integrated into data tuple, and with star-like
Model stores the data tuple；
Data Mart root is extracted from internet business information data warehouse and/or the Distributed Data Warehouse
Specify data in conversion load fairground；
Business intelligence system is needed from the Data Mart and/or the non-relational database according to data analysis
Obtain data needed for analyzing.
As a kind of prioritization scheme, the Data Mart includes exchanges, sale fairground and customer service fairground；
The exchanges, sale fairground need to extract from internet business information data warehouse according to business converts
It loads the fairground and specifies data；
The customer service fairground needs to extract the conversion load fairground from the Distributed Data Warehouse according to business and refers to
Fixed number evidence.
As a kind of prioritization scheme, also cached by MemCache between the customer service fairground and the business intelligence system
System carries out data transmission；
The business intelligence system first checks requested analysis when obtaining to the customer service fairground and analyzing required data
Whether required data are in MemCache caching system, if so, then directly obtaining from MemCache caching system, if not
Data needed for analyzing then are being obtained from the customer service fairground and are being cached in MemCache caching system a.
As a kind of prioritization scheme, the business intelligent system is used for data mining, enterprise diagnosis, customer analysis, data
File analysis and on-line analysis.
As a kind of prioritization scheme, the operation data storage function module includes three-decker:
Mapping layer: the field of the data original table of external data layer is mapped to the local number of operation data storage function module
According in library, association of the data from operation layer to analysis layer is completed；
Data prediction layer: pre-processing the trading information data, which includes integration, screening and increase
Contingency table；
Data cleansing layer: data cleansing operation is carried out for defective in quality trading information data.
As a kind of prioritization scheme, the fact that the Star Model is located at star centre table include several data tuples
Time address, domain addresses, store address, product address and the corresponding number of transaction of data tuple and transaction amount；
The dimension table of the Star Model includes shop dimension table, time dimension table, product dimension table and region dimension table；
The shop dimension table includes electric business platform information, platform board block message and shop management information；
The time dimension table includes temporal information；
The product dimension table includes name of product, product description, product price and product quality；
The region dimension table includes geographical location information, which includes country, province, city.
As a kind of prioritization scheme, the operation data storage function module uses full dose loading method, specifically:
S101 empties the object table of the operation data storage function module,
The full dose trading information data of external data layer is inserted into the object table by S102.
As a kind of prioritization scheme, the operation data storage function module uses step increment method mode, specifically:
S201 empties the interim table of the operation data storage function module,
The increment trading information data of external data layer is inserted into the interim table by S202,
S203, delete target table and the interim duplicate data of table,
S204, by rear return step S1 in the data insertion object table in interim table until data whole extraction terminates.
As a kind of prioritization scheme, the dimension table in internet business information data warehouse is step increment method mode, true
Table is full dose loading method；
The dimension table step increment method process in internet business information data warehouse specifically:
S301 calculates sequence according to the line number of dimension table historical data to incremental data in interim table,
S302 will be inserted into interim table with the incremental data of the sequence,
S303 empties the dimension table in internet business information data warehouse,
The tables of data of the operation data storage function module is associated with by S304 with the interim table, described will have institute
State the dimension table in the incremental data insertion internet business information data warehouse of sequence；
The step increment method process of true table specifically:
After emptying interim table, judge whether to be derived from different true tables there are also index；
If being derived from different true tables there are also index, repeat to take out the different business thing in true table in timestamp scope
The process of part, until all business event recycle after being all removed to be terminated；
If being derived from different true tables without index, the different business event in true table in timestamp scope is taken out,
The data of first interim table are aggregated into the second interim table according to dimension field, it is second interim with this in delete target fact table
The data of described second interim table are inserted into target fact table by the duplicate data of table.
Compared with prior art, the present invention have it is following the utility model has the advantages that
The present invention is directed to internet electronic business Transaction Information feature, designs application oriented, integrating, special with the time
Sign, stable data acquisition system, to provide support for transaction data processing, analysis and monitoring decision.
Detailed description of the invention
In order to illustrate the technical solution of the embodiments of the present invention more clearly, required use in being described below to embodiment
Attached drawing be briefly described, it is therefore apparent that drawings in the following description are only some embodiments of the invention, for ability
For field technique personnel, without creative efforts, it is also possible to obtain other drawings based on these drawings.It is attached
In figure:
Fig. 1 is the core data warehouse architecture diagram in the BI solution of traditional industries；
Fig. 2 is one of alternative embodiment distributed interconnection Transaction Information storage architecture schematic diagram；
Fig. 3 is the principle framework of data warehouse；
Fig. 4 is one of alternative embodiment distributed interconnection Transaction Information storage processing method schematic diagram；
Fig. 5 is data warehouse Star Model schematic diagram；
Fig. 6 is operation data storage function module (ODS) full dose load flow chart；
Fig. 7 is operation data storage function module (ODS) step increment method flow chart；
Fig. 8 is the dimension table step increment method flow chart in internet business information data warehouse (DW)；
The fact that Fig. 9 is DW table (DM) step increment method flow chart；
Figure 10 is operation data storage function module ODS, between internet business information data warehouse DW, true table DM
Relation schematic diagram.
Specific embodiment
Below in conjunction with attached drawing, the present invention is described in detail in a manner of specific embodiment.Following embodiment will be helpful to
Those skilled in the art further understands the present invention, but the invention is not limited in any way.It should be pointed out that can be with
Modification structurally and functionally is carried out using other embodiments, or to embodiment enumerated herein, without departing from this hair
Bright scope and spirit.
The present invention is directed to the application scenarios of internet business Data Analysis Services, in conjunction with internet electronic business analysis and early warning
The characteristics of on the basis of, mainly include leading portion Reports module, ETL module, data warehouse module, database management module, data tune
Spend module and Web service module composition.The present invention is directed to internet electronic business Transaction Information feature, designs application-oriented
, integrated, data acquisition systems with temporal characteristics, stable, to provide branch for transaction data processing, analysis and monitoring decision
It holds, overall architecture is as shown in Figure 2.
In a kind of embodiment of distributed interconnection Transaction Information storage processing method provided by the invention, such as Fig. 2 and
Shown in Fig. 4, comprising:
Trading information data of the operation data storage function module from external data layer drawing-out structure；
Distributed Data Warehouse and non-relational database read non-structured net from distributed file system respectively
Network daily record data；
Internet business information data warehouse is from the operation data storage function module and the Distributed Data Warehouse
Middle extracted after conversion loads the trading information data and network log data respectively carries out being integrated into data tuple, and with star-like
Model stores the data tuple；
Data Mart root is extracted from internet business information data warehouse and/or the Distributed Data Warehouse
Specify data in conversion load fairground；
Business intelligence system is needed from the Data Mart and/or the non-relational database according to data analysis
Obtain data needed for analyzing.
Extraction conversion is loaded as ETL, is the abbreviation of English Extract-Transform-Load, for describe by
Data are from source terminal by extracting (extract), conversion (transform), the process for loading (load) to destination.
With the development of e-commerce, need to carry out on-line analysis to the trading activity of user in real time, for example show some
All history access of shop on net and inquiry record, while real-time tracing shows that this shop is being interviewed on electric business platform
The information such as the page asked are difficult to meet such need using the relevant database of traditional support off-line analysis and complex query
It asks.Meanwhile semi-structured big data as the more and more web log files of e-commerce industry, user behavior is handled and is combined
Complicated data mining processing, therefore the present invention realizes mass data volume using the big data platform or Hadoop of power stone science and technology
Handling implement.Power stone science and technology big data processing core module mainly includes cloud database, cloud storage, search engine and data point
Analysis, energy processing structure, semi-structured, unstructured data, supports standard interface, provides the data portion of one-stop automation
The functions such as administration, migration, backup, recovery, disaster tolerance.The principle framework of data warehouse is as shown in Figure 3.
The data of data warehouse are by the data-interface of standard, to be originated from internet electronic business transaction platform and opening
To applications.Data warehouse is divided into three-decker according to data flow: data Layer, Information Level and analysis layer, as shown in Figure 4.
Data Layer
By the standard data interface of propelling movement type, model consistent with electric business platform database or the mode of middle table are used
Obtain the external data of electric business platform, then data pick-up data grabber in other words carried out by ODS, the format of extraction include XML and
TXT etc..
Information Level
Among data Layer and internet business information data depot layer increase operation data storage function module (ODS:
Operation Data Storage).Purpose is as a buffer pool, temporarily to one by the data integration of multiple data sources
It is used in buffer area for data warehouse, effectively to mitigate data source and the pressure of ETL.
Wherein, ODS includes three-decker:
Mapping layer: the field of the data original table of external data layer is mapped to the local number of operation data storage function module
According in library, complete association of the data from operation layer to analysis layer, for external data and by system administration to mapping layer into
Row is concentrated.
Data prediction layer: pre-processing the trading information data, which includes integration, screening and increase
Contingency table, it is therefore intended that simplify and promoted the work of ETL；
Data cleansing: data cleansing operation is carried out for defective in quality trading information data.
ODS storage is all the internet business information data grabbed from all kinds of electric business platforms.
Analysis layer
By BI system and Hadoop tool to the trading information data of all kinds of electric business platforms and non-structured website day
Will carries out the processing such as data mining, enterprise diagnosis, customer analysis, data support and on-line analysis.The business intelligent system is used
In data mining, enterprise diagnosis, customer analysis, data file analysis and on-line analysis.
For storing the easy information data warehouse in internet of processing structure trading information data using relational database, interior
The synthesis in deposit data library and distributed data base uses relational database and traditional BI service is suitble to carry out at analysis
Reason；Storage and real-time query analysis for a large amount of real time data is using the Hadoop distributed file system for supporting HBase
Non-relational database (Nosql) based on HDFS.
In embodiment as shown in Figure 4, the Data Mart includes exchanges, sale fairground and customer service fairground；
The exchanges, sale fairground need to extract from internet business information data warehouse according to business converts
It loads the fairground and specifies data；
The customer service fairground needs to extract the conversion load fairground from the Distributed Data Warehouse according to business and refers to
Fixed number evidence.
The exchanges are used to store the transaction related information of processing buyer and shop, such as exchange hour, number of transaction
Deng.
The sale fairground is used to store the sale related data in processing shop, such as shop amount of access, sales situation.
The customer service fairground is used to store both sides' interaction in the calling information and transaction of processing buyer.
Distributed Data Warehouse in the present embodiment is power stone cloud database, provides high-performance, the distributed of High Availabitity closes
It is type database all-in-one machine, can supports OLAP, OLTP and Combination application, support high-performance (distribution), High Availabitity, supports heat
Migration, warm back-up, heat are restored, and support stsndard SQL, support mainstream development language, support based on x86, Godson, soar, PowerPC
It is equal chips server, low to hardware requirement.
HBase, i.e. Hadoop Database can also be used, is a heightReliablyProperty, high-performance, towards column, it is scalable
'sDistributed memory system。
Distributed Data Warehouse and non-relational database are read from distributed file system unstructured respectively in Fig. 4
Network log data.When customer service needs to transfer transaction record and corresponding transaction data, directly pass through from HBASE
ETL mode extracts, and needs if it is third-party business intelligence system for statistical analysis to internet business data, is not necessarily to
Detailed Transaction Information then directly obtains the network log data of all kinds of electric business platforms from the NOSQL.Thus it improves
The speed of service, so that system storage processing is more efficient.
As one embodiment, also by MemCache caching system between the customer service fairground and the business intelligence system
System carries out data transmission；
The business intelligence system first checks requested analysis when obtaining to the customer service fairground and analyzing required data
Whether required data are in MemCache caching system, if so, then directly obtaining from MemCache caching system, if not
Data needed for analyzing then are being obtained from the customer service fairground and are being cached in MemCache caching system a.
Memcache is a high performance distributed memory object caching system, by safeguarding a system in memory
One huge hash table, it can be used to store the data of various formats, including image, video, file and database inspection
The result etc. of rope.It is briefly exactly then to be read from memory by data call into memory, to greatly improve reading speed
Degree.
The metadata memory module in internet business information data warehouse is using the star-like mould for being suitble to dimension and true separation
Type, as shown in Figure 5.Data have already passed through pretreatment, have been engaged on and pull out in reality to establish about true dimensional information
In corresponding dimension table.Process flow is: ODS layers are drawn into from operation layer about the relevant data of operation flow；In depot layer (DW
Layer) according to business function progress data Layer design (mainly including the common dimensions such as time dimension table, region dimension table)；True table
The fact that (DM layers) are historical datas, without repeating operational, correspond to business association relation table, prestige table,
Amount of access etc., true table are the cores of star structure, record the trunk content of main body.
In the embodiment as shown in fig.5, it includes several datas that the Star Model, which is located at the fact that star centre table,
The time address of tuple, domain addresses, store address, product address and the corresponding number of transaction of data tuple and transaction
The amount of money；
The dimension table of the Star Model includes shop dimension table, time dimension table, product dimension table and region dimension table；
The shop dimension table includes electric business platform information, platform board block message and shop management information；
The time dimension table includes temporal information；
The product dimension table includes name of product, product description, product price and product quality；
The region dimension table includes geographical location information, which includes country, province, city.
As one embodiment, as shown in fig. 6, the operation data storage function module uses full dose loading method, tool
Body are as follows:
S101 empties the object table of the operation data storage function module,
The full dose trading information data of external data layer is inserted into the object table by S102.
As one embodiment, as shown in fig. 7, the operation data storage function module uses step increment method mode, tool
Body are as follows:
S201 empties the interim table of the operation data storage function module,
The increment trading information data of external data layer is inserted into the interim table by S202,
S203, delete target table and the interim duplicate data of table,
S204, by rear return step S1 in the data insertion object table in interim table until data whole extraction terminates.
As one embodiment, as shown in figure 8, the dimension table in internet business information data warehouse is step increment method side
Formula, true table are full dose loading method；
The dimension table step increment method process in internet business information data warehouse specifically:
S301 calculates sequence according to the line number of dimension table historical data to incremental data in interim table,
S302 will be inserted into interim table with the incremental data of the sequence,
S303 empties the dimension table in internet business information data warehouse,
The tables of data of the operation data storage function module is associated with by S304 with the interim table, described will have institute
State the dimension table in the incremental data insertion internet business information data warehouse of sequence.
As one embodiment, as shown in figure 9, the step increment method process of true table specifically:
After emptying interim table, judge whether to be derived from different true tables there are also index；
If being derived from different true tables there are also index, repeat to take out the different business thing in true table in timestamp scope
The process of part, until all business event recycle after being all removed to be terminated；
If being derived from different true tables without index, the different business event in true table in timestamp scope is taken out,
The data of first interim table are aggregated into the second interim table according to dimension field, it is second interim with this in delete target fact table
The data of described second interim table are inserted into target fact table by the duplicate data of table.
Intelligently all kinds of full doses load the present embodiment or step increment method completion all further includes one and insertion situation is written later
The step of system log.
Using Multi-dimension on-line analytical process (OLAP:Online Analysis Processing) come according to different business
Demand, from different demand angles (such as sale, customer service, finance, time, region, industry visual angle) to from other data knots
The related data of structure carries out alternate analysis.It is presented by the multidimensional of analysis and leading portion system to data, realizes and internet is handed over
Easy information is shown and real-time trend analysis early warning.Dimension and indication information are extracted first；Secondly because dimension exists between each main body
It is all independent from each other when definition, for the consistency and incidence relation of data, the relationship of each analysis personnel dimension will be established
Information, realization body are extended and are associated with；After establishing data model, using ETL by the data in data warehouse according to client
Demand carries out corresponding statistics and summarizes to obtain multidimensional analysis data, eventually forms report.
The present invention devises reasonable system architecture and adaptable ETL, data warehouse and data dimension design, makes it
Storage suitable for internet business information is handled.
In view of the data to different electric business platforms are handled and stored, so the demand of application layer is different；It is right simultaneously
The changes in demand of user is not only that unitem is available, thus need to combine the demand of search efficiency and data dynamics with
And good scalability, using star-like Multidimensional Data Model as data warehouse model.
By pretreated data, the dimensional information about true table extracted from the fact is established in corresponding dimension
In table.Therefore, process layer functional module, which only needs to inquire true table, can obtain Transaction Information, substantially increase visit
The efficiency asked.Operation data storage function module ODS, internet business information data warehouse DW, the relationship between true table DM
Schematic diagram is as shown in Figure 10.
ODS layers of correspondence are drawn into the data about internet business main body and behavior from data Layer；DW layers are data warehouse
Layer, the public informations dimension table such as main storage and internet business correlation time dimension table, region dimension table；The corresponding true table of DM, also
It is the core of hub-and-spoke configuration, records the information such as the incidence relation of transaction.
ETL update mechanism, warehouse update at first, and fairground is updating；Dimension first updates, the mechanism that the fact updates again.
The foregoing is merely presently preferred embodiments of the present invention, and those skilled in the art know, is not departing from essence of the invention
In the case where mind and range, various changes or equivalent replacement can be carried out to these features and embodiment.In addition, of the invention
Under introduction, it can modify to these features and embodiment to adapt to particular situation and material without departing from of the invention
Spirit and scope.Therefore, the present invention is not limited to the particular embodiment disclosed, and the right of fallen with the application is wanted
The embodiment in range is asked to belong to protection scope of the present invention.
Claims (9)
1. a kind of distributed interconnection Transaction Information storage processing method characterized by comprising
Trading information data of the operation data storage function module from external data layer drawing-out structure；
Distributed Data Warehouse and non-relational database read non-structured network day from distributed file system respectively
Will data；
Internet business information data warehouse divides from the operation data storage function module and the Distributed Data Warehouse
It carries out being integrated into data tuple after the load trading information data and network log data Chou Qu not converted, and with Star Model
Store the data tuple；
Data Mart extracts conversion from internet business information data warehouse and/or the Distributed Data Warehouse and adds
It carries fairground and specifies data；
Business intelligence system needs to obtain from the Data Mart and/or the non-relational database according to data analysis
Data needed for analyzing；
Wherein, when customer service needs to transfer transaction record and corresponding transaction data, directly pass through ETL from Data Mart
Mode extracts, and business intelligence system need it is for statistical analysis to internet business data, be not necessarily to detailed Transaction Information when,
The network log data of all kinds of electric business platforms are then directly obtained from the non-relational database.
2. a kind of distributed interconnection Transaction Information storage processing method according to claim 1, which is characterized in that described
Data Mart includes exchanges, sale fairground and customer service fairground；
The exchanges, sale fairground need to extract conversion load from internet business information data warehouse according to business
Specify data in corresponding fairground；
The customer service fairground needs to extract the corresponding fairground of conversion load from the Distributed Data Warehouse according to business and specifies
Data.
3. a kind of distributed interconnection Transaction Information storage processing method according to claim 2, which is characterized in that described
Also carry out data transmission by MemCache caching system between customer service fairground and the business intelligence system；
Needed for the business intelligence system first checks requested analysis when obtaining to the customer service fairground and analyzing required data
Data whether in MemCache caching system, if so, then directly obtained from MemCache caching system, if not existing,
Data needed for analyzing are obtained from the customer service fairground and are cached in MemCache caching system a.
4. a kind of distributed interconnection Transaction Information storage processing method according to claim 1, which is characterized in that described
Business intelligence system is used for data mining, enterprise diagnosis, customer analysis, data file analysis and on-line analysis.
5. a kind of distributed interconnection Transaction Information storage processing method according to claim 1, which is characterized in that described
Operation data storage function module includes three-decker:
Mapping layer: the field of the data original table of external data layer is mapped to the local data base of operation data storage function module
In, complete association of the data from operation layer to analysis layer；
Data prediction layer: pre-processing the trading information data, which includes integrating, screen and increasing to be associated with
Table；
Data cleansing layer: data cleansing operation is carried out for defective in quality trading information data.
6. a kind of distributed interconnection Transaction Information storage processing method according to claim 1, which is characterized in that described
Star Model be located at the time address that the fact that star centre table includes several data tuples, domain addresses, store address, production
Product address and the corresponding number of transaction of data tuple and transaction amount；
The dimension table of the Star Model includes shop dimension table, time dimension table, product dimension table and region dimension table；
The shop dimension table includes electric business platform information, platform board block message and shop management information；
The time dimension table includes temporal information；
The product dimension table includes name of product, product description, product price and product quality；
The region dimension table includes geographical location information, which includes country, province, city.
7. a kind of distributed interconnection Transaction Information storage processing method according to claim 1, which is characterized in that described
Operation data storage function module uses full dose loading method, specifically:
S101 empties the object table of the operation data storage function module,
The full dose trading information data of external data layer is inserted into the object table by S102.
8. a kind of distributed interconnection Transaction Information storage processing method according to claim 1, which is characterized in that described
Operation data storage function module uses step increment method mode, specifically:
S201 empties the interim table of the operation data storage function module,
The increment trading information data of external data layer is inserted into the interim table by S202,
S203, delete target table and the interim duplicate data of table,
Rear return in data insertion object table in interim table is continued to believe from the transaction of external data layer drawing-out structure by S204
Data are ceased, until data, which all extract, to be terminated.
9. a kind of distributed interconnection Transaction Information storage processing method according to claim 1, which is characterized in that described
The dimension table in internet business information data warehouse is step increment method mode, and true table is full dose loading method；
The dimension table step increment method process in internet business information data warehouse specifically:
S301 calculates sequence according to the line number of dimension table historical data to incremental data in interim table,
S302 will be inserted into interim table with the incremental data of the sequence,
S303 empties the dimension table in internet business information data warehouse,
The tables of data of the operation data storage function module is associated with by S304 with the interim table, described will have the row
The dimension table in the incremental data insertion internet business information data warehouse of sequence；
The full dose loading procedure of true table specifically:
After emptying interim table, judge whether to be derived from different true tables there are also index；
If being derived from different true tables there are also index, repeat to take out the different business event in true table in timestamp scope
Process, until all business event recycle after being all removed to be terminated；
If being derived from different true tables without index, the different business event in true table in timestamp scope is taken out, by
The data of one interim table are aggregated into the second interim table according to dimension field, in delete target fact table with the second interim table weight
The data of described second interim table are inserted into target fact table by multiple data.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201510302559.4A CN104933112B (en) | 2015-06-04 | 2015-06-04 | Distributed interconnection Transaction Information storage processing method |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201510302559.4A CN104933112B (en) | 2015-06-04 | 2015-06-04 | Distributed interconnection Transaction Information storage processing method |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN104933112A CN104933112A (en) | 2015-09-23 |
| CN104933112B true CN104933112B (en) | 2018-12-21 |
Family
ID=54120280
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201510302559.4A Active CN104933112B (en) | 2015-06-04 | 2015-06-04 | Distributed interconnection Transaction Information storage processing method |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN104933112B (en) |
Families Citing this family (28)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| JP6646465B2 (en) * | 2015-03-10 | 2020-02-14 | 技研商事インターナショナル株式会社 | Trade area analysis system |
| CN105320757A (en) * | 2015-10-19 | 2016-02-10 | 杭州华量软件有限公司 | Business intelligent analysis method for quickly processing data |
| CN105590259A (en) * | 2015-11-04 | 2016-05-18 | 中国银联股份有限公司 | Device and method for diagnosis of transaction system |
| CN105589940A (en) * | 2015-12-16 | 2016-05-18 | 南京联成科技发展有限公司 | Safety management operation and maintenance service platform based on unstructured real-time database |
| CN105653696A (en) * | 2015-12-29 | 2016-06-08 | 台山核电合营有限公司 | Data processing method and system for nuclear power plant databases |
| CN105787660A (en) * | 2016-02-24 | 2016-07-20 | 国家电网公司 | Information management system for photovoltaic power distribution network |
| CN106227862A (en) * | 2016-07-29 | 2016-12-14 | 浪潮软件集团有限公司 | E-commerce data integration method based on distribution |
| CN108280084A (en) * | 2017-01-06 | 2018-07-13 | 上海前隆信息科技有限公司 | A kind of construction method of data warehouse, system and server |
| CN106934023A (en) * | 2017-03-13 | 2017-07-07 | 山东浪潮云服务信息科技有限公司 | A kind of data managing method and device |
| CN107944866B (en) * | 2017-10-17 | 2021-08-31 | 厦门市美亚柏科信息股份有限公司 | Transaction record duplication elimination method and computer-readable storage medium |
| CN107832392A (en) * | 2017-10-31 | 2018-03-23 | 链家网（北京）科技有限公司 | A kind of metadata management system |
| CN107958046A (en) * | 2017-11-24 | 2018-04-24 | 小花互联网金融服务（深圳）有限公司 | Internet finance big data warehouse analysis mining method |
| CN108733758B (en) * | 2018-04-11 | 2022-04-05 | 北京三快在线科技有限公司 | Hotel static data pushing method and device, electronic equipment and readable storage medium |
| CN108595685B (en) * | 2018-05-04 | 2021-06-15 | 北京顶象技术有限公司 | Data processing method and device |
| CN109325648A (en) * | 2018-06-29 | 2019-02-12 | 深圳市彬讯科技有限公司 | Multi-dimensional data stream statistics method, server and storage medium based on index |
| CN109189861A (en) * | 2018-06-29 | 2019-01-11 | 深圳市彬讯科技有限公司 | Data stream statistics method, server and storage medium based on index |
| CN109656910B (en) * | 2018-12-06 | 2021-04-13 | 哈尔滨工业大学 | Extensible large-scale biomedical sample management and visualization platform |
| CN112947844A (en) * | 2019-12-11 | 2021-06-11 | 北京金山云网络技术有限公司 | Data storage method and device, electronic equipment and medium |
| CN111581254A (en) * | 2020-05-03 | 2020-08-25 | 上海维信荟智金融科技有限公司 | ETL method and system based on internet financial data |
| CN112256523B (en) * | 2020-09-23 | 2023-01-06 | 贝壳技术有限公司 | Service data processing method and device |
| CN112380218B (en) * | 2020-11-18 | 2023-03-28 | 浪潮通信信息系统有限公司 | ETL-based automatic triggering method for summarizing data tables of data warehouse layers |
| CN112395345A (en) * | 2020-12-04 | 2021-02-23 | 江苏苏宁云计算有限公司 | HBase full data import method and device, computer equipment and storage medium |
| CN112416630A (en) * | 2020-12-10 | 2021-02-26 | 湖南新云网科技有限公司 | Data flow architecture and data processing method |
| WO2022133981A1 (en) * | 2020-12-25 | 2022-06-30 | 京东方科技集团股份有限公司 | Data processing method, platform, computer-readable storage medium, and electronic device |
| CN112650738B (en) * | 2020-12-31 | 2021-09-21 | 广西中科曙光云计算有限公司 | Construction method of open database |
| CN113362018A (en) * | 2021-05-25 | 2021-09-07 | 北京明略软件系统有限公司 | Conference time processing method and system |
| CN113515362B (en) * | 2021-07-12 | 2023-10-20 | 广州云从洪荒智能科技有限公司 | Data processing method, device, computer equipment and storage medium |
| CN113742320B (en) * | 2021-11-05 | 2022-03-01 | 亿景智联(北京)科技有限公司 | Management method and device of OLAP data warehouse |
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102043841A (en) * | 2010-12-10 | 2011-05-04 | 上海市城市建设设计研究院 | Multi-source information supplying method based on Web technology and integrated service system thereof |
| CN102867282A (en) * | 2012-09-13 | 2013-01-09 | 福建富士通信息软件有限公司 | Implementation method for mobile Internet-based customer service quality analysis system |
| CN103678665A (en) * | 2013-12-24 | 2014-03-26 | 焦点科技股份有限公司 | Heterogeneous large data integration method and system based on data warehouses |
| CN104298779A (en) * | 2014-11-04 | 2015-01-21 | 中国银行股份有限公司 | Processing method and system for massive data processing |
-
2015
- 2015-06-04 CN CN201510302559.4A patent/CN104933112B/en active Active
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102043841A (en) * | 2010-12-10 | 2011-05-04 | 上海市城市建设设计研究院 | Multi-source information supplying method based on Web technology and integrated service system thereof |
| CN102867282A (en) * | 2012-09-13 | 2013-01-09 | 福建富士通信息软件有限公司 | Implementation method for mobile Internet-based customer service quality analysis system |
| CN103678665A (en) * | 2013-12-24 | 2014-03-26 | 焦点科技股份有限公司 | Heterogeneous large data integration method and system based on data warehouses |
| CN104298779A (en) * | 2014-11-04 | 2015-01-21 | 中国银行股份有限公司 | Processing method and system for massive data processing |
Also Published As
| Publication number | Publication date |
|---|---|
| CN104933112A (en) | 2015-09-23 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN104933112B (en) | Distributed interconnection Transaction Information storage processing method | |
| US10936588B2 (en) | Self-described query execution in a massively parallel SQL execution engine | |
| Chaudhuri et al. | An overview of business intelligence technology | |
| US9542424B2 (en) | Lifecycle-based horizontal partitioning | |
| EP2270691B1 (en) | Computer-implemented method for operating a database and corresponding computer system | |
| CN102214176B (en) | Method for splitting and join of huge dimension table | |
| US7814045B2 (en) | Semantical partitioning of data | |
| Schaffner et al. | A hybrid row-column OLTP database architecture for operational reporting | |
| CN112148718A (en) | Big data support management system for city-level data middling station | |
| CN104239377A (en) | Platform-crossing data retrieval method and device | |
| CN106649687A (en) | Method and device for on-line analysis and processing of large data | |
| El Alami et al. | Supply of a key value database redis in-memory by data from a relational database | |
| CN102945270B (en) | Parallel distribution type network public opinion data management method and system | |
| CN208207819U (en) | A kind of big data analysis processing system based on extended node cluster | |
| CN116483822B (en) | Service data early warning method, device, computer equipment and storage medium | |
| CN115062028B (en) | Method for multi-table join query in OLTP field | |
| Dong et al. | Research on Architecture of Power Big Data High-Speed Storage System for Energy Interconnection | |
| CN114860780A (en) | Data warehouse, data processing system and computer device | |
| AU2010263721A1 (en) | Database management device using key-value store with attributes, and key-value-store structure caching-device therefor | |
| Fong et al. | Toward a scale-out data-management middleware for low-latency enterprise computing | |
| CN113779215A (en) | Data processing platform | |
| Ma et al. | Bank big data architecture based on massive parallel processing database | |
| CN111026814B (en) | Low-cost data storage method | |
| Alam | Data Migration: Relational Rdbms To Non-Relational Nosql | |
| Siddesh et al. | Driving big data with hadoop technologies |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| C10 | Entry into substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant | ||
| PE01 | Entry into force of the registration of the contract for pledge of patent right |
Denomination of invention: Distributed Internet transaction information storage and processing method Effective date of registration: 20200306 Granted publication date: 20181221 Pledgee: Huaxia Bank Co., Ltd. Hangzhou Yuhang sub branch Pledgor: Zhejiang Li Shi Science and Technology Co., Ltd. Registration number: Y2020330000080 |
|
| PE01 | Entry into force of the registration of the contract for pledge of patent right |