CN110442627A - Data transmission method and system between a kind of memory database system and data warehouse - Google Patents
Data transmission method and system between a kind of memory database system and data warehouse Download PDFInfo
- Publication number
- CN110442627A CN110442627A CN201910604930.0A CN201910604930A CN110442627A CN 110442627 A CN110442627 A CN 110442627A CN 201910604930 A CN201910604930 A CN 201910604930A CN 110442627 A CN110442627 A CN 110442627A
- Authority
- CN
- China
- Prior art keywords
- data
- memory database
- event
- database system
- warehouse
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/23—Updating
- G06F16/2379—Updates performed during online database operations; commit processing
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/245—Query processing
- G06F16/2455—Query execution
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/25—Integrating or interfacing systems involving database management systems
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Databases & Information Systems (AREA)
- Data Mining & Analysis (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Computational Linguistics (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention discloses the data transmission method and system between a kind of memory database system and data warehouse, the data transmission system includes memory database system, data warehouse, data loading module and SQL data management system；Data event is loaded into memory database system by the data loading module, data event is sent to the SQL data management system by the memory database system, the data event received is transferred to the data warehouse by the SQL data management system, the data warehouse stores the data event, and inquiry signal is sent to the memory database system, to inquire the not stored data event into the data warehouse.Advantage is: big data/rapid data system limitation can be overcome, to realize data analysis more preferably, more comprehensively and faster.
Description
Technical field
The present invention relates between data processing field more particularly to a kind of memory database system and data warehouse
Data transmission method and system.
Background technique
As more and more people and enterprise use computer, and as more services electronically provide, nowadays
Various common application programs need to receive, and handle and store mass data.In addition, common application program is also required to faster
Response speed and obtain more in order to do commercial decision-making needs various analysis dimensions information.It so just will appear some ask
Topic, not the shortcomings that the data processing of homologous ray itself, when these system interactions, the negative effect of these disadvantages can more
Greatly, it is less than the summation of its each section in some aspects so as to cause system.For example, realizing the product of in-system decryption (OLTP)
With the product of online analysis and processing (OLAP).
Data are synchronized in OLAP data warehouse from OLTP memory database, and OLTP memory database is usually to handle greatly
The system with affairs related application in real time is measured, multiple Real-time Transactions are received, the relatively easy and high efficiency of OLTP product
Characteristic determines that it is very sensitive to these interactions, so that them be allowed to support near real-time data analysis and decision.Therefore, this kind of production
Product are referred to as " rapid data "；But these " rapid data " systems are tended to can store in its memory database
The data of limited quantity are operated, and mass data is that OLTP product brings significant challenge.It is limited by memory database, it
Can not handle enough data needed for depth analysis.
Online analysis and processing (OLAP) product, such as data warehouse product.A greater amount of data can be stored and analyze, it is huge
Data storage allow it is more complicated, deeper into and the analysis of various dimensions.Therefore, this kind of product is sometimes referred to as " big data "；But
That as other scenes, big thing processing speed is all slow, the slow reason in OLAP data warehouse at least there are two.Firstly,
OLAP data warehouse always receives data and handles, such as the batch processing task in evening, for example, memory database can be specific
Many Transaction Informations are received during period (such as one day).Memory database these affairs are stored in it on the day of number
According in library.Then its being pushed to these batch datas in data warehouse at night.Since only timing receipt is more for data warehouse
Newly, so data warehouse is substantially delay disposal data, it is simply that the delay for receiving data means to handle data
Delay；Secondly, the framework of OLAP product determines that it runs slowly than memory system.
Big data and rapid data are the directions of two opposition, memory database and data warehouse the two system results
Different is that not homologous ray is stored in due to data, across different time dimension and the two to the analysis ability of data not
Together.Accordingly, there exist a problems, if it is desired to which the analysis of analysis in real time (or almost real-time) receives the storage in another system
Data, current system cannot all handle this demand, mainly since current system query performance is limited, the data of processing
Measure limited and data inaccessible property.Therefor it is required that big data/rapid data system limitation can be overcome
System and method, to realize data analysis more preferably, more comprehensively and faster.
Summary of the invention
The purpose of the present invention is to provide the transmission side datas between a kind of memory database system and data warehouse
Method and system, to solve foregoing problems existing in the prior art.
To achieve the goals above, The technical solution adopted by the invention is as follows:
A kind of data transmission method between memory database system and data warehouse, the memory database system
In include multiple parallel memory databases, also store in the memory database system and grasped parallel by multiple memory databases
At least one the memory database example made, includes the following steps,
S1, the memory database system in real time or near real-time receive multiple data events；
S2, the memory database example receive the notice of storing data event, and each data event is corresponded to and is stored
In corresponding queue；
S3, judge whether the data event in each queue meets update condition, when the data event in each queue is equal
When meeting update condition, data warehouse is updated；
S4, the data event stored in each queue is stored in the data warehouse, and to memory database
System sends inquiry signal；
S5, memory database system receive inquiry signal, and inquire the data thing being stored in the data warehouse
Part obtains the first query result；Meanwhile audit memory database instance, and obtain the second query result；
S6, first query result and second query result are compared, is obtained not stored into data warehouse
Data event in system, and the not stored data event into the data warehouse is sent to the memory database
System；
S7, the memory database system real-time reception should be to store into the data event in the data warehouse.
Preferably, the update condition is the quantity of received data event in designated time period and/or individual queue.
Preferably, quick acquisition module is provided in the memory database system, the quick acquisition module is for supervising
The received data event of the memory database system is listened, and is sent it in the data warehouse.
Preferably, it is provided at least one in the quick acquisition module and synchronizes monitor；When the memory database is real
Example receives the notice of storing data event, and each synchronous monitor divides the quick received data event of acquisition module
It is not stored in corresponding queue.
Preferably, the queue is proxy table, and the proxy table is for storing the received data of quick acquisition module
Event stores respectively, to ensure that each data event shares the load for being transmitted in the data warehouse.
Preferably, it is connected separately with a micro- batch of monitor in each proxy table, is provided in each micro- batch of monitor
Data event can be transferred to the parameter in data warehouse, and the parameter is batch size and/or the time of data event
Interval and/or data event rule of combination.
Preferably, different proxy table and micro- batch of prison is arranged in the type that data event is received according to memory database system
Listen device.
Preferably, the synchronous monitor can select corresponding proxy table storing data event, and detailed process is, described
The key of each data event is carried out hash and calculates its corresponding Hash codes modulus of acquisition by synchronous monitor, by each Hash codes
Modulus is multiplied with the quantity of proxy table respectively obtains the quantity for the data event that should be stored respectively in each proxy table, and will count
It is respectively stored in corresponding proxy table according to event.
The object of the invention is also to provide the data transmission between a kind of memory database system and data warehouse
System, the data transmission system include for realizing any of the above-described data transmission method, the data transmission system,
Memory database system, event for receiving data, and the data event is transferred to the data warehouse system
In system；
Data warehouse, the data event transmitted for receiving the memory database system；
Data loading module, the data loading module are used to the data event being loaded into the memory database system
In system；
SQL data management system；It is arranged between the memory database system and the data warehouse, being used for will
The received data event of memory database system is transferred in the Database Systems, and by load buffer to data warehouse
In, meanwhile, when the data warehouse accidental switches off, the SQL data management system can continue to operation and will be described
The data event sequence that memory database system transmits, and when data warehouse reopens, sequence will be sequenced
Data event is stored in the memory database system.
Preferably, be provided with partition table in the data transmission system, the partition table and the data loading module and
The synchronous monitor is connected；The partition table is used to distribute the load of the data event；When the data loading module obtains
The data event that takes is deleted, update or the partition table in creation new table clause when, the synchronous monitor is by data thing
The backup of part is respectively stored into corresponding proxy table, and each micro- batch of monitor monitors the number in coupled proxy table respectively
According to event, and satisfaction it will can be transferred to the data event of parameter in data warehouse in corresponding proxy table, be transferred to data
In warehouse system.
The beneficial effects of the present invention are: 1, big data/rapid data system limitation can be overcome, to realize more
Well, more comprehensively and faster data are analyzed.2, the data and progress of enormous amount, source dispersion, format multiplicity are capable of handling
Collect with association analysis etc., have the processing capacity of real-time high-efficiency and solves the problems, such as practical business.3, enterprise is provided to realize
Intelligent decision assists foundation.
Detailed description of the invention
Fig. 1 is data transmission method flow diagram in the embodiment of the present invention；
Fig. 2 is that data event is stored into the flow diagram in proxy table in the embodiment of the present invention；
Fig. 3 is data transmission procedure schematic diagram between the memory database system and data warehouse of the embodiment of the present invention two；
Fig. 4 is data transmission procedure signal between memory database system and data warehouse in the embodiment of the present invention three
Figure；
Fig. 5 is data transmission procedure signal between memory database system and data warehouse in the embodiment of the present invention four
Figure；
Fig. 6 is data transmission procedure signal between memory database system and data warehouse in the embodiment of the present invention five
Figure.
Fig. 7 is data transmission procedure schematic diagram between memory database system and data warehouse in the embodiment of the present invention six
Fig. 8 is data transmission system structural schematic diagram in the embodiment of the present invention；
Fig. 9 is the workflow schematic diagram of partition table in the embodiment of the present invention；
Figure 10 is the schematic diagram that data transmission system is realized based on holder in the embodiment of the present invention.
Specific embodiment
In order to make the objectives, technical solutions, and advantages of the present invention clearer, below in conjunction with attached drawing, to the present invention into
Row is further described.It should be appreciated that the specific embodiments described herein are only used to explain the present invention, it is not used to
Limit the present invention.
Embodiment one
As shown in Figure 1 to Figure 2, the number between a kind of memory database system and data warehouse is present embodiments provided
It include multiple parallel memory databases in the memory database system, in the memory database system according to transmission method
At least one memory database example by multiple memory database parallel work-flows is also store, is included the following steps,
S1, the memory database system in real time or near real-time receive multiple data events；
S2, the memory database example receive the notice of storing data event, and each data event is corresponded to and is stored
In corresponding queue；
S3, judge whether the data event in each queue meets update condition, when the data event in each queue is equal
When meeting update condition, data warehouse is updated；
S4, the data event stored in each queue is stored in the data warehouse, and to memory database
System sends inquiry signal；
S5, memory database system receive inquiry signal, and inquire the data thing being stored in the data warehouse
Part obtains the first query result；Meanwhile audit memory database instance, and obtain the second query result；
S6, first query result and second query result are compared, is obtained not stored into data warehouse
Data event in system, and the not stored data event into the data warehouse is sent to the memory database
System；
S7, the memory database system real-time reception should be to store into the data event in the data warehouse.
In the present embodiment, the update condition is the quantity of received data event in designated time period and/or individual queue.
In the present embodiment, quick acquisition module is provided in the memory database system, the quick acquisition module is used
In the monitoring received data event of memory database system, and send it in the data warehouse.
In the present embodiment, it is provided at least one in the quick acquisition module and synchronizes monitor；When the internal storage data
Library example receives the notice of storing data event, and each synchronous monitor is by the quick received data thing of acquisition module
Part is respectively stored in corresponding queue.
In the present embodiment, the queue is proxy table, and the proxy table is received for storing the quick acquisition module
Data event stores respectively, to ensure that each data event shares the load for being transmitted in the data warehouse.
In the present embodiment, it is connected separately with a micro- batch of monitor in each proxy table, is set in each micro- batch of monitor
Be equipped with the parameter that data event can be transferred in data warehouse, the parameter be data event batch size and/or
Time interval and/or data event rule of combination.
In the present embodiment, the type of data event is received according to memory database system, and different proxy table and micro- is set
Criticize monitor.
In the present embodiment, the synchronous monitor can select corresponding proxy table storing data event, and detailed process is,
The key of each data event is carried out hash and calculates its corresponding Hash codes modulus of acquisition by the synchronous monitor, by each Kazakhstan
Uncommon code modulus is multiplied with the quantity of proxy table respectively obtains the quantity for the data event that should be stored respectively in each proxy table, and
Data event is respectively stored in corresponding proxy table.
In the present embodiment, synchronous monitor selection or identification proxy table/queue placement data event method are specific as follows:
The key of each data event is subjected to hash first and calculates its corresponding Hash codes modulus of acquisition, the key can be data
The major key of event；Then by the way that the modulus of Hash codes is determined that each proxy table can receive data multiplied by the quantity of proxy table
The quantity of event, according to the following formula: the data event quantity of each proxy table storage=(carrying out Hash codes according to major key) % (generation
Manage table number).
In the present embodiment, if partition table lacks key, cyclic policy or some other is can be used in synchronous monitor
Load balance selects proxy table.Each proxy table has a micro- batch of monitor, this writes formula monitor after can also being referred to as,
Each miniature monitor monitored data event (associated proxy table and the data event in response to meeting condition), will be lined up
Data event be written data warehouse.Those skilled in the art will appreciate that it is micro to trigger that many condition/parameters can be used
Data warehouse is written in patch.For example, condition can be over time, become, certain amount has been had recorded in microcomputer group queue
After event or its common ground, its data event is sent data warehouse by microcomputer ticket queuing mechanism.Once data event is turned
It changes, micro- batch is removed.Data entry can have the key or label of instruction priority or service quality, can make data
Event is sent to data warehouse in a manner of accelerating or postpone, and depends on the circumstances.
Embodiment two
As shown in figure 3, the memory database system includes multiple memory databases in the present embodiment, the memory number
Form a memory database example parallel according to library, they are communicably coupled to data warehouse, and data warehouse includes
At least one data warehouse.Although each memory database includes synchronous monitor, the only one in memory database is real
Example.
In the present embodiment, although memory database system receives many input data events, so that any internal storage data
Library can receive data interaction, but is lined up and is only carried out by an example, i.e., based on memory database in the implementation；All teams
Column all reside in a single memory database.The synchronous prison of each of data event is received in associated memory database
It listens device that can complete to identify the process which queue receives data event, then transmits this information to queue appropriate.Such as this
Known to the those of ordinary skill of field, the resource in memory database system structure is usually registered to locator service, positioning
Device service is the registration table that resource can notice its position, therefore client and other repositioning devices can be found that movable money
Source.
Embodiment three
As shown in figure 4, memory database system includes multiple memory database examples, they are communicatedly in the present embodiment
It is coupled to data warehouse, data warehouse may include at least one data warehouse.Each memory database includes same
It walks monitor and further includes queue；It include a proxy table and relevant micro- batch of monitor in one memory database, with reality
Two difference of example is applied, the queue distribution in the embodiment is into multiple memory databases.One advantage of this configuration is queue
The resource in different system resources is occupied, whole handling capacity can be improved in this.In embodiment, network hop is in this system
Unnecessary in structure because for each event queue be occur event memory database defined in.Each
Data event is received in the synchronous associated memory database of monitor, and received data event is sent to team appropriate
Column.In the present embodiment, using locator or registration table, the synchronization monitor in a memory database can be easily positioned
And it is interacted with the queue resource in other memory databases.
Example IV
As shown in figure 5, a memory database system contains multiple memory database examples, each in the present embodiment
Database instance can be communicated with one or more data warehouses in data warehouse.Each memory database includes same
Monitor is walked, multiple queues are respectively further comprised, wherein each queue may include proxy table and associated micro- batch of monitor.It is this
One advantage of configuration is queue distribution in each memory database in resource, and handling capacity can be improved in this.The structure is not only
Reduce network hop as embodiment three, but also the load of micro- batch data is divided into single memory database example
Multiple threads in.Those skilled in the art should can be appreciated that the implementation with the framework in Fig. 3,4,5, and system is more and more flat
Row, and the throughput performance become better and better is provided.
In the present embodiment, with Fig. 3 and Fig. 4, the synchronous monitor of each of Fig. 5 is in associated memory database
When middle reception data event, the process of which queue reception data event of identification can be executed, is then transmitted to appropriate
Queue.Using locator or registration table, the synchronization monitor in a data in EMS memory library can be easily positioned and with it is another
Queue resource interaction in one data in EMS memory library.
In the present embodiment, as embodiment three, a benefit of the present embodiment is to improve the potentiality of throughput performance.This
Field is it should be recognized by those skilled in the art that the memory database system in the present embodiment can be configured to different embodiments, with reality
Existing various benefits.For example, in embodiment, one group of queue in an example can be the queue of one group of redundancy, as backup,
It goes wrong to prevent major queue group.Redundancy can be in same memory database, can also be simultaneous across memory database, or both
And there is it.
Embodiment five
As shown in fig. 6, one in data in EMS memory library group of queue can be used and further discriminate between data thing in the present embodiment
Part, to provide the update being more classified to data warehouse；Illustrate that the queue in the database in memory can be nested.Quick
Queue in acquisition module can be seen as nested queue.For example, all may be used with the associated all data events of a major key
To be sent to major queue, then data event can be separated further.In one embodiment, secondary key can be used and comes area
Data event is sent major queue by divided data event, then using load balancing (such as by illustrative and not limiting,
Recycle scheme) it is distributed to one of subqueue.
In the present embodiment, those skilled in the art will appreciate that the configuration of Fig. 6 can repeat on higher level, from
And there are the queuing levels of multiple nestings.
Embodiment six
As shown in fig. 7, described in the invention is " rapid data " rather than the duplication of the data of " big data " rank.Memory number
Tend to realize in RAM according to library, and data warehouse tends to realize using disk storage.Since RAM device is usually than being based on
The system of disk executes read-write operation faster, therefore in the present embodiment, execute data duplication be not in data warehouse rank, and
It is in memory database rank.By the way that reproduction process is moved to memory database, reproduction process can be performed faster.Cause
This, description executes data duplication in memory database rank in Fig. 7.
In the present embodiment, data in EMS memory library can be by network connection (for example, wide area network (WAN) is connected to for multiple
Another memory database of data processed).As previously mentioned, the data event in memory database is usually sent with a high-volume
To data warehouse.Sometimes the time needed for executing batch processing may be very long, it means that data warehouse has insufficient time to
This data is replicated to be backed up.Therefore, it does not execute duplication in data warehouse, executes in memory database rank.
In embodiment, can in memory database using one or more micro- batch queue embodiments, with previously described class
As mode send data event to the memory database of duplication, the memory database for just looking like the duplication is data warehouse one
Sample.
In the present embodiment, can memory database, duplication memory database or both in using one or more micro-
Batch queue embodiment.However, in the present embodiment, memory database can be used quick acquisition module, and the memory replicated
Database simply can send data warehouse for the data of duplication using traditional batch processing and back up.
Embodiment seven
It is provided as shown in Fig. 8 to Fig. 9, in the present embodiment between a kind of memory database system and data warehouse
Data transmission system, the data transmission system is for realizing data transmission method described above, the data transmission system
Including,
Memory database system, event for receiving data, and the data event is transferred to the data warehouse system
In system；
Data warehouse, the data event transmitted for receiving the memory database system；
Data loading module, the data loading module are used to the data event being loaded into the memory database system
In system；
SQL data management system；It is arranged between the memory database system and the data warehouse, being used for will
The received data event of memory database system is transferred in the Database Systems, and by load buffer to data warehouse
In, meanwhile, when the data warehouse accidental switches off, the SQL data management system can continue to operation and will be described
The data event sequence that memory database system transmits, and when data warehouse reopens, sequence will be sequenced
Data event is stored in the memory database system.
In the present embodiment, SQL data management system (VMware vFabr1c SQLF1re, it is characterized in that distributed, interior
Deposit, be unshared, having fault tolerance) in be provided with based on distributed shared drive mechanism, answered for the data-intensive modern times
Dynamic sealing ability and high performance distributed SQL database are provided with program.The internal memory optimization framework of SQLF1re is to greatest extent
Ground reduces the time for waiting disk access, this is the performance bottleneck of traditional database；SQL F1re can by memory pool come
Realize extension, CPU and across cluster, across geographical location storing data；Its data warehouse that can be used supports big data
Analysis, can manage and store the data of TB rank.
The weakness of low latency, is placed on front end for memory database system when in order to overcome memory database to handle data, will
Before SQL data management system is placed on data warehouse, user can extract, transmit, and by load buffer into data warehouse
(such as according to the time, according to the entry number of new data or according to event trigger) these all meet in processing business process by
Plan the mode waited.
In the present embodiment, when valuable data event to be loaded into data warehouse, SQL F1re is also to handle this
A little data provide a highly usable frame；If some calculate node fails, SQL F1re can be passed not closing data
Restore in the case where defeated system and rebalances data payload, in addition, if data warehouse is closed for some reason, SQL
F1re can be continued to run, sorting data event, and restart the slow of data event after restarting data warehouse
Punching.
In the present embodiment, partition table is provided in the data transmission system, the partition table and the data load mould
Block is connected with the synchronous monitor；The partition table is used to distribute the load of the data event；When the data load mould
When the data event that block obtains is deleted, creates new table clause in update or the partition table, the synchronous monitor will be counted
It is respectively stored into corresponding proxy table according to the backup of event, each micro- batch of monitor is monitored respectively in coupled proxy table
Data event, and will meet in corresponding proxy table and can be transferred to the data event of parameter in data warehouse, be transferred to
In data warehouse.
Embodiment eight
As shown in Figure 10, the present embodiment is for indicating the realization based on cloud of notebook data Transmission system.Quick acquisition module has
Help make the data in data warehouse preferably synchronous with memory database.
It include quick intake since data warehouse can update in real time (or almost in real time) now in the present embodiment
The data transmission system of module has " rapid data " and data warehouse (i.e. " big data " system of memory database system
System) in-depth analysis the advantages of.
In the present embodiment, data transmission system can support prioritized data to analyze, can add one or more labels and
The synchronous monitor of one or more, carries out priority ranking, the priority of inquiry in order to update to the data in data warehouse
It is ranked up, or priority ranking is carried out to the two simultaneously.Can have one or more secondary synchronous monitors check one or
Multiple labels are in order to handle them.
In the present embodiment, partition table may include the entry for marking (such as timestamp), which helps to track memory
Data event more new state between Database Systems and data warehouse.Therefore, if the inquiry executed needs reality simultaneously
When data event or near real-time data event and historical data event, then inquiry may be wanted with timestamp or comprising timestamp
It asks, and inquiry is not carried out before receiving the correct update based on data time stamp in data warehouse.Alternatively, data warehouse can
Data, analysis or both are all returned to memory database, memory database can be supplemented with the data event temporarily occurred
Data event, analysis or both, to realize real-time and big data analysis.As previously mentioned, data transmission system can also include
One or more label indicates whether and/or when should carry out priority ranking to inquiry, data event or both.Implementing
In example, priority, label or other indexs can be appointed as to a part of SQL query, which can correspond to memory number
According to library system, data warehouse or with it is both corresponding.
In the present embodiment, data transmission system includes central processing unit (CPU), it provides computing resource and controls calculating
Machine.CPU 1001 can be realized with microprocessor etc., can also include graphics processor and/or the floating-point association for mathematical computations
Processor.Data transmission system can also include system storage, can be random access memory (RAM) and read-only storage
The form of device (ROM).
In the present embodiment, some controllers and peripheral equipment can also be provided, input controller indicates to various inputs to set
Standby interface, such as keyboard, mouse or stylus.There may also be scanner controller, be communicated with scanner.Data transmission system is also
May include for the storage control with one or more storage device interfaces, each storage equipment include storage medium (such as
Tape or disk), or the optical medium of the program of operating system, utility program and application program instructions is store, the operation
System, utility program and application program may include the embodiment of program, in all fields using the present invention.Storage equipment also can be used
In the data for storing processed data or handling according to the present invention.Data transmission system can also include display controller, use
In provide display equipment interface, display equipment can be cathode-ray tube (CRT), thin film transistor (TFT) (TFT) display or its
The display of his type.Data transmission system can also include for the printer controller with printer communication.Communication control
Device can be docked with one or more communication device interfaces, this enable data transmission system by various networks (including
Internet, local area network (LAN), wide area network (WAN)) it is connected to remote equipment, or by any suitable electromagnetic carrier wave signal,
Remote equipment is connected to including infrared signal.
In the present embodiment, in shown data transmission system, all major system components may be coupled to bus, and bus can
To represent more than one physical bus.However, various system components can with or cannot physical access each other.For example, input number
According to and/or output data can be from a physical location remote transmission to another physical location.Furthermore it is possible to by network from
The program of various aspects of the invention is realized in remote location (such as server) access.These data and/or program can pass through
Any one of various machine readable medias include, including tape, disk, CD or transmitter, receiver.
In the present embodiment, it can be encoded in one or more non-transitory computer-readable mediums, this is computer-readable
Medium has the instruction for one or more processors, processing unit thereby executing step.It is noted that one or more non-
Temporary computer-readable medium should include volatile and non-volatile memory.It is further noted that alternate embodiments are possible
, including hardware realization or software/hardware realization.ASIC (s), programmable array, digital signal processing circuit etc. can be used
To reach the achievable function of hardware.Therefore, any " device " term is intended to cover software and hardware realization.Similarly, here
The term " computer-readable medium " used includes software and/or hardware or its group with the instruction repertorie realized on it
It closes.In view of going to realize these alternatives, it should be understood that the description of attached drawing and accompanying provides those skilled in the art and writes journey
Sequence code (i.e. software) and/or manufacture circuit (i.e. hardware) are with the functional information of execution.
By using above-mentioned technical proposal disclosed by the invention, following beneficial effect has been obtained:
The present invention by provide a kind of data transmission method between memory database system and data warehouse and
System can overcome big data/rapid data system limitation, to realize data analysis more preferably, more comprehensively and faster；
Be capable of handling enormous amount, source dispersion, format multiplicity data and collected with association analysis etc., have real-time high-efficiency
Processing capacity and solve the problems, such as practical business；This method and system are provided to enterprise simultaneously realizes intelligent decision auxiliary
Foundation.
The above is only a preferred embodiment of the present invention, it is noted that for the ordinary skill people of the art
For member, various improvements and modifications may be made without departing from the principle of the present invention, these improvements and modifications are also answered
Depending on protection scope of the present invention.
Claims (10)
1. the data transmission method between a kind of memory database system and data warehouse, in the memory database system
Including multiple parallel memory databases, also store in the memory database system by multiple memory database parallel work-flows
At least one memory database example, it is characterised in that: include the following steps,
S1, the memory database system in real time or near real-time receive multiple data events；
S2, the memory database example receive the notice of storing data event, and each data event correspondence is stored in phase
In the queue answered；
S3, judge whether the data event in each queue meets update condition, when the data event in each queue is all satisfied
When update condition, data warehouse is updated；
S4, the data event stored in each queue is stored in the data warehouse, and to memory database system
Send inquiry signal；
S5, memory database system receive inquiry signal, and inquire the data event being stored in the data warehouse, obtain
Take the first query result；Meanwhile audit memory database instance, and obtain the second query result；
S6, first query result and second query result are compared, is obtained not stored into data warehouse
In data event, and the not stored data event into the data warehouse is sent to the memory database system
System；
S7, the memory database system the real-time reception not stored data event into the data warehouse.
2. data transmission method its feature between memory database system according to claim 1 and data warehouse
Be: the update condition is the quantity of received data event in designated time period and/or individual queue.
3. data transmission method its feature between memory database system according to claim 1 and data warehouse
It is: is provided with quick acquisition module in the memory database system, the quick acquisition module is for monitoring the memory
The received data event of Database Systems, and send it in the data warehouse.
4. data transmission method its feature between memory database system according to claim 3 and data warehouse
It is: is provided at least one in the quick acquisition module and synchronizes monitor；It is deposited when the memory database example receives
The notice of data event is stored up, the quick received data event of acquisition module is respectively stored in phase by each synchronous monitor
In the queue answered.
5. according to benefit require 1 described in data transmission method its feature between memory database system and data warehouse exist
In: the queue is proxy table, and the proxy table stores respectively for storing the received data event of quick acquisition module,
To ensure that each data event shares the load for being transmitted in the data warehouse.
6. according to benefit require 5 described in data transmission method its feature between memory database system and data warehouse exist
In: a micro- batch of monitor is connected separately in each proxy table, being provided with data event in each micro- batch of monitor can
The parameter being transferred in data warehouse, the parameter are the batch size and/or time interval and/or data of data event
Composition of matter rule.
7. according to benefit require 6 described in data transmission method its feature between memory database system and data warehouse exist
In: the type of data event is received according to memory database system, and different proxy table and micro- batch of monitor are set.
8. according to benefit require 7 described in data transmission method its feature between memory database system and data warehouse exist
In: the synchronous monitor can select corresponding proxy table storing data event, and detailed process is that the synchronous monitor will
The key of each data event carry out hash calculate obtain its corresponding Hash codes modulus, by each Hash codes modulus respectively with generation
The quantity of reason table, which is multiplied, obtains the quantity for the data event that should be stored respectively in each proxy table, and data event is deposited respectively
Storage is in corresponding proxy table.
9. the data transmission system between a kind of memory database system and data warehouse, it is characterised in that: the data
Transmission system includes for realizing any data transmission method of the claims 1 to 8, the data transmission system,
Memory database system, event for receiving data, and the data event is transferred in the data warehouse；
Data warehouse, the data event transmitted for receiving the memory database system；
Data loading module, the data loading module are used to the data event being loaded into the memory database system
In；
SQL data management system；It is arranged between the memory database system and the data warehouse, being used for will be described
The received data event of memory database system is transferred in the Database Systems, and by load buffer into data warehouse,
Meanwhile when the data warehouse accidental switches off, the SQL data management system can continue to operation and by the memory
The data event sequence that Database Systems transmit, and when data warehouse reopens, the data of sequence will be sequenced
Event is stored in the memory database system.
10. the data transmission system between memory database system according to claim 9 and data warehouse, special
Sign is: being provided with partition table in the data transmission system, the partition table and the data loading module and described synchronous
Monitor is connected；The partition table is used to distribute the load of the data event；When the data that the data loading module obtains
Event is deleted, update or the partition table in creation new table clause when, the synchronous monitor is by the backup of data event
It being respectively stored into corresponding proxy table, each micro- batch of monitor monitors the data event in coupled proxy table respectively,
And satisfaction it will can be transferred to the data event of parameter in data warehouse in corresponding proxy table, be transferred to data warehouse
In.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910604930.0A CN110442627A (en) | 2019-07-05 | 2019-07-05 | Data transmission method and system between a kind of memory database system and data warehouse |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910604930.0A CN110442627A (en) | 2019-07-05 | 2019-07-05 | Data transmission method and system between a kind of memory database system and data warehouse |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN110442627A true CN110442627A (en) | 2019-11-12 |
Family
ID=68429432
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201910604930.0A Pending CN110442627A (en) | 2019-07-05 | 2019-07-05 | Data transmission method and system between a kind of memory database system and data warehouse |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN110442627A (en) |
Cited By (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111556066A (en) * | 2020-05-08 | 2020-08-18 | 国家计算机网络与信息安全管理中心 | Network behavior detection method and device |
| CN112988860A (en) * | 2019-12-18 | 2021-06-18 | 菜鸟智能物流控股有限公司 | Data acceleration processing method and device and electronic equipment |
| CN113468182A (en) * | 2021-07-14 | 2021-10-01 | 广域铭岛数字科技有限公司 | Data storage method and system |
| CN115329015A (en) * | 2022-10-14 | 2022-11-11 | 中孚安全技术有限公司 | Data warehouse system with hybrid architecture and implementation method |
Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101329685A (en) * | 2008-07-30 | 2008-12-24 | 烽火通信科技股份有限公司 | Implementing method of memory database on household gateway |
| US20120259809A1 (en) * | 2011-04-11 | 2012-10-11 | Sap Ag | In-Memory Processing for a Data Warehouse |
| CN103678665A (en) * | 2013-12-24 | 2014-03-26 | 焦点科技股份有限公司 | Heterogeneous large data integration method and system based on data warehouses |
| CN103942342A (en) * | 2014-05-12 | 2014-07-23 | 中国人民大学 | Memory database OLTP and OLAP concurrency query optimization method |
| CN106844703A (en) * | 2017-02-04 | 2017-06-13 | 中国人民大学 | A kind of internal storage data warehouse query processing implementation method of data base-oriented all-in-one |
| CN107329814A (en) * | 2017-06-16 | 2017-11-07 | 电子科技大学 | A kind of distributed memory database query engine system based on RDMA |
| US9934263B1 (en) * | 2012-12-04 | 2018-04-03 | Pivotal Software, Inc. | Big-fast data connector between in-memory database system and data warehouse system |
-
2019
- 2019-07-05 CN CN201910604930.0A patent/CN110442627A/en active Pending
Patent Citations (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101329685A (en) * | 2008-07-30 | 2008-12-24 | 烽火通信科技股份有限公司 | Implementing method of memory database on household gateway |
| US20120259809A1 (en) * | 2011-04-11 | 2012-10-11 | Sap Ag | In-Memory Processing for a Data Warehouse |
| EP2523124A1 (en) * | 2011-04-11 | 2012-11-14 | Sap Ag | In-memory processing for a data warehouse |
| US9934263B1 (en) * | 2012-12-04 | 2018-04-03 | Pivotal Software, Inc. | Big-fast data connector between in-memory database system and data warehouse system |
| CN103678665A (en) * | 2013-12-24 | 2014-03-26 | 焦点科技股份有限公司 | Heterogeneous large data integration method and system based on data warehouses |
| CN103942342A (en) * | 2014-05-12 | 2014-07-23 | 中国人民大学 | Memory database OLTP and OLAP concurrency query optimization method |
| CN106844703A (en) * | 2017-02-04 | 2017-06-13 | 中国人民大学 | A kind of internal storage data warehouse query processing implementation method of data base-oriented all-in-one |
| CN107329814A (en) * | 2017-06-16 | 2017-11-07 | 电子科技大学 | A kind of distributed memory database query engine system based on RDMA |
Cited By (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112988860A (en) * | 2019-12-18 | 2021-06-18 | 菜鸟智能物流控股有限公司 | Data acceleration processing method and device and electronic equipment |
| CN112988860B (en) * | 2019-12-18 | 2023-09-26 | 菜鸟智能物流控股有限公司 | Data acceleration processing method and device and electronic equipment |
| CN111556066A (en) * | 2020-05-08 | 2020-08-18 | 国家计算机网络与信息安全管理中心 | Network behavior detection method and device |
| CN113468182A (en) * | 2021-07-14 | 2021-10-01 | 广域铭岛数字科技有限公司 | Data storage method and system |
| CN115329015A (en) * | 2022-10-14 | 2022-11-11 | 中孚安全技术有限公司 | Data warehouse system with hybrid architecture and implementation method |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN110442627A (en) | Data transmission method and system between a kind of memory database system and data warehouse | |
| US10698891B2 (en) | MxN dispatching in large scale distributed system | |
| US8584136B2 (en) | Context-aware request dispatching in clustered environments | |
| US9589041B2 (en) | Client and server integration for replicating data | |
| US9934263B1 (en) | Big-fast data connector between in-memory database system and data warehouse system | |
| US7761556B2 (en) | Performance monitoring within an enterprise software system | |
| CN107038162A (en) | Real time data querying method and system based on database journal | |
| US10817532B2 (en) | Scientific computing process management system | |
| US20170286485A1 (en) | High Performance Query Processing and Data Analytics | |
| US20130346540A1 (en) | Storing and Moving Data in a Distributed Storage System | |
| CN103984761A (en) | Massive isomerous data storage method and system | |
| CN102541858A (en) | Data equality processing method, device and system based on mapping and protocol | |
| CN109146381A (en) | Logistics data monitoring method, device, electronic equipment and computer storage medium | |
| CN109478973A (en) | For task schedule, the SDN controller of resource granting and service offer, system and method | |
| CN110737643A (en) | big data analysis, processing and management center station based on catering information management system | |
| CN102761602A (en) | Hadoop-based mass data real-time analyzing and processing method | |
| US11822556B2 (en) | Exactly-once performance from a streaming pipeline in a fault-vulnerable system | |
| Belo et al. | Restructuring dynamically analytical dashboards based on usage profiles | |
| CN107276914B (en) | Self-service resource allocation scheduling method based on CMDB | |
| CN105892957B (en) | A kind of distributed transaction execution method based on Dynamic Program Slicing | |
| Abdelhamid et al. | Cruncher: Distributed in-memory processing for location-based services | |
| CN110389817A (en) | Dispatching method, device and the computer program product of cloudy system | |
| US20230185817A1 (en) | Multi-model and clustering database system | |
| US9203692B1 (en) | Optimized event routing in distributed data management | |
| JP3712791B2 (en) | Database management method and information processing apparatus therefor |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20191112 |