CN107870946A - Real time data dynamic connection method and device - Google Patents
Real time data dynamic connection method and device Download PDFInfo
- Publication number
- CN107870946A CN107870946A CN201610859240.6A CN201610859240A CN107870946A CN 107870946 A CN107870946 A CN 107870946A CN 201610859240 A CN201610859240 A CN 201610859240A CN 107870946 A CN107870946 A CN 107870946A
- Authority
- CN
- China
- Prior art keywords
- real time
- time data
- data
- destination node
- rule
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/245—Query processing
- G06F16/2458—Special types of queries, e.g. statistical queries, fuzzy queries or distributed queries
- G06F16/2471—Distributed queries
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/245—Query processing
- G06F16/2453—Query optimisation
- G06F16/24534—Query rewriting; Transformation
- G06F16/24542—Plan optimisation
- G06F16/24544—Join order optimisation
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/27—Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
Abstract
The invention discloses a kind of real time data dynamic connection method and device.Methods described includes：Receive the real time data from multiple data sources；According to association routing rule, the real time data received is routed to corresponding destination node；Each destination node for receiving route real time data, judge that whether possessed real time data meets condition of contact in each destination node, obtains corresponding judged result；When the judged result is certainly, the real time data in respective objects node is attached according to concatenate rule.
Description
Technical field
The present invention relates to data processing field, more particularly to real time data dynamic connection method and device.
Background technology
With the arrival of the rapid development of internet industry, especially big data epoch, collection, storage to data, place
Reason and processing seem important all the more, the focus that real time data even more wherein receives much attention.Nowadays, to the processing of magnanimity real time data
Tend in real-time streaming calculates carry out, final result is stored into database.Due to the data calculated in real time be it is instantaneous,
, it is necessary to which data are immediately transmitted into downstream after being handled at node, in addition, arrival opportunity that also can not be to data is carried out
Control.
Data connection (data JOIN) is the important content of Data processing.When the data in database come from multiple data
During source, generally require to be combined data according to the association between these different types of data.This in database
With reference to being static combination.But in real time data processing, static combination can not be realized calculates remittance in real time to the various dimensions of data
Always, and the data volume that is finally persisted in database is bigger, and inquiry velocity is slower.
The content of the invention
In order to solve the above-mentioned problems in the prior art, the present invention proposes a kind of real time data dynamic connection method
And device.
According to an aspect of the present invention, it is proposed that a kind of real time data dynamic connection method.Methods described includes：Receive
Real time data from multiple data sources；According to association routing rule, the real time data received is routed to corresponding target
Node；Each destination node for receiving route real time data, judge possessed real-time in each destination node
Whether data meet condition of contact, obtain corresponding judged result；When the judged result is certainly, according to concatenate rule pair
Real time data in respective objects node is attached.
Preferably, methods described also includes：When the judged result is negates, will be route recently in respective objects node
Real time data stored together with the real time data stored before.
Preferably, methods described also includes：It is real-time to what is received before the real time data to being received is route
Data are filtered, to remove the real time data repeated.
Preferably, association routing rule, condition of contact and the concatenate rule are pre-configured with.
Preferably, the association routing rule is the mapping table between data source and destination node.
Preferably, the condition of contact provides at least one in the following：Possessed number in real time in destination node
According to from data source, the quantity of real time data from different data sources.
Preferably, the concatenate rule provides at least one in the following：Real time data from different data sources
The order of connection, the connection quantity of real time data from different data sources, connection type.
According to another aspect of the invention, it is proposed that a kind of real time data Dynamic link library device.Described device includes using
In the one or more destination nodes for accommodating the real time data by route.Described device also includes：Receiving module, for receiving
Real time data from multiple data sources；Routing module, for being route the real time data received according to association routing rule
Corresponding destination node into one or more of destination nodes；Judge module, for for receiving route reality
When data each destination node judge that whether possessed real time data meets condition of contact in each destination node, obtains phase
The judged result answered；And link block, for when the judged result for certainly when according to concatenate rule to respective objects section
Real time data in point is attached.
Preferably, described device also includes：Memory module, for when the judged result for negative when by respective objects section
The real time data being route recently in point is stored together with the real time data stored before.
Preferably, described device also includes：Filtering module, it is connected between the receiving module and the routing module,
For being filtered to the real time data received, to remove the real time data repeated.
Preferably, association routing rule, condition of contact and the concatenate rule are pre-configured with.
Preferably, the association routing rule is the mapping table between data source and destination node.
Preferably, the condition of contact provides at least one in the following：Possessed number in real time in destination node
According to from data source, the quantity of real time data from different data sources.
Preferably, the concatenate rule provides at least one in the following：Real time data from different data sources
The order of connection, the connection quantity of real time data from different data sources, connection type.
By using real time data dynamic connection method proposed by the invention and device, can realize to multiple streaming numbers
According to real-time merging, it is possible thereby to downstream carry out various dimensions System calculating.In addition, enter caused by the solution of the present invention
Storehouse data are reduced, and can be alleviated the pressure of database and be improved inquiry velocity.
Brief description of the drawings
Fig. 1 shows the flow chart of real time data dynamic connection method according to an embodiment of the invention.
Fig. 2 shows the block diagram of real time data Dynamic link library device according to an embodiment of the invention.
Embodiment
The specific embodiment of the present invention is described more fully below, it should be noted that the embodiments described herein is served only for illustrating
Illustrate, be not intended to limit the invention.In the following description, in order to provide thorough understanding of the present invention, a large amount of spies are elaborated
Determine details.It will be apparent, however, to one skilled in the art that：This hair need not be carried out using these specific details
It is bright.In other instances, in order to avoid obscuring the present invention, known circuit, material or method are not specifically described.
Throughout the specification, meaning is referred to " one embodiment ", " embodiment ", " example " or " example "
：It is comprised in reference to special characteristic, structure or the characteristic that the embodiment or example describe at least one embodiment of the present invention.
Therefore, each local phrase " in one embodiment " occurred in entire disclosure, " in embodiment ", " example "
Or " example " is not necessarily all referring to the same embodiment or example.Furthermore, it is possible to will be specific with any appropriate combination and/or sub-portfolio
Feature, structure or property combination in one or more embodiments or example.In addition, those of ordinary skill in the art should manage
Solution, accompanying drawing is provided to the purpose of explanation provided herein, and accompanying drawing is not necessarily drawn to scale.Art used herein
Language "and/or" includes any and all combination for the project that one or more correlations are listed.
Herein, the operation such as the reception of real time data and route can be realized by Storm systems.Storm systems
It is a kind of distributed real-time streaming computing system, it can be with horizontal extension, automatic fault tolerant, and can realize the delay of Millisecond,
It is usually used in the field such as analysis in real time and online machine learning.
Herein, the operation such as judgement, connection and storage of real time data can be realized by Redis systems.Redis systems
System is a kind of high performance key-value memory database systems.
Herein, " connection " of data is also referred to as " JOIN " of data, refers to two or more data by specified
The connection or combination that condition is carried out.
The present invention is specifically described below with reference to accompanying drawing.
Fig. 1 shows the flow chart of real time data dynamic connection method 100 according to an embodiment of the invention.It is described real-time
Data dynamic connection method 100 starts from step S110, receives the real time data from multiple data sources.Then, in step
In S120, according to association routing rule, the real time data received is routed to corresponding destination node.Next, in step
In S130, each destination node for receiving route real time data, possessed reality in each destination node is judged
When data whether meet condition of contact, obtain corresponding judged result.Finally, in step S140, when the judged result is
When certainly, the real time data in respective objects node is attached according to concatenate rule.
In step s 110, the real time data from multiple data sources is received.
In one embodiment, there can be different types of data from different data sources.In other embodiments,
Can from the data of different data sources same types, wherein, the data of the same type from different data sources can be with
For different objects, for different purposes or suitable for different operations.In an alternative embodiment, from different data sources
The repetition that the data of same type can be only for preventing loss of data and carrying out receives.In another embodiment, in difference
At the time of, can be from the different types of real time data of identical data sources.
In the step s 120, according to association routing rule, the real time data received is routed to corresponding destination node.
In one embodiment, the association routing rule is the mapping table between data source and destination node.So as to press
The real time data from different data sources is routed to corresponding destination node according to the mapping relations in mapping table.In another implementation
In example, real time correlation routing rule is formulated according to data type, accordingly it is desirable to first to from the real-time of particular source
The data type of data is judged, the real time data then is routed into corresponding destination node according to data type.
In another embodiment, can be first real-time to what is received before the real time data to being received is route
Data are filtered, and remove the real time data of repetition, so as to mitigate the burden of system resource.
In step s 130, for receiving each destination node of route real time data, each target section is judged
Whether possessed real time data meets condition of contact in point, obtains corresponding judged result.
The real time data of storage (comes from before possessed real time data is included in the destination node in each destination node
In real-time Data Transmission before or the data being permanently kept in node) and this reception in received it is newest real-time
Data.
In one embodiment, the condition of contact provide possessed real time data in destination node from data
Source and/or the quantity of real time data from different data sources.
In step S140, when the judged result is certainly, according to concatenate rule to the reality in respective objects node
When data be attached.Now, the condition that data fit included in destination node is attached, then according to specific connection
Rule performs connection and by the obtained data distributing by connecting to downstream, carries out subsequent operation.
In one embodiment, the concatenate rule provides at least one in the following：From different data sources
The order of connection of real time data, the connection quantity of real time data from different data sources, connection type.Wherein, connection type
Explicit connection, implicit connection, equal connection, Nature Link, interconnection etc. can be included.
In one embodiment, methods described 100 is further comprising the steps of：When the judged result is negates, by phase
The real time data being route recently in destination node is answered to be stored together with the real time data stored before.The real-time number stored
According to that can be stored in corresponding destination node, can also be stored in for the destination node on outside or remote server,
But all can be described as is the real time data for destination node storage.In subsequent process, when continuing to for this
During the real time data of respective nodes, then real time data of this storage have become " real time data stored before " one
Part.
Preferably, association routing rule, condition of contact and the concatenate rule are pre-configured with.
Fig. 2 shows the block diagram of real time data Dynamic link library device 200 according to an embodiment of the invention.Described device
There are the one or more destination nodes for being used for accommodating the real time data by route in 200.In addition, described device 200 includes
Receiving module 210, routing module 220, judge module 230 and link block 240.Specifically, receiving module 210, which is used to receive, comes
From the real time data of multiple data sources.Routing module 220 is used to be route the real time data received according to association routing rule
Corresponding destination node into one or more of destination nodes.Judge module 230 is used for for receiving what is route
Each destination node of real time data judges that whether possessed real time data meets condition of contact in each destination node, obtains
Corresponding judged result.Link block 240 is used for when the judged result is certainly according to concatenate rule to respective objects section
Real time data in point is attached.
In one embodiment, described device 200 may also include memory module, and the memory module is used to judge to tie when described
Fruit is stored the real time data being route recently in respective objects node when being negative together with the real time data stored before.
In one embodiment, described device 200 may also include filtering module, and the filtering module is connected to the reception mould
Between block and the routing module, for being filtered to the real time data received, to remove the real time data repeated.
The real time data Dynamic link library device 200 in Fig. 2 and the phase of real time data dynamic connection method 100 in Fig. 1
It is corresponding.Device 200 is equally applicable to other specific descriptions of method 100 and explanation above, will not be repeated here.
Detailed description above has elaborated inspection method and system by using schematic diagram, flow chart and/or example
Numerous embodiments.In the case where this schematic diagram, flow chart and/or example include one or more functions and/or operation,
It will be understood by those skilled in the art that each function and/or operation in this schematic diagram, flow chart or example can be by various
Structure, hardware, software, firmware or they substantial any combination to realize individually and/or jointly.In one embodiment,
If the stem portion of theme described in embodiments of the invention can pass through application specific integrated circuit (ASIC), field programmable gate array
(FPGA), digital signal processor (DSP) or other integrated forms are realized.However, those skilled in the art will appreciate that
The some aspects of embodiments disclosed herein can be realized equally in integrated circuits on the whole or partly, be embodied as
The one or more computer programs run on one or more computer are (for example, be embodied as in one or more computer
The one or more programs run in system), it is embodied as the one or more program (examples run on the one or more processors
Such as, it is embodied as the one or more programs run in one or more microprocessors), it is embodied as firmware, or it is substantially real
It is now any combination of aforesaid way, and those skilled in the art will be provided with designing circuit and/or write-in is soft according to the disclosure
The ability of part and/or firmware code.In addition, it would be recognized by those skilled in the art that the mechanism of theme described in the disclosure can be made
It is distributed for the program product of diversified forms, and no matter is actually used for performing the particular type of the signal bearing medium of distribution
How, the exemplary embodiment of theme described in the disclosure is applicable.The example of signal bearing medium includes but is not limited to：It is recordable
Type medium, such as floppy disk, hard disk drive, compact-disc (CD), digital universal disc (DVD), digital magnetic tape, computer storage；
And transmission type media, such as numeral and/or analogue communication medium are (for example, optical fiber cable, waveguide, wired communications links, channel radio
Believe link etc.).
Although exemplary embodiment describing the present invention with reference to several, it is to be understood that, term used is explanation and shown
Example property and nonrestrictive term.Because the present invention can be embodied without departing from the spiritual or real of invention in a variety of forms
Matter, it should therefore be appreciated that above-described embodiment is not limited to any foregoing details, and should be in the spirit that appended claims are limited
With widely explained in scope, therefore the whole changes fallen into claim or its equivalent scope and remodeling all should be the power of enclosing
Profit requires to be covered.
Claims (14)
1. a kind of real time data dynamic connection method, including：
Receive the real time data from multiple data sources；
According to association routing rule, the real time data received is routed to corresponding destination node；
Each destination node for receiving route real time data, judge possessed number in real time in each destination node
According to whether condition of contact is met, corresponding judged result is obtained；
When the judged result is certainly, the real time data in respective objects node is attached according to concatenate rule.
2. the method according to claim 11, in addition to：
It is when the judged result is negates, the real time data being route recently in respective objects node is real-time with storing before
Data are stored together.
3. method according to claim 1 or 2, in addition to：Before the real time data to being received is route, to institute
The real time data of reception is filtered, to remove the real time data repeated.
4. method according to claim 1 or 2, association routing rule, condition of contact and the concatenate rule are to match somebody with somebody in advance
Put.
5. method according to claim 1 or 2, the association routing rule is the mapping between data source and destination node
Table.
6. method according to claim 1 or 2, the condition of contact provides at least one in the following：Target section
In point possessed real time data from data source, the quantity of real time data from different data sources.
7. method according to claim 1 or 2, the concatenate rule provides at least one in the following：From not
The order of connection with the real time data of data source, the connection quantity of the real time data from different data sources, connection type.
8. a kind of real time data Dynamic link library device, including for accommodating one or more targets of the real time data by route
Node, described device also include：
Receiving module, for receiving the real time data from multiple data sources；
Routing module, for the real time data received to be routed into one or more of target sections according to association routing rule
Corresponding destination node in point；
Judge module, for judging institute in each destination node for receiving each destination node of route real time data
Whether the real time data having meets condition of contact, obtains corresponding judged result；And
Link block, for when the judged result for certainly when according to concatenate rule to the real time data in respective objects node
It is attached.
9. device according to claim 8, in addition to：
Memory module, for when the judged result for negative when by the real time data being route recently in respective objects node therewith
The real time data of preceding storage is stored together.
10. device according to claim 8 or claim 9, in addition to：Filtering module, it is connected to the receiving module and the road
By between module, for being filtered to the real time data received, to remove the real time data repeated.
11. device according to claim 8 or claim 9, association routing rule, condition of contact and the concatenate rule are to match somebody with somebody in advance
Put.
12. device according to claim 8 or claim 9, the association routing rule is reflecting between data source and destination node
Firing table.
13. device according to claim 8 or claim 9, the condition of contact provides at least one in the following：Target section
In point possessed real time data from data source, the quantity of real time data from different data sources.
14. device according to claim 8 or claim 9, the concatenate rule provides at least one in the following：From not
The order of connection with the real time data of data source, the connection quantity of the real time data from different data sources, connection type.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201610859240.6A CN107870946A (en) | 2016-09-28 | 2016-09-28 | Real time data dynamic connection method and device |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201610859240.6A CN107870946A (en) | 2016-09-28 | 2016-09-28 | Real time data dynamic connection method and device |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN107870946A true CN107870946A (en) | 2018-04-03 |
Family
ID=61761109
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201610859240.6A Pending CN107870946A (en) | 2016-09-28 | 2016-09-28 | Real time data dynamic connection method and device |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN107870946A (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110096514A (en) * | 2019-04-01 | 2019-08-06 | 跬云(上海)信息科技有限公司 | Data query method and apparatus |
Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103152287A (en) * | 2013-03-27 | 2013-06-12 | 恒生电子股份有限公司 | Method and device for reliably receiving real-time data |
| CN103377236A (en) * | 2012-04-26 | 2013-10-30 | 中兴通讯股份有限公司 | Connection query method and system for distributed database |
| CN103905833A (en) * | 2013-07-12 | 2014-07-02 | 吉首大学 | Distributed network video data mining and collecting system based on cloud calculation |
| CN104885078A (en) * | 2012-12-29 | 2015-09-02 | 华为技术有限公司 | Method for two-stage query optimization in massively parallel processing database clusters |
| CN105071994A (en) * | 2015-08-27 | 2015-11-18 | 许继集团有限公司 | Mass data monitoring system |
| US20160197862A1 (en) * | 2009-01-15 | 2016-07-07 | Social Communications Company | Context based virtual area creation |
| CN105786808A (en) * | 2014-12-15 | 2016-07-20 | 阿里巴巴集团控股有限公司 | Method and apparatus for executing relation type calculating instruction in distributed way |
-
2016
- 2016-09-28 CN CN201610859240.6A patent/CN107870946A/en active Pending
Patent Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20160197862A1 (en) * | 2009-01-15 | 2016-07-07 | Social Communications Company | Context based virtual area creation |
| CN103377236A (en) * | 2012-04-26 | 2013-10-30 | 中兴通讯股份有限公司 | Connection query method and system for distributed database |
| CN104885078A (en) * | 2012-12-29 | 2015-09-02 | 华为技术有限公司 | Method for two-stage query optimization in massively parallel processing database clusters |
| CN103152287A (en) * | 2013-03-27 | 2013-06-12 | 恒生电子股份有限公司 | Method and device for reliably receiving real-time data |
| CN103905833A (en) * | 2013-07-12 | 2014-07-02 | 吉首大学 | Distributed network video data mining and collecting system based on cloud calculation |
| CN105786808A (en) * | 2014-12-15 | 2016-07-20 | 阿里巴巴集团控股有限公司 | Method and apparatus for executing relation type calculating instruction in distributed way |
| CN105071994A (en) * | 2015-08-27 | 2015-11-18 | 许继集团有限公司 | Mass data monitoring system |
Non-Patent Citations (1)
| Title |
|---|
| 刘学军等: "分布式数据流连接查询算法", 《计算机工程》 * |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110096514A (en) * | 2019-04-01 | 2019-08-06 | 跬云(上海)信息科技有限公司 | Data query method and apparatus |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US10749668B2 (en) | Reduction in storage usage in blockchain | |
| CN110998558B (en) | Delayed updating of database hash codes in a blockchain | |
| US9952940B2 (en) | Method of operating a shared nothing cluster system | |
| US9954939B2 (en) | Processing a message received by a message server | |
| CN109508326B (en) | Method, device and system for processing data | |
| CN109542352B (en) | Method and apparatus for storing data | |
| CN105530272A (en) | Method and device for application data synchronization | |
| US20140297769A1 (en) | Retrieving messages in order in a distributed publish/subscribe system | |
| US9946808B2 (en) | Using vertex self-information scores for vertices in an entity graph to determine whether to perform entity resolution on the vertices in the entity graph | |
| CN109005208B (en) | Method and device for pushing information | |
| CN114185678A (en) | Data storage method, device, equipment and storage medium | |
| US11341842B2 (en) | Metering data management system and computer readable recording medium | |
| CN107870946A (en) | Real time data dynamic connection method and device | |
| US7996499B2 (en) | Intelligent processing of web service requests for backward compatibility | |
| CN109218338B (en) | Information processing system, method and device | |
| CN113572560A (en) | Method, electronic device, and storage medium for determining clock synchronization accuracy | |
| CN111405037A (en) | Block synchronization method, device and storage medium | |
| US10705755B2 (en) | Method, apparatus and computer program product for data backup | |
| RU2698766C1 (en) | Method and device for transmitting, sending and receiving information | |
| CN109542981A (en) | A kind of system and method, electronic equipment, storage medium that data synchronize | |
| CN115099972A (en) | Transaction data processing method, device and equipment based on event-driven architecture | |
| CN110413427B (en) | Subscription data pulling method, device, equipment and storage medium | |
| CN109712011B (en) | Community discovery method and device | |
| CN108959302B (en) | Method for data processing and apparatus therefor | |
| US10929356B2 (en) | Detection of hidden data co-occurrence relationships |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20180403 |