CN104408132A - Data push method and system - Google Patents
Data push method and system Download PDFInfo
- Publication number
- CN104408132A CN104408132A CN201410706244.1A CN201410706244A CN104408132A CN 104408132 A CN104408132 A CN 104408132A CN 201410706244 A CN201410706244 A CN 201410706244A CN 104408132 A CN104408132 A CN 104408132A
- Authority
- CN
- China
- Prior art keywords
- database
- data
- incremental
- server
- message queue
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/25—Integrating or interfacing systems involving database management systems
- G06F16/256—Integrating or interfacing systems involving database management systems in federated or virtual databases
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/50—Network services
- H04L67/55—Push-based network services
Landscapes
- Engineering & Computer Science (AREA)
- Databases & Information Systems (AREA)
- Theoretical Computer Science (AREA)
- Data Mining & Analysis (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Computer Networks & Wireless Communication (AREA)
- Signal Processing (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention relates to a data push method and system. The data push method comprises the following steps of task creation extraction processing, dumping data from a source end database to a database increment log analysis server with a database embodiment being served as a boundary, obtaining incremental data through database incremental log analysis and temporarily storing the obtained incremental data into the memory of the database incremental log analysis server; information distribution processing, distributing the obtained incremental data to one or a plurality of corresponding message queue of a distributed message queue server with corresponding themes and marks according to a required rule, wherein the themes represent logical library names, the marks represent the logical table names, every distributed incremental data which is served as a message comprises a corresponding theme and mark; subscription push processing, enabling the distributed message queue server to push one or a plurality of message queues to a subscription end based on the subscripted themes of the subscription tasks.
Description
Technical field
The present invention relates to the large-scale data stream real-time consumption technology of resolving based on database Incremental Log, more specifically, relate to a kind of data push method and system.
Background technology
Along with the development of corporate business, the demand for large-scale data pick-up is strong all the more.Data pick-up refers to from source database and extracts the data needed.In practical application, the more employing of source database be relational database.Common data pick-up mode comprises: (one) full dose data pick-up, and full dose data pick-up is similar to Data Migration or data Replica, and it extracts intact for the data of the table in source or view from database; (2) incremental data extracts, and incremental data extracts data that are newly-increased in the table only extracting and will extract in database since last time is extracted or amendment.Incremental data extracts comparatively full dose data pick-up and applies wider.The data of how to catch change are the keys that incremental data extracts.Generally there are 2 requirements to catching method: accuracy, the delta data in operation system can be captured exactly; Performance, reduces as far as possible and causes too large pressure to operation system, affect existing business.
When the mode of traditional timing dump (dump) incremental data can not satisfy the demands, the acquisition of real-time incremental data becomes the trend addressed this problem.Data-pushing platform is a set of total solution realizing the data-pushing realizing increasing/full dose between heterogeneous database.Increment pushes the propelling movement incremental data that is intended to solve extensive lower efficient stable to subscribing to client to meet the object of business demand.
The extraction mode of traditional incremental data is that timing dump incremental data generates corresponding data file, but this mode exists following problem: need to increase timestamp field in source traffic table, pollute source traffic table to a certain extent; Timing performs and causes data existence time delay to a certain degree; Dump data file out and business association tension, not easily reused; Timed task often causes larger instantaneous pressure to source database and network when starting.
And solve the problems referred to above well based on the incremental data stream that database Incremental Log is resolved, can to source traffic table without intrusion, quasi real time (as long as subscription end consumption rate is enough fast) of incremental data, extracts and subscription data as required, h falls apart, and not easily occurs instantaneous peak value.Such as, the canal open source projects adopted in prior art is exactly resolve based on database Incremental Log, and provides increment subscription and consumption function.Canal can be good at fulfillment database Incremental Log parsing work, and based on the function of canal, we can extract incremental data by configuration extraction task from source and be temporary in internal memory, then according to the subscription client of business demand exploitation customization.
Fig. 1 schematically illustrates the view of the data delivery system according to prior art.The data delivery system 100 of prior art comprises: source database 101; Database Incremental Log resolution server 103, resolves based on database Incremental Log, provides increment to subscribe to and consumption function, database Incremental Log resolution server 103 such as canal server; Subscription end 105, for generation of subscription task.
Although canal provides the basic function that incremental data is resolved, but still faces following problem:
(1). data pick-up and data subscription cannot realize one-to-many
Because canal incremental data is not landed the design of (that is, data temporary storage is in internal memory) and limit, canal cannot accomplish only extract a secondary data and subscribed to by multi-client.This also just means extraction and subscribes to and must occur in pairs, the pressure that too much extraction task increases extraction source database, the load-bearing capacity and the handling capacity that waste a large amount of network traffics, reduce separate unit canal server.
(2). the incremental data holding time is not controlled
This problem is also relevant with above-mentioned design, because incremental data is not landed, and the daily record holding time of source database can not by data-pushing platform courses, so data-pushing platform also cannot promise to undertake the incremental data pot-life to the subscription end in downstream.This reduces the ability that data-pushing platform externally provides service.
(3). subscription end with extract hold relied on strong
Here dependence is divided into two aspects.
System level: extracting end with subscription end is based on the serial processing under long connection, and the consumption rate of subscription end directly affects the extraction speed extracting end.
Service layer: extracting end task take database instance as border, cannot shield the information that the subscription end such as source point storehouse submeter originally should not be concerned about, also just result in coupling stronger between extraction task and subscription task to subscription end.
(4). subscription end lacks common function support
Canal itself is only absorbed in and the extraction of data and subscription, and it is more weak for some conventional function supports of subscription end, such as: although the structured data of increment can be obtained by the subscription end SDK (SoftwareDevelopment Kit: SDK (Software Development Kit)) of canal, support not enough for structured data being resolved, changing into corresponding DML (Data Manipulation Language: data manipulation language (DML)) and finally insert this modal application scenarios of target DB (Database: database).Another open source projects otter (chained address of increasing income of project: https: //github.com/alibaba/otter) although support above-mentioned partial function, but still cannot meet the customized demand of Complex Flexible.
Summary of the invention
According to an aspect of the present invention, provide a kind of data push method, comprise: extract task creation process, with database instance be border by data from source database dump to database Incremental Log resolution server, resolved by database Incremental Log and obtain incremental data, and the incremental data of acquisition is temporary in the internal memory of database Incremental Log resolution server; Message distribution process, by the incremental data rule as required that obtains using corresponding theme and mark as message distribution in the corresponding one or more message queue in Distributed Message Queue server, theme represents the library name of logic, mark represents the table name of logic, and the incremental data that every bar is distributed as message all comprises corresponding theme and mark; And subscribe to and push process, the theme that Distributed Message Queue server is subscribed to based on each subscription task of subscription end and described one or more message queue is pushed to subscription end.
According to a further aspect in the invention, provide a kind of data delivery system, comprising: source database; Database Incremental Log resolution server; Distributed Message Queue server; And subscription end, wherein, with database instance be border by data from source database dump to database Incremental Log resolution server, database Incremental Log resolution server is resolved by database Incremental Log and is obtained incremental data, and the incremental data of acquisition is temporary in the internal memory of database Incremental Log resolution server; Database Incremental Log resolution server by the incremental data rule as required that obtains using corresponding theme and mark as message distribution in the corresponding one or more message queue in Distributed Message Queue server, theme represents the library name of logic, mark represents the table name of logic, and the incremental data that every bar is distributed as message all comprises corresponding theme and mark; And the theme subscribed to based on each subscription task of subscription end of Distributed Message Queue server and described one or more message queue is pushed to subscription end.
The present invention is by introducing persistent message queue means, can realize only extracting a secondary data and being subscribed to by multi-client, that is extract and subscribe to and need not occur in pairs, thus also greatly reduce the flow of network, make incremental data be become possibility by many subscription end consumption, greatly improve load-bearing capacity and the service efficiency of server.In addition, subscription end, by using the special SDK after encapsulating based on RocketMQ client, can realize the customized development of subscription end, thus meet the customized demand of Complex Flexible.
Accompanying drawing explanation
Fig. 1 schematically illustrates the view of the data delivery system according to prior art.
Fig. 2 schematically illustrates the view of data delivery system according to an embodiment of the invention.
Fig. 3 schematically illustrates the process flow diagram of data push method according to an embodiment of the invention.
Embodiment
Explain technical scheme according to an embodiment of the invention in detail below with reference to accompanying drawings.
Term " dump " (" dump ") implication refers to and the data (can contain data structure) in MySQL is exported to text, to import on other database instance or as other application or the input data of system.
Fig. 2 schematically illustrates the view of data delivery system according to an embodiment of the invention.As shown in Figure 2, data delivery system 200 comprises: source database 201, database Incremental Log resolution server 203, Distributed Message Queue server 205, subscription end 207.For purposes of illustration, three subscription task 1,2,3 that illustrate only three message queues 1,2,3 in Fig. 2 and correspond respectively, but it will be appreciated by those skilled in the art that and the present invention is not limited thereto to have more or less message queue and subscription task.
Source database 201 is not limited to certain specific type of data storehouse, such as, can be MySQL database." MySQL " is the small-sized correlation data base management system (DBMS) of an open source code.
Database Incremental Log resolution server 203, for resolving based on database Incremental Log, provides increment to subscribe to and consumption function.Database Incremental Log resolution server 203 can be such as canal server." canal " resolves based on MySQL database Incremental Log, provides incremental data to subscribe to and consumption function.The chained address of increasing income of canal project: https: //github.com/alibaba/canal.
Distributed Message Queue server 205 for providing distributed, persistent message queue service, it has the guarantee order of message and the feature of message duration, thus can meet the feature (strong order) of incremental data and the feature of persistence (persistence) demand.Can according to the remember history disposing Distributed Message Queue server 205 disk size situation external commitment data, if data in the term of validity retained, subscription end can from optional position consumption data.Distributed Message Queue server 205 can be such as RocketMQ server." RocketMQ " is the message-oriented middleware of a distributed, queuing model, has following characteristics: support strict message sequence; Support theme (Topic) and queue (Queue) two kinds of patterns; Hundred million grades of message pile up ability; More friendly distributed nature; Support to push away (Push) and pull (Pull) mode to consume message simultaneously.The chained address of increasing income of RocketMQ project: https: //github.com/alibaba/RocketMQ.By the introducing of the Distributed Message Queue middleware of above-mentioned such as RocketMQ, substantially increase the extendability of system, the distribution situation of incremental data in different distributions formula Message Queuing server can be adjusted in time, to reach the object of Quick Extended according to the size of business datum amount.Term " middleware " is one independently system software or service routine, and Distributed Application software is by this software shared resource between different technology.
Subscription end 207, for generation of subscription task.Although Fig. 2 schematically illustrates a subscription end 207 and have subscribed three subscription task 1,2,3, but those skilled in the art understand, the present invention is not limited thereto, the number of subscription end can be one or more, and the subscription task of each subscription end can be one or more.If subscription end 207 is modal databases push scene, only need does some and the most simply subscribe to configuration (theme and target DB as subscription) and subscription end 207 can be started and without the need to developing again.If subscription end 207 needs customized development, so can use the special SDK after encapsulating based on such as RocketMQ client, after having specified topic of subscription, can incremental data be obtained.Multiple subscription end can subscribe to the incremental data of same subject simultaneously, and does not affect each other.By developing general module for subscription end, the object reducing subscription end exploitation and maintenance cost can be reached.General module exploitation the cost of development of subscription end under common scene is reduced greatly, and make subscription end hold decoupling zero with extracting, perception need not extract point storehouse submeter information of end, make overall plan be easier to expand.By modular design and to the effective boundary demarcation of extraction subscription task, coordinate existing robotization deployment system, management maintenance can be carried out to subscription extraction task easily, greatly reduce the maintenance work amount on line.
Fig. 3 schematically illustrates the process flow diagram of data push method according to an embodiment of the invention.
As shown in Figure 3, in step 301, database Incremental Log resolution server 203 creates extraction task, be that data are dumped to database Incremental Log resolution server 203 from source database 201 by border with database instance, database Incremental Log resolution server 203 is resolved by database Incremental Log and is obtained incremental data, and the incremental data of acquisition is temporary in the internal memory of database Incremental Log resolution server 203.
In step 303, database Incremental Log resolution server 203 by the incremental data rule as required that obtains using corresponding theme and mark as message distribution in the message queue 1,2,3 in Distributed Message Queue server 205, every bar all comprises respective theme and mark so that subscription end 207 is subscribed to and filtered as the incremental data of message, described theme represents the library name of logic, and described mark represents the table name of logic.The rule of needs mentioned here refers to the mapping ruler of the storehouse table information of source incremental data to queue theme and mark, modal application scenarios is that source has carried out point storehouse submeter, as library name: product_pop_1, product_pop_2 ... .product_pop_N the product_pop theme of queue can be mapped to, in like manner, table name sku_N is mapped to mark sku.This rule freely can define according to service needed.
In step 305, the theme subscribed to based on each subscription task 1,2,3 of subscription end 207 of Distributed Message Queue server 205 and message queue 1,2,3 is pushed to subscription end 207.
The extraction task in past is not in units of database instance, in units of the logical base table of being correlated with by one group of business, extraction task in the past a database instance has a lot of logical base tables, so can open the connection of multiple extracted data on a database instance according to the number of business library table group.This namely in Fig. 1 the connection (dump 1-3) of source database 101 and database Incremental Log resolution server 103 have 3, and the connection of source database 201 in Fig. 2 and database Incremental Log resolution server 203 only has the reason of 1.In contrast, data delivery system according to the above embodiment of the present invention and method, persistent message queue means are used by making the propelling movement of incremental data, because the establishment of the task of extraction take database instance as border, compare extraction task state in the past, directly reduce the pressure of source database, thus link 1 that is reduced to as shown in Figure 2 by schematically illustrating 3 (for dozens or even hundreds of in practical application scene) dumps as Fig. 1, and extract and subscribe to and need not occur in pairs again, thus also greatly reduce the flow of network, incremental data is made to be become possibility by many subscription end consumption, greatly improve load-bearing capacity and the service efficiency of server.
Above-described embodiment is only the preferred embodiments of the present invention, is not limited to the present invention.It will be apparent for a person skilled in the art that without departing from the spirit and scope of the present invention, various amendment and change can be carried out to embodiments of the invention.Therefore, the invention is intended to contain all such amendments within the scope of the present invention as defined by the appended claims of falling into or modification.
Claims (6)
1. a data push method, comprising:
Extract task creation process, with database instance be border by data from source database dump to database Incremental Log resolution server, resolved by database Incremental Log and obtain incremental data, and the incremental data of acquisition is temporary in the internal memory of database Incremental Log resolution server;
Message distribution process, by the incremental data rule as required that obtains using corresponding theme and mark as message distribution in the corresponding one or more message queue in Distributed Message Queue server, theme represents the library name of logic, mark represents the table name of logic, and the incremental data that every bar is distributed as message all comprises corresponding theme and mark; And
Subscribe to and push process, the theme that Distributed Message Queue server is subscribed to based on each subscription task of subscription end and described one or more message queue is pushed to subscription end.
2. data push method according to claim 1,
Wherein, described source database is MySQL database, and described database Incremental Log resolution server is canal server, and described Distributed Message Queue server is RocketMQ server.
3. data push method according to claim 2,
Wherein, described subscription end be by use based on RocketMQ client encapsulate after special SDK and customized development.
4. a data delivery system, comprising:
Source database;
Database Incremental Log resolution server;
Distributed Message Queue server; With
Subscription end,
Wherein,
With database instance be border by data from source database dump to database Incremental Log resolution server, database Incremental Log resolution server is resolved by database Incremental Log and is obtained incremental data, and the incremental data of acquisition is temporary in the internal memory of database Incremental Log resolution server;
Database Incremental Log resolution server by the incremental data rule as required that obtains using corresponding theme and mark as message distribution in the corresponding one or more message queue in Distributed Message Queue server, theme represents the library name of logic, mark represents the table name of logic, and the incremental data that every bar is distributed as message all comprises corresponding theme and mark; And
The theme that Distributed Message Queue server is subscribed to based on each subscription task of subscription end and described one or more message queue is pushed to subscription end.
5. data delivery system according to claim 4,
Wherein, described source database is MySQL database, and described database Incremental Log resolution server is canal server, and described Distributed Message Queue server is RocketMQ server.
6. data delivery system according to claim 5,
Wherein, described subscription end be by use based on RocketMQ client encapsulate after special SDK and customized development.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201410706244.1A CN104408132B (en) | 2014-11-28 | 2014-11-28 | Data push method and system |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201410706244.1A CN104408132B (en) | 2014-11-28 | 2014-11-28 | Data push method and system |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN104408132A true CN104408132A (en) | 2015-03-11 |
| CN104408132B CN104408132B (en) | 2018-03-02 |
Family
ID=52645763
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201410706244.1A Active CN104408132B (en) | 2014-11-28 | 2014-11-28 | Data push method and system |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN104408132B (en) |
Cited By (38)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104679888A (en) * | 2015-03-17 | 2015-06-03 | 浪潮集团有限公司 | Device and method for data synchronization |
| CN105071966A (en) * | 2015-08-10 | 2015-11-18 | 浪潮(北京)电子信息产业有限公司 | Log information management method and log extraction server |
| CN105100250A (en) * | 2015-07-31 | 2015-11-25 | 北京奇虎科技有限公司 | Message distribution method, message distribution server and message distribution system |
| CN105119752A (en) * | 2015-09-08 | 2015-12-02 | 北京京东尚科信息技术有限公司 | Distributed log acquisition method, device and system |
| CN105677774A (en) * | 2015-12-30 | 2016-06-15 | 百度在线网络技术（北京）有限公司 | Data event pushing method and device |
| CN105847063A (en) * | 2016-05-12 | 2016-08-10 | 中国联合网络通信集团有限公司 | Core network data management method and system |
| CN106301972A (en) * | 2016-11-16 | 2017-01-04 | 北京集奥聚合科技有限公司 | A kind of monitoring data push away method and system in real time |
| CN106354434A (en) * | 2016-08-31 | 2017-01-25 | 中国人民大学 | Log data storing method and system |
| WO2017049927A1 (en) * | 2015-09-22 | 2017-03-30 | 乐视控股（北京）有限公司 | Message dispatching method, device and system |
| WO2017114393A1 (en) * | 2015-12-31 | 2017-07-06 | 中兴通讯股份有限公司 | Http streaming media transmission method and device |
| CN107180116A (en) * | 2017-06-28 | 2017-09-19 | 努比亚技术有限公司 | A kind of data synchronizing processing method, mobile terminal and computer-readable recording medium |
| CN107341062A (en) * | 2017-06-28 | 2017-11-10 | 百度在线网络技术（北京）有限公司 | A kind of data push method, device, equipment and storage medium |
| CN107678856A (en) * | 2017-09-20 | 2018-02-09 | 苏宁云商集团股份有限公司 | The method and device of increment information in a kind of processing business entity |
| CN108614852A (en) * | 2018-03-14 | 2018-10-02 | 广州市优普科技有限公司 | A kind of data map generation method based on big data |
| CN108737568A (en) * | 2018-06-22 | 2018-11-02 | 重庆助康医学检验实验室有限公司 | A kind of data interaction system and method based on Message Queuing server |
| CN108874834A (en) * | 2017-05-16 | 2018-11-23 | 北京嘀嘀无限科技发展有限公司 | A kind of data processing method, processing system and computer installation |
| CN109325008A (en) * | 2018-09-07 | 2019-02-12 | 国电南瑞科技股份有限公司 | The cross-domain data synchronous method and system of Message Oriented Middleware |
| CN109783573A (en) * | 2018-12-18 | 2019-05-21 | 北京华夏电通科技有限公司 | The method of data synchronization and terminal of multichannel push |
| CN110019123A (en) * | 2017-11-10 | 2019-07-16 | 北京京东尚科信息技术有限公司 | A kind of data migration method and device |
| CN110209554A (en) * | 2019-06-14 | 2019-09-06 | 上海中通吉网络技术有限公司 | Database journal distribution method, device and equipment |
| CN110875935A (en) * | 2018-08-30 | 2020-03-10 | 阿里巴巴集团控股有限公司 | Message publishing, processing and subscribing method, device and system |
| CN110995566A (en) * | 2019-10-30 | 2020-04-10 | 深圳震有科技股份有限公司 | Message data pushing method, system and device |
| CN111371672A (en) * | 2020-02-14 | 2020-07-03 | 贝壳技术有限公司 | Message pushing method and device |
| CN111367873A (en) * | 2018-12-26 | 2020-07-03 | 深圳市优必选科技有限公司 | Log data storage method and device, terminal and computer storage medium |
| CN111556019A (en) * | 2020-03-27 | 2020-08-18 | 天津市普迅电力信息技术有限公司 | Vehicle-mounted machine data encryption transmission and processing method under distributed environment |
| CN111669313A (en) * | 2020-05-29 | 2020-09-15 | 广西东信互联科技有限公司 | Method for storing and synchronizing instant communication system message |
| CN111831748A (en) * | 2020-06-30 | 2020-10-27 | 北京小米松果电子有限公司 | Data synchronization method, device and storage medium |
| CN111858273A (en) * | 2020-07-02 | 2020-10-30 | 五八有限公司 | Display method and device for processing logs |
| CN111985944A (en) * | 2019-05-21 | 2020-11-24 | 北京沃东天骏信息技术有限公司 | Method, device and equipment for processing material data and storage medium |
| US10891226B1 (en) | 2019-08-01 | 2021-01-12 | International Business Machines Corporation | Virtual address space dump in a computer system |
| CN112836236A (en) * | 2021-02-04 | 2021-05-25 | 航天广通科技(深圳)有限公司 | Method and system for subscribing database table resources of data sharing exchange platform |
| CN113067853A (en) * | 2021-03-12 | 2021-07-02 | 北京金山云网络技术有限公司 | Data pushing method and device, electronic equipment and storage medium |
| CN113098969A (en) * | 2021-04-09 | 2021-07-09 | 薪得付信息技术(上海)有限公司 | Data distribution method, device and system and electronic equipment |
| CN113934793A (en) * | 2021-09-07 | 2022-01-14 | 华数传媒网络有限公司 | Distributed data management system and method for operation system |
| CN114567633A (en) * | 2022-03-01 | 2022-05-31 | 上海浦东发展银行股份有限公司 | Cloud platform system supporting full life cycle of multi-stack database and management method |
| CN114827171A (en) * | 2022-03-23 | 2022-07-29 | 青岛中科曙光科技服务有限公司 | Information synchronization method and device, computer equipment and storage medium |
| CN116980475A (en) * | 2023-07-31 | 2023-10-31 | 深圳市亲邻科技有限公司 | Data pushing system based on binlog and double annular buffer areas |
| CN111831748B (en) * | 2020-06-30 | 2024-04-30 | 北京小米松果电子有限公司 | Data synchronization method, device and storage medium |
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| JPH09200257A (en) * | 1996-01-23 | 1997-07-31 | Toshiba Corp | Electronic information notifying device |
| CN102890631A (en) * | 2012-09-13 | 2013-01-23 | 新浪网技术(中国)有限公司 | Method for transmitting message based on persistent message queue and message transmission device |
| CN103581250A (en) * | 2012-07-31 | 2014-02-12 | 盛趣信息技术（上海）有限公司 | Data processing, issuing and obtaining method and system |
| CN103607476A (en) * | 2013-12-05 | 2014-02-26 | 上海普坤信息科技有限公司 | System and method for distributed multi-stage persistence of huge number of messages |
-
2014
- 2014-11-28 CN CN201410706244.1A patent/CN104408132B/en active Active
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| JPH09200257A (en) * | 1996-01-23 | 1997-07-31 | Toshiba Corp | Electronic information notifying device |
| CN103581250A (en) * | 2012-07-31 | 2014-02-12 | 盛趣信息技术（上海）有限公司 | Data processing, issuing and obtaining method and system |
| CN102890631A (en) * | 2012-09-13 | 2013-01-23 | 新浪网技术(中国)有限公司 | Method for transmitting message based on persistent message queue and message transmission device |
| CN103607476A (en) * | 2013-12-05 | 2014-02-26 | 上海普坤信息科技有限公司 | System and method for distributed multi-stage persistence of huge number of messages |
Cited By (45)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104679888A (en) * | 2015-03-17 | 2015-06-03 | 浪潮集团有限公司 | Device and method for data synchronization |
| CN105100250A (en) * | 2015-07-31 | 2015-11-25 | 北京奇虎科技有限公司 | Message distribution method, message distribution server and message distribution system |
| CN105071966A (en) * | 2015-08-10 | 2015-11-18 | 浪潮(北京)电子信息产业有限公司 | Log information management method and log extraction server |
| CN105119752A (en) * | 2015-09-08 | 2015-12-02 | 北京京东尚科信息技术有限公司 | Distributed log acquisition method, device and system |
| WO2017049927A1 (en) * | 2015-09-22 | 2017-03-30 | 乐视控股（北京）有限公司 | Message dispatching method, device and system |
| CN105677774A (en) * | 2015-12-30 | 2016-06-15 | 百度在线网络技术（北京）有限公司 | Data event pushing method and device |
| WO2017114393A1 (en) * | 2015-12-31 | 2017-07-06 | 中兴通讯股份有限公司 | Http streaming media transmission method and device |
| CN105847063A (en) * | 2016-05-12 | 2016-08-10 | 中国联合网络通信集团有限公司 | Core network data management method and system |
| CN106354434A (en) * | 2016-08-31 | 2017-01-25 | 中国人民大学 | Log data storing method and system |
| CN106354434B (en) * | 2016-08-31 | 2019-07-23 | 中国人民大学 | The storage method and system of daily record data |
| CN106301972A (en) * | 2016-11-16 | 2017-01-04 | 北京集奥聚合科技有限公司 | A kind of monitoring data push away method and system in real time |
| CN108874834A (en) * | 2017-05-16 | 2018-11-23 | 北京嘀嘀无限科技发展有限公司 | A kind of data processing method, processing system and computer installation |
| CN107341062B (en) * | 2017-06-28 | 2021-01-08 | 百度在线网络技术（北京）有限公司 | Data pushing method, device, equipment and storage medium |
| CN107341062A (en) * | 2017-06-28 | 2017-11-10 | 百度在线网络技术（北京）有限公司 | A kind of data push method, device, equipment and storage medium |
| CN107180116A (en) * | 2017-06-28 | 2017-09-19 | 努比亚技术有限公司 | A kind of data synchronizing processing method, mobile terminal and computer-readable recording medium |
| CN107678856A (en) * | 2017-09-20 | 2018-02-09 | 苏宁云商集团股份有限公司 | The method and device of increment information in a kind of processing business entity |
| CN110019123B (en) * | 2017-11-10 | 2021-10-15 | 北京京东尚科信息技术有限公司 | Data migration method and device |
| CN110019123A (en) * | 2017-11-10 | 2019-07-16 | 北京京东尚科信息技术有限公司 | A kind of data migration method and device |
| CN108614852A (en) * | 2018-03-14 | 2018-10-02 | 广州市优普科技有限公司 | A kind of data map generation method based on big data |
| CN108737568A (en) * | 2018-06-22 | 2018-11-02 | 重庆助康医学检验实验室有限公司 | A kind of data interaction system and method based on Message Queuing server |
| CN110875935A (en) * | 2018-08-30 | 2020-03-10 | 阿里巴巴集团控股有限公司 | Message publishing, processing and subscribing method, device and system |
| CN109325008A (en) * | 2018-09-07 | 2019-02-12 | 国电南瑞科技股份有限公司 | The cross-domain data synchronous method and system of Message Oriented Middleware |
| CN109783573A (en) * | 2018-12-18 | 2019-05-21 | 北京华夏电通科技有限公司 | The method of data synchronization and terminal of multichannel push |
| CN111367873A (en) * | 2018-12-26 | 2020-07-03 | 深圳市优必选科技有限公司 | Log data storage method and device, terminal and computer storage medium |
| CN111985944A (en) * | 2019-05-21 | 2020-11-24 | 北京沃东天骏信息技术有限公司 | Method, device and equipment for processing material data and storage medium |
| CN110209554A (en) * | 2019-06-14 | 2019-09-06 | 上海中通吉网络技术有限公司 | Database journal distribution method, device and equipment |
| CN110209554B (en) * | 2019-06-14 | 2023-08-11 | 上海中通吉网络技术有限公司 | Database log distribution method, device and equipment |
| US10891226B1 (en) | 2019-08-01 | 2021-01-12 | International Business Machines Corporation | Virtual address space dump in a computer system |
| CN110995566A (en) * | 2019-10-30 | 2020-04-10 | 深圳震有科技股份有限公司 | Message data pushing method, system and device |
| CN111371672A (en) * | 2020-02-14 | 2020-07-03 | 贝壳技术有限公司 | Message pushing method and device |
| CN111371672B (en) * | 2020-02-14 | 2022-04-08 | 贝壳技术有限公司 | Message pushing method and device |
| CN111556019A (en) * | 2020-03-27 | 2020-08-18 | 天津市普迅电力信息技术有限公司 | Vehicle-mounted machine data encryption transmission and processing method under distributed environment |
| CN111556019B (en) * | 2020-03-27 | 2022-06-14 | 天津市普迅电力信息技术有限公司 | Vehicle-mounted machine data encryption transmission and processing method under distributed environment |
| CN111669313A (en) * | 2020-05-29 | 2020-09-15 | 广西东信互联科技有限公司 | Method for storing and synchronizing instant communication system message |
| CN111831748B (en) * | 2020-06-30 | 2024-04-30 | 北京小米松果电子有限公司 | Data synchronization method, device and storage medium |
| CN111831748A (en) * | 2020-06-30 | 2020-10-27 | 北京小米松果电子有限公司 | Data synchronization method, device and storage medium |
| CN111858273A (en) * | 2020-07-02 | 2020-10-30 | 五八有限公司 | Display method and device for processing logs |
| CN112836236A (en) * | 2021-02-04 | 2021-05-25 | 航天广通科技(深圳)有限公司 | Method and system for subscribing database table resources of data sharing exchange platform |
| CN113067853A (en) * | 2021-03-12 | 2021-07-02 | 北京金山云网络技术有限公司 | Data pushing method and device, electronic equipment and storage medium |
| CN113098969A (en) * | 2021-04-09 | 2021-07-09 | 薪得付信息技术(上海)有限公司 | Data distribution method, device and system and electronic equipment |
| CN113934793A (en) * | 2021-09-07 | 2022-01-14 | 华数传媒网络有限公司 | Distributed data management system and method for operation system |
| CN114567633A (en) * | 2022-03-01 | 2022-05-31 | 上海浦东发展银行股份有限公司 | Cloud platform system supporting full life cycle of multi-stack database and management method |
| CN114827171A (en) * | 2022-03-23 | 2022-07-29 | 青岛中科曙光科技服务有限公司 | Information synchronization method and device, computer equipment and storage medium |
| CN114827171B (en) * | 2022-03-23 | 2023-12-15 | 青岛中科曙光科技服务有限公司 | Information synchronization method, apparatus, computer device and storage medium |
| CN116980475A (en) * | 2023-07-31 | 2023-10-31 | 深圳市亲邻科技有限公司 | Data pushing system based on binlog and double annular buffer areas |
Also Published As
| Publication number | Publication date |
|---|---|
| CN104408132B (en) | 2018-03-02 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN104408132A (en) | Data push method and system | |
| CN105933376A (en) | Data manipulation method, server and storage system | |
| CN101937474A (en) | Mass data query method and device | |
| CN102508919B (en) | Data processing method and system | |
| CN103744724A (en) | Timed task clustering method and device thereof | |
| CN101923566A (en) | Data increment extraction method based on trigger | |
| CN106951552A (en) | A kind of user behavior data processing method based on Hadoop | |
| CN105320680A (en) | Data synchronization method and device | |
| CN105205105A (en) | Data ETL (Extract Transform Load) system based on storm and treatment method based on storm | |
| CN104778188A (en) | Distributed device log collection method | |
| CN104699723A (en) | Data exchange adapter and system and method for synchronizing data among heterogeneous systems | |
| CN104050276A (en) | Cache processing method and system of distributed database | |
| CN104516989A (en) | Incremental data pushing system and method | |
| CN102467525A (en) | Document associating method and system | |
| CN112214453B (en) | Large-scale industrial data compression storage method, system and medium | |
| CN108256820A (en) | A kind of PBOM methods of adjustment under three-dimensional assembled view based on MBD | |
| CN102147809A (en) | Parallel file system and management method thereof | |
| CN106156319A (en) | Telescopic distributed resource description framework data storage method and device | |
| CN107870982A (en) | Data processing method, system and computer-readable recording medium | |
| CN109800069B (en) | Method and device for realizing data management | |
| CN104881475A (en) | Method and system for randomly sampling big data | |
| CN102436501A (en) | Parallel file managing system based on web | |
| CN105159925A (en) | Database cluster data distribution method and system | |
| CN108287889B (en) | A kind of multi-source heterogeneous date storage method and system based on elastic table model | |
| CN107807977B (en) | Object attribute metadata extraction system based on configuration |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| C10 | Entry into substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant | ||
| TR01 | Transfer of patent right | ||
| TR01 | Transfer of patent right |
Effective date of registration: 20191128 Address after: 100176 room 222, 2f, building C, No. 18, Kechuang 11th Street, Beijing Economic and Technological Development Zone, Daxing District, Beijing Patentee after: Beijing Jingdong three hundred and sixty degree e-commerce Co., Ltd. Address before: 100080 Beijing city Haidian District xingshikou Road No. 65 west Shan Creative Park District 11C four floor East West 1-4 layer 1-4 layer Co-patentee before: Beijing Jingdong Century Commerce Co., Ltd. Patentee before: Beijing Jingdong Shangke Information Technology Co., Ltd. |