CN108491332A - A kind of real-time buffering updating method and system based on Redis - Google Patents
A kind of real-time buffering updating method and system based on Redis Download PDFInfo
- Publication number
- CN108491332A CN108491332A CN201810300592.7A CN201810300592A CN108491332A CN 108491332 A CN108491332 A CN 108491332A CN 201810300592 A CN201810300592 A CN 201810300592A CN 108491332 A CN108491332 A CN 108491332A
- Authority
- CN
- China
- Prior art keywords
- database
- data
- information
- binlog
- real
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F12/00—Accessing, addressing or allocating within memory systems or architectures
- G06F12/02—Addressing or allocation; Relocation
- G06F12/08—Addressing or allocation; Relocation in hierarchically structured memory systems, e.g. virtual memory systems
- G06F12/0802—Addressing of a memory level in which the access to the desired data or data block requires associative addressing means, e.g. caches
- G06F12/0804—Addressing of a memory level in which the access to the desired data or data block requires associative addressing means, e.g. caches with main memory updating
Abstract
The present invention relates to Data cache technology fields, disclose a kind of real-time buffering updating method and system based on Redis, include the following steps, S1, user in real are to the modification informations of the data in database；S2, the modification that data and correspondence database data in database are changed based on modification information generate binlog information；S3, the binlog information for receiving database push, binlog information is parsed；S4, acquisition simultaneously convert the binlog information after parsing to fresh information corresponding with data in cache database；S5, the data in cache database are changed based on fresh information；By obtaining the modification information to the data of database, the data in real time modifying database, to complete the data modification in database in real time；The logic of data in synchronous logic and update cache database is separated, two logics are divided into.The modification for first completing database, then changes the data in cache database according to the modification of database, and chaotic probability occurs to reduce logic.
Description
Technical field
The present invention relates to Data cache technology fields, and in particular to a kind of real-time buffering updating method based on Redis and
System.
Background technology
Caching is exactly the buffering area of data exchange, and when a certain hardware will read data, can be searched first from caching needs
The data wanted, if having found directly execute, can not find if if looked for from memory.Since the speed of service of caching compares memory
Faster, therefore the effect of caching is just to aid in hardware and quickly runs.
It, or can be text after being finished because caching often uses RAM (the impermanent storage that power-off is fallen)
Part is sent in the memories such as hard disk and permanently stores.Maximum caching is exactly memory bar in computer, that most fast is the L1 that is inlayed on CPU with
L2 is cached, and the video memory of video card is that also have the caching of 16M or 32M on hard disk to the caching of video card operation chip.
Redis is a key-value storage system.Similar with Memcached, it supports the value type phases of storage
To more, including string (character string), list (chained list), set (set) and zset (ordered set).These data types are all
It supports push/pop, add/remove and takes intersection union and difference set and more rich operation, and these operations are all atoms
Property.On this basis, redis supports the sequence of various different modes.As memcached, for guaranteed efficiency, data
All it is to cache in memory.Difference be redis can periodically newer data be written disk or modification operation write
Enter additional record file, and it is synchronous to realize master-slave (principal and subordinate) on this basis.
The principle of canal is that simulation Slave sends request to Master, and canal parses binlog, but not by analysis result
Persistence, and be stored in memory, as soon as there is client to read time message every time, delete the message.
Message queue (MQ) is a kind of communication means of application program to application program.Application program is by writing and retrieving
Enter the data (message) for application program lined up to communicate, them are linked without special connection.What message transmission referred to
It is to be communicated by transmission data in the message between program, rather than communicated by directly invoking each other, directly invokes
Typically it is used for the technology of such as remote procedure call.Queuing refers to application program by queue to communicate.The use of queue
It eliminates and sends and receivees the requirement that application program is performed simultaneously.
During real cache, it will usually there are following several schemes to synchronize the content in caching.
Scheme one, the scheme of Timing Synchronization caching.The data of database are synchronized to caching by a preset time
In server.The preset time, otherwise computer can be in working condition for a long time usually more than the second, waste the property of computer
Energy.Such scheme, in this preset time period, the data of user's change cannot timely be shown.
Scheme two, the function of embedded synchronization caching in back-stage management service.In this scheme, user's modification needs to carry out
The data in cache server are changed while the data of caching.Such scheme, cache synchronization logic can be coupling in business generation
In code logic, the data stored in cache database are potentially stored in multiple key, the number in changing cache database
According to when, need to call all key for being stored with the data and be modified.When data are more, cache logic and synchronous logic are mixed
It is combined, the logic of caching becomes larger, and be easy to cause the confusion of synchronous logic and service logic.
Invention content
In view of the deficiencies of the prior art, the present invention intends to provide it is a kind of based on Redis it is real-time caching more
New method.
To achieve the above object, the present invention provides following technical solutions：
A kind of real-time buffering updating method based on Redis, includes the following steps,
S1, user in real are to the modification informations of the data in database；
S2, the modification that data and correspondence database data in database are changed based on modification information generate binlog information；
S3, the binlog information for receiving database push, binlog information is parsed；
S4, acquisition simultaneously convert the binlog information after parsing to fresh information corresponding with data in cache database；
S5, the data in cache database are changed based on fresh information.
By using above-mentioned technical proposal, according to the prior art, simultaneously by the data in database and cache database
Modification, due in database and the mode difference for storing data in cache database, a corresponding data in cache database
Multiple key be might have to store, when data are more, need the amount changed big in cache database, then cache logic is big, is easy
Lead to the confusion of cache logic and synchronous logic.
By obtaining the modification information to the data of database, the data in real time modifying database, to complete in real time
Data modification in database.
By obtaining the binlog information after parsing, and convert and the number in cache database the binlog after parsing to
According to corresponding fresh information, the information in update cache database is corresponded to based on fresh information, to by synchronous logic and more
The logic of data in new cache database separates, and is divided into two logics.The modification for first completing database, then according to database
Modification change the data in cache database, chaotic probability occurs to reduce logic.
The further setting of the present invention is：Step S3 receives and parses through binlog information by canal.
By using above-mentioned technical proposal, canal disguise as databases from library server, so as to listen to data
Binlog information is parsed by canal and pushes the binlog information after parsing by binlog information in library.
The further setting of the present invention is：It is provided between step S3 and S4 for receiving and storing the binlog after parsing
The message queue of information.
By using above-mentioned technical proposal, message queue is used to receive and store the binlog information after parsing.By disappearing
Breath queue stores the binlog information after parsing.It is asked since the equipment such as computer are likely to occur in executing operating process
Topic, if the binlog information storages after parsing can be removed number in cache database when capacity is low in cache database
According to may be by the binlog erasings of information after parsing.So being deposited to the binlog information after parsing by message queue
Storage, to reduce the probability that binlog data are lost.
The further setting of the present invention is：The message queue takes MQ.
By using above-mentioned technical proposal, message queue MQ is powerful, stable, technology maturation, applicable operation
System is more.
In view of the deficiencies of the prior art, the present invention intends to provide it is a kind of based on Redis it is real-time caching more
New system.
To achieve the above object, the present invention provides following technical solutions：
A kind of real-time buffer update system based on Redis, comprises the following modules,
Acquisition module, for user in real to the modification information of the data in database；
Modified module, the modification that data and correspondence database data in database are changed based on modification information generate binlog letters
Breath；
Parsing module, the binlog information for receiving database push, binlog information is parsed；
Conversion module obtains and converts the binlog information after parsing to update corresponding with data in cache database and believes
Breath；
Update module changes the data in cache database based on fresh information.
By using above-mentioned technical proposal, according to the prior art, simultaneously by the data in database and cache database
Modification, due in database and the mode difference for storing data in cache database, a corresponding data in cache database
Multiple key be might have to store, when data are more, need the amount changed big in cache database, then cache logic is big, is easy
Lead to the confusion of cache logic and synchronous logic.
By obtaining the modification information to the data of database, the data in real time modifying database, to complete in real time
Data modification in database.
By obtaining the binlog information after parsing, and convert and the number in cache database the binlog after parsing to
According to corresponding fresh information, the information in update cache database is corresponded to based on fresh information, to by synchronous logic and more
The logic of data in new cache database separates, and is divided into two logics.The modification for first completing database, then according to database
Modification change the data in cache database, chaotic probability occurs to reduce logic.
The further setting of the present invention is：Parsing module receives and parses through binlog information by canal.
By using above-mentioned technical proposal, canal disguise as databases from library server, so as to listen to data
Binlog information is parsed by canal and pushes the binlog information after parsing by binlog information in library.
The further setting of the present invention is：After being provided between parsing module and conversion module for receiving and storing parsing
Binlog information message queue.
By using above-mentioned technical proposal, message queue is used to receive and store the binlog information after parsing.By disappearing
Breath queue stores the binlog information after parsing.It is asked since the equipment such as computer are likely to occur in executing operating process
Topic, if the binlog information storages after parsing can be removed number in cache database when capacity is low in cache database
According to may be by the binlog erasings of information after parsing.So being deposited to the binlog information after parsing by message queue
Storage, to reduce the probability that binlog data are lost.
The further setting of the present invention is：The message queue takes MQ.
By using above-mentioned technical proposal, message queue MQ is powerful, stable, technology maturation, applicable operation
System is more.
The present invention has the following advantages：
1, by obtaining the modification information to the data of database, the data in real time modifying database, to complete number in real time
According to the data modification in library；
2, the logic of the data in synchronous logic and update cache database is separated, is divided into two logics.First complete database
Modification, the data in cache database are then changed according to the modification of database, to reduce logic occur it is chaotic
Probability；
3, canal disguise as database passes through from library server so as to listen to binlog information in database
Binlog information is parsed and pushes the binlog information after parsing by canal.
Description of the drawings
Fig. 1 is the flow chart of the real-time buffering updating method based on Redis；
Fig. 2 is the flow chart of the real-time buffer update system based on Redis.
Specific implementation mode
With reference to attached drawing, the present invention will be further described.
Embodiment 1, as shown in Figure 1：A kind of real-time buffering updating method based on Redis, includes the following steps,
S1, in real time modification information of the acquisition to the data in database.
S2, the modification that data and correspondence database data in database are changed based on modification information generate binlog letters
Breath.
According to the prior art, the data in database and cache database are changed simultaneously, due to gentle in database
The mode that data are stored in deposit data library is different, and a corresponding data might have multiple key to store in cache database,
When data are more, need the amount changed big in cache database, then cache logic is big, is easy to cause cache logic and synchronous logic
Confusion.
By obtaining the modification information to the data of database, the data in real time modifying database, to complete in real time
Data modification in database.
S3, the binlog information that database push is received by canal, binlog information is parsed.
Canal disguise as databases pass through from library server so as to listen to binlog information in database
Binlog information is parsed and pushes to breath queue MQ by canal.
Message queue MQ is used to receive and store the binlog information after parsing.By message queue MQ to parsing after
Binlog information is stored.Since the equipment such as computer are likely to occur problem in executing operating process, if after parsing
Binlog information storages in cache database, can clear data, may will be parsed when capacity is low in cache database
Binlog erasings of information afterwards.So being stored to the binlog information after parsing by message queue MQ, to reduce
The probability that binlog data are lost.
S4, acquisition are stored in the binlog information after the parsing in message queue MQ, and the binlog after parsing is believed
Breath is converted into fresh information corresponding with data in cache database.
S5, the data in cache database are changed based on fresh information.
By the binlog information after the parsing in acquisition message queue MQ, passes through the binlog information after parsing and convert
For fresh information corresponding with data in cache database, due to the position He storage side of file in cache database and database
Formula etc. difference, by conversion, to obtain fresh information, and according to fresh information by data change in cache database.
Due to completed in computer the process be Millisecond, reduce in this approach data cannot show in time it is general
Rate.
Embodiment 2, as shown in Figure 2：A kind of real-time buffer update system based on Redis, including acquisition module, modification mould
Block, parsing module, conversion module, update module.
Acquisition module for obtaining the modification information to the data in database in real time.
The modification that modified module changes data and correspondence database data in database based on modification information generates
Binlog information.
According to the prior art, the data in database and cache database are changed simultaneously, due to gentle in database
The mode that data are stored in deposit data library is different, and a corresponding data might have multiple key to store in cache database,
When data are more, need the amount changed big in cache database, then cache logic is big, is easy to cause cache logic and synchronous logic
Confusion.
By obtaining the modification information to the data of database, the data in real time modifying database, to complete in real time
Data modification in database.
Parsing module receives the binlog information of database push by canal, and binlog information is parsed.Canal is pseudo-
Dress is database from library server, so as to listen to binlog information in database, by canal by binlog information
It parses and pushes to breath queue MQ.
Message queue MQ is used to receive and store the binlog information after parsing.By message queue MQ to parsing after
Binlog information is stored.Since the equipment such as computer are likely to occur problem in executing operating process, if after parsing
Binlog information storages in cache database, can clear data, may will be parsed when capacity is low in cache database
Binlog erasings of information afterwards.So being stored to the binlog information after parsing by message queue MQ, to reduce
The probability that binlog data are lost.
Conversion module acquisition is stored in the binlog information after the parsing in message queue MQ, and will be after parsing
Binlog information is converted into fresh information corresponding with data in cache database.
Update module changes the data in cache database based on fresh information.
By the binlog information after the parsing in acquisition message queue MQ, passes through the binlog information after parsing and convert
For fresh information corresponding with data in cache database, due to the position He storage side of file in cache database and database
Formula is different, by conversion, to obtain fresh information, and according to fresh information by data change in cache database.
Due to completed in computer the process be Millisecond, so this system reduces data cannot show in time it is general
Rate.
The above is only a preferred embodiment of the present invention, protection scope of the present invention is not limited merely to above-mentioned implementation
Example, all technical solutions belonged under thinking of the present invention all belong to the scope of protection of the present invention.It should be pointed out that for the art
Those of ordinary skill for, several improvements and modifications without departing from the principles of the present invention, these improvements and modifications
It should be regarded as protection scope of the present invention.
Claims (8)
1. a kind of real-time buffering updating method based on Redis, it is characterized in that：Include the following steps,
S1, user in real are to the modification informations of the data in database；
S2, the modification that data and correspondence database data in database are changed based on modification information generate binlog information；
S3, the binlog information for receiving database push, binlog information is parsed；
S4, acquisition simultaneously convert the binlog information after parsing to fresh information corresponding with data in cache database；
S5, the data in cache database are changed based on fresh information.
2. a kind of real-time buffering updating method based on Redis according to claim 1, it is characterized in that：Step S3 passes through
Canal receives and parses through binlog information.
3. a kind of real-time buffering updating method based on Redis according to claim 2, it is characterized in that：Step S3 and S4
Between be provided with for receive and store parsing after binlog information message queue.
4. a kind of real-time buffering updating method based on Redis according to claim 3, it is characterized in that：The message team
Row take MQ.
5. a kind of real-time buffer update system based on Redis, it is characterized in that：It comprises the following modules,
Acquisition module, for user in real to the modification information of the data in database；
Modified module, the modification that data and correspondence database data in database are changed based on modification information generate binlog letters
Breath；
Parsing module, the binlog information for receiving database push, binlog information is parsed；
Conversion module obtains and converts the binlog information after parsing to update corresponding with data in cache database and believes
Breath；
Update module changes the data in cache database based on fresh information.
6. a kind of real-time buffer update system based on Redis according to claim 5, it is characterized in that：Parsing module is logical
It crosses canal and receives and parses through binlog information.
7. a kind of real-time buffer update system based on Redis according to claim 6, it is characterized in that：Parsing module and
The message queue for receiving and storing the binlog information after parsing is provided between conversion module.
8. a kind of real-time buffer update system based on Redis according to claim 7, it is characterized in that：The message team
Row take MQ.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201810300592.7A CN108491332A (en) | 2018-04-04 | 2018-04-04 | A kind of real-time buffering updating method and system based on Redis |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201810300592.7A CN108491332A (en) | 2018-04-04 | 2018-04-04 | A kind of real-time buffering updating method and system based on Redis |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN108491332A true CN108491332A (en) | 2018-09-04 |
Family
ID=63314865
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201810300592.7A Pending CN108491332A (en) | 2018-04-04 | 2018-04-04 | A kind of real-time buffering updating method and system based on Redis |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN108491332A (en) |
Cited By (9)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109710688A (en) * | 2018-11-26 | 2019-05-03 | 福建天泉教育科技有限公司 | A kind of real-time Inspection method of data and message-oriented middleware |
| CN109857812A (en) * | 2019-02-27 | 2019-06-07 | 珠海天燕科技有限公司 | A kind of method and apparatus handling data in caching |
| CN109871388A (en) * | 2019-02-19 | 2019-06-11 | 北京字节跳动网络技术有限公司 | Data cache method, device, whole electronic equipment and storage medium |
| CN110704401A (en) * | 2019-10-10 | 2020-01-17 | 北京字节跳动网络技术有限公司 | Data processing method and device, electronic equipment and storage medium |
| CN111046106A (en) * | 2019-12-19 | 2020-04-21 | 杭州中恒电气股份有限公司 | Cache data synchronization method, device, equipment and medium |
| CN111127252A (en) * | 2019-12-24 | 2020-05-08 | 华中科技大学 | Data management method of water resource management decision support system |
| CN111563102A (en) * | 2020-04-10 | 2020-08-21 | 中国联合网络通信集团有限公司 | Cache updating method, server, system and storage medium |
| CN112419018A (en) * | 2020-01-03 | 2021-02-26 | 上海哔哩哔哩科技有限公司 | General data reconciliation method, server and storage medium in distributed environment |
| CN113138993A (en) * | 2020-01-19 | 2021-07-20 | 上海臻客信息技术服务有限公司 | Hotel information updating method, device, equipment and medium based on redis |
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104572689A (en) * | 2013-10-17 | 2015-04-29 | 腾讯科技（深圳）有限公司 | Data synchronizing method, device and system |
| CN106484869A (en) * | 2016-10-12 | 2017-03-08 | 北京集奥聚合科技有限公司 | A kind of distributed caching method based on mysql binlog and system |
| CN107341212A (en) * | 2017-06-26 | 2017-11-10 | 努比亚技术有限公司 | A kind of buffering updating method and equipment |
| CN107783975A (en) * | 2016-08-24 | 2018-03-09 | 北京京东尚科信息技术有限公司 | The method and apparatus of distributed data base synchronization process |
-
2018
- 2018-04-04 CN CN201810300592.7A patent/CN108491332A/en active Pending
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104572689A (en) * | 2013-10-17 | 2015-04-29 | 腾讯科技（深圳）有限公司 | Data synchronizing method, device and system |
| CN107783975A (en) * | 2016-08-24 | 2018-03-09 | 北京京东尚科信息技术有限公司 | The method and apparatus of distributed data base synchronization process |
| CN106484869A (en) * | 2016-10-12 | 2017-03-08 | 北京集奥聚合科技有限公司 | A kind of distributed caching method based on mysql binlog and system |
| CN107341212A (en) * | 2017-06-26 | 2017-11-10 | 努比亚技术有限公司 | A kind of buffering updating method and equipment |
Cited By (10)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109710688A (en) * | 2018-11-26 | 2019-05-03 | 福建天泉教育科技有限公司 | A kind of real-time Inspection method of data and message-oriented middleware |
| CN109871388A (en) * | 2019-02-19 | 2019-06-11 | 北京字节跳动网络技术有限公司 | Data cache method, device, whole electronic equipment and storage medium |
| CN109857812A (en) * | 2019-02-27 | 2019-06-07 | 珠海天燕科技有限公司 | A kind of method and apparatus handling data in caching |
| CN110704401A (en) * | 2019-10-10 | 2020-01-17 | 北京字节跳动网络技术有限公司 | Data processing method and device, electronic equipment and storage medium |
| CN111046106A (en) * | 2019-12-19 | 2020-04-21 | 杭州中恒电气股份有限公司 | Cache data synchronization method, device, equipment and medium |
| CN111127252A (en) * | 2019-12-24 | 2020-05-08 | 华中科技大学 | Data management method of water resource management decision support system |
| CN112419018A (en) * | 2020-01-03 | 2021-02-26 | 上海哔哩哔哩科技有限公司 | General data reconciliation method, server and storage medium in distributed environment |
| CN112419018B (en) * | 2020-01-03 | 2022-11-18 | 上海哔哩哔哩科技有限公司 | General data reconciliation method, server and storage medium in distributed environment |
| CN113138993A (en) * | 2020-01-19 | 2021-07-20 | 上海臻客信息技术服务有限公司 | Hotel information updating method, device, equipment and medium based on redis |
| CN111563102A (en) * | 2020-04-10 | 2020-08-21 | 中国联合网络通信集团有限公司 | Cache updating method, server, system and storage medium |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN108491332A (en) | A kind of real-time buffering updating method and system based on Redis | |
| CN102831156B (en) | Distributed transaction processing method on cloud computing platform | |
| US7783607B2 (en) | Decentralized record expiry | |
| US20120254249A1 (en) | Database Management System | |
| CN110175213A (en) | A kind of oracle database synchronization system and method based on SCN mode | |
| CN107888657A (en) | Low latency distributed memory system | |
| CN108509462B (en) | Method and device for synchronizing activity transaction table | |
| CN109743137B (en) | Distributed delay message queue processing system supporting updating | |
| CN101901237A (en) | Type-Index-Value distributed database built based on SQIite | |
| CN106599104A (en) | Mass data association method based on redis cluster | |
| CN112307037A (en) | Data synchronization method and device | |
| CN107302569B (en) | Cloud platform-oriented security monitoring data acquisition and storage method | |
| CN112612799B (en) | Data synchronization method and terminal | |
| US20220035786A1 (en) | Distributed database management system with dynamically split b-tree indexes | |
| CN103164525A (en) | Method and device for WEB application release | |
| CN104317957A (en) | Open platform and system for processing reports and report processing method | |
| CN112000649B (en) | Method and device for synchronizing incremental data based on map reduce | |
| CN110750372A (en) | Log system based on shared memory and log management method | |
| CN111966748A (en) | Distributed space-based simulation operation control management method | |
| CN113076304A (en) | Distributed version management method, device and system | |
| CN106294205A (en) | caching data processing method and device | |
| CN111797119B (en) | Caching device, system and caching method | |
| WO2023159976A1 (en) | Data segmented writing method, data reading method and apparatus | |
| CN111046106A (en) | Cache data synchronization method, device, equipment and medium | |
| CN116028505A (en) | Method for realizing high available sequence in distributed database |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20180904 |