CN108259252A - A kind of Redis cluster methods based on twemproxy - Google Patents
A kind of Redis cluster methods based on twemproxy Download PDFInfo
- Publication number
- CN108259252A CN108259252A CN201810343035.3A CN201810343035A CN108259252A CN 108259252 A CN108259252 A CN 108259252A CN 201810343035 A CN201810343035 A CN 201810343035A CN 108259252 A CN108259252 A CN 108259252A
- Authority
- CN
- China
- Prior art keywords
- redis
- twemproxy
- host
- request
- client
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L41/00—Arrangements for maintenance, administration or management of data switching networks, e.g. of packet switching networks
- H04L41/08—Configuration management of networks or network elements
- H04L41/0803—Configuration setting
- H04L41/0813—Configuration setting characterised by the conditions triggering a change of settings
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L41/00—Arrangements for maintenance, administration or management of data switching networks, e.g. of packet switching networks
- H04L41/06—Management of faults, events, alarms or notifications
- H04L41/0654—Management of faults, events, alarms or notifications using network fault recovery
- H04L41/0663—Performing the actions predefined by failover planning, e.g. switching to standby network elements
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L41/00—Arrangements for maintenance, administration or management of data switching networks, e.g. of packet switching networks
- H04L41/06—Management of faults, events, alarms or notifications
- H04L41/0654—Management of faults, events, alarms or notifications using network fault recovery
- H04L41/0668—Management of faults, events, alarms or notifications using network fault recovery by dynamic selection of recovery network elements, e.g. replacement by the most appropriate element after failure
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/50—Network services
- H04L67/56—Provisioning of proxy services
- H04L67/568—Storing data temporarily at an intermediate stage, e.g. caching
Landscapes
- Engineering & Computer Science (AREA)
- Computer Networks & Wireless Communication (AREA)
- Signal Processing (AREA)
- Debugging And Monitoring (AREA)
Abstract
The invention discloses a kind of Redis cluster methods based on twemproxy, are related to caching establishment field.Including the following contents：Business side is connected by redis client requests, transmission data；Lvs does four layers of load, receives client request back loading to a Lvs host；After Lvs receives request, according to loading rule, a twemproxy host is sent data to；Twemproxy receives the fragment rule being configured after request according to client and finds a redis host, transmission data；After Redis hosts receive request, data are stored, and be synchronized to slave, returned.Redis group schemes the present invention is based on twemproxy propose a kind of new redis Cluster Solutions, environment preliminary test stability is preferable on line at present, with certain disaster tolerance, reduce server and delay machine risk, be suitble to rely on than heavier distributed environment redis.
Description
Technical field
Embodiments of the present invention are related to caching establishment field, more specifically, embodiments of the present invention are related to a kind of base
In the Redis cluster methods of twemproxy.
Background technology
With the explosive growth of development of Mobile Internet technology, big data, artificial intelligence are fast-developing, the day of multiple terminals exploitation
Beneficial popular, the continuous improvement of business complexity, the requirement of real-time of server-side is also higher and higher, and traditional unit services increasingly
The market demand can not be met, " three papers " of Google opens the tide of big data industry, it is proposed that with common unit service
The distributed type assemblies of device composition solve the pressure that explosion type data increase, and caching becomes one of them indispensable establishment.
At present caching set up have redis, memcached, ehcache etc., in fact redis because efficient using flexible most by
It welcomes, but redis exploitation early stages do not propose distributed thought, so existing in a manner of single thread.It is popular at present
Redis group schemes have client fragment, twemproxy, redis official cluster, twemproxy clusters.
1st, client fragment
Client fragment is that the logic of fragment is placed on Redis clients to realize, is pre-defined by Redis clients
Routing rule, being forwarded in different Redis examples to the access of Key, finally collect returning the result.
The advantages of client fragmentation schema is that all logics are all controllable, does not depend on third party's logic, but shortcoming is this
Kind fragment is a kind of static fragment, needs, in client manual operations, to greatly improve development difficulty, and when redis clusters
Stripping strategy is adjusted during dilatation.
2、twemproxy
Twemproxy is acted on behalf of by the Redis that Twitter increases income, and basic principle is：Redis clients send out request
Twemproxy is sent to, Twemproxy is sent to correct Redis examples according to routing rule, and last Twemproxy is result
Collect and return to client.Multiple Redis examples are managed collectively, make by introducing an Agent layer by Twemproxy
Redis clients only need to be operated on Twemproxy, without how many Redis examples below be concerned about, so as to
Realize Redis clusters.
Twemproxy advantages are that exploitation is simple, using twemproxy with using redis, client unaware,
Twemproxy supports most redis orders at present, but twemproxy just can not after reading configuration file is started
It reads again, therefore additions and deletions redis collection group agent can not perceive, this brings dilatation very big problem.
3、codis
Codis is a redis clustered softwares of pea pods independent research, it is route using zookeeper storage data,
Perfect to realize redis dynamic capacity-expandings, the migrating data of energy safety transparent supports a variety of Data fragment algorithms.
The shortcomings that Codis clusters are very perfect for redis, but maximum is that codis is to use go language developments, for very
Development teams mostly using java as basic language bring very big difficulty, the O＆M of codis is needed to spend very big manpower into
This.
4th, redis3.0 clusters
3.0 clusters of Redis employ the pattern of P2P, complete decentralization.All Key are divided into 16384 by Redis
A slot, each Redis examples are responsible for a portion slot.All information (node, port, slot etc.) in cluster, all
By the way that regularly data exchange updates between node.
Redis clients send out request in any one Redis example, if required data not in this example, pass through
It redirects order guiding client and accesses required example.
Redis official websites cluster is that deployment is simple a little, because of non-stop layer, deployment complexity is very low, and shortcoming is
Redis3.0 starts have modification to redis agreements, and client will also upgrade, there is presently no passing through to put into practice certification, to current
Switch for the environment run on warp pretty troublesome.
Invention content
The purpose of the present invention is to propose to a kind of cluster method based on twemproxy, in the base for ensureing original twemproxy
This function increases following functions simultaneously：(1) the quasi- smooth expansion of dynamic and change stripping strategy, (3) increase front end load, pass through the whistle
Soldier detects the switching of redis main memories.
In order to reach above-mentioned technique effect, the present invention takes following technical scheme：
A kind of Redis cluster methods based on twemproxy, including the following contents：
Business side is connected by redis client requests, transmission data；
Lvs does four layers of load, receives client request back loading to a Lvs host；
After Lvs receives request, according to loading rule, a twemproxy host is sent data to；
Twemproxy receives the fragment rule being configured after request according to client and finds a redis host, sends number
According to；
After Redis hosts receive request, data are stored, and be synchronized to slave, returned.
Further technical solution is：After redis hosts delay machine, zookeeper clients detect that redis hosts are delayed
Machine sends data to ZK servers；It is host that sentry's machine discovery host, which delays and slave is actively triggered after machine,；Zookeeper takes
After business end detects client request, send a command to twemproxy and replace configuration file；Twemproxy is received
Script is triggered after zookeeper orders, configuration file is read in restarting.
Further technical solution is：After redis hosts restore, redis hosts delay and are actively set as slave after machine, extensive
It is still slave after multiple.
Further technical solution is：Twemproxy configurations are monitored by zookeeper, once find that configuration file goes out
It now changes, zookeeper forces twemproxy to be restarted, and twexproxy reads latest configuration after startup, again to redis dilatations
With change stripping strategy.
Compared with prior art, the present invention with following advantageous effect：The present invention is based on the redis collection of twemproxy
Group's scheme proposes a kind of new redis Cluster Solutions, and environment preliminary test stability is preferable on line at present, has certain
Disaster tolerance reduces server and delays machine risk, is suitble to rely on than heavier distributed environment redis.
Description of the drawings
Fig. 1 shows flow diagram according to an embodiment of the invention.
Specific embodiment
In order to make the purpose , technical scheme and advantage of the present invention be clearer, with reference to embodiments, to the present invention
It is further elaborated.It should be appreciated that the specific embodiments described herein are merely illustrative of the present invention, it is not used to
Limit the present invention.
Embodiment
As shown in Figure 1, a kind of Redis cluster methods based on twemproxy, including the following contents：
Data flow
1st, business side is connected by redis client requests, transmission data.
2nd, Lvs does four layers of load, receives and is loaded to a lvs host according to High Availabitity rule after client request.
3rd, after Lvs receives request, according to loading rule, a twemproxy host is sent data to.
4th, Twemproxy receives the fragment rule being configured after request according to client and finds a redis host, sends number
According to.
5th, after Redis hosts receive request, data are stored, and be synchronized to slave, returned.
After redis hosts delay machine
1st, zookeeper clients detect that redis hosts are delayed machine, send data to ZK servers.
2nd, it is host that sentry's machine discovery host, which delays and slave is actively triggered after machine,.
3rd, it after Zookeeper server-sides detect client request, sends a command to twemproxy and replaces configuration file.
4th, Twemproxy triggers script after receiving zookeeper orders, and configuration file is read in restarting
After redis hosts restore
1redis hosts delay and are actively set as slave after machine, are still slave after recovery.
The present invention increases following functions simultaneously in the basic function for ensureing original twemproxy：
1. the quasi- smooth expansion of dynamic and change stripping strategy
Twemproxy configurations are monitored by zookeeper, once finding that configuration file is changed, zookeeper is forced
Twemproxy is restarted, because twemproxy is set up with the agency of c language developments, is started speed and is exceedingly fast, test is general at present
0.5 second or so, twexproxy read latest configuration after startup, again to redis dilatations and change stripping strategy.
2. increase front end load
Twemproxy is that more stable agency is set up, but machine risk of also delaying, by having increased load both at front end TP layers
It does not influence original data transmissions and can greatly reduce twemproxy again to delay machine risk.
3. it detects redis slaves by sentry to switch
Redis can be started with sentinel patterns, and monitoring host computer and slave can be completed after host delays machine with moment
Slave is replaced, but zookeeper cooperations is needed to replace twemproxy configuration files when completing slave and replacing.
Although reference be made herein to invention has been described for explanatory embodiment of the invention, however, it is to be understood that ability
Field technique personnel can be designed that a lot of other modifications and embodiment, these modifications and embodiment will be fallen in the application public affairs
Within the spirit and spirit opened.It more specifically, can be to the group of theme combination layout in range disclosed in the present application
A variety of variations and modifications are carried out into component and/or layout.In addition to variations and improvements to the component parts and or layout,
To those skilled in the art, other purposes also will be apparent.
Claims (4)
1. a kind of Redis cluster methods based on twemproxy, which is characterized in that including the following contents：
Business side is connected by redis client requests, transmission data；
Lvs does four layers of load, receives client request back loading to a Lvs host；
After Lvs receives request, according to loading rule, a twemproxy host is sent data to；
Twemproxy receives the fragment rule being configured after request according to client and finds a redis host, transmission data；
After Redis hosts receive request, data are stored, and be synchronized to slave, returned.
2. a kind of Redis cluster methods based on twemproxy according to claim 1, which is characterized in that work as redis
Host is delayed after machine, and zookeeper clients detect that redis hosts are delayed machine, send data to ZK servers；Sentry's machine discovery
It is host that host, which delays and slave is actively triggered after machine,；After Zookeeper server-sides detect client request, send a command to
Twemproxy replaces configuration file；Twemproxy triggers script after receiving zookeeper orders, restarts, and reads
Configuration file.
3. a kind of Redis cluster methods based on twemproxy according to claim 2, which is characterized in that work as redis
After host restores, redis hosts delay and are actively set as slave after machine, are still slave after recovery.
4. a kind of Redis cluster methods based on twemproxy according to claim 1, which is characterized in that its feature exists
In：Twemproxy configurations are monitored by zookeeper, once finding that configuration file is changed, zookeeper is forced
Twemproxy is restarted, and twexproxy reads latest configuration after startup, again to redis dilatations and change stripping strategy.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201810343035.3A CN108259252A (en) | 2018-04-17 | 2018-04-17 | A kind of Redis cluster methods based on twemproxy |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201810343035.3A CN108259252A (en) | 2018-04-17 | 2018-04-17 | A kind of Redis cluster methods based on twemproxy |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN108259252A true CN108259252A (en) | 2018-07-06 |
Family
ID=62748395
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201810343035.3A Pending CN108259252A (en) | 2018-04-17 | 2018-04-17 | A kind of Redis cluster methods based on twemproxy |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN108259252A (en) |
Cited By (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110290163A (en) * | 2018-08-28 | 2019-09-27 | 新华三技术有限公司 | A kind of data processing method and device |
| CN111639061A (en) * | 2020-05-26 | 2020-09-08 | 深圳壹账通智能科技有限公司 | Data management method, device, medium and electronic equipment in Redis cluster |
| CN112346899A (en) * | 2020-11-06 | 2021-02-09 | 北京北信源软件股份有限公司 | Method and device for optimizing microservice performance |
| CN114785713A (en) * | 2022-03-31 | 2022-07-22 | 度小满科技(北京)有限公司 | Method and proxy middleware for realizing high availability of Redis cluster |
Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103336815A (en) * | 2013-06-27 | 2013-10-02 | 北京京东尚科信息技术有限公司 | System and method of pushing network advertising |
| CN106055698A (en) * | 2016-06-14 | 2016-10-26 | 智者四海（北京）技术有限公司 | Data migration method, agent node and database instance |
| US20170163478A1 (en) * | 2015-12-02 | 2017-06-08 | Le Holdings (Beijing) Co., Ltd. | Method,electronic device and system for updating client configuration in key-value pair database |
-
2018
- 2018-04-17 CN CN201810343035.3A patent/CN108259252A/en active Pending
Patent Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103336815A (en) * | 2013-06-27 | 2013-10-02 | 北京京东尚科信息技术有限公司 | System and method of pushing network advertising |
| US20170163478A1 (en) * | 2015-12-02 | 2017-06-08 | Le Holdings (Beijing) Co., Ltd. | Method,electronic device and system for updating client configuration in key-value pair database |
| CN106055698A (en) * | 2016-06-14 | 2016-10-26 | 智者四海（北京）技术有限公司 | Data migration method, agent node and database instance |
Non-Patent Citations (2)
| Title |
|---|
| DARWIN BOYLE等: "《Visual Basic 4 开发人员指南》", 30 April 1997 * |
| 王心妍等: "基于twemproxy的Redis集群解决方案的设计与实现.王心妍等", 《电子测试》 * |
Cited By (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110290163A (en) * | 2018-08-28 | 2019-09-27 | 新华三技术有限公司 | A kind of data processing method and device |
| CN111639061A (en) * | 2020-05-26 | 2020-09-08 | 深圳壹账通智能科技有限公司 | Data management method, device, medium and electronic equipment in Redis cluster |
| CN111639061B (en) * | 2020-05-26 | 2023-03-17 | 深圳壹账通智能科技有限公司 | Data management method, device, medium and electronic equipment in Redis cluster |
| CN112346899A (en) * | 2020-11-06 | 2021-02-09 | 北京北信源软件股份有限公司 | Method and device for optimizing microservice performance |
| CN112346899B (en) * | 2020-11-06 | 2024-02-09 | 北京北信源软件股份有限公司 | Micro-service performance optimization method and device |
| CN114785713A (en) * | 2022-03-31 | 2022-07-22 | 度小满科技(北京)有限公司 | Method and proxy middleware for realizing high availability of Redis cluster |
| CN114785713B (en) * | 2022-03-31 | 2024-02-23 | 度小满科技(北京)有限公司 | Method and proxy middleware for realizing high availability of Redis cluster |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US11349949B2 (en) | Method of using path signatures to facilitate the recovery from network link failures | |
| US11704290B2 (en) | Methods, devices and systems for maintaining consistency of metadata and data across data centers | |
| CN108259252A (en) | A kind of Redis cluster methods based on twemproxy | |
| US10530855B2 (en) | Lock state synchronization for non-disruptive persistent operation | |
| US8495250B2 (en) | Asynchronous file operations in a scalable multi-node file system cache for a remote cluster file system | |
| US8195769B2 (en) | Rule based aggregation of files and transactions in a switched file system | |
| US8533231B2 (en) | Cloud storage system with distributed metadata | |
| CN102523234B (en) | A kind of application server cluster implementation method and system | |
| CA2512312C (en) | Metadata based file switch and switched file system | |
| CN102571991B (en) | Multistage-mapping-based large-scale multi-copy distributed storage system and application method thereof | |
| US20090144338A1 (en) | Asynchronously replicated database system using dynamic mastership | |
| CN106294585A (en) | A kind of storage method under cloud computing platform | |
| CN106156359A (en) | A kind of data synchronization updating method under cloud computing platform | |
| AU2015241457A1 (en) | Geographically-distributed file system using coordinated namespace replication | |
| US8572201B2 (en) | System and method for providing a directory service network | |
| US8478898B2 (en) | System and method for routing directory service operations in a directory service network | |
| CN104935668A (en) | Distributed file system and data synchronization method therefor | |
| CN109753244A (en) | A kind of application method of Redis cluster | |
| CN106850724B (en) | Data pushing method and device | |
| JP2016530656A (en) | Distributed disaster recovery file synchronization server system | |
| CN112866406B (en) | Data storage method, system, device, equipment and storage medium | |
| US9922031B2 (en) | System and method for efficient directory performance using non-persistent storage | |
| JP5756379B2 (en) | Data transfer system, transfer source system, transfer destination system, and program | |
| Junping | Analysis of key technologies of distributed file system based on big data [J] | |
| Riley | MusicNode: A Music File and Playback Synchronization Tool |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20180706 |
|
| RJ01 | Rejection of invention patent application after publication |