CN104363303A - Method for synchronizing asynchronously cached data - Google Patents

Abstract

The invention relates to the technical field of networks and specifically relates to a method for synchronizing asynchronously cached data. The method comprises the steps of firstly putting MySQL data into Gearman, and then synchronizing the data into Redis by use of a programmed PHP Gearman Worker. The method for synchronizing the asynchronously cached data is used for solving the problems of large expenditure of static caching and Redis processing, and difficult satisfaction of data reliability, and can be used for performing the synchronous processing on the asynchronously cached data.

Description

The method of asynchronous buffer data syn-chronization

Technical field

The present invention relates to networking technology area, be specifically related to a kind of method of asynchronous buffer data syn-chronization.

Background technology

Today of network technology high speed development, caching technology has become a key technology of large-scale website and Mobile solution.The speed of a website visiting of caching design quality direct relation, and purchase the quantity of server, even have influence on the experience of user.

For change frequency data quickly, if also select traditional static cache mode (Memcached, File System etc.) display data, very large expense may be had in the access of buffer memory, can not well satisfy the demand.And Redis is like this based on the NoSQL database of internal memory, be just cut out for very much the container of real time data.

But often we have again the demand of data reliability.

Summary of the invention

The technical problem that the present invention solves is a kind of method providing data cache synchronization, effectively solves static cache, Redis process exists that expense is large, the implacable problem of data reliability.

The technical scheme that the present invention solves the problems of the technologies described above is:

First MySQL data are put into Gearman, and the PHP GearmanWorker then write by, by data syn-chronization to Redis.

Described method comprises the following steps:

One, Gearman is installed;

After installing, check the operation conditions of Gearman; When returning gearmand is running, Gearman successful installation is described;

Two, PHP Gearman expands;

Employing compile mode is installed;

Three, by MySQL UDF+Trigger synchrodata to Gearman;

Use the combination of lib_mysqludf_json and gearman-mysql-udf, allow MySQL import data into Gearman;

First lib_mysqludf_json is installed;

Then, the plug-in unit installation path of MySQL is checked;

Again by lib_mysqludf_json.so file copy to correspondence position:

Finally login MySQL and run statement registration UDF function.

The Redis part of the corresponding Resque of Job Server of described described Gearman, the Queue operation of the corresponding Resque of Client, Worker and Job of the corresponding Resque of Worker.

During described PHP Gearman expands, select legacy version expansion is installed:

pecl install gearman-1.0.3

Gearman+PHP example

To carry out file process,

First a Gearman Client is write and called after client.php

Run associated documents, after analog subscriber request Web page, information process terminated returns user,

Can see that output is

writeLog 1 0 0

The task of establishing a writeLog by name in Gearman is described, and has 1 task in queue is waited for;

And 4 row above represent the running status of current Gearman respectively: task names, the task in waiting list, running of task, the Worker process run;

Watch can be used to monitor in real time:

Then a Gearman Worker called after worker.php is write:

Worker uses a while endless loop to realize finger daemon, runs php worker.php and can see that Gearman state becomes:

writeLog 0 0 1

Check gearman.log under same catalogue, content should be the value Log conte imported into from Client simultaneously.The invention has the beneficial effects as follows: the present invention adopts MySQL as data storage engines, and Redis is then as Cache; Adopt the data Replica of combination asynchronous implement MySQL to the Redis of Gearman+PHP+MySQL UDF.It is lower that the present invention realizes cost, more easily operates.

Accompanying drawing explanation

Below in conjunction with accompanying drawing, the present invention is further described:

Fig. 1 is overall framework figure of the present invention.

Embodiment

In the process of asynchronous buffer data, need to consider performance, stability and availability issue.

Performance---that respective data storage is got up to avoid data repeats establishment, process and transmission, effectively can improve performance.Such as immovable data buffer storage is got up, such as list of countries etc., the reaction speed of web program can be significantly improved like this.

Stability---in same application, recurrent during repeatedly request to same data, logic function and user interface.When user base number is very large, if each request all processes, the resource of consumption is very large waste, also causes the instability of system simultaneously.Such as, in web application, the effective saving resource of buffer memory energy is carried out to the rendering content of some static pages, improves stability.And the data cached access times that also can reduce database, reduce the burden of database and improve the service ability of database.

Availability---sometimes, provide the service of data message surprisingly to stop, if employ caching technology, the support to end user still normally can be provided within a certain period of time, improve the availability of system.

Key point of the present invention is:

Key point 1, understands state

Before deeply introducing caching technology, need there is a understanding, because buffer memory can be described as the framework of condition managing to state.The implication of understanding state and its some characteristics---such as life cycle and scope---are to determining whether buffer memory has very great help with the suitable caching technology of selection.State refers to some data, on certain time point of application system, and the state of data and condition.These data may be permanent storage in a database, may be only stop in internal memory for a moment, also may be that according to the survival of certain logic (such as how long discharging afterwards), its range of application may be all user-accessibles, may be that unique user has permission;

Key point 2, the life cycle of state;

Refer to that data maintain the time interval of validity life cycle, namely from being created to the time interval removed.Common life cycle has following several:

The permanent data that permanent state Permanent State---application program uses;

Process status Process State---only effective within the process cycle;

Session status Session State---relevant with specific user conversation;

Message status Message State---the time processing certain message is effectively interior;

Key point 3, status data outmoded;

The status data of buffer memory is the snapshot of master data (Master State Data), because data source may be modified, so status data just has characteristic that can be outmoded.This characteristic of Appropriate application and negative effect outmoded for data to be minimized be a vital task of buffer status data.You can define the outmoded foundation of data with a under type:

The possibility of master data change---along with the propelling of time, may whether greatly increasing of master data change? the outmoded of buffer status data is decided according to this point;

Does the not corresponding renewal of status data of buffer memory impact the use of system when the correlation of change---master data upgrades? such as, the appearance style changing system can't make a big impact to business.

Key point 4, the tolerance that status data is outmoded;

The outmoded impact on operation flow of buffer status data is called tolerance, application system can for can't stand (No Tolerance) and tolerance to a certain degree (some Tolerance), the former must with master data synchronized update, the latter allows the outmoded of certain hour or certain limit, and criterion is exactly the disturbance degree to operation flow.

Based on to the understanding that the present invention relates to problem, no matter MySQL or Redis, self is all with the mechanism of data syn-chronization, the Master/Slave pattern of the MySQL that picture is relatively more conventional, the binlog analyzing Master is held to realize by Slave, such data Replica in fact or an asynchronous procedure, only when server is all in same Intranet, almost can ignore by asynchronous delay.

So also can use the same manner, analyze the binlog file of MySQL and data are inserted Redis.But this needs there is very deep understanding to binlog file and MySQL, simultaneously because binlog exists Statement/Row/Mixedlevel various ways, it is very large that analysis binlog realizes synchronous workload.

Therefore the present invention selects the mode that a kind of development cost is cheaper, uses the MySQLUDF of comparative maturity, and first MySQL data are put into Gearman, and the PHP GearmanWorker then oneself write by, by data syn-chronization to Redis.Add many flow processs than the mode analyzing binlog, but it is lower to realize cost, more easily operates.

As shown in Figure 1, below in conjunction with instantiation, the present invention will be described.

(1) installation and use of Gearman

Gearman is one and supports distributed task distribution framework.Design is succinct, obtains support widely.A typical Gearman application comprises these parts following:

Gearman Job Server:Gearman kernel program, needs compiling install and operate in backstage with finger daemon form

Gearman Client: the addressee person that can be understood as task, such as I will perform the task that sends mail on backstage, a Gearman Client can be called in a program and import the information of mail into, then just execution result can be showed user immediately, and task itself can slowly at running background.Gearman Worker: the real executor of task, general needs oneself is write concrete logic and is run by finger daemon mode, after Gearman Worker receives the task definition of Gearman Client transmission, can processed in sequence.

The Redis part of Gearman Job Server: corresponding Resque

The Queue operation of Gearman Client: corresponding Resque

Gearman Worker: Worker and Job of corresponding Resque

Here Gearman instead of Resque why is selected to be that workload is less because Gearman provides relatively good MySQL UDF.

(2) Gearman and PHP Gearman expansion is installed

apt-get install gearman gearman-server libgearman-dev

Check the operation conditions of Gearman:

/etc/init.d/gearman-job-server status

When returning gearmand is running, Gearman successful installation is described.

(3) the Gearman expansion of PHP

pecl install gearman

echo″extension＝gearman.so″>/etc/php5/conf.d/gearman.ini

service php5-fpm restart

But actual measurement finds that the gearman version of ubuntu default installation is too low, directly running peclinstall gearman can report an error

configure：error：libgearman vetsion 1.1.0or later required

Therefore Gearman+PHP expansion suggestion is installed by compile mode, here in order to simple declaration, selects to install legacy version expansion:

pecl install gearman-1.0.3

Gearman+PHP example

In order to be easier to understand the operational process of hereinafter Gearman, might as well illustrate from a Gearman example the simplest here, such as we will carry out the operation of a file process, first write a Gearman Client and called after client.php

<？php

$client＝new GearmanClient()；

$client->addServer()；

$client->doBackground(’writeLog’，’Log content’)；

Echo ' file is in consistency operation ';

Run this file, after being equivalent to analog subscriber request Web page, information process terminated returns user:

php client.php

Look over the situation of Gearman:

(echo status；sleep 0.1)|netcat 127.0.0.1 4730

Can see that output is

writeLog 1 0 0

Illustrate that we have established the task of a writeLog by name in Gearman, and have 1 task in queue is waited for.

And 4 row above represent the running status of current Gearman respectively:

Task names

Task in waiting list

Running of task

The Worker process run

Watch can be used to monitor in real time:

watch-n l″(echo status；sleep 0.1)|nc 127.0.0.1 4730″

Then we need to write a Gearman Worker called after worker.php:

Worker uses a while endless loop to realize finger daemon, runs php worker.php and can see that Gearman state becomes:

writeLog 0 0 1

Check gearman.log under same catalogue, content should be the value Log content imported into from Client simultaneously.

By MySQL UDF+Trigger synchrodata to Gearman

MySQL will realize still being realized by MySQL UDF (MySQL userdefined functions) with the best way of external program intercommunication.In order to allow MySQL import data into Gearman, be used herein the combination of lib_mysqludf_json and gearman-mysql-udf.

Lib_mysqludf_json is installed

Using the reason of lib_mysqludf_json to be because Gearman only accepts character string as suction parameter, can be JSON character string by the data encoding in MySQL by lib_mysqludf_json.

apt-get install libmysqlclient-dev

wget

https：//github.com/mysqludf/lib_mysqludf_json/archive/master.zip

unzip master.zip

cd l ib_mysqludf_json-master/

rm l ib_mysqludf_json.so

gcc$(mysql_config--cflags)-shared-fPIC-o lib_mysqludf_json.solib_mysqludf_json.c

Can see that recompility generates lib_mysqludf_json.so file, now need the plug-in unit installation path checking MySQL:

mysql-u root-pPASSWORD--execute＝″show variables like’％plugin％’；″

Then by lib_mysqludf_json.so file copy to correspondence position:

cp lib_mysqludf_json.so/usr/lib/mysql/plugin/

Finally login MySQL and run statement registration UDF function:

CREATE FUNCTION json_object RETURNS STRING SONAME’lib_mysqludf_json.so’。

Claims (5)

1. the method for asynchronous buffer data syn-chronization, is characterized in that: first MySQL data are put into Gearman, and the PHP Gearman Worker then write by, by data syn-chronization to Redis.

2. the method for asynchronous buffer data syn-chronization according to claim 1, is characterized in that: described method comprises the following steps:

One, Gearman is installed;

After installing, check the operation conditions of Gearman; When returning gearmand is running, Gearman successful installation is described;

Two, PHP Gearman expands;

Employing compile mode is installed;

Three, by MySQL UDF+Trigger synchrodata to Gearman;

Use the combination of lib_mysqludf_json and gearman-mysql-udf, allow MySQL import data into Gearman;

First lib_mysqludf_json is installed;

Then, the plug-in unit installation path of MySQL is checked;

Again by lib_mysqludf_json.so file copy to correspondence position:

Finally login MySQL and run statement registration UDF function.

3. the method for asynchronous buffer data syn-chronization according to claim 1, it is characterized in that: the Redis part of the corresponding Resque of Job Server of described described Gearman, the Queue operation of the corresponding Resque of Client, Worker and Job of the corresponding Resque of Worker.

4. the method for asynchronous buffer data syn-chronization according to claim 2, it is characterized in that: the Redis part of the corresponding Resque of Job Server of described described Gearman, the Queue operation of the corresponding Resque of Client, Worker and Job of the corresponding Resque of Worker.

5. the method for the asynchronous buffer data syn-chronization according to claim 2 or 4, is characterized in that: during described PHP Gearman expands, and selects to install legacy version expansion:

pecl install gearman-1.0.3

Gearman+PHP example

To carry out file process,

First a Gearman Client is write and called after client.php

Run associated documents, after analog subscriber request Web page, information process terminated returns user,

Can see that output is

writeLog 1 0 0

The task of establishing a writeLog by name in Gearman is described, and has 1 task in queue is waited for;

And 4 row above represent the running status of current Gearman respectively: task names, the task in waiting list, running of task, the Worker process run;

Watch can be used to monitor in real time:

Then a Gearman Worker called after worker.php is write:

Worker uses a while endless loop to realize finger daemon, runs php worker.php and can see that Gearman state becomes:

writeLog 0 0 1

Check gearman.log under same catalogue, content should be the value Log conte imported into from C/ient simultaneously.