CN114398453A - Data access method based on Redis cache technology - Google Patents

Abstract

The invention discloses a data access method based on Redis cache technology, which comprises the following steps of firstly dividing a MYSQL database into a plurality of sub-databases; then distributing the condition requirements of the dictionary table to each sub-database; then each sub-database adopts a redis distributed lock to process the corresponding condition requirement; and finally, when the sub-database can meet the storage condition of the corresponding redis distributed lock, storing the data in the corresponding redis distributed lock into a MYSQLDB persistent database of the sub-database. The invention improves the convenience of system development and the reliability of operation, has wide support degree, can greatly reduce the pressure of frequently refreshing pages on the database for high concurrency, particularly for a large number of requests for writing, and improves the stability of system operation while ensuring the accuracy of data.

Description

Data access method based on Redis cache technology

Technical Field

The invention relates to the technical field of micro service architecture, in particular to a data access method based on a Redis cache technology.

Background

The existing data access method generally comprises a data storage and import module, a Mysql database, an Hbase database and a Query module, wherein the storage and import module (Importer module) analyzes all fields during data storage, extracts retrieval fields and display fields, stores the retrieval fields in the Mysql database, and stores the display fields in the Hbase; the Query module is responsible for performing lexical analysis and syntactic analysis on the Query statement, and reading related information from Mysql and Hbase according to an analysis result, as shown in fig. 1.

In the scheme, data Query is carried out through a Query module, and data display is carried out on a field corresponding to Hbase. Then if it is to be stored: and the Importer carries out value taking of the data field and stores the data field into a Mysql persistent database. If 3000 requests per second come from the system peak period, the system architecture is still above, and thus MYSQL will be down.

Disclosure of Invention

The technical problem to be solved by the invention is to provide a data access method based on a Redis cache technology aiming at the defects involved in the background technology.

The invention adopts the following technical scheme for solving the technical problems:

a data access method based on Redis cache technology comprises the following steps:

step 1), dividing the MYSQL database into a plurality of sub-databases;

step 2), distributing the condition requirements of the dictionary table to each sub-database;

step 3), each sub-database adopts a redis distributed lock to process the corresponding condition requirement;

and 4) when the sub-database can meet the storage condition of the corresponding redis distributed lock, storing the data in the corresponding redis distributed lock into a MYSQLDB persistent database of the sub-database.

As a further optimization scheme of the data access method based on the Redis caching technology, the release and loss of the lock are prevented by setting lock timeout in the step 3) and the step 4).

Compared with the prior art, the invention adopting the technical scheme has the following technical effects:

the invention discloses a method for realizing information cache reading/writing based on Redis cache technology, which divides an original database into a plurality of sub-databases, each sub-database writes data requirements simultaneously, the original database can only complete one writing requirement every 20ms, each sub-database in the plurality of sub-databases can complete one writing requirement every 8ms, the processing speed of table alignment depends on the number of the divided sub-databases.

Drawings

FIG. 1 is a schematic diagram of the principles of the present invention;

fig. 2 is a schematic diagram of the lock timeout solution of the present invention.

Detailed Description

The technical scheme of the invention is further explained in detail by combining the attached drawings:

the present invention may be embodied in many different forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the scope of the invention to those skilled in the art. In the drawings, components are exaggerated for clarity.

As shown in fig. 1, the present invention discloses a data access method based on the Redis caching technology, which comprises the following steps:

step 1), dividing the MYSQL database into a plurality of sub-databases;

step 2), distributing the condition requirements of the dictionary table to each sub-database;

step 3), each sub-database adopts a redis distributed lock to process the corresponding condition requirement;

and 4) when the sub-database can meet the storage condition of the corresponding redis distributed lock, storing the data in the corresponding redis distributed lock into a MYSQLDB persistent database of the sub-database.

When data are written in, Redis caches the data, the data are not stored in the MYSQL persistent database, if a large amount of data are stored in the MYSQL, the MYSQL persistent database is down, the amount of the cached data is larger than the amount of the cached data or the amount of the cached data is stored in the MYSQL persistent database at regular time, therefore, the access times of the MYSQL database can be reduced, and the performance loss caused by long-term occupation of i/o can be solved.

Redis may set a lock timeout, which may prevent the release and loss of the lock, such as for example, the A process and the B process: the quartz/elastic-joba/xxl-joba sets a timeout, the A process is not run, the lock is not released, and the lock is directly taken away in case of collapse, so that the lock can not be taken by anyone in the system, and the operation behind the B process can not be given.

The problem of lock is solved as shown in fig. 2, in the figure, the time T5 is changed into lock overtime, how long the time is set by Redis to release, the process B can take the lock at T6, the operation of the process a is completed, and the lock is released by calling a delete at the end. This is a solution to the lock out problem in Redis.

The detailed steps are as follows:

1. there is no such method in the two operations atomic spring data RedisTemplate to guarantee setting of lock and expiration time, but there is such method of atomic operation in jedis;

acquiring an operation command object in jedis by an execute method of Redistemperlate;

NX indicates that set was successful only if the locked resource did not exist. By utilizing the atomicity of Redis, the method ensures that only the thread of the first request can obtain the lock, and then other threads cannot obtain the lock before the locking resource is released; xpire represents the automatic expiration time of the locked resource in milliseconds. The specific expiration time depends on the actual scenario. The atomicity of the Redis value and the automatic expiration time is guaranteed through the command setting of set NX and PX, the phenomenon that a thread is hung when a setIfAbsent method is called and deadlock is caused due to the fact that the expiration time is not set is avoided, and a lock cannot be released

try {

String result = redisTemplate.execute(new RedisCallback() {

@Override

public String doInRedis(RedisConnection connection) throws DataAccessException {

JedisCommands commands = (JedisCommands) connection.getNativeConnection();

String uuid = UUID.randomUUID().toString();

Set (uuid); locked resource

return commands.set(key, uuid, "NX", "PX", expire);

}

});

return !StringUtils.isEmpty(result);

} catch (Exception e) {

System.out.println(e.getMessage());

}

return false;

}

4. The above method guarantees atomicity of the Redis value and the automatic expiration time by setting set NX, PX commands, but there is a problem that if thread A acquires the lock, but blocks for some reason for a long time when processing A service, the set expiration time is up, thread B acquires the lock, and after thread A finishes operating, the lock is released (the lock of B is released), that is, there is no protection mechanism for providing the lock on the thread of B. Therefore, an owner identifier needs to be locked, that is, a random and non-unique string is generated and placed in the current thread each time when the lock is acquired, and whether the corresponding value is the same as the value in the thread or not is judged when the lock is released.

public boolean releaseLock(String key) { }

5. When the lock is released, the execution time of the method after the lock is held is larger than the validity period of the lock, and the lock may be held by another thread at the moment, so that the lock cannot be deleted directly.

try {

List keys = new ArrayList();

keys.add(key);

List args = new ArrayList();

args.add(lockFlag.get());

}

6. And deleting the key matched with the value in the redis by using the lua script, so that the problem that locks of other threads are mistakenly deleted when the redis lock is automatically expired and invalid due to overlong execution time of the method can be avoided.

In the spring self-contained script execution method, the cluster mode directly throws the exception which does not support the script execution, so the script can only be executed by taking the connection of the original redis.

Long result = redisTemplate.execute(new RedisCallback() {

public Long doInRedis(RedisConnection connection) throws DataAccessException {

Object nativeConnection = connection.getNativeConnection();

}

8. Although the method of executing the script is the same as that of the cluster mode and the single mode, the cluster mode and the single mode have no common interface and can only be executed separately.

9. Cluster mode

if (nativeConnection instanceof JedisCluster) {

return (Long) ((JedisCluster) nativeConnection).eval(UNLOCK_LUA, keys, args);

}

10. Single machine mode

else if (nativeConnection instanceof Jedis) {

return (Long) ((Jedis) nativeConnection).eval(UNLOCK_LUA, keys, args);

}

return 0L;

}

});

return result != null && result > 0;

} catch (Exception e) {

e.printStackTrace();

}

return false;

}

The method includes the steps of utilizing the establishment of a java development environment to obtain various data sources in the system, classifying the obtained data, storing the data into a MYSQL persistent database, caching the data by adopting Redis according to a SpringBoot2.5.8 framework, supporting a native asynchronous view, adopting a middleware technology, avoiding downtime of the MYSQL database in the process of reading and writing a large amount of data, improving the efficiency of writing the data, and generally adopting a caching mechanism to load the data in the database into a memory or store the data in a storage medium with higher processing access speed for a long time when a high concurrency request is solved by the system so as to reduce the number of times that the MYSQL persistent database is accessed and further reduce performance loss caused by long-term occupation of i/o of the database in a large amount of reading actions.

It will be understood by those skilled in the art that, unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention belongs. It will be further understood that terms, such as those defined in commonly used dictionaries, should be interpreted as having a meaning that is consistent with their meaning in the context of the prior art and will not be interpreted in an idealized or overly formal sense unless expressly so defined herein.

The above-mentioned embodiments, objects, technical solutions and advantages of the present invention are further described in detail, it should be understood that the above-mentioned embodiments are only illustrative of the present invention and are not intended to limit the present invention, and any modifications, equivalents, improvements and the like made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims (2)

1. A data access method based on Redis cache technology is characterized by comprising the following steps:

step 1), dividing the MYSQL database into a plurality of sub-databases;

step 2), distributing the condition requirements of the dictionary table to each sub-database;

step 3), each sub-database adopts a redis distributed lock to process the corresponding condition requirement;

and 4) when the sub-database can meet the storage condition of the corresponding redis distributed lock, storing the data in the corresponding redis distributed lock into a MYSQLDB persistent database of the sub-database.

2. The Redis caching technique based data access method according to claim 1, wherein the step 3) and the step 4) are implemented to prevent the release and the loss of the lock by setting a lock timeout.

Images