CN112711730A

CN112711730A - Data caching method and device, storage medium and electronic equipment

Info

Publication number: CN112711730A
Application number: CN202110081588.8A
Authority: CN
Inventors: 沙烨; 金仲伟; 张垒; 李乾猛
Original assignee: Shanghai Observer Information Technology Co ltd
Current assignee: Shanghai Observer Information Technology Co ltd
Priority date: 2021-01-21
Filing date: 2021-01-21
Publication date: 2021-04-27
Anticipated expiration: 2041-01-21
Also published as: CN112711730B

Abstract

The invention discloses a data caching method and device, a storage medium and electronic equipment. The method comprises the following steps: receiving an access request sent by a client, and acquiring a request address of the access request; accessing cdn a cache, and if the access request is hit, returning a request result; if not, acquiring the version number of the access request, taking the version number and the request address as a request identifier to access a redis cache, and if the request address is hit, returning a request result; if not, locking the access request by using the request identifier, and taking the request identifier as a lock identifier; and accessing mysql database cache by the lock identifier, refreshing the redis cache if the mysql database cache is hit, and returning a request result. The technical scheme disclosed by the invention can improve the performance of the front-end server and simultaneously reduce the use cost of the redis and the mysql.

Description

Data caching method and device, storage medium and electronic equipment

Technical Field

The present invention relates to the field of internet technologies, and in particular, to a data caching method and apparatus, a storage medium, and an electronic device.

Background

With the rapid development of the internet, the way of people to surf the internet becomes very convenient, the demands for network access are increasing day by day, and great pressure is brought to the website server, and the network access generally requires real-time processing of data and subsequent processing and analysis. The problem that how to reduce the load of the server, quickly respond to the request of the client and process and store the data in real time is solved by facing various devices and high concurrent access of a plurality of users. Therefore, how to improve the performance of the front-end server and reduce the use cost of the database is a technical problem to be solved urgently by those skilled in the art.

Disclosure of Invention

In view of the above drawbacks of the prior art, the technical problem to be solved by the present invention is that the concurrent access number increases the pressure on the front-end server and the use cost for redis and mysql is high. According to the scheme, three levels of caches are arranged, the first level adopts cdn caches, the second level adopts redis caches, and the third level adopts mysql database caches, so that the performance of a front-end server is improved, and the use cost of redis and mysql is reduced.

In order to achieve the above object, the present invention provides a data caching method, device, storage medium and electronic device.

In a preferred embodiment of the present invention, an embodiment of the present application provides a data caching method, where the method is executed by a front-end server, and the method includes:

receiving an access request sent by a client, and acquiring a request address of the access request;

accessing cdn a cache, and if the access request is hit, returning a request result;

if not, acquiring the version number of the access request, taking the version number and the request address as a request identifier to access a redis cache, and if the request address is hit, returning a request result;

if not, locking the access request by using the request identifier, and taking the request identifier as a lock identifier;

and accessing mysql database cache by the lock identifier, refreshing the redis cache if the mysql database cache is hit, and returning a request result.

Further, after the accessing the mysql database cache with the lock identification, the method further comprises:

and if not, sending the access request to a back-end server for the back-end server to obtain a request result through a network and returning the request result.

Further, after sending the access request to a back-end server for the back-end server to obtain a request result through a network and return the request result, the method further includes:

updating the mysql database cache according to the returned request result; and updates the redis cache according to the returned request result.

Further, after the locking processing is performed on the access request by using the request identifier and the request identifier is used as a lock identifier, the method further includes:

determining the validity period of the lock;

if no data is returned in the validity period of the lock, an error message is returned, and the lock resource is released.

Further, the method further comprises:

if an adding event of new data is detected, determining the type of the new data as a timely data type or a non-timely data type;

if the data type is the timely data type, the new data is synchronously added to an cdn cache, a redis cache and a mysql database cache;

and if the new data type is a non-timely data type, adding the new data into a processing queue to asynchronously add the new data.

Further, the cdn cache time is a preset fixed time length;

if the data to be cached is new data, the caching time of the redis cache is a first duration; and if the data to be cached is the data updated from the mysql database cache to the redis cache, the caching time of the redis cache is the second duration.

Further, the method further comprises:

acquiring whether the starting state of the interface is changed to be overdue;

if the interface is not expired, the caching time of the interface in the mysql database cache is permanent;

if the interface is changed to be overdue, the caching time of the interface in the mysql database is preset caching duration.

In another preferred embodiment of the present invention, an apparatus for caching data is configured in a front-end server, and the apparatus includes:

the access request receiving module is used for receiving an access request sent by a client and acquiring a request address of the access request;

cdn cache access module, for accessing cdn cache, if the access request is hit, returning request result;

the redis cache access module is used for acquiring the version number of the access request if the request is not hit, accessing the redis cache by taking the version number and the request address as request identifiers, and returning a request result if the request is hit;

the locking processing module is used for locking the access request by using the request identifier if the access request is not hit yet, and using the request identifier as a lock identifier;

and the mysql database cache access module is used for accessing the mysql database cache by using the lock identifier, refreshing the redis cache if the lock identifier is hit, and returning a request result.

Optionally, the apparatus is further configured to:

after the accessing a mysql database cache with the lock identification, the method further comprises:

Further, after sending the access request to a back-end server for the back-end server to obtain a request result through a network and return the request result, the apparatus is further configured to:

Further, after the locking processing is performed on the access request by using the request identifier and using the request identifier as a lock identifier, the apparatus is further configured to:

determining the validity period of the lock;

Further, the apparatus is further configured to:

Further, the cdn cache time is a preset fixed time length;

Further, the apparatus is further configured to:

acquiring whether the starting state of the interface is changed to be overdue;

In another preferred embodiment of the present invention, the present application provides a computer-readable storage medium, on which a computer program is stored, and the computer program, when executed by a processor, implements the data caching method according to the present application.

In another preferred embodiment of the present invention, an electronic device is provided, which includes a memory, a processor, and a computer program stored in the memory and executable by the processor, and when the processor executes the computer program, the processor implements the method for caching server-side data according to the embodiment of the present invention.

The technical scheme provided by the invention has the following technical effects:

according to the invention, three levels of caches are arranged, wherein the first level adopts cdn cache, the second level adopts redis cache, and the third level adopts mysql database cache, so that the performance of the front-end server is improved, and the use cost of redis and mysql is reduced.

The conception, the specific structure and the technical effects of the present invention will be further described with reference to the accompanying drawings to fully understand the objects, the features and the effects of the present invention.

Drawings

Fig. 1 is a schematic diagram of a data caching method according to an embodiment of the present application;

fig. 2 is a schematic diagram of a data caching apparatus according to an embodiment of the present application;

fig. 3 is a schematic structural diagram of an electronic device according to an embodiment of the present application.

Detailed Description

The embodiments of the present invention are described below with reference to specific embodiments, and other advantages and effects of the present invention will be easily understood by those skilled in the art from the disclosure of the present specification. The invention is capable of other and different embodiments and of being practiced or of being carried out in various ways, and its several details are capable of modification in various respects, all without departing from the spirit and scope of the present invention. It is to be noted that the features in the following embodiments and examples may be combined with each other without conflict.

It should be noted that the drawings provided in the following embodiments are only for illustrating the basic idea of the present invention, and the drawings only show the components related to the present invention rather than the number, shape and size of the components in actual implementation, and the type, quantity and proportion of the components in actual implementation may be changed freely, and the layout of the components may be more complicated.

Some exemplary embodiments of the invention have been described for illustrative purposes, and it is to be understood that the invention may be practiced otherwise than as specifically described.

Fig. 1 is a schematic diagram of a data caching method according to an embodiment of the present application, which is applicable to a case where a front-end server returns data. As shown in fig. 1, the method is executed by a front-end server, and the data caching method includes:

s110, receiving an access request sent by a client, and acquiring a request address of the access request.

The client may be in communication connection with the front-end server, the client may be an access request issued according to an operation instruction of a user, and the method request may be a request for accessing a page, data, or the like in an interface manner. The request address may be an address input by a user making a request, such as a url address.

And S120, accessing cdn a cache, and if the access request is hit, returning a request result.

In the scheme, cdn cache is the first level cache, and the client requests to access cdn cache, and if the client hits cdn cache, the client returns directly. The http cache is a browser-side cache and cdn is a server-side cache. The function of cdn is illustrated by way of example: cdn are agents. The manufacturer delivers goods to the merchant, and the merchant is cdn, which is convenient for you to buy goods from the merchant. Similar to Http, when a client requests data, the client searches from a local cache first, and if the requested data is not expired, the client takes the data for use, and if the requested data is expired, the client initiates a request to a CDN edge node. The CDN detects whether the requested data is expired, returns the data to the client if the requested data is not expired, and sends a request to the source station to obtain new data if the requested data is expired. The CDN edge node caching mechanism generally complies with an http standard protocol, and sets the data caching time of a CDN edge node through the fields of Cache-Control and max-age in an http response header.

And S130, if the request is not hit, obtaining the version number of the access request, using the version number and the request address as a request identifier to access the redis cache, and if the request is hit, returning a request result.

In the scheme, if the data is not acquired, the data is requested back to the source, and cdn automatically caches the data returned by the original station.

In this scheme, the request to return to the source is processed by using nginx + lua. Among them, nginx (engine x) is a high-performance HTTP and reverse proxy web server. Lua is a lightweight and compact scripting language written in the standard C language and open in source code form, designed to be embedded in applications to provide flexible extension and customization of applications.

And acquiring version number version of the access request by utilizing nginx, maintaining and managing a connection pool of the redis by adopting a lua-reserve-redis module, requesting redis cache data by taking url + version as an identifier, and directly returning the data if the cache is hit.

And S140, if the access request is not hit, locking the access request by using the request identifier, and using the request identifier as a lock identifier.

In this scheme, optionally, after the locking processing is performed on the access request by using the request identifier and the request identifier is used as a lock identifier, the method further includes:

determining the validity period of the lock;

Specifically, mysql request processing is performed on a request which does not hit in the redis cache, and the processing steps are as follows:

a. and managing the connection pool of the mysql by adopting a lua-restore-mysql module.

b. And adopting lua-reserve-lock to perform requested locking processing, wherein url + version is the identifier of the lock, setting the timeout time of the lock, and returning error information and releasing the lock resource if no data is returned within the timeout time of the lock.

c. The request for acquiring the lock for the first time is connected with a mysql database, the url + version is used as an identifier to acquire cache content, if the cache content is hit, a redis cache is refreshed, and data is returned after the lock resource is released; subsequent requests with the url + version may again get the redis cache directly.

d. And requesting the application program for the request which does not hit the mysql cache, and then performing the operation of the step c on the result returned by the application program.

In this embodiment, optionally, after the mysql database cache is accessed by using the lock identifier, the method further includes:

As in step d above.

In this scheme, optionally, after sending the access request to a back-end server for the back-end server to obtain a request result through a network and return the request result, the method further includes:

As in step c above.

determining the validity period of the lock;

As in step b above.

The principle of key generation for cache function of mysql is as follows: and generating a unique key by the select statement according to a certain hash rule, and generating a value by the result of the select, namely the key > value. So for cache, the select statement is case-specific, and also space-specific. The two select statements must be completely identical to each other to obtain the same cache.

After the cache is generated, all relevant caches are deleted as long as there is any data change in the table involved in the select (insert, update, delete operation, etc.). Therefore, only a table with little data change makes sense to introduce mysql cache.

Therefore, the cache function of mysql is only applicable to the following situations: less data variation, more select tables.

S150, accessing mysql database cache by the lock identifier, refreshing the redis cache if hit, and returning a request result.

In the scheme, after the mysql database access cache is hit, the redis cache can be cached according to the hit content, so that the request result is directly returned from the redis cache during subsequent access. Therefore, the service performance of the redis cache and the mysql database cache is improved.

According to the technical scheme, cdn cache is adopted in the first level, redis cache is adopted in the second level, mysql database cache is adopted in the third level, performance of a front-end server is improved, and using cost of redis and mysql is reduced.

On the basis of the above technical solutions, optionally, the method further includes:

Specifically, the cache preheating is processed by adopting two methods, namely event triggering and batch.

The event triggering method is mainly used for adding or changing new data, the data is divided into two types, namely timely data and non-technical data, the timely data is preheated in a synchronous mode, and a message queue for transmitting the non-timely data is asynchronously preheated. If a new news is added, at the time of releasing the news, the news content is synchronously preheated, and the channel page of the news is sent to the queue for asynchronous preheating.

The batch preheating is mainly used for issuing a new interface and modifying interface logic, and can be carried out according to parameters such as a time range, a self-increment ID range, a page number range, designated parameters and the like.

On the basis of the above technical solutions, optionally, the cache time cached by cdn is a preset fixed time length;

For cdn caching, a fixed caching time may be set according to different interfaces.

For the redis cache, the time for the redis cache may be set to 7 days in the process of preheating, which is the first time length, and the cache time may be set to 5 minutes when the redis data is refreshed in the second level, which is the second time length.

acquiring whether the starting state of the interface is changed to be overdue;

For mysql data caching, there may be no expiration time for the interface in the enabled state, and the expired interface is set to be 5 minutes of expiration time, that is, preset caching duration, and after 5 minutes, the background process will regularly clear the expired cache, and the enabled and expired states may be controlled by the interface management system.

Fig. 2 is a schematic diagram of a data caching apparatus according to an embodiment of the present application, and as shown in fig. 2, the apparatus is configured in a front-end server, and the apparatus includes:

an access request receiving module 210, configured to receive an access request sent by a client, and obtain a request address of the access request;

cdn cache access module 220, configured to access cdn cache, and return a request result if the access request is hit;

a redis cache access module 230, configured to, if the request is not hit, obtain a version number of the access request, access the redis cache by using the version number and the request address as a request identifier, and if the request is hit, return a request result;

a locking processing module 240, configured to perform locking processing on the access request by using the request identifier if the access request is still missed, and use the request identifier as a lock identifier;

and a mysql database cache access module 250, configured to access the mysql database cache with the lock identifier, if the lock identifier is hit, refresh the redis cache, and return a request result.

The product can execute the method provided by the embodiment of the application, and has the corresponding functional modules and beneficial effects of the execution method.

Embodiments of the present application also provide a storage medium containing computer-executable instructions that, when executed by a computer processor, perform a method for providing power based on a solar panel, the method comprising:

Storage medium-any of various types of memory devices or storage devices. The term "storage medium" is intended to include: mounting media such as CD-ROM, floppy disk, or tape devices; computer system memory or random access memory such as DRAM, DDR RAM, SRAM, EDO RAM, Lanbas (Rambus) RAM, etc.; non-volatile memory such as flash memory, magnetic media (e.g., hard disk or optical storage); registers or other similar types of memory elements, etc. The storage medium may also include other types of memory or combinations thereof. In addition, the storage medium may be located in the computer system in which the program is executed, or may be located in a different second computer system connected to the computer system through a network (such as the internet). The second computer system may provide the program instructions to the computer for execution. The term "storage medium" may include two or more storage media that may reside in different locations, such as in different computer systems that are connected by a network. The storage medium may store program instructions (e.g., embodied as a computer program) that are executable by one or more processors.

Of course, the storage medium provided in the embodiments of the present application contains computer-executable instructions, and the computer-executable instructions are not limited to the operations of the data caching method described above, and may also perform related operations in the data caching method provided in any embodiments of the present application.

Fig. 3 is a schematic structural diagram of an electronic device according to an embodiment of the present application. The electronic device may be a server or a terminal.

Referring now to FIG. 3, shown is a schematic diagram of an electronic device 300 suitable for use in implementing embodiments of the present application. The electronic device in the embodiment of the present application may be an electronic device for providing an information display function. The electronic device shown in fig. 3 is only an example, and should not bring any limitation to the functions and the scope of use of the embodiments of the present application.

As shown in fig. 3, the electronic device 300 may include a processing means (e.g., a central processing unit, a graphics processor, etc.) 301 that may perform various appropriate actions and processes in accordance with a program stored in a Read Only Memory (ROM)302 or a program loaded from a storage means 308 into a Random Access Memory (RAM) 303. In the RAM 303, various programs and data necessary for the operation of the electronic apparatus 300 are also stored. The processing device 301, the ROM 302, and the RAM 303 are connected to each other via a bus 304. An input/output (I/O) interface 305 is also connected to bus 304.

Generally, the following devices may be connected to the I/O interface 305: input devices 306 including, for example, a touch screen, touch pad, keyboard, mouse, camera, microphone, accelerometer, gyroscope, etc.; an output device 307 including, for example, a Liquid Crystal Display (LCD), a speaker, a vibrator, and the like; storage devices 308 including, for example, magnetic tape, hard disk, etc.; and a communication device 309. The communication means 309 may allow the electronic device 300 to communicate wirelessly or by wire with other electronic devices to exchange data. While fig. 3 illustrates an electronic device 300 having various means, it is to be understood that not all illustrated means are required to be implemented or provided. More or fewer devices may alternatively be implemented or provided.

In particular, according to embodiments of the application, the processes described above with reference to the flow diagrams may be implemented as computer software programs. For example, embodiments of the present application include a computer program product comprising a computer program embodied on a computer readable medium, the computer program comprising program code for performing the method illustrated by the flow chart. In such an embodiment, the computer program may be downloaded and installed from a network through the communication means 309, or installed from the storage means 308, or installed from the ROM 302. The computer program, when executed by the processing device 301, performs the above-described functions defined in the methods of the embodiments of the present application.

It should be noted that the computer readable medium mentioned above in the present application may be a computer readable signal medium or a computer readable medium or any combination of the two. A computer readable medium may be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination of the foregoing. More specific examples of the computer readable medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer diskette, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. In the present application, a computer readable medium may be any tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device. In this application, however, a computer readable signal medium may include a propagated data signal with computer readable program code embodied therein, for example, in baseband or as part of a carrier wave. Such a propagated data signal may take many forms, including, but not limited to, electro-magnetic, optical, or any suitable combination thereof. A computer readable signal medium may be any computer readable medium that is not a computer readable medium and that can communicate, propagate, or transport a program for use by or in connection with an instruction execution system, apparatus, or device. Program code embodied on a computer readable medium may be transmitted using any appropriate medium, including but not limited to: electrical wires, optical cables, RF (radio frequency), etc., or any suitable combination of the foregoing.

The computer readable medium may be embodied in the electronic device; or may exist separately without being assembled into the electronic device.

The computer readable medium carries one or more programs which, when executed by the electronic device, cause the electronic device to perform:

Computer program code for carrying out operations for aspects of the present application may be written in any combination of one or more programming languages, including an object oriented programming language such as Java, Smalltalk, C + + or the like and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The program code may execute entirely on the passenger computer, partly on the passenger computer, as a stand-alone software package, partly on the passenger computer and partly on a remote computer or entirely on the remote computer or electronic device. In the case of a remote computer, the remote computer may be connected to the passenger computer through any type of network, including a Local Area Network (LAN) or a Wide Area Network (WAN), or the connection may be made to an external computer.

The flowchart and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods and computer program products according to various embodiments of the present application. In this regard, each block in the flowchart or block diagrams may represent a module, segment, or portion of code, which comprises one or more executable instructions for implementing the specified logical function(s). It should also be noted that, in some alternative implementations, the functions noted in the block may occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams and/or flowchart illustration, and combinations of blocks in the block diagrams and/or flowchart illustration, can be implemented by special purpose hardware-based systems which perform the specified functions or acts, or combinations of special purpose hardware and computer instructions.

The units described in the embodiments of the present application may be implemented by software or hardware. The names of the modules and units do not limit the modules and units in some cases.

The above description is only a preferred embodiment of the application and is illustrative of the principles of the technology employed. It will be appreciated by those skilled in the art that the scope of the disclosure herein is not limited to the particular combination of features described above, but also encompasses other arrangements formed by any combination of the above features or their equivalents without departing from the spirit of the disclosure. For example, the above features may be replaced with (but not limited to) features having similar functions disclosed in the present application.

The foregoing embodiments are merely illustrative of the principles and utilities of the present invention and are not intended to limit the invention. Any person skilled in the art can modify or change the above-mentioned embodiments without departing from the spirit and scope of the present invention. Accordingly, it is intended that all equivalent modifications or changes which can be made by those skilled in the art without departing from the spirit and technical spirit of the present invention be covered by the claims of the present invention.

Claims

1. A method for caching data, the method being performed by a front-end server, the method comprising:

2. The method of claim 1, wherein after the accessing the mysql database cache with the lock identification, the method further comprises:

3. The method of claim 2, wherein after sending the access request to a backend server for the backend server to obtain request results over a network and return request results, the method further comprises:

4. The method of claim 1, wherein after the locking the access request with the request identification and with the request identification as a lock identification, the method further comprises:

determining the validity period of the lock;

5. The method of claim 1, wherein the method further comprises:

6. The method as claimed in claim 5, wherein the cdn buffer time is a preset fixed time;

7. The method of claim 1, wherein the method further comprises:

acquiring whether the starting state of the interface is changed to be overdue;

8. An apparatus for caching data, the apparatus being configured in a front-end server, the apparatus comprising:

9. A computer-readable storage medium, on which a computer program is stored, which, when being executed by a processor, carries out a method of caching data according to any one of claims 1 to 7.

10. An electronic device comprising a memory, a processor and a computer program stored on the memory and executable on the processor, characterized in that the processor implements a method of caching data according to any one of claims 1 to 7 when executing the computer program.