CN110636121A - Data acquisition method and system - Google Patents
Data acquisition method and system Download PDFInfo
- Publication number
- CN110636121A CN110636121A CN201910857631.8A CN201910857631A CN110636121A CN 110636121 A CN110636121 A CN 110636121A CN 201910857631 A CN201910857631 A CN 201910857631A CN 110636121 A CN110636121 A CN 110636121A
- Authority
- CN
- China
- Prior art keywords
- data
- version number
- request
- cache
- current
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
- 238000000034 method Methods 0.000 title claims abstract description 56
- 230000008859 change Effects 0.000 claims description 11
- 238000012508 change request Methods 0.000 claims description 6
- 230000008569 process Effects 0.000 description 12
- 238000010586 diagram Methods 0.000 description 11
- 239000002966 varnish Substances 0.000 description 7
- 230000006870 function Effects 0.000 description 6
- 238000012986 modification Methods 0.000 description 6
- 230000004048 modification Effects 0.000 description 6
- 238000004590 computer program Methods 0.000 description 5
- 230000007246 mechanism Effects 0.000 description 5
- 238000003860 storage Methods 0.000 description 4
- 239000011347 resin Substances 0.000 description 3
- 229920005989 resin Polymers 0.000 description 3
- 230000009286 beneficial effect Effects 0.000 description 2
- 238000005516 engineering process Methods 0.000 description 2
- 230000000149 penetrating effect Effects 0.000 description 2
- 230000035515 penetration Effects 0.000 description 2
- 238000009826 distribution Methods 0.000 description 1
- 238000002474 experimental method Methods 0.000 description 1
- 230000003993 interaction Effects 0.000 description 1
- 238000004519 manufacturing process Methods 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 230000002035 prolonged effect Effects 0.000 description 1
Images
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/50—Network services
- H04L67/56—Provisioning of proxy services
- H04L67/568—Storing data temporarily at an intermediate stage, e.g. caching
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/50—Network services
- H04L67/60—Scheduling or organising the servicing of application requests, e.g. requests for application data transmissions using the analysis and optimisation of the required network resources
- H04L67/63—Routing a service request depending on the request content or context
Abstract
The invention discloses a data acquisition method and a data acquisition system, and belongs to the technical field of information. The method comprises the following steps: the front end sends the user data request to a proxy server; the proxy server generates a new data acquisition request according to the user data request and a current data version number of data corresponding to the user data request, wherein the current data version number is generated after the back-end data is updated; and the cache server judges whether the data corresponding to the current data version number exists or not, and if so, directly acquires the corresponding data from the cache server. The data acquisition method and the data acquisition system provided by the invention ensure that the data is updated faster, and the pressure of back-source database searching is reduced, so that the cache hit rate is improved, and meanwhile, the sufficiently long cache time can be ensured.
Description
Technical Field
The present invention relates to the field of information technologies, and in particular, to a data acquisition method and system.
Background
At present, most systems adopt a cache technology in order to reduce the number of times of source searching, so that most data requests can not penetrate back to the source, and data with low change rate can be taken. Specifically, the passive caching is performed for a certain time generally based on redis, varnish, cdn and the like, so that the data pressure of real-time library searching at the back end is reduced.
According to the existing cache strategy, the cache time is fixed, the latest data can be obtained when the time is up, that is, the obtained data is always the same no matter how the request is made in the cache period, the data cannot be changed and updated in real time, and in order to ensure that the data updating speed is high, the cache time is usually sacrificed, so that the latest data can be obtained more quickly, and thus, the pressure on real-time database searching at the rear end is increased, and the cache hit rate is influenced.
Disclosure of Invention
In order to solve the problems of the prior art, embodiments of the present invention provide a data acquisition method and system, according to a paradox relationship that a cache hit rate, an update speed, and a cache time cannot be simultaneously increased, a data version number is updated on real-time updated data by a back end, so that a proxy server only needs to penetrate data after the data version number is updated once, the data update is ensured to be faster, a backsource database searching pressure is reduced, the cache hit rate is increased, and a sufficiently long cache time can be ensured. The technical scheme is as follows:
in one aspect, a data acquisition method is provided, and the method includes:
the front end sends the user data request to a proxy server;
the proxy server generates a new data acquisition request according to the user data request and a current data version number of data corresponding to the user data request, and sends the new data acquisition request to a cache server, wherein the current data version number is generated after the back-end data is updated;
and the cache server judges whether the data corresponding to the current data version number exists or not, and if so, directly acquires the corresponding data from the cache server.
Further, the method further comprises:
and if the judgment result is negative, acquiring the data corresponding to the current data version number from the rear end.
Further, the method further comprises:
the back end sets a default value for the initial data version number of the data;
and when the data is changed, the back end updates the data version number of the data according to a preset numbering rule and sends the updated data version number to the proxy server.
Further, when the data is changed, the back end updates the data version number of the data according to a preset numbering rule, and sends the updated data version number to the proxy server, including:
and the rear end receives a change request of the data change, updates the data version number of the data according to the preset numbering rule of the data subject name and the Arabic numerals which are sequentially numbered, and sends the updated data version number to the proxy server.
Further, the proxy server generates a new data acquisition request according to the user data request and a current data version number of data corresponding to the user data request, including:
splicing the current data version number with the user data request to generate a new data acquisition request;
the method further comprises the following steps:
and the cache server analyzes the data acquisition request to acquire the data corresponding to the current data version number.
Further, the data corresponding to the current data version number stored by the cache server has preset cache time.
On the other hand, the data acquisition system comprises a front end, a proxy server, a cache server and a back end, wherein the front end sends a user data request to the proxy server; the proxy server generates a new data acquisition request according to the user data request and a current data version number of data corresponding to the user data request, and sends the new data acquisition request to a cache server, wherein the current data version number is generated after the back-end data is updated; and the cache server judges whether the data corresponding to the current data version number exists or not, and if so, directly acquires the corresponding data from the cache server.
Further, if the judgment result is negative, the cache server obtains the data corresponding to the current data version number from the back end.
Further, the back end sets a default value for the initial data version number of the data; and when the data is changed, the back end updates the data version number of the data according to a preset numbering rule and sends the updated data version number to the proxy server.
Further, the back end receives a change request of the data change, updates the data version number of the data according to the preset numbering rule of the data subject name and the Arabic numerals which are sequentially numbered, and sends the updated data version number to the proxy server.
Further, the proxy server generates a new data acquisition request according to the user data request and a current data version number of data corresponding to the user data request, including: splicing the current data version number with the user data request to generate a new data acquisition request;
and the cache server analyzes the data acquisition request to obtain the current data version number.
Further, the data corresponding to the current data version number stored by the cache server has preset cache time.
The technical scheme provided by the embodiment of the invention has the following beneficial effects:
1. first, consider based on the following technique: how to change the current state of the prior art, ensure the updating speed, guarantee the caching time, improve the cache hit rate, through adding the version number to the data, and achieve the real-time update version number, guarantee the data version number after changing is different from previous historical version number, when the data of rear end changes, notify the reverse proxy layer in the front of the cache, the data version number in the update request, only need to penetrate the cache once to obtain the newest data, still read the cache subsequently, thus achieve and can cache for a long time, improve the cache hit rate, reduce the pressure of looking over the database back, can upgrade the data in real time;
2. secondly, under the data acquisition mechanism, enough long caching time can be guaranteed, the caching time of the data needing caching is set to the longest period of the service life of the data, and the data almost never penetrates through the back end;
3. and correspondingly, a mechanism for penetrating the cache according to the service data caching time and the updated data version number can help to detect possible infringement and further improve the data acquisition service performance.
Drawings
In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings needed to be used in the description of the embodiments will be briefly introduced below, and it is obvious that the drawings in the following description are only some embodiments of the present invention, and it is obvious for those skilled in the art to obtain other drawings based on these drawings without creative efforts.
Fig. 1 is a flowchart of a data acquisition method provided in embodiment 1 of the present invention;
fig. 2 is a flowchart of a data acquisition method provided in embodiment 2 of the present invention;
FIG. 3 is a schematic diagram of a data acquisition system according to embodiment 3 of the present invention;
fig. 4 is a service diagram of a preferred implementation of the data acquisition method and system according to the embodiment of the present invention.
Detailed Description
In order to make the objects, technical solutions and advantages of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention, and it is obvious that the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention. In the description of the present invention, "a plurality" means two or more unless specifically defined otherwise.
Aiming at the current situation of data acquisition through cache in the prior art, the inventor of the application discovers three important technical indexes: the present invention relates to a method for updating a cache of a computer system, and more particularly to a method for updating a cache of a computer system, which comprises a hit rate, an update rate and a cache time, wherein the hit rate is hooked with a back-end library lookup pressure, and the inventor further finds that the hit rate, the update rate and the cache time cannot be simultaneously increased, because the cache time is inevitably sacrificed and the cache hit rate is reduced when the update rate is ensured, and when the cache time is increased and the cache hit rate is increased, the update rate, which is the update rate, of a user is. Therefore, the inventor proposes a problem how to change the technical status of the cache, so that the cache time can be ensured and the cache hit rate can be improved while the update speed is ensured. Based on the technical consideration, the embodiment of the invention provides a data acquisition method and a data acquisition system, and the data version number of the real-time updated data is updated and set through the back end, so that the data after the data version number is updated is acquired by the proxy server and is stored in the cache only through one time, the data updating is faster, the pressure of source return database searching is reduced, the cache hit rate is improved, and meanwhile, under the data acquisition mechanism, the enough cache time can be ensured. Therefore, the data acquisition method and the data acquisition system can be widely popularized in various application scenes related to data acquisition.
The following describes the data acquisition method and system provided by the embodiments of the present invention in detail with reference to the specific embodiments and the accompanying drawings.
Example 1
Fig. 1 is a flowchart of a data acquisition method provided in embodiment 1 of the present invention, and as shown in fig. 1, the data acquisition method includes the following steps:
101. the front end sends the user data request to the proxy server.
Specifically, a user sends a user data request from the front end, and then the front end sends the user data request to a proxy server of the reverse proxy layer. The front end only refers to a local page of a user, and only needs to initiate a data request, that is, a local cache, and the cache data layer may adopt any possible technical scheme in varnish, Memcache, redis or other prior art, which is not particularly limited by the embodiment of the present invention. The proxy server of the reverse proxy layer may use other web servers besides Nginx, such as Apache, IIS, Nginx, Resin, Tomcat, etc., or use any other proxy server possible in the prior art, and the embodiment of the present invention is not particularly limited thereto.
It should be noted that, the process of step 101 may be implemented in other ways besides the way described in the above steps, and the embodiment of the present invention does not limit the specific way.
102. And the proxy server generates a new data acquisition request according to the user data request and the current data version number of the data corresponding to the user data request, and sends the new data acquisition request to the cache server, wherein the current data version number is generated after the update according to the backend data.
In order to ensure the cache hit rate and not reduce the cache time, and simultaneously achieve the continuous updating of data returned to a user, data version numbers are arranged on all data at a back end, namely corresponding data version numbers are arranged according to the continuously updated version of each piece of data, and the corresponding data version numbers of each piece of data are sent to a proxy server for storage.
As for the data version number setting problem, it can be preferably implemented in the following manner: the data acquisition method provided by the embodiment of the invention further comprises the steps of setting and updating the data version number, and specifically, the back end sets a default value for the initial data version number of the data; and when the data is changed, the back end updates the data version number of the data according to a preset numbering rule and sends the updated data version number to the proxy server. Further preferably, this process is implemented as: and the rear end receives a change request of data change, updates the data version number of the data according to the preset numbering rule of the data subject name combined with the Arabic numerals for numbering in sequence, and sends the updated data version number to the proxy server.
Because the proxy server stores the data version number of the data which is pre-inquired by the user, when the proxy server receives the user data request from the front end, the current data version number of the data is inquired according to the data corresponding to the user data request (namely the pre-acquired data), and then a new data acquisition request with the information of the current data version number is generated according to the acquired current data version number and the user data request. Preferably, the current data version number is spliced with the user data request to generate a new data acquisition request, and then the cache server analyzes the data acquisition request to obtain data corresponding to the current data version number.
It should be noted that, the process of step 102 may be implemented in other ways besides the way described in the above steps, and the embodiment of the present invention does not limit the specific way.
103. And the cache server judges whether the data corresponding to the current data version number exists or not, and if so, the cache server directly acquires the corresponding data.
Specifically, according to the new user data acquisition request, the cache device or the cache server first determines whether data of the current data version number is stored, and if so, directly returns the data queried from the cache device or the cache server. That is to say, if the data of the current latest data version number exists in the cache device or the cache server, the cache device or the cache server does not need to penetrate the source-finding library, and the data to be updated is preferentially obtained from the cache.
In addition, preferably, if the determination result is negative, the data corresponding to the current data version number is acquired from the back end. That is, only in this case, it is necessary to obtain the data corresponding to the current data version number from the backend through one pass, and perform caching. And preferably, the current data version number stored by the cache server has a preset cache time corresponding to the data, that is, a cache valid time is set, and due to the existence of the data version number accompanying the data update, the cache time can be set to be long enough, so that the backend is hardly penetrated in the longest cache time ever. Preferably, the maximum buffer time here is slightly longer than the average or maximum update period, for example, if one service data is updated every hour, the buffer time may be set to 5 minutes per hour, if the memory of the buffer device is sufficient, the buffer time may be set to be longer than the maximum update period, if the memory is limited, the buffer time may be set to be slightly longer than the average period.
It should be noted that, the process of step 103 may be implemented in other ways besides the way described in the above steps, and the embodiment of the present invention does not limit the specific way.
Example 2
Fig. 2 is a flowchart of a data acquisition method provided in embodiment 2 of the present invention, and as shown in fig. 2, the data acquisition method includes the following steps:
201. the front end sends the user data request to the proxy server.
Specifically, a user sends a user data request from the front end, and then the front end sends the user data request to a proxy server of the reverse proxy layer. Here, the front end includes a cache server, that is, a local cache, and the cache data layer may adopt any possible technical solution in varnish, Memcache, redis, or other prior art, which is not particularly limited by the embodiment of the present invention. The proxy server of the reverse proxy layer may use other web servers besides Nginx, such as Apache, IIS, Nginx, Resin, Tomcat, etc., or use any other proxy server possible in the prior art, and the embodiment of the present invention is not particularly limited thereto.
It should be noted that, the process of step 201 may be implemented in other ways besides the way described in the above steps, and the embodiment of the present invention does not limit the specific way.
202. And the proxy server generates a new data acquisition request according to the user data request and the current data version number of the data corresponding to the user data request, wherein the current data version number is generated after the backend data is updated.
In order to ensure the cache hit rate and not reduce the cache time, and simultaneously achieve the continuous updating of data returned to a user, data version numbers are arranged on all data at a back end, namely corresponding data version numbers are arranged according to the continuously updated version of each piece of data, and the corresponding data version numbers of each piece of data are sent to a proxy server for storage.
The data version number setting can be preferably realized in the following way: the data acquisition method provided by the embodiment of the invention further comprises the steps of setting and updating the data version number, and specifically, the back end sets a default value for the initial data version number of the data; and when the data is changed, the back end updates the data version number of the data according to a preset numbering rule and sends the updated data version number to the proxy server. Further preferably, this process is implemented as: and the rear end receives a change request of data change, updates the data version number of the data according to the preset numbering rule of the data subject name combined with the Arabic numerals for numbering in sequence, and sends the updated data version number to the proxy server.
Because the proxy server stores the data version number of the data which is pre-inquired by the user, when the proxy server receives the user data request from the front end, the current data version number of the data is inquired according to the data corresponding to the user data request (namely the pre-acquired data), and then a new data acquisition request with the information of the current data version number is generated according to the acquired current data version number and the user data request. Preferably, the current data version number is spliced with the user data request to generate a new data acquisition request, and then the cache server analyzes the data acquisition request to obtain the current data version number.
It should be noted that, the process of step 202 may be implemented in other ways besides the way described in the above steps, and the embodiment of the present invention does not limit the specific way.
203. And the cache server judges whether the data corresponding to the current data version number exists or not, if so, the corresponding data is directly acquired from the cache server, and if not, the data corresponding to the current data version number is acquired from the rear end.
Specifically, according to a new user data acquisition request, the cache server first determines whether data of the current data version number is stored, and if so, the data queried from the cache server is directly returned. That is, if the data of the current latest data version number exists in the cache server, it is not necessary to penetrate back to the source lookup database, and the currently updated data is preferentially acquired from the cache. And only if the judgment result is negative, the data corresponding to the current data version number needs to be acquired from the back end through one-time penetration, and caching is carried out at the same time. And preferably, the current data version number stored by the cache server has a preset cache time corresponding to the data, that is, a cache valid time is set, and due to the existence of the data version number accompanying the data update, the cache time can be set to be long enough, so that the backend is hardly penetrated in the longest cache time ever.
It should be noted that, the process of step 203 may be implemented in other ways besides the way described in the above step, and the embodiment of the present invention does not limit the specific way.
Example 3
Fig. 3 is a schematic diagram illustrating a data acquisition system according to embodiment 3 of the present invention, and as shown in fig. 3, the data acquisition system according to the embodiment of the present invention includes a front end 31, a proxy server 32, a cache server 33, and a back end 34, where the front end 31 sends a user data request to the proxy server 32; the proxy server 32 preferably splices the current data version number with the user data request according to the user data request and the current data version number of the data corresponding to the user data request to generate a new data acquisition request, wherein the current data version number is generated after the data is updated according to the back end 33; preferably, the cache server 33 parses the data obtaining request to obtain the current data version number, determines whether data corresponding to the current data version number exists, and directly obtains corresponding data from the cache server 33 if the determination result is yes.
In addition, if the determination result is negative, the data corresponding to the current data version number is preferably acquired from the backend 34 and cached at the same time. Preferably, the cache server 33 stores the current data version number corresponding to the data with a preset cache time.
In addition, the backend 34 preferably sets a default value for the initial data version number of the data; when the data is changed, the backend 34 updates the data version number of the data according to the preset numbering rule, and sends the updated data version number to the proxy server 32. Further preferably, the back end 34 receives a change request for changing data, updates the data version number of the data according to a preset numbering rule that the data subject name is sequentially numbered in combination with the arabic numerals, and sends the updated data version number to the proxy server 32.
Fig. 4 shows a business process of a preferred embodiment of the data acquisition method and system provided by the embodiment of the present invention, specifically demonstrating an operation process and interaction between the data acquisition business and the data acquisition business, however, any other possible implementation may be adopted without departing from the inventive concept, and the embodiment of the present invention is not particularly limited thereto.
The following description further describes the operation flow of the data acquisition method and system provided in the embodiments of the present invention, with reference to a specific application example.
In this example, a nginx + varnish + DATA back-end implementation manner is adopted, a DATA request DATA1-URL link first passes through a nginx cluster, nginx obtains a VERSION number DATA1-VERSION ═ 0 (the VERSION number N takes a value as a default value, such as 0, for the first time) of the DATA in a local memory (front-end memory), and the VERSION numbers are spliced into DATA1-URL-N to request varnish to obtain the DATA.
And (3) obtaining cache DATA by varnish according to URL dimension DATA1-URL-N, directly returning a result if the cache exists, continuously requesting the DATA back end to inquire latest DATA in real time if the cache does not exist, caching the DATA for a long time by the URL with the DATA version number after the DATA is obtained, and returning the latest DATA to nginx.
When the application back-end DATA1 is changed, nginx is notified to change the VERSION number DATA1-VERSION of the DATA, the change mode can be increment, decrement and the like, for example, DATA1-VERSION is DATA1-VERSION +1, as long as the changed DATA VERSION number is different from the previous historical VERSION number.
Similarly, other DATA may be set, such as the DATA2 VERSION number DATA2-VERSION ═ 0, and changed DATA2-VERSION ═ 1, stored in nginx memory.
In this example, the reverse proxy layer may use other web servers besides Nginx, such as Apache, IIS, Nginx, Resin, Tomcat, etc.; the data layer varnish may also be replaced by other related caching schemes, such as Memcache, redis, etc.
It should be noted that: in the data acquisition system provided in the foregoing embodiment, when triggering a data acquisition service, only the division of each functional module is described as an example, and in practical applications, the function distribution may be completed by different functional modules according to needs, that is, the internal structure of the system is divided into different functional modules to complete all or part of the functions described above. In addition, the data acquisition system and the data acquisition method provided by the above embodiments belong to the same concept, and specific implementation processes thereof are described in the method embodiments for details, which are not described herein again.
All the above-mentioned optional technical solutions can be combined arbitrarily to form the optional embodiments of the present invention, and are not described herein again.
In summary, the data acquisition method and system provided in the embodiments of the present invention have the following beneficial effects, compared with the prior art:
1. first, consider based on the following technique: how to change the current state of the prior art, ensure the updating speed, guarantee the caching time, improve the cache hit rate, through adding the version number to the data, and achieve the real-time update version number, guarantee the data version number after changing is different from previous historical version number, when the data of rear end changes, notify the reverse proxy layer in the front of the cache, the data version number in the update request, only need to penetrate the cache once to obtain the newest data, still read the cache subsequently, thus achieve and can cache for a long time, improve the cache hit rate, reduce the pressure of looking over the database back, can upgrade the data in real time;
2. secondly, under the data acquisition mechanism, enough long caching time can be guaranteed, the caching time of the data needing caching is set to the longest period of the service life of the data, and the data almost never penetrates through the back end;
3. and correspondingly, a mechanism for penetrating the cache according to the service data caching time and the updated data version number can help to detect possible infringement and further improve the data acquisition service performance.
Moreover, practice proves that compared with the prior art, the data acquisition scheme provided by the embodiment of the invention has different improvements in data caching time, data caching hit rate and data accuracy index, and the details are shown in the following service experiment and data in table 1.
Assuming that a service scenario allows data delay of 5 minutes at most, the following data are observed according to the actual request amount:
| Prior Art | Scheme of embodiment of the invention | |
| Data buffer time | 5 minutes | Maximum data life |
| Data cache hit rate | Less than 50 percent | 99％ |
| Accuracy of data | Cannot be updated in real time | 100％ |
TABLE 1
By adopting the data acquisition scheme provided by the embodiment of the invention, the service data caching time can be infinitely prolonged, the data penetration rate is reduced, the hit rate is improved, and the data can be updated quickly in real time.
It will be understood by those skilled in the art that all or part of the steps for implementing the above embodiments may be implemented by hardware, or may be implemented by a program instructing relevant hardware, where the program may be stored in a computer-readable storage medium, and the above-mentioned storage medium may be a read-only memory, a magnetic disk or an optical disk, etc.
Embodiments of the present application are described with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems) and computer program products according to embodiments of the application. It will be understood that each flow and/or block of the flow diagrams and/or block diagrams, and combinations of flows and/or blocks in the flow diagrams and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general purpose computer, special purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instruction means which implement the function specified in the flowchart flow or flows and/or block diagram block or blocks.
These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
While preferred embodiments of the present application have been described, additional variations and modifications in those embodiments may occur to those skilled in the art once they learn of the basic inventive concepts. Therefore, it is intended that the appended claims be interpreted as including the preferred embodiment and all changes and modifications that fall within the true scope of the embodiments of the present application.
It will be apparent to those skilled in the art that various changes and modifications may be made in the present invention without departing from the spirit and scope of the invention. Thus, if such modifications and variations of the present invention fall within the scope of the claims of the present invention and their equivalents, the present invention is also intended to include such modifications and variations.
The above description is only for the purpose of illustrating the preferred embodiments of the present invention and is not to be construed as limiting the invention, and any modifications, equivalents, improvements and the like that fall within the spirit and principle of the present invention are intended to be included therein.
Those not described in detail in this specification are within the skill of the art.
Claims (10)
1. A method for data acquisition, the method comprising:
the front end sends the user data request to a proxy server;
the proxy server generates a new data acquisition request according to the user data request and a current data version number of data corresponding to the user data request, and sends the new data acquisition request to a cache server, wherein the current data version number is generated after the update of backend data;
and the cache server judges whether the data corresponding to the current data version number exists or not, and if so, directly acquires the corresponding data from the cache server.
2. The method of claim 1, further comprising:
and if the judgment result is negative, acquiring the data corresponding to the current data version number from the rear end.
3. The method of claim 1, further comprising:
the back end sets a default value for the initial data version number of the data;
and when the data is changed, the back end updates the data version number of the data according to a preset numbering rule and sends the updated data version number to the proxy server.
4. The method of claim 3, wherein when the data is changed, the backend updates a data version number of the data according to a preset numbering rule, and sends the updated data version number to the proxy server, and the method comprises:
and the rear end receives a change request of the data change, updates the data version number of the data according to the preset numbering rule of the data subject name and the Arabic numerals which are sequentially numbered, and sends the updated data version number to the proxy server.
5. The method of claim 1,
the proxy server generates a new data acquisition request according to the user data request and the current data version number of the data corresponding to the user data request, and the method comprises the following steps:
splicing the current data version number with the user data request to generate a new data acquisition request;
the method further comprises the following steps:
and the cache server analyzes the data acquisition request to acquire the data corresponding to the current data version number.
6. The method of claim 1, wherein the current data version number stored by the cache server corresponds to data having a preset cache time.
7. A data acquisition system is characterized by comprising a front end, a proxy server, a cache server and a back end, wherein the front end sends a user data request to the proxy server; the proxy server generates a new data acquisition request according to the user data request and a current data version number of data corresponding to the user data request, and sends the new data acquisition request to a cache server, wherein the current data version number is generated after the back-end data is updated; and the cache server judges whether the data corresponding to the current data version number exists or not, and if so, directly acquires the corresponding data from the cache server.
8. The system according to claim 7, wherein if the determination result is negative, the data corresponding to the current data version number is obtained from the backend.
9. The system of claim 7, wherein the backend sets a default value for an initial data version number of the data; and when the data is changed, the back end updates the data version number of the data according to a preset numbering rule and sends the updated data version number to the proxy server.
10. The system of claim 7, wherein the current data version number stored by the cache server corresponds to data having a preset cache time.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910857631.8A CN110636121B (en) | 2019-09-09 | 2019-09-09 | Data acquisition method and system |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910857631.8A CN110636121B (en) | 2019-09-09 | 2019-09-09 | Data acquisition method and system |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN110636121A true CN110636121A (en) | 2019-12-31 |
| CN110636121B CN110636121B (en) | 2022-07-05 |
Family
ID=68972547
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201910857631.8A Active CN110636121B (en) | 2019-09-09 | 2019-09-09 | Data acquisition method and system |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN110636121B (en) |
Citations (9)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US6535911B1 (en) * | 1999-08-06 | 2003-03-18 | International Business Machines Corporation | Viewing an information set originated from a distribution media and updating using a remote server |
| US20040268344A1 (en) * | 2001-03-08 | 2004-12-30 | Sridhar Obilisetty | Centrally managed and distributed applications |
| CN101795392A (en) * | 2009-12-25 | 2010-08-04 | 深圳市同洲电子股份有限公司 | Application data-downloading method, digital television receiving terminal and system |
| CN102098344A (en) * | 2011-02-21 | 2011-06-15 | 中国科学院计算技术研究所 | Method and device for synchronizing editions during cache management and cache management system |
| CN102780773A (en) * | 2012-07-16 | 2012-11-14 | 西安电子科技大学 | Method for keeping cache consistency in network using content as center |
| CN102946442A (en) * | 2012-12-06 | 2013-02-27 | 网宿科技股份有限公司 | Method and system for updating and releasing file based on smart refreshing |
| CN104537013A (en) * | 2014-12-18 | 2015-04-22 | 北京京东尚科信息技术有限公司 | Data processing method and device |
| WO2017020743A1 (en) * | 2015-08-06 | 2017-02-09 | 阿里巴巴集团控股有限公司 | Method and device for sharing cache data |
| CN106489145A (en) * | 2015-12-28 | 2017-03-08 | 华为技术有限公司 | The access method of web site, device and Web site system |
-
2019
- 2019-09-09 CN CN201910857631.8A patent/CN110636121B/en active Active
Patent Citations (9)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US6535911B1 (en) * | 1999-08-06 | 2003-03-18 | International Business Machines Corporation | Viewing an information set originated from a distribution media and updating using a remote server |
| US20040268344A1 (en) * | 2001-03-08 | 2004-12-30 | Sridhar Obilisetty | Centrally managed and distributed applications |
| CN101795392A (en) * | 2009-12-25 | 2010-08-04 | 深圳市同洲电子股份有限公司 | Application data-downloading method, digital television receiving terminal and system |
| CN102098344A (en) * | 2011-02-21 | 2011-06-15 | 中国科学院计算技术研究所 | Method and device for synchronizing editions during cache management and cache management system |
| CN102780773A (en) * | 2012-07-16 | 2012-11-14 | 西安电子科技大学 | Method for keeping cache consistency in network using content as center |
| CN102946442A (en) * | 2012-12-06 | 2013-02-27 | 网宿科技股份有限公司 | Method and system for updating and releasing file based on smart refreshing |
| CN104537013A (en) * | 2014-12-18 | 2015-04-22 | 北京京东尚科信息技术有限公司 | Data processing method and device |
| WO2017020743A1 (en) * | 2015-08-06 | 2017-02-09 | 阿里巴巴集团控股有限公司 | Method and device for sharing cache data |
| CN106489145A (en) * | 2015-12-28 | 2017-03-08 | 华为技术有限公司 | The access method of web site, device and Web site system |
Non-Patent Citations (2)
| Title |
|---|
| ""23502_CR0668_5GS_Ph1_(Rel-15)_S2-187952_TS23.502_Correction on the service name for UE Subscriber Data Update Notification v1"", 《3GPP TSG_SA\TSG_SA》 * |
| NOKIA等: "S2-1907401 "UDR service for mapping IMS Public Identity to HSS Group ID for HSS selection"", 《3GPP TSG_SA\WG2_ARCH》 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN110636121B (en) | 2022-07-05 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US9898477B1 (en) | Writing to a site cache in a distributed file system | |
| CN107197359B (en) | Video file caching method and device | |
| CN104573128A (en) | Business data processing method, a business data processing device and server | |
| CN111414389B (en) | Data processing method and device, electronic equipment and storage medium | |
| US9864684B2 (en) | Performing efficient cache invalidation | |
| CN111221469B (en) | Method, device and system for synchronizing cache data | |
| KR102098415B1 (en) | Cache management | |
| CN107809684B (en) | Video segment generation method and device, cache server | |
| CN103136338A (en) | Metadata distribution method and device based on directory partition | |
| CN106970974B (en) | Version-based APP low-flow data updating system and method | |
| CN111506604A (en) | Method, apparatus and computer program product for accessing data | |
| CN103248666A (en) | System, method and device for offline resource download | |
| US20230239341A1 (en) | Proactive conditioned prefetching and origin flooding mitigation for content delivery | |
| CN105677754A (en) | Method, apparatus and system for acquiring subitem metadata in file system | |
| CN103036948A (en) | Network file processing method and execution node and software as a service (SaaS) platform | |
| US20150032802A1 (en) | Shared data de-duplication method and system | |
| CN110636121B (en) | Data acquisition method and system | |
| CN102298527B (en) | A kind of method of CPE parallel upgrades version and realize system | |
| CN106022161B (en) | A kind of data processing method and device | |
| CN111291083B (en) | Webpage source code data processing method and device and computer equipment | |
| CN111182053A (en) | Method and device for downloading file, computer storage medium and terminal | |
| EP3220607B1 (en) | Service data processing methods and devices | |
| CN104636384A (en) | Document processing method and device | |
| CN108415976B (en) | Method for rapidly publishing website file based on CMS system | |
| CN105812894A (en) | Video file processing method and device based on intelligent terminal |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant | ||
| CP03 | Change of name, title or address |
Address after: No.1-1 Suning Avenue, Xuzhuang Software Park, Xuanwu District, Nanjing, Jiangsu Province, 210000 Patentee after: Jiangsu Suning cloud computing Co.,Ltd. Country or region after: China Address before: No.1-1 Suning Avenue, Xuzhuang Software Park, Xuanwu District, Nanjing, Jiangsu Province, 210000 Patentee before: Suning Cloud Computing Co.,Ltd. Country or region before: China |
|
| CP03 | Change of name, title or address | ||
| TR01 | Transfer of patent right |
Effective date of registration: 20240315 Address after: 210000, 1-5 story, Jinshan building, 8 Shanxi Road, Nanjing, Jiangsu. Patentee after: SUNING.COM Co.,Ltd. Country or region after: China Address before: No.1-1 Suning Avenue, Xuzhuang Software Park, Xuanwu District, Nanjing, Jiangsu Province, 210000 Patentee before: Jiangsu Suning cloud computing Co.,Ltd. Country or region before: China |
|
| TR01 | Transfer of patent right |