CN115174546B - Data list caching method and device - Google Patents
Data list caching method and device Download PDFInfo
- Publication number
- CN115174546B CN115174546B CN202211082438.XA CN202211082438A CN115174546B CN 115174546 B CN115174546 B CN 115174546B CN 202211082438 A CN202211082438 A CN 202211082438A CN 115174546 B CN115174546 B CN 115174546B
- Authority
- CN
- China
- Prior art keywords
- data list
- static file
- list
- active data
- active
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/01—Protocols
- H04L67/02—Protocols based on web technology, e.g. hypertext transfer protocol [HTTP]
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/22—Indexing; Data structures therefor; Storage structures
- G06F16/2282—Tablespace storage structures; Management thereof
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/245—Query processing
- G06F16/2455—Query execution
- G06F16/24552—Database cache management
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/90—Details of database functions independent of the retrieved data types
- G06F16/95—Retrieval from the web
- G06F16/958—Organisation or management of web site content, e.g. publishing, maintaining pages or automatic linking
- G06F16/972—Access to data in other repository systems, e.g. legacy data or dynamic Web page generation
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/90—Details of database functions independent of the retrieved data types
- G06F16/95—Retrieval from the web
- G06F16/958—Organisation or management of web site content, e.g. publishing, maintaining pages or automatic linking
- G06F16/986—Document structures and storage, e.g. HTML extensions
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Databases & Information Systems (AREA)
- General Physics & Mathematics (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- Data Mining & Analysis (AREA)
- Software Systems (AREA)
- Computational Linguistics (AREA)
- Computer Networks & Wireless Communication (AREA)
- Signal Processing (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
- Information Transfer Between Computers (AREA)
Abstract
The invention discloses a caching method and a caching device for a data list, which are used for acquiring a data list reading request sent by a Web client, acquiring an active data list from a Web cache service or a database based on the request, acquiring a static file identifier from the Web cache service, returning the active data list and the static file identifier to the Web client, returning a data packet carrying a static file to the Web client after receiving the static file request sent by the Web client, combining data objects in the active data list and the static data list in the static file into a complete data list after the data objects in the active data list and the static data list in the static file are deduplicated by the Web client, and setting the caching configuration of HTTP (hyper text transport protocol) cache at the head of the data packet. The Web cache service and the database belong to resources in the station, the HTTP cache belongs to resources outside the station, and the cost of the source station production hardware is not counted.
Description
Technical Field
The present invention relates to the field of data storage technologies, and in particular, to a method and an apparatus for caching a data list.
Background
To support the amount of concurrency of user access, web services interfaces typically have a cache built into the interface. These caches typically employ common Web caching services such as redis, memcached, and the like. In order to support concurrent access, the Web cache service places the data list as a cached object in a memory of a device where the Web cache service is located.
However, the more data that is cached in memory, the greater the consumption. And the hardware cost of the memory is higher compared to other storage hardware (such as a hard disk). Therefore, caching the data list in memory requires a certain hardware cost. When the data list is too long, the required cache memory consumption also increases. Meanwhile, the Web cache service is usually not deployed on the same server as the Web service, so that too long a data list also causes the transmission data packets of the Web cache service and the Web service before to become larger and larger, thereby forming an intranet traffic bottleneck.
In summary, how to reduce the usage amount of the Web cache service and achieve normal issuing of the large data list while reducing the hardware cost becomes a technical problem that needs to be solved urgently by the technical personnel in the field.
Disclosure of Invention
In view of this, the present invention discloses a method and an apparatus for caching a data list, so as to ensure normal issuing of a large data list while reducing hardware cost.
A caching method of a data list is applied to a Web server, and comprises the following steps:
acquiring a data list reading request sent by a Web client, wherein the data list reading request carries a data list ID;
acquiring a corresponding active data list from a Web cache service or a database based on the data list reading request, wherein the active data list is a data list consisting of latest incremental data;
acquiring a corresponding static file identifier from the Web cache service based on the data list reading request;
returning the active data list and the static file identification to the Web client;
acquiring a static file request sent by the Web client, wherein the static file request is generated when the Web client determines that a static file exists based on the fact that the value of the static file identifier is greater than a first numerical value, and the static file request carries the data list ID and the static file identifier;
and returning a data packet carrying the static file to the Web client, and combining the data objects in the active data list and the data objects in the static data list in the static file into a complete data list by the Web client after de-duplication based on the data object ID, wherein the head of the data packet is provided with the cache configuration of the HTTP cache.
Optionally, the obtaining a corresponding active data list from a Web cache service or a database based on the data list ID includes:
determining an active data list key value according to the data list ID;
judging whether the active data list can be inquired in the Web cache service or not based on the active data list key value;
and if so, directly acquiring the active data list from the Web cache service.
Optionally, the method further includes:
if not, sending an active data list query request to a database based on the data list ID;
and acquiring an active data list returned by the database, and writing the active data list into the Web cache service based on the key value of the active data list, wherein data objects in the active data list are sorted in the sequence of IDs from large to small, and the maximum length is the preset length limit value.
Optionally, after the obtaining the corresponding active data list from the Web cache service or the database based on the data list reading request, the method further includes:
and when the length of the active data list is smaller than the preset length limit value, returning the active data list and the static file identifier with the value as a first numerical value to the Web client, and enabling the Web client to determine to only use the active data list based on the first numerical value, wherein the first numerical value indicates that the static file does not exist.
Optionally, after the obtaining the corresponding active data list from the Web cache service or the database based on the data list reading request, the method further includes:
and when the length of the active data list is equal to the preset length limit value, querying a Web cache service based on a static file key value to obtain the static file identifier, wherein the static file key value is determined based on the data list ID.
Optionally, after the obtaining the corresponding active data list from the Web cache service or the database based on the data list reading request, the method further includes:
when the length of the active data list is equal to a preset length limit value, checking a static file identifier, and when the cache of the static file identifier does not exist, submitting the data list ID to a task pool to generate a corresponding static file;
setting the value of the static file identifier as a second numerical value, wherein the second numerical value is used for representing that a static file is being generated;
and returning the active data list and the static file identifier to the Web client, and determining whether to use the static file by the Web client based on the value of the static file identifier.
Optionally, the method further includes:
when the length of the active data list is equal to the preset length limit value and the cache of the static file identifier exists, judging whether the ID of the minimum data object in the active data list is smaller than the value of the static file identifier or not;
if so, determining that the active data list is overlapped with the data in the static file;
if not, determining that the active data list is not completely overlapped with the data in the static file, and submitting the data list ID to the task pool to regenerate the static file.
Optionally, the submitting the data list ID to a task pool to generate a corresponding static file includes:
sending a full data list reading request to a database based on the data list ID, and creating a static file with empty content for the full data list;
acquiring the full data list returned by the database, and writing the full data list into the static file;
writing the maximum data object ID in the full data list into Web cache service as the value of the static file identifier;
and taking the values of the data list ID and the static file identification as the unique file name identification of the created static file.
A caching device of a data list is applied to a Web server, and comprises:
the system comprises a list request acquisition unit, a data list reading unit and a data processing unit, wherein the list request acquisition unit is used for acquiring a data list reading request sent by a Web client, and the data list reading request carries a data list ID;
the list acquisition unit is used for acquiring a corresponding active data list from a Web cache service or a database based on the data list reading request, wherein the active data list is a data list consisting of latest incremental data;
a first static file identifier obtaining unit, configured to obtain a corresponding static file identifier from the Web cache service based on the data list reading request;
the first returning unit is used for returning the active data list and the static file identifier to the Web client;
a file request obtaining unit, configured to obtain a static file request sent by the Web client, where the static file request is generated when the Web client determines that a static file exists based on that a value of the static file identifier is greater than a first numerical value, and the static file request carries the data list ID and the static file identifier;
and the second returning unit is used for returning a data packet carrying the static file to the Web client, and the Web client combines the data object in the active data list and the data object in the static data list in the static file into a complete data list after de-duplication based on the data object ID, wherein the head of the data packet is provided with cache configuration of HTTP cache.
Optionally, the list acquiring unit includes:
a key value determining subunit, configured to determine an active data list key value according to the data list ID;
a judging subunit, configured to judge, based on the active data list key value, whether the active data list can be queried in the Web cache service;
a first obtaining subunit, configured to, in a case where the determining subunit determines that the data is the active data list, directly obtain the active data list from the Web cache service.
Optionally, the list obtaining unit further includes:
a sending subunit, configured to send an active data list query request to a database based on the data list ID if the determining subunit determines that the data list ID is negative;
and the second acquiring subunit is configured to acquire an active data list returned by the database, and write the active data list into the Web cache service based on the active data list key value, where data objects in the active data list are sorted in an order from IDs to IDs, and the maximum length is the preset length limit.
Optionally, the method further includes:
and a third returning unit, configured to, after the list obtaining unit obtains a corresponding active data list from a Web cache service or a database based on the data list reading request, return the active data list and the static file identifier having a first value to the Web client when the length of the active data list is smaller than the preset length limit value, so that the Web client determines to use only the active data list based on the first value, where the first value indicates that the static file does not exist.
Optionally, the method further includes:
a first static file identifier obtaining unit, configured to, after the list obtaining unit obtains a corresponding active data list from a Web cache service or a database based on the data list read request, query the Web cache service based on a static file key value to obtain the static file identifier when the length of the active data list is equal to the preset length limit, where the static file key value is determined based on the data list ID.
Optionally, the method further includes:
the generating unit is used for checking a static file identifier when the length of the active data list is equal to a preset length limit value after the list acquiring unit acquires the corresponding active data list from a Web cache service or a database based on the data list reading request, and submitting the data list ID to a task pool to generate a corresponding static file when the cache of the static file identifier does not exist;
the setting unit is used for setting the value of the static file identifier as a second numerical value, and the second numerical value is used for representing that the static file is being generated;
and the fourth returning unit is used for returning the active data list and the static file identifier to the Web client, and the Web client determines whether to use the static file or not based on the value of the static file identifier.
Optionally, the method further includes:
a judging unit, configured to, when the length of the active data list is equal to the preset length limit and a cache of the static file identifier exists, judge whether an ID of a minimum data object in the active data list is smaller than a value of the static file identifier;
the coincidence determining unit is used for determining that the active data list and the data in the static file are coincided under the condition that the judging unit judges that the active data list and the data in the static file are coincided;
and the incomplete superposition determining unit is used for determining that the data in the active data list and the data in the static file are not completely superposed under the condition that the judging unit judges that the data in the active data list and the data in the static file are not completely superposed, and submitting the data list ID to the task pool to regenerate the static file.
Optionally, the generating unit is specifically configured to:
sending a full data list reading request to a database based on the data list ID, and creating a static file with empty content for the full data list;
acquiring the full data list returned by the database, and writing the full data list into the static file;
writing the maximum data object ID in the full data list into Web cache service as the value of the static file identifier;
and taking the values of the data list ID and the static file identification as the unique file name identification of the created static file.
The technical scheme includes that a data list reading request sent by a Web client is obtained, a corresponding active data list is obtained from a Web cache service or a database based on the data list reading request, a corresponding static file identifier is obtained from the Web cache service based on the data list reading request, the active data list and the static file identifier are returned to the Web client, a data packet carrying a static file is returned to the Web client after the static file request sent by the Web client is received, a data object in the active data list and a data object in the static data list in the static file are combined into a complete data list after data object ID is removed by the Web client, and the head of the data packet is provided with HTTP cache configuration. When the data list is cached, the data list is divided into two parts, one part is placed in the active data list and stored in the Web cache service and the database, and the other part is placed in the static data list and stored in the static file. Since the Web cache service and the database belong to the resources in the station, and the HTTP cache belongs to the resources outside the station, the cost of the hardware produced by the source station is not counted. Therefore, the invention reasonably splits the data list, ensures the issuing of the large data list, reduces the usage amount of Web cache service to the maximum extent and reduces the hardware cost of the memory.
Drawings
In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly described below, it is obvious that the drawings in the following description are only embodiments of the present invention, and for those skilled in the art, other drawings can be obtained according to the disclosed drawings without creative efforts.
Fig. 1 is a flowchart of a data list caching method according to an embodiment of the present invention;
fig. 2 is a flowchart of a method for acquiring an active data list according to an embodiment of the present invention;
FIG. 3 is a flowchart of a method for generating a corresponding static file by submitting a data list ID to a task pool according to an embodiment of the present invention;
FIG. 4 is a schematic diagram illustrating that the content in the static file and the content in the active data list completely overlap according to an embodiment of the disclosure;
FIG. 5 is a schematic diagram illustrating that the content in the static file and the content in the active data list are not completely overlapped according to an embodiment of the present invention;
FIG. 6 is a diagram illustrating a comparison of contents in an active data list, contents in an old static file, and contents in a new static file according to an embodiment of the present invention;
fig. 7 is a schematic structural diagram of a data list caching apparatus according to an embodiment of the present invention.
Detailed Description
The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.
The embodiment of the invention discloses a caching method and a caching device for a data list, wherein a data list reading request sent by a Web client is obtained, a corresponding active data list is obtained from a Web cache service or a database based on the data list reading request, a corresponding static file identifier is obtained from the Web cache service based on the data list reading request, the active data list and the static file identifier are returned to the Web client, a data packet carrying a static file is returned to the Web client after the static file request sent by the Web client is received, a data object in the active data list and a data object in the static data list in the static file are combined into a complete data list after de-duplication based on a data object ID by the Web client, and the head of the data packet is provided with caching configuration of HTTP caching. When the data list is cached, the data list is divided into two parts, one part is placed in the active data list and stored in the Web cache service and the database, and the other part is placed in the static data list and stored in the static file. Since the Web cache service and the database belong to the resources in the station, and the HTTP cache belongs to the resources outside the station, the cost of the hardware produced by the source station is not counted. Therefore, the invention reasonably splits the data list, ensures the issuing of the large data list, reduces the usage amount of Web cache service to the maximum extent and reduces the hardware cost of the memory.
Referring to fig. 1, a flowchart of a caching method for a data list according to an embodiment of the present invention is applied to a Web server, and the caching method includes:
step S101, acquiring a data list reading request sent by a Web client;
the data list reading request carries a data list ID.
The data list ID is a unique identifier of the data list, and if the data list is a video list of an album, the ID is the album ID; if the data list is a bullet screen list of a certain video, the ID is a video ID; if the data list is a bill list of a certain user, the ID is the user ID.
In this embodiment, the data list ID is used to identify the data object, and in practical application, other fields may also be used to identify the data object, for example, the creation time is only required to ensure that the field identifying the data object satisfies: the field value of the newly added data is certainly larger than the field values corresponding to all the existing data in the data list, and the field values are unique and not repeated.
In practical applications, the Web client may send a data list reading request through a browser, a mobile phone APP or an applet, and the like.
Step S102, acquiring a corresponding active data list from a Web cache service or a database based on the data list reading request;
wherein the active data list is a data list composed of latest incremental data.
Step S103, acquiring a corresponding static file identifier from the Web cache service based on the data list reading request;
when the static file identification is equal to the first numerical value, the static file does not exist, and at the moment, the Web client only uses the active data list;
when the static file identification is equal to a second numerical value, the static file is generated, and at the moment, the Web client only uses the active data list;
and when the static file identification is larger than the first numerical value, determining that the static file exists, and at the moment, using the active data list and the static data list in the static file by the Web client side at the same time.
It should be noted that the first value and the second value are different in value, for example, the first value is 0, and the second value is-1, which is determined according to actual needs, and the present invention is not limited herein.
Step S104, returning the active data list and the static file identification to the Web client;
step S105, acquiring a static file request sent by the Web client;
the static file request is generated when the Web client determines that a static file exists based on the fact that the value of the static file identifier is larger than a first numerical value, and the static file request carries the data list ID and the static file identifier.
Specifically, when the Web client determines that the static file identifier is larger than a first value, the Web client determines that the static file exists, and at this time, the Web client sends a static file request to the Web server.
And step S106, returning a data packet carrying the static file to the Web client, and combining the data objects in the active data list and the data objects in the static data list in the static file into a complete data list after de-duplication by the Web client based on the data object ID.
And the header of the data packet sets the cache configuration of the HTTP cache.
Specifically, after receiving a static file request sent by a Web client, the Web server returns a data packet to the Web client, where the data packet includes a header and a static file, and the header is provided with a cache configuration of an HTTP cache, for example, a cache duration, so that after sending a static file request with other same parameters, the Web client can directly obtain the corresponding static file from an intermediate network node without being at a request source station.
After receiving the data packet, the Web client searches a static data list from a static file of the data packet, and combines a data object in the static data list and a data object in the previously received active data list into a complete data list after de-duplication based on the data object ID.
To sum up, the invention discloses a data list caching method, which comprises the steps of obtaining a data list reading request sent by a Web client, obtaining a corresponding active data list from a Web cache service or a database based on the data list reading request, obtaining a corresponding static file identifier from the Web cache service based on the data list reading request, returning the active data list and the static file identifier to the Web client, returning a data packet carrying a static file to the Web client after receiving the static file request sent by the Web client, combining a data object in the active data list and a data object in the static data list in the static file into a complete data list after de-duplicating the data object based on a data object ID by the Web client, and setting the caching configuration of HTTP caching at the head of the data packet. When the data list is cached, the data list is divided into two parts, one part is placed in the active data list and stored in the Web cache service and the database, and the other part is placed in the static data list and stored in the static file. Since the Web cache service and the database belong to the resources in the station, and the HTTP cache belongs to the resources outside the station, the cost of the hardware produced by the source station is not counted. Therefore, the invention reasonably splits the data list, ensures the issuing of the large data list, reduces the usage amount of Web cache service to the maximum extent and reduces the hardware cost of the memory.
It should be noted that, the Web server usually sets the cache configuration of the HTTP cache of the header of the returned data packet as: the CACHE field value NO-CACHE. Therefore, during the data transmission process, the intermediate network nodes do not cache the request information into the storage medium of the intermediate network nodes, and all the requests are finally requested to the source station. And when new data is added into the data list and is successfully stored, the Web server deletes the active data list cache associated with the ID of the data list. And after receiving a next data list reading request sent by the Web client, the Web server continues to execute the step S102 to the step S106 so as to ensure that the Web client can acquire the latest incremental data. If no incremental data is heated to the data list, the cache of the active data list exists all the time, and the Web server reads the request according to the data list.
To further optimize the above embodiment, referring to fig. 2, a flowchart of a method for acquiring an activity data list disclosed in the embodiment of the present invention, that is, step S102 may specifically include:
step S201, determining an active data list key value according to a data list ID;
after receiving a data list reading request sent by a Web client, a Web server queries a cache of an active data list through key = active _ list + data list ID, wherein the key represents an active data list key value, and the active _ list represents a uniform character string prefix.
Step S202, judging whether an active data list can be inquired in the Web cache service or not based on the key value of the active data list, and if so, executing step S203;
step S203, directly acquiring the active data list from the Web cache service.
To further optimize the above embodiment, when the step S202 determines no, the method further includes:
step S204, sending an active data list inquiry request to a database based on the data list ID;
when the Web cache service does not have cache of the active data list, the Web server sends an active data list query request to the database according to the ID of the data list, the database returns to the active data list, the data objects in the active data list are sorted from big to small according to the ID, and the maximum length is the preset length limit value n. The Web server then writes the active data list to the Web caching service for subsequent use.
When writing an active data list to the Web caching service, key = active _ list + data list ID, value = active data list.
Step S205, obtaining an active data list returned by the database, and writing the active data list into the Web cache service based on the active data list key value.
And sorting the data objects in the active data list according to the sequence of IDs from large to small, wherein the maximum length is the preset length limit value n, and the ID is an attribute of the data object.
To further optimize the above embodiment, after step S102, the method may further include:
and when the length of the active data list is smaller than a preset length limit value n, returning the active data list and the static file identifier with the value of a first numerical value to the Web client side, and enabling the Web client side to determine that only the active data list is used based on the first numerical value.
The first value indicates that the static file does not exist, and the value of the first value may be 0, which is determined according to actual needs.
Specifically, when the Web server determines that the length of the active data list is smaller than a preset length limit value n, the Web server returns the active data list and a static file identifier (equal to a first numerical value) to the Web client, and when the Web client determines that the static file does not exist based on the fact that the value of the static file identifier is equal to the first numerical value, the Web client is only applicable to the active data list.
To further optimize the above embodiment, after step S102, the method may further include:
when the length of the active data list is equal to a preset length limit value, inquiring Web cache service based on a static file key value to obtain a static file identifier;
wherein the static file key is determined based on the data list ID. Static file key = static _ file + data list ID.
The generation method of the static file key value comprises the following steps: appointing a uniform character string prefix (namely static _ file), and splicing the prefix with the data list ID to obtain the data list. The prefixes of the key values of the static files corresponding to the bullet screen lists of different videos are the same, and the difference is the ID of the data list spliced at the back.
To further optimize the above embodiment, after step S102, the method may further include:
when the length of the active data list is equal to a preset length limit value, checking a static file identifier, and when the cache of the static file identifier does not exist, submitting the data list ID to a task pool to generate a corresponding static file;
setting the value of the static file identifier as a second numerical value, wherein the second numerical value is used for representing that a static file is being generated;
and returning the active data list and the static file identifier to the Web client, and determining whether to use the static file by the Web client based on the value of the static file identifier.
In this embodiment, the value of the second value is determined according to actual needs, for example, the value of the second value is-1, and the present invention is not limited herein.
To further optimize the above embodiment, the caching method may further include:
when the length of an active data list is equal to a preset length limit value and a cache of a static file identifier exists (namely the value of the static file identifier is greater than a first numerical value), judging whether the ID of the minimum data object in the active data list is smaller than the value of the static file identifier;
if yes, determining that the active data list is overlapped with the data in the static file;
if not, determining that the active data list is not completely overlapped with the data in the static file, and submitting the data list ID to the task pool to regenerate the static file.
Specifically, when the length of the active data list is equal to the preset length limit and the cache of the static file identifier exists, the value of the static file identifier is compared with the minimum data object ID in the active data list acquired from the Web cache service or the database, and the result is as follows:
and when the minimum data object ID is smaller than the value of the static file identifier, indicating that the active data list is overlapped with the data in the static file, and not carrying out subsequent processing.
When the minimum data object ID is not smaller than the value of the static file identifier, it indicates that the data in the active data list and the data in the static file are not completely overlapped, and at this time, the data list ID needs to be submitted to the task pool to regenerate the static file.
The Web server creates an asynchronous thread that loops through the task pool, reads one or more data list IDs when the task pool has data, and removes them from the task pool. And the Web server generates a corresponding static file according to each data list ID.
Therefore, to further optimize the above embodiment, referring to fig. 3, a flowchart of a method for submitting a data list ID to a task pool to generate a corresponding static file disclosed in the embodiment of the present invention includes:
step S301, sending a full data list reading request to a database based on a data list ID, and creating a static file with empty content for the full data list;
step S302, a full data list returned by the database is obtained and written into a static file;
step S303, writing the maximum data object ID in the full data list into Web cache service as the value of the static file identifier;
specifically, key = static _ file + data list ID, value = maximum data object ID.
And step S304, taking the values of the data list ID and the static file identification as the unique file name identification of the created static file.
For example, { data list ID } - { value of static file identification }. File suffix. So that the static file can be uniquely queried when the user client initiates the request B.
For the data list caching method disclosed by the invention, the invention is exemplified as follows:
suppose that the data objects in the active data list are sorted in the order of ID from large to small, and the maximum length is a preset length limit n.
After the nth data is inserted into the data list, the Web client requests the active data list for the first time, the Web server submits a task of generating a static file, and the returned result of the Web client is as follows: list of 1-n pieces of data + static file identification (equal to-1).
Wherein, when the static file identification is equal to-1, it indicates that the static file is being generated, and only the active data list is used.
After the static file is generated, the Web client requests the active data list again to obtain the list of the 1 st to n th data and the static file identifier (equal to the ID of the n th data). The Web client side initiates a static file request to obtain a static file, wherein the static file comprises 1 st to n th data. The content in the static file and the content in the active data list are completely overlapped, and the 1 st to n th data are still obtained after the overlapping and the combination are removed. See figure 4 for details.
And after the (n + 1) th data is inserted into the data list, the Web client requests the active data list again, and the latest data list is obtained as the (2-n + 1) th data plus the static file identifier (equal to the ID of the (n) th data). And the Web client side initiates a request B to obtain a static file, wherein the static file comprises 1 st to n th data. The contents in the static file and the contents in the active data list are not completely overlapped, and the data are 1 st to n +1 st data after de-overlapping and merging, and are shown in detail in fig. 5.
And continuously inserting the data until the (n + n) th data is inserted, requesting an active data list by the Web client, and acquiring the (n + 1-n + n) th data + static file identification (equal to the ID of the n) th data in the latest data list. The Web client initiates a static file request to obtain a static file (the old static file in fig. 6), wherein the static file (the old static file in fig. 6) includes the 1 st to n th data. The two lists are merged into the 1 st to n + n data. Because n +1> < n >, the contents of the active data list and the static file (the old static file in fig. 6) are not completely overlapped, and the active data list and the static file are executed according to the interactive process between the Web client and the Web server in fig. 1, and the Web server will submit the generation task of the static file when processing the data list reading request sent by the Web client, so as to obtain the new static file in fig. 6.
After the Web task generates the static file, the Web client requests the active data list again, and the latest data list is acquired as the (n + 1) -n + n data + the static file identifier (equal to the ID of the (n + n) data). The Web client side initiates a static file request to obtain a static file, wherein the static file comprises 1-n + n data. And at the moment, the data of the two files are completely overlapped again until the incremental data exceeds n pieces again, and the static file is generated for the third time.
Corresponding to the embodiment of the method, the invention also discloses a cache device of the data list.
Referring to fig. 7, a schematic structural diagram of a cache device for a data list disclosed in an embodiment of the present invention is applied to a Web server, where the cache device includes:
a list request obtaining unit 401, configured to obtain a data list reading request sent by a Web client, where the data list reading request carries a data list ID;
a list obtaining unit 402, configured to obtain a corresponding active data list from a Web cache service or a database based on the data list reading request, where the active data list is a data list composed of latest incremental data;
a first static file identifier obtaining unit 403, configured to obtain, based on the data list reading request, a corresponding static file identifier from the Web cache service;
a first returning unit 404, configured to return the active data list and the static file identifier to the Web client;
a file request obtaining unit 405, configured to obtain a static file request sent by the Web client, where the static file request is generated when the Web client determines that a static file exists based on that a value of the static file identifier is greater than a first numerical value, and the static file request carries the data list ID and the static file identifier;
a second returning unit 406, configured to return a data packet carrying the static file to the Web client, where the Web client combines the data object in the active data list and the data object in the static data list in the static file into a complete data list after deduplication based on a data object ID, where a header of the data packet sets a cache configuration of an HTTP cache.
To sum up, the invention discloses a data list caching device, which acquires a data list reading request sent by a Web client, acquires a corresponding active data list from a Web caching service or a database based on the data list reading request, acquires a corresponding static file identifier from the Web caching service based on the data list reading request, returns the active data list and the static file identifier to the Web client, returns a data packet carrying a static file to the Web client after receiving the static file request sent by the Web client, combines a data object in the active data list and a data object in the static data list in the static file into a complete data list after de-duplicating the data object based on a data object ID by the Web client, and sets the caching configuration of HTTP caching at the head of the data packet. When the data list is cached, the data list is divided into two parts, one part is placed in the active data list and stored in the Web cache service and the database, and the other part is placed in the static data list and stored in the static file. Since the Web cache service and the database belong to the resources in the station, and the HTTP cache belongs to the resources outside the station, the cost of the hardware produced by the source station is not counted. Therefore, the invention reasonably splits the data list, ensures the issuing of the large data list, reduces the usage amount of Web cache service to the maximum extent and reduces the hardware cost of the memory.
To further optimize the above embodiment, the list obtaining unit 402 may include:
a key value determining subunit, configured to determine an active data list key value according to the data list ID;
a judging subunit, configured to judge whether the active data list can be queried in the Web cache service based on the active data list key value;
a first obtaining subunit, configured to, if the determining subunit determines that the data is the active data, directly obtain the active data list from the Web cache service.
To further optimize the above embodiment, the list obtaining unit 402 may further include:
a sending subunit, configured to send an active data list query request to a database based on the data list ID if the determining subunit determines that the data list ID is negative;
and the second acquiring subunit is configured to acquire an active data list returned by the database, and write the active data list into the Web cache service based on the active data list key value, where data objects in the active data list are sorted in an order from IDs to IDs, and the maximum length is the preset length limit.
To further optimize the above embodiment, the caching apparatus may further include:
a third returning unit, configured to, after the list obtaining unit obtains a corresponding active data list from a Web cache service or a database based on the data list reading request, return the active data list and the static file identifier whose value is a first numerical value to the Web client when the length of the active data list is smaller than the preset length limit value, so that the Web client determines to use only the active data list based on the first numerical value, where the first numerical value indicates that the static file does not exist.
To further optimize the above embodiment, the caching apparatus may further include:
a first static file identifier obtaining unit, configured to, after the list obtaining unit obtains a corresponding active data list from a Web cache service or a database based on the data list read request, query, when the length of the active data list is equal to the preset length limit, the Web cache service based on a static file key to obtain the static file identifier, where the static file key is determined based on the data list ID.
To further optimize the above embodiment, the caching apparatus may further include:
a generating unit, configured to, after the list obtaining unit 402 obtains a corresponding active data list from a Web cache service or a database based on the data list reading request, check a static file identifier when the length of the active data list is equal to a preset length limit, and submit the data list ID to a task pool to generate a corresponding static file when a cache of the static file identifier does not exist;
the setting unit is used for setting the value of the static file identifier as a second numerical value, and the second numerical value is used for representing that the static file is being generated;
and the fourth returning unit is used for returning the active data list and the static file identifier to the Web client, and the Web client determines whether to use the static file or not based on the value of the static file identifier.
To further optimize the above embodiment, the caching apparatus may further include:
a judging unit, configured to, when the length of the active data list is equal to the preset length limit and a cache of the static file identifier exists, judge whether an ID of a minimum data object in the active data list is smaller than a value of the static file identifier;
the overlapping determining unit is used for determining that the active data list and the data in the static file overlap under the condition that the judging unit judges that the active data list and the data in the static file overlap;
and the incomplete overlapping determining unit is used for determining that the data in the active data list and the data in the static file are not completely overlapped under the condition that the judging unit judges that the data in the active data list and the data in the static file are not completely overlapped, and submitting the data list ID to the task pool to regenerate the static file.
To further optimize the above embodiment, the generating unit may specifically be configured to:
sending a full data list reading request to a database based on the data list ID, and creating a static file with empty content for the full data list;
acquiring the full data list returned by the database, and writing the full data list into the static file;
writing the maximum data object ID in the full data list into Web cache service as the value of the static file identifier;
and taking the values of the data list ID and the static file identification as the unique file name identification of the created static file.
It should be noted that, for the specific working principle of each component in the device embodiment, please refer to the corresponding part of the method embodiment, which is not described herein again.
Finally, it should also be noted that, in this document, relational terms such as first and second, and the like are used solely to distinguish one entity or action from another entity or action without necessarily requiring or implying any actual such relationship or order between such entities or actions. Also, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrases "comprising one of 8230; \8230;" 8230; "does not exclude the presence of additional like elements in a process, method, article, or apparatus that comprises the element.
The embodiments in the present description are described in a progressive manner, each embodiment focuses on differences from other embodiments, and the same and similar parts among the embodiments are referred to each other.
The previous description of the disclosed embodiments is provided to enable any person skilled in the art to make or use the present invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the generic principles defined herein may be applied to other embodiments without departing from the spirit or scope of the invention. Thus, the present invention is not intended to be limited to the embodiments shown herein but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.
Claims (16)
1. A caching method for a data list is applied to a Web server, and comprises the following steps:
acquiring a data list reading request sent by a Web client, wherein the data list reading request carries a data list ID;
acquiring a corresponding active data list from a Web cache service or a database based on the data list reading request, wherein the active data list is a data list consisting of latest incremental data, and the length of the active data list is not greater than a preset length limit value;
acquiring a corresponding static file identifier from the Web cache service based on the data list reading request;
returning the active data list and the static file identification to the Web client;
acquiring a static file request sent by the Web client, wherein the static file request is generated when the Web client determines that a static file exists based on the fact that the value of the static file identifier is greater than a first numerical value, and the static file request carries the data list ID and the static file identifier;
and returning a data packet carrying the static file to the Web client, and combining the data objects in the active data list and the data objects in the static data list in the static file into a complete data list by the Web client after de-duplication based on the data object ID, wherein the head of the data packet is provided with the cache configuration of the HTTP cache.
2. The caching method according to claim 1, wherein the obtaining a corresponding active data list from a Web caching service or a database based on the data list read request comprises:
determining an active data list key value according to the data list ID;
judging whether the active data list can be inquired in the Web cache service or not based on the active data list key value;
and if so, directly acquiring the active data list from the Web cache service.
3. The caching method according to claim 2, further comprising:
if not, sending an active data list query request to a database based on the data list ID;
and acquiring an active data list returned by the database, and writing the active data list into the Web cache service based on the key value of the active data list, wherein data objects in the active data list are sorted in the sequence of IDs from large to small, and the maximum length is the preset length limit value.
4. The caching method according to claim 1, further comprising, after the obtaining a corresponding active data list from a Web caching service or a database based on the data list read request:
and when the length of the active data list is smaller than the preset length limit value, returning the active data list and the static file identifier with the value as a first numerical value to the Web client, and enabling the Web client to determine to only use the active data list based on the first numerical value, wherein the first numerical value indicates that the static file does not exist.
5. The caching method according to claim 1, wherein after said obtaining the corresponding active data list from the Web caching service or the database based on the data list read request, further comprising:
and when the length of the active data list is equal to the preset length limit value, querying a Web cache service based on a static file key value to obtain the static file identifier, wherein the static file key value is determined based on the data list ID.
6. The caching method according to claim 1, further comprising, after the obtaining a corresponding active data list from a Web caching service or a database based on the data list read request:
when the length of the active data list is equal to the preset length limit value, checking a static file identifier, and when the cache of the static file identifier does not exist, submitting the data list ID to a task pool to generate a corresponding static file;
setting the value of the static file identifier as a second numerical value, wherein the second numerical value is used for representing that a static file is being generated;
and returning the active data list and the static file identifier to the Web client, and determining whether to use the static file by the Web client based on the value of the static file identifier.
7. The caching method according to claim 1, further comprising:
when the length of the active data list is equal to the preset length limit value and the cache of the static file identifier exists, judging whether the minimum data object ID in the active data list is smaller than the value of the static file identifier;
if yes, determining that the active data list is overlapped with the data in the static file;
if not, determining that the active data list is not completely overlapped with the data in the static file, and submitting the data list ID to a task pool to regenerate the static file.
8. The caching method according to claim 6, wherein the submitting the data list IDs to a task pool generates a corresponding static file, and comprises:
sending a full data list reading request to a database based on the data list ID, and creating a static file with empty content for the full data list;
acquiring the full data list returned by the database, and writing the full data list into the static file;
writing the maximum data object ID in the full data list into Web cache service as the value of the static file identifier;
and taking the values of the data list ID and the static file identification as the unique file name identification of the created static file.
9. A data list caching apparatus, applied to a Web server, the caching apparatus comprising:
the Web client side comprises a list request acquisition unit and a list reading unit, wherein the list request acquisition unit is used for acquiring a data list reading request sent by the Web client side, and the data list reading request carries a data list ID;
the list acquisition unit is used for acquiring a corresponding active data list from a Web cache service or a database based on the data list reading request, wherein the active data list is a data list consisting of latest incremental data, and the length of the active data list is not more than a preset length limit value;
a first static file identifier obtaining unit, configured to obtain a corresponding static file identifier from the Web cache service based on the data list reading request;
the first returning unit is used for returning the active data list and the static file identifier to the Web client;
a file request obtaining unit, configured to obtain a static file request sent by the Web client, where the static file request is generated when the Web client determines that a static file exists based on that a value of the static file identifier is greater than a first numerical value, and the static file request carries the data list ID and the static file identifier;
and a second returning unit, configured to return a data packet carrying the static file to the Web client, and combine, by the Web client, the data object in the active data list and the data object in the static data list in the static file into a complete data list after deduplication based on a data object ID, where a header of the data packet sets a cache configuration of an HTTP cache.
10. The caching apparatus according to claim 9, wherein the list acquisition unit includes:
a key value determining subunit, configured to determine an active data list key value according to the data list ID;
a judging subunit, configured to judge whether the active data list can be queried in the Web cache service based on the active data list key value;
a first obtaining subunit, configured to, if the determining subunit determines that the data is the active data, directly obtain the active data list from the Web cache service.
11. The cache apparatus according to claim 10, wherein the list obtaining unit further comprises:
a sending subunit, configured to send an active data list query request to a database based on the data list ID if the judging subunit judges no;
and the second acquiring subunit is configured to acquire an active data list returned by the database, and write the active data list into the Web cache service based on the active data list key value, where data objects in the active data list are sorted in an order from IDs to IDs, and the maximum length is the preset length limit.
12. The caching apparatus of claim 9, further comprising:
and a third returning unit, configured to, after the list obtaining unit obtains a corresponding active data list from a Web cache service or a database based on the data list reading request, return the active data list and the static file identifier having a first value to the Web client when the length of the active data list is smaller than the preset length limit value, so that the Web client determines to use only the active data list based on the first value, where the first value indicates that the static file does not exist.
13. The caching apparatus of claim 9, further comprising:
a first static file identifier obtaining unit, configured to, after the list obtaining unit obtains a corresponding active data list from a Web cache service or a database based on the data list read request, query the Web cache service based on a static file key value to obtain the static file identifier when the length of the active data list is equal to the preset length limit, where the static file key value is determined based on the data list ID.
14. The caching apparatus of claim 9, further comprising:
the generating unit is used for checking a static file identifier when the length of the active data list is equal to a preset length limit value after the list acquiring unit acquires a corresponding active data list from a Web cache service or a database based on the data list reading request, and submitting the data list ID to a task pool to generate a corresponding static file when the cache of the static file identifier does not exist;
the setting unit is used for setting the value of the static file identifier as a second numerical value, and the second numerical value is used for representing that the static file is being generated;
and the fourth returning unit is used for returning the active data list and the static file identifier to the Web client, and the Web client determines whether to use the static file or not based on the value of the static file identifier.
15. The caching apparatus of claim 9, further comprising:
a judging unit, configured to, when the length of the active data list is equal to the preset length limit and a cache of the static file identifier exists, judge whether an ID of a minimum data object in the active data list is smaller than a value of the static file identifier;
the overlapping determining unit is used for determining that the active data list and the data in the static file overlap under the condition that the judging unit judges that the active data list and the data in the static file overlap;
and the incomplete superposition determining unit is used for determining that the data in the active data list and the data in the static file are completely superposed under the condition that the judging unit judges that the data in the active data list and the data in the static file are not completely superposed, and submitting the data list ID to a task pool to regenerate the static file.
16. The caching device of claim 14, wherein the generating unit is specifically configured to:
sending a full data list reading request to a database based on the data list ID, and creating a static file with empty content for the full data list;
acquiring the full data list returned by the database, and writing the full data list into the static file;
writing the maximum data object ID in the full data list into Web cache service as the value of the static file identifier;
and taking the values of the data list ID and the static file identification as the unique file name identification of the created static file.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202211082438.XA CN115174546B (en) | 2022-09-06 | 2022-09-06 | Data list caching method and device |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202211082438.XA CN115174546B (en) | 2022-09-06 | 2022-09-06 | Data list caching method and device |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN115174546A CN115174546A (en) | 2022-10-11 |
| CN115174546B true CN115174546B (en) | 2023-01-06 |
Family
ID=83481312
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202211082438.XA Active CN115174546B (en) | 2022-09-06 | 2022-09-06 | Data list caching method and device |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN115174546B (en) |
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102655517A (en) * | 2011-03-03 | 2012-09-05 | 苏州同程旅游网络科技有限公司 | Cloud computing based network load balancing method |
| CN103857063A (en) * | 2012-11-29 | 2014-06-11 | 广州市千钧网络科技有限公司 | Spatial state connection establishment method and device based on mobile terminals |
| CN105488078A (en) * | 2014-10-10 | 2016-04-13 | 阿里巴巴集团控股有限公司 | Web page data caching method and equipment |
| CN110727720A (en) * | 2019-10-21 | 2020-01-24 | 腾讯科技（深圳）有限公司 | List display and query method and device, storage medium and computer equipment |
Family Cites Families (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2016106155A1 (en) * | 2014-12-22 | 2016-06-30 | Zenedge, Inc. | Dynamic content caching system |
| US11265257B2 (en) * | 2020-04-16 | 2022-03-01 | Cisco Technology, Inc. | Rapid network traffic telemetry exports with split templates and flow records |
| CN113886735A (en) * | 2021-09-24 | 2022-01-04 | 浪潮云信息技术股份公司 | Nginx-based static resource and data caching method and system |
| CN114143376A (en) * | 2021-11-18 | 2022-03-04 | 青岛聚看云科技有限公司 | Server for loading cache, display equipment and resource playing method |
-
2022
- 2022-09-06 CN CN202211082438.XA patent/CN115174546B/en active Active
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102655517A (en) * | 2011-03-03 | 2012-09-05 | 苏州同程旅游网络科技有限公司 | Cloud computing based network load balancing method |
| CN103857063A (en) * | 2012-11-29 | 2014-06-11 | 广州市千钧网络科技有限公司 | Spatial state connection establishment method and device based on mobile terminals |
| CN105488078A (en) * | 2014-10-10 | 2016-04-13 | 阿里巴巴集团控股有限公司 | Web page data caching method and equipment |
| CN110727720A (en) * | 2019-10-21 | 2020-01-24 | 腾讯科技（深圳）有限公司 | List display and query method and device, storage medium and computer equipment |
Non-Patent Citations (1)
| Title |
|---|
| 《大数据文件存储策略探索》;屈美娟等;《科技创新与应用》;20190428(第12期);正文1-3页 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN115174546A (en) | 2022-10-11 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US11586506B2 (en) | System and method for indexing image backups | |
| US7793112B2 (en) | Access to content addressable data over a network | |
| CN105224546B (en) | Data storage and query method and equipment | |
| CN106302595B (en) | Method and equipment for carrying out health check on server | |
| US8312222B1 (en) | Event-driven regeneration of pages for web-based applications | |
| CN108829787B (en) | Metadata distributed system | |
| CN111078667B (en) | Data migration method and related device | |
| US20070192401A1 (en) | System and method for synchronizing syndicated content over multiple locations | |
| CN109167840B (en) | Task pushing method, node autonomous server and edge cache server | |
| US7376650B1 (en) | Method and system for redirecting a request using redirection patterns | |
| CN105760380A (en) | Database query method, device and system | |
| WO2019076102A1 (en) | Data rollback method and system, device, and computer readable storage medium | |
| CN110399348A (en) | File deletes method, apparatus, system and computer readable storage medium again | |
| CN108540510B (en) | Cloud host creation method and device and cloud service system | |
| CN109558378A (en) | File management method, device, equipment and storage medium | |
| CN104348859A (en) | File synchronizing method, device, server, terminal and system | |
| CN109062500B (en) | Metadata management server, data storage system and data storage method | |
| CN110795395A (en) | File deployment system and file deployment method | |
| CN111782707A (en) | Data query method and system | |
| CN108243207B (en) | A kind of date storage method of network cloud disk | |
| CN107357557B (en) | Information updating method and device | |
| CN111198885A (en) | Data processing method and device | |
| CN111143126A (en) | Data copying method, system and related components of distributed file system | |
| CN115174546B (en) | Data list caching method and device | |
| WO2021037072A1 (en) | Buffer information updating method and apparatus, device, and medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |