CN113760982B - Data processing method and device - Google Patents
Data processing method and device Download PDFInfo
- Publication number
- CN113760982B CN113760982B CN202110062487.6A CN202110062487A CN113760982B CN 113760982 B CN113760982 B CN 113760982B CN 202110062487 A CN202110062487 A CN 202110062487A CN 113760982 B CN113760982 B CN 113760982B
- Authority
- CN
- China
- Prior art keywords
- data
- target data
- query
- acquired
- query information
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000003672 processing method Methods 0.000 title claims abstract description 29
- 238000000034 method Methods 0.000 claims abstract description 21
- 238000012545 processing Methods 0.000 claims description 26
- 238000004590 computer program Methods 0.000 claims description 9
- 238000012163 sequencing technique Methods 0.000 claims description 8
- 238000012544 monitoring process Methods 0.000 claims description 5
- 230000015556 catabolic process Effects 0.000 abstract description 25
- 238000010586 diagram Methods 0.000 description 15
- 230000000694 effects Effects 0.000 description 15
- 230000008030 elimination Effects 0.000 description 10
- 238000003379 elimination reaction Methods 0.000 description 10
- 230000006870 function Effects 0.000 description 8
- 238000004891 communication Methods 0.000 description 7
- 238000012423 maintenance Methods 0.000 description 6
- 230000003287 optical effect Effects 0.000 description 4
- 238000012360 testing method Methods 0.000 description 4
- 238000007405 data analysis Methods 0.000 description 3
- 238000012986 modification Methods 0.000 description 3
- 230000004048 modification Effects 0.000 description 3
- 230000002085 persistent effect Effects 0.000 description 3
- 230000008569 process Effects 0.000 description 3
- 230000008901 benefit Effects 0.000 description 2
- 238000010276 construction Methods 0.000 description 2
- 239000000835 fiber Substances 0.000 description 2
- 230000000644 propagated effect Effects 0.000 description 2
- 239000004065 semiconductor Substances 0.000 description 2
- 238000004458 analytical method Methods 0.000 description 1
- 230000008859 change Effects 0.000 description 1
- 230000003247 decreasing effect Effects 0.000 description 1
- 238000013461 design Methods 0.000 description 1
- 238000001514 detection method Methods 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 239000004973 liquid crystal related substance Substances 0.000 description 1
- 230000007246 mechanism Effects 0.000 description 1
- 239000013307 optical fiber Substances 0.000 description 1
- 230000035755 proliferation Effects 0.000 description 1
- 230000002441 reversible effect Effects 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/245—Query processing
- G06F16/2455—Query execution
- G06F16/24552—Database cache management
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/23—Updating
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/27—Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Databases & Information Systems (AREA)
- Data Mining & Analysis (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Computing Systems (AREA)
- Computational Linguistics (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention discloses a data processing method and device, and relates to the technical field of computers. One embodiment of the method comprises the following steps: acquiring query information corresponding to the data to be acquired in a query period; updating the target data set according to the query information; the target data set is constructed according to query information corresponding to the historical data; and storing the target data in the updated target data set to a local cache. According to the embodiment, the target data is updated in real time, so that the cache breakdown phenomenon is effectively solved, the applicable scene is expanded, and the user experience is improved.
Description
Technical Field
The present invention relates to the field of computer technologies, and in particular, to a data processing method and apparatus.
Background
In the case of a cache mechanism, when a large number of requests come in, the cache is invalidated at a certain limit moment, so that all requests are sent to a background database or other devices with frequently operated Input/Output (I/O) by bypassing the cache, and the system is down (also referred to as cache breakdown). With the proliferation of internet users, how to avoid cache breakdown is a problem to be solved.
The prior art has at least the following problems:
In the existing data processing method, due to hysteresis of the set hot key (namely target data), the effect of solving the cache breakdown is poor, the applicable scene is less, and the user experience is poor.
Disclosure of Invention
In view of the above, the embodiment of the invention provides a data processing method and device, which can effectively avoid the cache breakdown phenomenon by updating target data in real time, expand the applicable scene and promote the user experience.
To achieve the above object, according to a first aspect of an embodiment of the present invention, there is provided a data processing method, including:
acquiring query information corresponding to the data to be acquired in a query period;
Updating the target data set according to the query information; the target data set is constructed according to query information corresponding to the historical data;
and storing the target data in the updated target data set to a local cache.
Further, the constructing step of the target data set includes:
Acquiring query information corresponding to historical data, wherein the query information comprises query quantity and query time;
and sequencing the historical data according to the query information, determining target data according to the sequencing result and preset conditions, and constructing a target data set.
Further, the method further comprises the following steps:
responding to a data acquisition request initiated by a user based on the data to be acquired, and judging whether the data to be acquired is target data;
if yes, acquiring data to be acquired from a local cache; if not, the data to be acquired is acquired from the database.
Further, the query information includes a query number and a query rate; updating the target data set according to the query information, and further comprising:
Monitoring the query quantity and the query rate corresponding to the data to be acquired, which are acquired from the database, and determining first data according to a first query quantity threshold value and a first query rate threshold value;
Reporting query information corresponding to the first data;
And updating the target data set according to the query information corresponding to the reported first data and the preset condition.
Further, storing the key value pair list corresponding to the target data to a local cache, and judging whether the data to be acquired is the target data, and further comprising:
and traversing the key value pair list according to the primary key corresponding to the data to be acquired so as to judge whether the data to be acquired is target data.
Further, the method further comprises the following steps:
and acquiring an application number corresponding to the data acquisition request, updating a preset condition corresponding to the application number according to the service requirement corresponding to the application number, and updating target data in the target data set according to the updated preset condition.
Further, the method further comprises the following steps:
and acquiring the update information corresponding to the target data set, and storing the update information.
According to a second aspect of an embodiment of the present invention, there is provided a data processing apparatus including:
the information acquisition module is used for acquiring query information corresponding to the data to be acquired in the query period;
the updating module is used for updating the target data set according to the query information; the target data set is constructed according to query information corresponding to the historical data;
And the storage module is used for storing the target data in the updated target data set to the local cache.
According to a third aspect of an embodiment of the present invention, there is provided an electronic apparatus including:
one or more processors;
Storage means for storing one or more programs,
When the one or more programs are executed by the one or more processors, the one or more processors are caused to implement any of the data processing methods described above.
According to a fourth aspect of embodiments of the present invention, there is provided a computer readable medium having stored thereon a computer program which when executed by a processor implements a data processing method as any one of the above.
One embodiment of the above invention has the following advantages or benefits: because the query information corresponding to the data to be acquired in the query period is acquired; updating the target data set according to the query information; the target data set is constructed according to query information corresponding to the historical data; the technical means of storing the target data in the updated target data set into the local cache is overcome, so that the technical effects of poor cache breakdown solving effect, less applicable scenes and poor user experience existing in the existing data processing method due to hysteresis of the set hot key (namely the target data) are overcome, the target data are updated in real time, the cache breakdown phenomenon is effectively avoided, the applicable scenes are expanded, and the user experience is improved.
Further effects of the above-described non-conventional alternatives are described below in connection with the embodiments.
Drawings
The drawings are included to provide a better understanding of the invention and are not to be construed as unduly limiting the invention. Wherein:
FIG. 1 is a schematic diagram of the main flow of a data processing method according to a first embodiment of the present invention;
FIG. 2a is a schematic diagram of the main flow of a data processing method according to a second embodiment of the present invention;
FIG. 2b is a schematic diagram of a main flow of determining whether the data to be acquired is the target data in the method shown in FIG. 2 a;
FIG. 2c is a schematic diagram of the main flow of data retrieval from a local cache or database in the method of FIG. 2 a;
FIG. 3 is a schematic diagram of the main modules of a data processing apparatus provided according to an embodiment of the present invention;
FIG. 4 is an exemplary system architecture diagram in which embodiments of the present invention may be applied;
fig. 5 is a schematic diagram of a computer system suitable for use in implementing an embodiment of the invention.
Detailed Description
Exemplary embodiments of the present invention will now be described with reference to the accompanying drawings, in which various details of the embodiments of the present invention are included to facilitate understanding, and are to be considered merely exemplary. Accordingly, those of ordinary skill in the art will recognize that various changes and modifications of the embodiments described herein can be made without departing from the scope and spirit of the invention. Also, descriptions of well-known functions and constructions are omitted in the following description for clarity and conciseness.
FIG. 1 is a schematic diagram of the main flow of a data processing method according to a first embodiment of the present invention; as shown in fig. 1, the data processing method provided by the embodiment of the present invention mainly includes:
Step S101, acquiring query information corresponding to data to be acquired in a query period.
According to the embodiment of the invention, the Query information comprises the corresponding Query number and Query time of each data to be acquired in the Query period, and the Query rate (such as QPS (Query Per Second) Per Second) of the data to be acquired in the latest time period can be determined according to the information, so that the target data (hot key) can be more accurately determined, and the cache breakdown phenomenon can be effectively avoided.
The hot key refers to a large amount of requests (specific data is set according to the carrying capacity of the accessed database (such as redis, mySQL, etc.) in a short time to access a specific key (data) in the database. At this time, the access amount is too concentrated, so that the upper limit of the physical network card is reached, and the phenomenon of downtime of the system is often caused.
Specifically, according to an embodiment of the present invention, the method further includes:
responding to a data acquisition request initiated by a user based on the data to be acquired, and judging whether the data to be acquired is target data;
if yes, acquiring data to be acquired from a local cache; if not, the data to be acquired is acquired from the database.
Through the setting, after receiving a data acquisition request initiated by a user based on application, first selecting whether the data to be acquired indicated by the data acquisition request belongs to a hot key or not, and if so, directly going to a local cache for acquisition; if the data does not belong to the hot key, namely the data is not stored in the local cache, the data is obtained in the database. It should be noted that the database may be a secondary cache database (e.g., redis) or a persistent database (e.g., mySQL). Through the arrangement, the hot key is directly obtained from the local cache, so that the cache breakdown phenomenon is effectively avoided, and the user experience is improved.
Further, according to an embodiment of the present invention, the storing the key value pair list corresponding to the target data in the local cache, to determine whether the data to be acquired is the target data, further includes:
and traversing the key value pair list according to the primary key corresponding to the data to be acquired so as to judge whether the data to be acquired is target data.
Specifically, for the stored data (whether stored in the local cache or the database), a key-value pair list corresponding to the stored data is built according to key-value pairs, a specific key-value pair list is built for target data (hot key) and stored in the local cache, and by the arrangement, whether the data to be acquired is the target data or not is determined quickly, and data processing efficiency is improved.
Step S102, updating the target data set according to the query information; the target data set is constructed according to query information corresponding to the historical data.
Through the arrangement, the target data set is updated according to the query information corresponding to the data to be acquired, the update of the target data according to the real-time information is realized, the target data can be monitored and determined in real time, the real-time performance of the target data in the target data set is ensured, the cache breakdown phenomenon is effectively avoided, and the application scene is expanded.
Further, according to an embodiment of the present invention, the constructing step of the target data set includes:
Acquiring query information corresponding to historical data, wherein the query information comprises query quantity and query time;
and sequencing the historical data according to the query information, determining target data according to the sequencing result and preset conditions, and constructing a target data set.
Through the arrangement, the target data is determined according to the query information corresponding to the historical data, and the target data is more representative in a longer time range. Meanwhile, according to the target data determined by the query information corresponding to the historical data, a preset condition can be formulated in a targeted manner, and the type and the number of queries (namely, the preset condition is specifically set differently according to time, promotion activities, promotion products and the like) which are possibly the target data can be judged.
Preferably, according to an embodiment of the present invention, the query information includes a query number and a query rate; updating the target data set according to the query information, and further comprising:
Monitoring the query quantity and the query rate corresponding to the data to be acquired, which are acquired from the database, and determining first data according to a first query quantity threshold value and a first query rate threshold value;
Reporting query information corresponding to the first data;
And updating the target data set according to the query information corresponding to the reported first data and the preset condition.
Specifically, according to the embodiment of the present invention, if the data to be acquired is acquired from the database, that is, the data to be acquired is not the target data when the acquisition request is initiated, the query number and the query rate of the data to be acquired are determined according to the first query number threshold and the first query rate threshold, so that the target data are updated, thereby further ensuring the instantaneity of the target data stored in the local cache and improving the user experience.
Step S103, storing the target data in the updated target data set to a local cache.
The data in the local cache is fast in reading speed, and the access quantity which can be simultaneously accommodated is large. The target data (hot key) is stored in the local cache, so that the cache breakdown phenomenon can be effectively avoided, and the system maintenance cost is reduced.
Further, according to an embodiment of the present invention, the method further includes:
and acquiring an application number corresponding to the data acquisition request, updating a preset condition corresponding to the application number according to the service requirement corresponding to the application number, and updating target data in the target data set according to the updated preset condition.
Through the arrangement, different preset conditions are formulated according to the service requirements corresponding to the application numbers, so that the target data is updated, and the application scene of the data processing method is expanded.
Preferably, according to an embodiment of the present invention, the method further includes:
and acquiring the update information corresponding to the target data set, and storing the update information.
The updated information of each target data in the target data set is stored, so that the subsequent data display, data analysis and operation and maintenance are convenient; the preset conditions can be adjusted according to the update information corresponding to the target data, so that the real-time target data can be accurately determined, the user experience is further improved, and the data processing efficiency is improved.
Illustratively, according to an embodiment of the present invention, the method further comprises:
Setting a destiny strategy, and carrying out timing destiny on target data stored in a local buffer according to the destiny strategy.
The elimination strategy can include one or more of elimination period, minimum inquiry rate, priority elimination of the first added data when the cache space is insufficient, and the like, so that the local cache space is effectively released through the arrangement, and the flexibility of target data stored in the local cache is ensured.
According to the technical scheme of the embodiment of the invention, the query information corresponding to the data to be acquired in the query period is acquired; updating the target data set according to the query information; the target data set is constructed according to query information corresponding to the historical data; the technical means of storing the target data in the updated target data set into the local cache is overcome, so that the technical effects of poor cache breakdown solving effect, less applicable scenes and poor user experience existing in the existing data processing method due to hysteresis of the set hot key (namely the target data) are overcome, the target data are updated in real time, the cache breakdown phenomenon is effectively avoided, the applicable scenes are expanded, and the user experience is improved.
FIG. 2a is a schematic diagram of the main flow of a data processing method according to a second embodiment of the present invention; as shown in fig. 2a, the data processing method provided in the embodiment of the present invention mainly includes:
step S201, query information corresponding to the historical data is obtained, wherein the query information comprises query quantity and query time.
Specifically, according to the embodiment of the present invention, the computing processing machine may be dynamically increased or decreased according to the amount of QPS (Query Per Second) by using a hotspot data computing service cluster (such as a computing distributed computing service), so as to perform computing to determine hotspot data (i.e. a hotkey), and push the computing result to an application cluster at the same time to obtain corresponding data.
Step S202, sorting the historical data according to the query information, determining target data according to the sorting result and preset conditions, and constructing a target data set.
Through the arrangement, the target data is determined according to the query information corresponding to the historical data, and the target data is more representative in a longer time range. Meanwhile, according to the target data determined by the query information corresponding to the historical data, a preset condition can be formulated in a targeted manner, and the type and the number of queries (namely, the preset condition is specifically set differently according to time, promotion activities, promotion products and the like) which are possibly the target data can be judged.
According to the embodiment of the invention, the preset conditions are set and stored through the established ETCD cluster (the distributed key-value storage developed by CoreOS based on Raft can be used for service discovery, shared configuration and consistency guarantee, and the existing cluster capable of realizing the functions can also be adopted, such as Zookeep er cluster).
Step S203, responding to a data acquisition request initiated by a user based on the data to be acquired, traversing a key value pair list according to a main key corresponding to the data to be acquired, so as to judge whether the data to be acquired is target data; the target data and the corresponding key value pair list are stored in a local cache.
Specifically, as shown in fig. 2b, according to an embodiment of the present invention, an application is registered in an ETCD cluster, and by Netty (which is a java open source framework provided by JBOSS and is now an independent item on Github, netty provides an asynchronous, event-driven network application framework and tools for quickly developing a high-performance, high-reliability network server and client program) long-connection hotspot data computing service cluster (simply referred to as computing service, after the computing service is started, the computing service will register itself in the ETCD cluster service, and start Netty service port application end connection and heartbeat detection), and at the same time, a timer is started, a computing service List < String > is pulled from the ETCD at regular time, and then connected with the client of the computing service, and when the connection fails, the computing service will push a prompt; until the connection is successful. After the connection of the application to the computing service is successful, the channel (channel) and the Netty address name of the connection are stored in a key-value form, so that the data (mainly query information) corresponding to the application can be acquired later and interacted with the computing service. And, start the thread pool and send the application heartbeat regularly, update the unconnected service of channel in real time, deposit the form to be CopyOnWriteArrayList < Map < Key, value > >. After the application is started, the preset conditions stored in the ETCD cluster are stored in the current local cache (i.e., the target data and the corresponding key value pair list are stored in the local cache).
Step S204, if yes, acquiring data to be acquired from a local cache; if not, the data to be acquired is acquired from the database.
Through the setting, after receiving a data acquisition request initiated by a user based on application, first selecting whether the data to be acquired indicated by the data acquisition request belongs to a hot key or not, and if so, directly going to a local cache for acquisition; if the data does not belong to the hot key, namely the data is not stored in the local cache, the data is obtained in the database. It should be noted that the database may be a secondary cache database (e.g., redis) or a persistent database (e.g., mySQL). Through the arrangement, the hot key is directly obtained from the local cache, so that the cache breakdown phenomenon is effectively avoided, and the user experience is improved. Specifically, as shown in fig. 2c, the hotspot rule is the preset condition, and redis refers to the cache database. When the A user searches redis data according to the key, whether the key accords with the hot spot rule is judged according to preset conditions set in the ETCD and the determined target data, if so, the data is obtained from the local cache, and the local cache can be used CAFFEINECACHE. If the data to be acquired is nearly expired or the data is not acquired in the local cache (namely, the data is acquired from the database), counting statistics (counting statistics on query information) are reported to the computing service at regular time.
Step S205, monitoring the query quantity and the query rate corresponding to the data to be acquired, which are acquired from the database, and determining first data according to a first query quantity threshold value and a first query rate threshold value;
According to a specific implementation manner of the embodiment of the present invention, the application starts a thread watch to monitor the number of queries and the query rate corresponding to the data to be acquired, and also monitors whether the preset condition and the target data set in the ETCD cluster change, if the preset condition changes, the preset condition and the target data cached locally need to be updated immediately.
Step S206, the query information corresponding to the first data is reported.
The application starting thread periodically reports the query information of the target data accessed to the cache within a certain period of time of the current application to the computing service (here, there is a local count, and the query information is not required to be sent to the computing service every time, for example, when the query number corresponding to a certain data exceeds 10 or when the query rate per second corresponding to a certain data exceeds 2 (the above values are only examples), the query information corresponding to the data is reported to the computing service), so that the computing service calculates whether the data is a hot key (target data), and then updates the target data. According to a specific implementation manner of the embodiment of the present invention, when the query information is reported, MOD (residual function) is calculated according to the data of the current computing service, so that the data to be computed is uniformly marked on the computing service, thereby improving the computing efficiency.
Step S207, updating the target data set according to the query information corresponding to the reported first data and the preset condition.
Specifically, according to the embodiment of the present invention, if the data to be acquired is acquired from the database, that is, the data to be acquired is not the target data when the acquisition request is initiated, the query number and the query rate of the data to be acquired are determined according to the first query number threshold and the first query rate threshold, so that the target data are updated, thereby further ensuring the instantaneity of the target data stored in the local cache and improving the user experience.
According to a specific implementation manner of the embodiment of the present invention, after the computing service receives the query information statistics data of the hotspot data reported by the application end, the computing service may directly write the updated target data into the key-value list in the cache period according to the application name+key+preset condition name, where the value is the hotspot data amount (total query amount), when a certain data amount (the first query amount threshold or the first query rate threshold) is reached and the updated target data is determined, and then call the ETCD cluster to perform a process of updating the target data stored in the local cache. If the test hot spot data sent by the application layer is obtained, judging whether the test hot spot data is a hot spot according to the sliding window, and if the data quantity exceeds a certain quantity in a certain period of time, the test hot spot data is the hot spot data, wherein the time intervals and the thread quantity are configurable.
Step S208, storing the target data in the updated target data set to a local cache.
The data in the local cache is fast in reading speed, and the access quantity which can be simultaneously accommodated is large. The target data (hot key) is stored in the local cache, so that the cache breakdown phenomenon can be effectively avoided, and the system maintenance cost is reduced.
Further, according to an embodiment of the present invention, the method further includes:
and acquiring an application number corresponding to the data acquisition request, updating a preset condition corresponding to the application number according to the service requirement corresponding to the application number, and updating target data in the target data set according to the updated preset condition.
Through the arrangement, different preset conditions are formulated according to the service requirements corresponding to the application numbers, so that the target data is updated, and the application scene of the data processing method is expanded.
Preferably, according to an embodiment of the present invention, the method further includes:
and acquiring the update information corresponding to the target data set, and storing the update information.
The updated information of each target data in the target data set is stored, so that the subsequent data display, data analysis and operation and maintenance are convenient; the preset conditions can be adjusted according to the update information corresponding to the target data, so that the real-time target data can be accurately determined, the user experience is further improved, and the data processing efficiency is improved.
Specifically, according to the embodiment of the invention, the update information about the target data set in the current ETCD cluster can be analyzed by utilizing the link (an open source stream processing framework), and the update information is subjected to data statistics of various dimensions in real time and stored in a database so as to be convenient for report display.
Illustratively, according to an embodiment of the present invention, the method further comprises:
Setting a destiny strategy, and carrying out timing destiny on target data stored in a local buffer according to the destiny strategy.
The elimination strategy can include one or more of elimination period, minimum inquiry rate, priority elimination of the first added data when the cache space is insufficient, and the like, so that the local cache space is effectively released through the arrangement, and the flexibility of target data stored in the local cache is ensured.
According to the technical scheme of the embodiment of the invention, the query information corresponding to the data to be acquired in the query period is acquired; updating the target data set according to the query information; the target data set is constructed according to query information corresponding to the historical data; the technical means of storing the target data in the updated target data set into the local cache is overcome, so that the technical effects of poor cache breakdown solving effect, less applicable scenes and poor user experience existing in the existing data processing method due to hysteresis of the set hot key (namely the target data) are overcome, the target data are updated in real time, the cache breakdown phenomenon is effectively avoided, the applicable scenes are expanded, and the user experience is improved.
FIG. 3 is a schematic diagram of the main modules of a data processing apparatus provided according to an embodiment of the present invention; as shown in fig. 3, a data processing apparatus 300 according to an embodiment of the present invention mainly includes:
the information obtaining module 301 is configured to obtain query information corresponding to data to be obtained in a query period.
According to the embodiment of the invention, the Query information comprises the corresponding Query number and Query time of each data to be acquired in the Query period, and the Query rate (such as QPS (Query Per Second) Per Second) of the data to be acquired in the latest time period can be determined according to the information, so that the target data (hot key) can be more accurately determined, and the cache breakdown phenomenon can be effectively avoided.
The hot key refers to a large amount of requests (specific data is set according to the carrying capacity of the accessed database (such as redis, mySQL, etc.) in a short time to access a specific key (data) in the database. At this time, the access amount is too concentrated, so that the upper limit of the physical network card is reached, and the phenomenon of downtime of the system is often caused.
Specifically, according to an embodiment of the present invention, the data processing apparatus 300 further includes a determining module, configured to:
responding to a data acquisition request initiated by a user based on the data to be acquired, and judging whether the data to be acquired is target data;
if yes, acquiring data to be acquired from a local cache; if not, the data to be acquired is acquired from the database.
Through the setting, after receiving a data acquisition request initiated by a user based on application, first selecting whether the data to be acquired indicated by the data acquisition request belongs to a hot key or not, and if so, directly going to a local cache for acquisition; if the data does not belong to the hot key, namely the data is not stored in the local cache, the data is obtained in the database. It should be noted that the database may be a secondary cache database (e.g., redis) or a persistent database (e.g., mySQL). Through the arrangement, the hot key is directly obtained from the local cache, so that the cache breakdown phenomenon is effectively avoided, and the user experience is improved.
Further, according to an embodiment of the present invention, the above-mentioned list of key value pairs corresponding to the target data is stored in a local cache; the judging module is further used for:
and traversing the key value pair list according to the primary key corresponding to the data to be acquired so as to judge whether the data to be acquired is target data.
Specifically, for the stored data (whether stored in the local cache or the database), a key-value pair list corresponding to the stored data is built according to key-value pairs, a specific key-value pair list is built for target data (hot key) and stored in the local cache, and by the arrangement, whether the data to be acquired is the target data or not is determined quickly, and data processing efficiency is improved.
An updating module 302, configured to update the target data set according to the query information; the target data set is constructed according to query information corresponding to the historical data.
Through the arrangement, the target data set is updated according to the query information corresponding to the data to be acquired, the update of the target data according to the real-time information is realized, the target data can be monitored and determined in real time, the real-time performance of the target data in the target data set is ensured, the cache breakdown phenomenon is effectively avoided, and the application scene is expanded.
Further, according to an embodiment of the present invention, the data processing apparatus 300 further includes a target data set construction module, configured to:
Acquiring query information corresponding to historical data, wherein the query information comprises query quantity and query time;
and sequencing the historical data according to the query information, determining target data according to the sequencing result and preset conditions, and constructing a target data set.
Through the arrangement, the target data is determined according to the query information corresponding to the historical data, and the target data is more representative in a longer time range. Meanwhile, according to the target data determined by the query information corresponding to the historical data, a preset condition can be formulated in a targeted manner, and the type and the number of queries (namely, the preset condition is specifically set differently according to time, promotion activities, promotion products and the like) which are possibly the target data can be judged.
Preferably, according to an embodiment of the present invention, the update module 302 is further configured to:
Monitoring the query quantity and the query rate corresponding to the data to be acquired, which are acquired from the database, and determining first data according to a first query quantity threshold value and a first query rate threshold value;
Reporting query information corresponding to the first data;
And updating the target data set according to the query information corresponding to the reported first data and the preset condition.
Specifically, according to the embodiment of the present invention, if the data to be acquired is acquired from the database, that is, the data to be acquired is not the target data when the acquisition request is initiated, the query number and the query rate of the data to be acquired are determined according to the first query number threshold and the first query rate threshold, so that the target data are updated, thereby further ensuring the instantaneity of the target data stored in the local cache and improving the user experience.
Further, according to an embodiment of the present invention, the update module 302 is further configured to:
and acquiring an application number corresponding to the data acquisition request, updating a preset condition corresponding to the application number according to the service requirement corresponding to the application number, and updating target data in the target data set according to the updated preset condition.
Through the arrangement, different preset conditions are formulated according to the service requirements corresponding to the application numbers, so that the target data is updated, and the application scene is expanded.
And the storage module 303 is configured to store the target data in the updated target data set to a local cache.
The data in the local cache is fast in reading speed, and the access quantity which can be simultaneously accommodated is large. The target data (hot key) is stored in the local cache, so that the cache breakdown phenomenon can be effectively avoided, and the system maintenance cost is reduced.
Further, according to an embodiment of the present invention, the data processing apparatus 300 further includes an update information obtaining module, configured to:
and acquiring the update information corresponding to the target data set, and storing the update information.
The updated information of each target data in the target data set is stored, so that the subsequent data display, data analysis and operation and maintenance are convenient; the preset conditions can be adjusted according to the update information corresponding to the target data, so that the real-time target data can be accurately determined, the user experience is further improved, and the data processing efficiency is improved.
Preferably, according to an embodiment of the present invention, the data processing apparatus 300 further includes a elimination module, configured to:
Setting a destiny strategy, and carrying out timing destiny on target data stored in a local buffer according to the destiny strategy.
The elimination strategy can include one or more of elimination period, minimum inquiry rate, priority elimination of the first added data when the cache space is insufficient, and the like, so that the local cache space is effectively released through the arrangement, and the flexibility of target data stored in the local cache is ensured.
According to the technical scheme of the embodiment of the invention, the query information corresponding to the data to be acquired in the query period is acquired; updating the target data set according to the query information; the target data set is constructed according to query information corresponding to the historical data; the technical means of storing the target data in the updated target data set into the local cache is overcome, so that the technical effects of poor cache breakdown solving effect, less applicable scenes and poor user experience existing in the existing data processing method due to hysteresis of the set hot key (namely the target data) are overcome, the target data are updated in real time, the cache breakdown phenomenon is effectively avoided, the applicable scenes are expanded, and the user experience is improved.
FIG. 4 illustrates an exemplary system architecture 400 in which a data processing method or data processing apparatus of an embodiment of the present invention may be applied.
As shown in fig. 4, a system architecture 400 may include terminal devices 401, 402, 403, a network 404, and a server 405 (this architecture is merely an example, and the components contained in a particular architecture may be tailored to the application specific case). The network 404 is used as a medium to provide communication links between the terminal devices 401, 402, 403 and the server 405. The network 404 may include various connection types, such as wired, wireless communication links, or fiber optic cables, among others.
A user may interact with the server 405 via the network 404 using the terminal devices 401, 402, 403 to receive or send messages or the like. Various communication client applications, such as a data processing class application, a browser application, a search class application, an instant messaging tool, a mailbox client, social platform software, etc., may be installed on the terminal devices 401, 402, 403, as just examples.
The terminal devices 401, 402, 403 may be various electronic devices having a display screen and supporting web browsing, including but not limited to smartphones, tablets, laptop and desktop computers, and the like.
The server 405 may be a server providing various services, such as a server (by way of example only) that utilizes terminal devices 401, 402, 403 (for testing/data processing) for a user. The server may perform analysis and other processing on the received data such as the query information, and feedback the processing result (e.g., the updated target data set—only an example) to the terminal device.
It should be noted that, the data processing method provided in the embodiment of the present invention is generally executed by the server 405, and accordingly, the data processing apparatus is generally disposed in the server 405.
It should be understood that the number of terminal devices, networks and servers in fig. 4 is merely illustrative. There may be any number of terminal devices, networks, and servers, as desired for implementation.
Referring now to FIG. 5, there is illustrated a schematic diagram of a computer system 500 suitable for use in implementing a terminal device or server in accordance with an embodiment of the present invention. The terminal device or server shown in fig. 5 is only an example, and should not impose any limitation on the functions and scope of use of the embodiments of the present invention.
As shown in fig. 5, the computer system 500 includes a Central Processing Unit (CPU) 501, which can perform various appropriate actions and processes according to a program stored in a Read Only Memory (ROM) 502 or a program loaded from a storage section 508 into a Random Access Memory (RAM) 503. In the RAM 503, various programs and data required for the operation of the system 500 are also stored. The CPU 501, ROM 502, and RAM 503 are connected to each other through a bus 504. An input/output (I/O) interface 505 is also connected to bus 504.
The following components are connected to the I/O interface 505: an input section 506 including a keyboard, a mouse, and the like; an output portion 507 including a Cathode Ray Tube (CRT), a Liquid Crystal Display (LCD), and the like, and a speaker, and the like; a storage portion 508 including a hard disk and the like; and a communication section 509 including a network interface card such as a LAN card, a modem, or the like. The communication section 509 performs communication processing via a network such as the internet. The drive 510 is also connected to the I/O interface 505 as needed. A removable medium 511 such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, or the like is mounted on the drive 510 as needed so that a computer program read therefrom is mounted into the storage section 508 as needed.
In particular, according to embodiments of the present disclosure, the processes described above with reference to flowcharts may be implemented as computer software programs. For example, embodiments of the present disclosure include a computer program product comprising a computer program embodied on a computer readable medium, the computer program comprising program code for performing the method shown in the flow chart. In such an embodiment, the computer program may be downloaded and installed from a network via the communication portion 509, and/or installed from the removable media 511. The above-described functions defined in the system of the present invention are performed when the computer program is executed by a Central Processing Unit (CPU) 501.
The computer readable medium shown in the present invention may be a computer readable signal medium or a computer readable storage medium, or any combination of the two. The computer readable storage medium can be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or a combination of any of the foregoing. More specific examples of the computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer diskette, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. In the context of this document, a computer readable storage medium may be any tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device. In the present invention, however, the computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, with the computer-readable program code embodied therein. Such a propagated data signal may take any of a variety of forms, including, but not limited to, electro-magnetic, optical, or any suitable combination of the foregoing. A computer readable signal medium may also be any computer readable medium that is not a computer readable storage medium and that can communicate, propagate, or transport a program for use by or in connection with an instruction execution system, apparatus, or device. Program code embodied on a computer readable medium may be transmitted using any appropriate medium, including but not limited to: wireless, wire, fiber optic cable, RF, etc., or any suitable combination of the foregoing.
The flowcharts and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods and computer program products according to various embodiments of the present invention. In this regard, each block in the flowchart or block diagrams may represent a module, segment, or portion of code, which comprises one or more executable instructions for implementing the specified logical function(s). It should also be noted that, in some alternative implementations, the functions noted in the block may occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams or flowchart illustration, and combinations of blocks in the block diagrams or flowchart illustration, can be implemented by special purpose hardware-based systems which perform the specified functions or acts, or combinations of special purpose hardware and computer instructions.
The modules involved in the embodiments of the present invention may be implemented in software or in hardware. The described modules may also be provided in a processor, for example, as: a processor includes an information acquisition module, an update module, and a storage module. The names of these modules do not in any way limit the module itself, and for example, the information acquisition module may also be described as "a module for acquiring query information corresponding to data to be acquired in a query period".
As another aspect, the present invention also provides a computer-readable medium that may be contained in the apparatus described in the above embodiments; or may be present alone without being fitted into the device. The computer readable medium carries one or more programs which, when executed by a device, cause the device to include: acquiring query information corresponding to the data to be acquired in a query period; updating the target data set according to the query information; the target data set is constructed according to query information corresponding to the historical data; and storing the target data in the updated target data set to a local cache.
According to the technical scheme of the embodiment of the invention, the query information corresponding to the data to be acquired in the query period is acquired; updating the target data set according to the query information; the target data set is constructed according to query information corresponding to the historical data; the technical means of storing the target data in the updated target data set into the local cache is overcome, so that the technical effects of poor cache breakdown solving effect, less applicable scenes and poor user experience existing in the existing data processing method due to hysteresis of the set hot key (namely the target data) are overcome, the target data are updated in real time, the cache breakdown phenomenon is effectively avoided, the applicable scenes are expanded, and the user experience is improved.
The above embodiments do not limit the scope of the present invention. It will be apparent to those skilled in the art that various modifications, combinations, sub-combinations and alternatives can occur depending upon design requirements and other factors. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the scope of the present invention.
Claims (9)
1. A method of data processing, comprising:
acquiring query information corresponding to the data to be acquired in a query period; the query information comprises query quantity and query rate;
Acquiring an application number corresponding to a data acquisition request initiated by a user, and updating a preset condition corresponding to the application number according to a service requirement corresponding to the application number;
Updating the target data set according to the query information and the updated preset conditions; the target data set is constructed according to query information corresponding to historical data;
and storing the target data in the updated target data set to a local cache.
2. The data processing method according to claim 1, wherein the constructing step of the target data set includes:
Acquiring query information corresponding to historical data, wherein the query information comprises query quantity and query time;
And sequencing the historical data according to the query information, determining target data according to a sequencing result and preset conditions, and constructing the target data set.
3. The data processing method according to claim 1, characterized by further comprising:
responding to a data acquisition request initiated by a user based on the data to be acquired, and judging whether the data to be acquired is target data;
If yes, acquiring the data to be acquired from a local cache; if not, the data to be acquired is acquired from the database.
4. The data processing method according to claim 3, wherein the updating the target data set according to the query information and the updated preset condition further comprises:
Monitoring the query quantity and the query rate corresponding to the data to be acquired, which are acquired from the database, and determining first data according to a first query quantity threshold value and a first query rate threshold value;
reporting query information corresponding to the first data;
And updating the target data set according to the query information corresponding to the reported first data and the preset condition.
5. The data processing method according to claim 3, wherein the storing the key value pair list corresponding to the target data in the local cache, the determining whether the data to be acquired is the target data, further comprises:
And traversing the key value pair list according to the primary key corresponding to the data to be acquired so as to judge whether the data to be acquired is target data.
6. The data processing method according to claim 1, characterized by further comprising:
And acquiring the update information corresponding to the target data set, and storing the update information.
7. A data processing apparatus, comprising:
the information acquisition module is used for acquiring query information corresponding to the data to be acquired in the query period; the query information comprises query quantity and query rate;
The updating module is used for acquiring an application number corresponding to a data acquisition request initiated by a user, updating a preset condition corresponding to the application number according to a service requirement corresponding to the application number, and updating a target data set according to the query information and the updated preset condition; the target data set is constructed according to query information corresponding to historical data;
And the storage module is used for storing the target data in the updated target data set to the local cache.
8. An electronic device, comprising:
one or more processors;
Storage means for storing one or more programs,
When executed by the one or more processors, causes the one or more processors to implement the method of any of claims 1-6.
9. A computer readable medium, on which a computer program is stored, characterized in that the program, when being executed by a processor, implements the method according to any of claims 1-6.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110062487.6A CN113760982B (en) | 2021-01-18 | 2021-01-18 | Data processing method and device |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110062487.6A CN113760982B (en) | 2021-01-18 | 2021-01-18 | Data processing method and device |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN113760982A CN113760982A (en) | 2021-12-07 |
| CN113760982B true CN113760982B (en) | 2024-05-17 |
Family
ID=78786473
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202110062487.6A Active CN113760982B (en) | 2021-01-18 | 2021-01-18 | Data processing method and device |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN113760982B (en) |
Families Citing this family (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN114969107A (en) * | 2022-05-07 | 2022-08-30 | 广东联想懂的通信有限公司 | Data processing method, device and equipment for multi-label object and storage medium |
| CN115840770B (en) * | 2023-02-10 | 2023-05-05 | 广东徐工汉云工业互联网有限公司 | Local cache data processing method and related equipment based on distributed environment |
Citations (13)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN106202581A (en) * | 2016-08-30 | 2016-12-07 | 北京奇艺世纪科技有限公司 | A kind of data search method and device |
| CN107918617A (en) * | 2016-10-10 | 2018-04-17 | 北京京东尚科信息技术有限公司 | Data query method and apparatus |
| CN108132958A (en) * | 2016-12-01 | 2018-06-08 | 阿里巴巴集团控股有限公司 | A kind of multi-level buffer data storage, inquiry, scheduling and processing method and processing device |
| CN109101580A (en) * | 2018-07-20 | 2018-12-28 | 北京北信源信息安全技术有限公司 | A kind of hot spot data caching method and device based on Redis |
| CN109656958A (en) * | 2018-12-18 | 2019-04-19 | 北京小米移动软件有限公司 | Data query method and system |
| CN109657174A (en) * | 2018-12-25 | 2019-04-19 | 北京百度网讯科技有限公司 | Method and apparatus for more new data |
| CN109684381A (en) * | 2018-12-20 | 2019-04-26 | 恒生电子股份有限公司 | Data hot statistics method and device |
| CN109885574A (en) * | 2019-02-22 | 2019-06-14 | 广州荔支网络技术有限公司 | A kind of data query method and device |
| CN110598138A (en) * | 2018-06-12 | 2019-12-20 | 北京京东尚科信息技术有限公司 | Cache-based processing method and device |
| CN110990372A (en) * | 2019-11-06 | 2020-04-10 | 苏宁云计算有限公司 | Dimensional data processing method and device and data query method and device |
| CN111259034A (en) * | 2020-02-17 | 2020-06-09 | 腾讯科技（深圳）有限公司 | Cache data management method and related device |
| CN111324615A (en) * | 2018-12-13 | 2020-06-23 | 北京京东尚科信息技术有限公司 | Data processing method, device, medium and electronic equipment |
| CN112506973A (en) * | 2020-12-14 | 2021-03-16 | 中国银联股份有限公司 | Method and device for managing stored data |
-
2021
- 2021-01-18 CN CN202110062487.6A patent/CN113760982B/en active Active
Patent Citations (13)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN106202581A (en) * | 2016-08-30 | 2016-12-07 | 北京奇艺世纪科技有限公司 | A kind of data search method and device |
| CN107918617A (en) * | 2016-10-10 | 2018-04-17 | 北京京东尚科信息技术有限公司 | Data query method and apparatus |
| CN108132958A (en) * | 2016-12-01 | 2018-06-08 | 阿里巴巴集团控股有限公司 | A kind of multi-level buffer data storage, inquiry, scheduling and processing method and processing device |
| CN110598138A (en) * | 2018-06-12 | 2019-12-20 | 北京京东尚科信息技术有限公司 | Cache-based processing method and device |
| CN109101580A (en) * | 2018-07-20 | 2018-12-28 | 北京北信源信息安全技术有限公司 | A kind of hot spot data caching method and device based on Redis |
| CN111324615A (en) * | 2018-12-13 | 2020-06-23 | 北京京东尚科信息技术有限公司 | Data processing method, device, medium and electronic equipment |
| CN109656958A (en) * | 2018-12-18 | 2019-04-19 | 北京小米移动软件有限公司 | Data query method and system |
| CN109684381A (en) * | 2018-12-20 | 2019-04-26 | 恒生电子股份有限公司 | Data hot statistics method and device |
| CN109657174A (en) * | 2018-12-25 | 2019-04-19 | 北京百度网讯科技有限公司 | Method and apparatus for more new data |
| CN109885574A (en) * | 2019-02-22 | 2019-06-14 | 广州荔支网络技术有限公司 | A kind of data query method and device |
| CN110990372A (en) * | 2019-11-06 | 2020-04-10 | 苏宁云计算有限公司 | Dimensional data processing method and device and data query method and device |
| CN111259034A (en) * | 2020-02-17 | 2020-06-09 | 腾讯科技（深圳）有限公司 | Cache data management method and related device |
| CN112506973A (en) * | 2020-12-14 | 2021-03-16 | 中国银联股份有限公司 | Method and device for managing stored data |
Also Published As
| Publication number | Publication date |
|---|---|
| CN113760982A (en) | 2021-12-07 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN110598138A (en) | Cache-based processing method and device | |
| CN113760982B (en) | Data processing method and device | |
| CN107426336B (en) | Method and device for adjusting push message opening rate | |
| CN108764866B (en) | Method and equipment for allocating resources and drawing resources | |
| CN112118352B (en) | Method and device for processing notification trigger message, electronic equipment and computer readable medium | |
| CN113364887A (en) | File downloading method based on FTP, proxy server and system | |
| CN111831503B (en) | Monitoring method based on monitoring agent and monitoring agent device | |
| CN113360348B (en) | Abnormal request processing method and device, electronic equipment and storage medium | |
| KR101810180B1 (en) | Method and apparatus for distributed processing of big data based on user equipment | |
| CN112910855B (en) | Sample message processing method and device | |
| CN113138943B (en) | Method and device for processing request | |
| CN112749204B (en) | Method and device for reading data | |
| CN112866179B (en) | Current limiting method and current limiting device | |
| CN112688982B (en) | User request processing method and device | |
| CN113722193A (en) | Method and device for detecting page abnormity | |
| CN113238919A (en) | Statistical method, device and system for user access number | |
| CN109213815B (en) | Method, device, server terminal and readable medium for controlling execution times | |
| CN110019671B (en) | Method and system for processing real-time message | |
| CN108984431B (en) | Method and apparatus for flushing stale caches | |
| CN111581930A (en) | Online form data processing method and device, electronic equipment and readable medium | |
| CN113778909B (en) | Method and device for caching data | |
| CN113362097B (en) | User determination method and device | |
| CN112783716B (en) | Monitoring method and device | |
| CN113760972B (en) | Data searching method and device | |
| CN112307060B (en) | Method and device for processing picking task |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |