CN113760982A - Data processing method and device - Google Patents
Data processing method and device Download PDFInfo
- Publication number
- CN113760982A CN113760982A CN202110062487.6A CN202110062487A CN113760982A CN 113760982 A CN113760982 A CN 113760982A CN 202110062487 A CN202110062487 A CN 202110062487A CN 113760982 A CN113760982 A CN 113760982A
- Authority
- CN
- China
- Prior art keywords
- data
- target data
- query
- acquired
- query information
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
- 238000003672 processing method Methods 0.000 title claims abstract description 31
- 238000000034 method Methods 0.000 claims abstract description 19
- 238000012545 processing Methods 0.000 claims description 21
- 238000004590 computer program Methods 0.000 claims description 9
- 238000012544 monitoring process Methods 0.000 claims description 5
- 230000015556 catabolic process Effects 0.000 abstract description 25
- 230000008030 elimination Effects 0.000 description 18
- 238000003379 elimination reaction Methods 0.000 description 18
- 238000010586 diagram Methods 0.000 description 15
- 230000000694 effects Effects 0.000 description 15
- 230000006870 function Effects 0.000 description 8
- 238000004891 communication Methods 0.000 description 7
- 238000012423 maintenance Methods 0.000 description 6
- 238000012163 sequencing technique Methods 0.000 description 6
- 230000003287 optical effect Effects 0.000 description 4
- 238000007405 data analysis Methods 0.000 description 3
- 230000004048 modification Effects 0.000 description 3
- 238000012986 modification Methods 0.000 description 3
- 230000002085 persistent effect Effects 0.000 description 3
- 230000008569 process Effects 0.000 description 3
- 238000012360 testing method Methods 0.000 description 3
- 230000008901 benefit Effects 0.000 description 2
- 239000000835 fiber Substances 0.000 description 2
- 230000000644 propagated effect Effects 0.000 description 2
- 239000004065 semiconductor Substances 0.000 description 2
- 230000000295 complement effect Effects 0.000 description 1
- 238000010276 construction Methods 0.000 description 1
- 230000003247 decreasing effect Effects 0.000 description 1
- 238000013461 design Methods 0.000 description 1
- 238000001514 detection method Methods 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 238000005111 flow chemistry technique Methods 0.000 description 1
- 230000006872 improvement Effects 0.000 description 1
- 230000003993 interaction Effects 0.000 description 1
- 239000004973 liquid crystal related substance Substances 0.000 description 1
- 230000007246 mechanism Effects 0.000 description 1
- 239000013307 optical fiber Substances 0.000 description 1
- 230000035755 proliferation Effects 0.000 description 1
- 230000002441 reversible effect Effects 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/245—Query processing
- G06F16/2455—Query execution
- G06F16/24552—Database cache management
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/23—Updating
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/27—Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
- Y02D10/00—Energy efficient computing, e.g. low power processors, power management or thermal management
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Databases & Information Systems (AREA)
- Data Mining & Analysis (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Computational Linguistics (AREA)
- Computing Systems (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention discloses a data processing method and device, and relates to the technical field of computers. One embodiment of the method comprises: acquiring corresponding query information of data to be acquired in a query period; updating the target data set according to the query information; the target data set is constructed according to query information corresponding to historical data; and storing the target data in the updated target data set to a local cache. According to the implementation mode, the target data is updated in real time, the cache breakdown phenomenon is effectively solved, the applicable scene is expanded, and the user experience is improved.
Description
Technical Field
The present invention relates to the field of computer technologies, and in particular, to a data processing method and apparatus.
Background
In the case of a cache mechanism, when a large number of requests come in, the cache fails at a certain limit time, so that all the requests bypass the cache and are sent to a background database or other devices frequently operated by I/O (Input/Output), and a system downtime is caused (also called cache breakdown). With the proliferation of internet users, how to avoid cache breakdown is an urgent problem to be solved.
The prior art has at least the following problems:
in the existing data processing method, the set hot key (namely target data) has hysteresis, so that the effect of solving cache breakdown is poor, the applicable scenes are few, and the user experience is poor.
Disclosure of Invention
In view of this, embodiments of the present invention provide a data processing method and apparatus, which can effectively avoid a cache breakdown phenomenon, expand an applicable scenario, and improve user experience by updating target data in real time.
To achieve the above object, according to a first aspect of embodiments of the present invention, there is provided a data processing method including:
acquiring corresponding query information of data to be acquired in a query period;
updating the target data set according to the query information; the target data set is constructed according to query information corresponding to historical data;
and storing the target data in the updated target data set to a local cache.
Further, the constructing of the target data set comprises:
acquiring query information corresponding to historical data, wherein the query information comprises query quantity and query time;
and sequencing the historical data according to the query information, determining target data according to a sequencing result and a preset condition, and constructing a target data set.
Further, still include:
responding to a data acquisition request initiated by a user based on data to be acquired, and judging whether the data to be acquired is target data;
if yes, acquiring data to be acquired from a local cache; and if not, acquiring the data to be acquired from the database.
Further, the query information includes query quantity and query rate; updating the target data set according to the query information, further comprising:
monitoring the query quantity and the query rate corresponding to the data to be acquired, which are acquired from the database, and determining first data according to a first query quantity threshold and a first query rate threshold;
reporting query information corresponding to the first data;
and updating the target data set according to the reported query information corresponding to the first data and preset conditions.
Further, storing the key-value pair list corresponding to the target data in a local cache, and determining whether the data to be acquired is the target data, the method further includes:
and traversing the key value pair list according to the main key corresponding to the data to be acquired so as to judge whether the data to be acquired is the target data.
Further, still include:
and acquiring an application number corresponding to the data acquisition request, updating a preset condition corresponding to the application number according to a service requirement corresponding to the application number, and updating target data in the target data set according to the updated preset condition.
Further, still include:
and acquiring the updating information corresponding to the target data set, and storing the updating information.
According to a second aspect of the embodiments of the present invention, there is provided a data processing apparatus including:
the information acquisition module is used for acquiring corresponding query information of the data to be acquired in a query period;
the updating module is used for updating the target data set according to the query information; the target data set is constructed according to query information corresponding to historical data;
and the storage module is used for storing the target data in the updated target data set to a local cache.
According to a third aspect of embodiments of the present invention, there is provided an electronic apparatus, including:
one or more processors;
a storage device for storing one or more programs,
when the one or more programs are executed by the one or more processors, the one or more processors are caused to implement any of the data processing methods described above.
According to a fourth aspect of embodiments of the present invention, there is provided a computer-readable medium on which a computer program is stored, the program, when executed by a processor, implementing any one of the data processing methods described above.
One embodiment of the above invention has the following advantages or benefits: the corresponding query information of the data to be acquired in the query period is acquired; updating the target data set according to the query information; the target data set is constructed according to query information corresponding to historical data; the technical means of storing the target data in the updated target data set to the local cache overcomes the technical problems that the effect of solving cache breakdown is poor, the applicable scenes are few and the user experience is poor due to hysteresis of the set hot key (namely the target data) in the existing data processing method, and further the target data is updated in real time, the cache breakdown phenomenon is effectively avoided, the applicable scenes are expanded, and the technical effect of the user experience is improved.
Further effects of the above-mentioned non-conventional alternatives will be described below in connection with the embodiments.
Drawings
The drawings are included to provide a better understanding of the invention and are not to be construed as unduly limiting the invention. Wherein:
fig. 1 is a schematic diagram of a main flow of a data processing method provided according to a first embodiment of the present invention;
FIG. 2a is a schematic diagram of a main flow of a data processing method according to a second embodiment of the present invention;
FIG. 2b is a schematic diagram illustrating a main process of determining whether the data to be acquired is the target data in the method illustrated in FIG. 2 a;
FIG. 2c is a schematic diagram of the main flow of data retrieval from a local cache or database in the method of FIG. 2 a;
FIG. 3 is a schematic diagram of the main blocks of a data processing apparatus according to an embodiment of the present invention;
FIG. 4 is an exemplary system architecture diagram in which embodiments of the present invention may be employed;
fig. 5 is a schematic block diagram of a computer system suitable for use in implementing a terminal device or server of an embodiment of the invention.
Detailed Description
Exemplary embodiments of the present invention are described below with reference to the accompanying drawings, in which various details of embodiments of the invention are included to assist understanding, and which are to be considered as merely exemplary. Accordingly, those of ordinary skill in the art will recognize that various changes and modifications of the embodiments described herein can be made without departing from the scope and spirit of the invention. Also, descriptions of well-known functions and constructions are omitted in the following description for clarity and conciseness.
Fig. 1 is a schematic diagram of a main flow of a data processing method provided according to a first embodiment of the present invention; as shown in fig. 1, the data processing method provided in the embodiment of the present invention mainly includes:
step S101, acquiring corresponding query information of the data to be acquired in a query period.
According to the embodiment of the invention, the Query information includes the corresponding Query quantity and Query time of each data to be acquired in the Query cycle, and the Query rate (such as QPS, Query Per Second, Query rate Per Second) of the data to be acquired in the latest period of time can be determined according to the information, so that the target data (hot key) can be determined more accurately, and the cache breakdown phenomenon is effectively avoided.
The hot key refers to a large number of requests (specific data is set according to the carrying capacity of the accessed database (such as redis, MySQL, and the like)) in a short time to access a specific key (data) in the database. At this time, the access amount is too concentrated, and reaches the upper limit of the physical network card, which often causes the system to be down.
Specifically, according to an embodiment of the present invention, the method further includes:
responding to a data acquisition request initiated by a user based on data to be acquired, and judging whether the data to be acquired is target data;
if yes, acquiring data to be acquired from a local cache; and if not, acquiring the data to be acquired from the database.
Through the arrangement, after a data acquisition request initiated by a user based on an application is received, whether the data to be acquired indicated by the data acquisition request belongs to a hot key is firstly judged, and if yes, the data to be acquired is directly sent to a local cache for acquisition; if the data does not belong to the hot key, the data is not stored in the local cache, and then the data is acquired in the database. It should be noted that the database may be a secondary cache database (e.g., redis) or a persistent database (e.g., MySQL). Through the setting, the hot keys are directly acquired from the local cache, so that the cache breakdown phenomenon is effectively avoided, and the user experience is improved.
Further, according to the embodiment of the present invention, the storing the key-value pair list corresponding to the target data in the local cache, and determining whether the data to be acquired is the target data, further includes:
and traversing the key value pair list according to the main key corresponding to the data to be acquired so as to judge whether the data to be acquired is the target data.
Specifically, for the stored data (whether stored in the local cache or the database), a key-value pair list corresponding to the stored data is constructed according to the key-value pairs, and a specific key-value pair list is constructed for the target data (hot key) and stored in the local cache.
Step S102, updating the target data set according to the query information; and the target data set is constructed according to the query information corresponding to the historical data.
Through the arrangement, the target data set is updated according to the query information corresponding to the data to be acquired, the target data is updated according to the real-time information, the target data can be monitored and determined in real time, the real-time performance of the target data in the target data set is guaranteed, the cache breakdown phenomenon is effectively avoided, and the applicable scene is expanded.
Further, according to an embodiment of the present invention, the constructing of the target data set includes:
acquiring query information corresponding to historical data, wherein the query information comprises query quantity and query time;
and sequencing the historical data according to the query information, determining target data according to a sequencing result and a preset condition, and constructing a target data set.
Through the arrangement, the target data is determined according to the query information corresponding to the historical data, and the target data is more representative in a longer time range. Meanwhile, according to the target data determined by the query information corresponding to the historical data, a preset condition can be formulated in a targeted manner, and which type and the query quantity reaching the large quantity can be judged in advance (namely, the preset condition is specifically set differently according to time, sales promotion activities, sales promotion products and the like).
Preferably, according to the embodiment of the present invention, the query information includes a query number and a query rate; updating the target data set according to the query information, further comprising:
monitoring the query quantity and the query rate corresponding to the data to be acquired, which are acquired from the database, and determining first data according to a first query quantity threshold and a first query rate threshold;
reporting query information corresponding to the first data;
and updating the target data set according to the reported query information corresponding to the first data and preset conditions.
Specifically, according to the embodiment of the present invention, if the data to be acquired is acquired from the database, that is, the data to be acquired is not the target data when the acquisition request is initiated, the query number and the query rate of the data to be acquired are determined according to the first query number threshold and the first query rate threshold, so as to update the target data, further ensure the real-time performance of the target data stored in the local cache, and improve the user experience.
And step S103, storing the target data in the updated target data set to a local cache.
The data in the local cache is fast to read, and the access quantity which can be accommodated simultaneously is large. The target data (hot key) is stored in the local cache, so that the cache breakdown phenomenon can be effectively avoided, and the system maintenance cost is reduced.
Further, according to an embodiment of the present invention, the method further includes:
and acquiring an application number corresponding to the data acquisition request, updating a preset condition corresponding to the application number according to a service requirement corresponding to the application number, and updating target data in the target data set according to the updated preset condition.
Through the setting, different preset conditions are set according to the business requirements corresponding to the application numbers aiming at the application numbers, so that the target data is updated, and the application scene of the data processing method is expanded.
Preferably, according to an embodiment of the present invention, the method further includes:
and acquiring the updating information corresponding to the target data set, and storing the updating information.
The updating information of each target data in the target data set is stored, so that the subsequent data display, data analysis and operation and maintenance are facilitated; the preset conditions can be adjusted according to the updating information corresponding to the target data, so that the real-time target data can be accurately determined, the user experience is further improved, and the data processing efficiency is improved.
Illustratively, according to an embodiment of the present invention, the method further includes:
and setting a elimination strategy, and carrying out timed elimination on the target data stored in the local buffer according to the elimination strategy.
The elimination strategy can comprise one or more of an elimination period, the lowest limit of the query rate, the prior elimination of the data added firstly when the cache space is insufficient, and the like, and the local cache space is effectively released through the setting, so that the flexibility of the target data stored in the local cache is ensured.
According to the technical scheme of the embodiment of the invention, the query information corresponding to the data to be acquired in the query period is acquired; updating the target data set according to the query information; the target data set is constructed according to query information corresponding to historical data; the technical means of storing the target data in the updated target data set to the local cache overcomes the technical problems that the effect of solving cache breakdown is poor, the applicable scenes are few and the user experience is poor due to hysteresis of the set hot key (namely the target data) in the existing data processing method, and further the target data is updated in real time, the cache breakdown phenomenon is effectively avoided, the applicable scenes are expanded, and the technical effect of the user experience is improved.
FIG. 2a is a schematic diagram of a main flow of a data processing method according to a second embodiment of the present invention; as shown in fig. 2a, the data processing method provided in the embodiment of the present invention mainly includes:
step S201, obtaining query information corresponding to the historical data, where the query information includes query quantity and query time.
Specifically, according to the embodiment of the present invention, a hotspot data computing service cluster (such as a call distributed computing service, which is mainly used for dynamically increasing or decreasing a computing processing machine according to the quantity of a QPS (Query Per Second, Query rate Per Second) to perform computation to determine hotspot data (i.e., a hot key), and simultaneously push a computation result to an application cluster) to obtain corresponding data.
Step S202, sorting the historical data according to the query information, determining target data according to a sorting result and preset conditions, and constructing a target data set.
Through the arrangement, the target data is determined according to the query information corresponding to the historical data, and the target data is more representative in a longer time range. Meanwhile, according to the target data determined by the query information corresponding to the historical data, a preset condition can be formulated in a targeted manner, and which type and the query quantity reaching the large quantity can be judged in advance (namely, the preset condition is specifically set differently according to time, sales promotion activities, sales promotion products and the like).
According to the embodiment of the invention, the preset conditions are set and stored through the built ETCD cluster (distributed key-value storage developed by CoreOS based on Raft, which can be used for service discovery, shared configuration and consistency guarantee), or the existing cluster capable of realizing the functions, such as zookeeper cluster.
Step S203, responding to a data acquisition request initiated by a user based on the data to be acquired, traversing the key value pair list according to a main key corresponding to the data to be acquired so as to judge whether the data to be acquired is target data; and storing the target data and the corresponding key-value pair list in a local cache.
Specifically, as shown in fig. 2b, according to a specific implementation manner of the embodiment of the present invention, an application is registered in an etc cluster, and a hotspot data computing service cluster (computing service for short) is long-connected through a Netty (which is a java open source framework provided by JBOSS and is now an independent item on Github), wherein the Netty provides an asynchronous, event-driven network application framework and a tool for rapidly developing a high-performance and high-reliability network server and a client program, and after the computing service is started, the computing service will register itself in the etc cluster service and start a Netty service port for application end connection and heartbeat detection, and at the same time, a timer is started, and a computing service List < String > is periodically pulled from the etc cluster d and then connected to a client of the computing service, and when the connection fails, the computing service will push out a prompt; until the connection is successful. After the connection success of the application to the computing service is ensured, the connected channel (channel) and the Netty address name are stored in a key-value form, so that the interaction between the data (mainly query information) corresponding to the application and the computing service is conveniently obtained subsequently. And starting a thread pool to send application heartbeats at regular time, updating the service of which the channel is not connected in real time, and storing the service in a form of CopyOnWriteArrayList < Map < Key, Value > >. After the application is started, the preset conditions stored in the ETCD cluster are stored in the current local cache (namely, the target data and the key-value pair list corresponding to the target data are stored in the local cache).
Step S204, if yes, acquiring data to be acquired from a local cache; and if not, acquiring the data to be acquired from the database.
Through the arrangement, after a data acquisition request initiated by a user based on an application is received, whether the data to be acquired indicated by the data acquisition request belongs to a hot key is firstly judged, and if yes, the data to be acquired is directly sent to a local cache for acquisition; if the data does not belong to the hot key, the data is not stored in the local cache, and then the data is acquired in the database. It should be noted that the database may be a secondary cache database (e.g., redis) or a persistent database (e.g., MySQL). Through the setting, the hot keys are directly acquired from the local cache, so that the cache breakdown phenomenon is effectively avoided, and the user experience is improved. Specifically, as shown in fig. 2c, wherein the hotspot rule is the preset condition, and redis refers to a cache database. When the user A searches for redis data according to the key, whether the key meets the hotspot rule is judged according to preset conditions set in the ETCD and determined target data, if yes, data is obtained from a local cache, and the local cache can use the cafeienecache. If the data to be acquired is near to expire or the data is not acquired in the local cache (namely the data is acquired from the database), counting statistics (counting statistics on the query information) is carried out, and the data is reported to the computing service at regular time.
Step S205, monitoring the query quantity and the query rate corresponding to the data to be acquired, which are acquired from the database, and determining first data according to a first query quantity threshold and a first query rate threshold;
according to a specific implementation manner of the embodiment of the invention, the application starts the thread watch (monitors) the query quantity and the query rate corresponding to the data to be acquired, and simultaneously monitors whether the preset condition and the target data set in the ETCD cluster are changed, if the preset condition is changed, the locally cached preset condition and the target data need to be updated immediately.
Step S206, reporting the query information corresponding to the first data.
The application starts a thread to report query information of target data accessing the cache in a certain period of time to a computing service (there is a local count, which is not required to be sent to the computing service every time, and if the query quantity corresponding to a certain data exceeds 10 or the query rate per second corresponding to a certain data exceeds 2 (the above numerical values are only examples), the query information corresponding to the data is reported to the computing service), so that the computing service can calculate whether the data is a hot key (target data), and then update the target data. According to a specific implementation manner of the embodiment of the present invention, when the query information is reported, MOD (complementary function) is obtained according to the data of the current computing service, so as to uniformly print the data to be computed on the computing service, thereby improving the computing efficiency.
And step S207, updating the target data set according to the query information corresponding to the reported first data and the preset condition.
Specifically, according to the embodiment of the present invention, if the data to be acquired is acquired from the database, that is, the data to be acquired is not the target data when the acquisition request is initiated, the query number and the query rate of the data to be acquired are determined according to the first query number threshold and the first query rate threshold, so as to update the target data, further ensure the real-time performance of the target data stored in the local cache, and improve the user experience.
According to a specific implementation manner of the embodiment of the present invention, after receiving query information statistical data of hotspot data reported by an application terminal, a computing service may directly make a key of a map according to an application name + key + a preset condition name, where value is a hotspot data volume (total query volume), and after determining updated target data when a certain data volume (a first query volume threshold or a first query rate threshold) is reached, call an etc cluster to write the updated target data into a key-value list in a cache period, and update the target data stored in a local cache. If the acquired test hotspot data sent by the application layer is acquired, whether the acquired test hotspot data is a hotspot is judged according to the sliding window, and within a certain thread number within a certain time, the data volume exceeds a certain amount, the data is the hotspot data, and of course, the time interval and the thread number are configurable.
And step S208, storing the target data in the updated target data set to a local cache.
The data in the local cache is fast to read, and the access quantity which can be accommodated simultaneously is large. The target data (hot key) is stored in the local cache, so that the cache breakdown phenomenon can be effectively avoided, and the system maintenance cost is reduced.
Further, according to an embodiment of the present invention, the method further includes:
and acquiring an application number corresponding to the data acquisition request, updating a preset condition corresponding to the application number according to a service requirement corresponding to the application number, and updating target data in the target data set according to the updated preset condition.
Through the setting, different preset conditions are set according to the business requirements corresponding to the application numbers aiming at the application numbers, so that the target data is updated, and the application scene of the data processing method is expanded.
Preferably, according to an embodiment of the present invention, the method further includes:
and acquiring the updating information corresponding to the target data set, and storing the updating information.
The updating information of each target data in the target data set is stored, so that the subsequent data display, data analysis and operation and maintenance are facilitated; the preset conditions can be adjusted according to the updating information corresponding to the target data, so that the real-time target data can be accurately determined, the user experience is further improved, and the data processing efficiency is improved.
Specifically, according to the embodiment of the invention, the Flink (an open source flow processing framework) can be utilized to analyze the update information about the target data set in the current ETCD cluster, perform data statistics of various dimensions on the update information in real time, and store the update information in the database so as to facilitate subsequent report display.
Illustratively, according to an embodiment of the present invention, the method further includes:
and setting a elimination strategy, and carrying out timed elimination on the target data stored in the local buffer according to the elimination strategy.
The elimination strategy can comprise one or more of an elimination period, the lowest limit of the query rate, the prior elimination of the data added firstly when the cache space is insufficient, and the like, and the local cache space is effectively released through the setting, so that the flexibility of the target data stored in the local cache is ensured.
According to the technical scheme of the embodiment of the invention, the query information corresponding to the data to be acquired in the query period is acquired; updating the target data set according to the query information; the target data set is constructed according to query information corresponding to historical data; the technical means of storing the target data in the updated target data set to the local cache overcomes the technical problems that the effect of solving cache breakdown is poor, the applicable scenes are few and the user experience is poor due to hysteresis of the set hot key (namely the target data) in the existing data processing method, and further the target data is updated in real time, the cache breakdown phenomenon is effectively avoided, the applicable scenes are expanded, and the technical effect of the user experience is improved.
FIG. 3 is a schematic diagram of the main blocks of a data processing apparatus according to an embodiment of the present invention; as shown in fig. 3, a data processing apparatus 300 according to an embodiment of the present invention mainly includes:
the information obtaining module 301 is configured to obtain query information corresponding to data to be obtained in a query period.
According to the embodiment of the invention, the Query information includes the corresponding Query quantity and Query time of each data to be acquired in the Query cycle, and the Query rate (such as QPS, Query Per Second, Query rate Per Second) of the data to be acquired in the latest period of time can be determined according to the information, so that the target data (hot key) can be determined more accurately, and the cache breakdown phenomenon is effectively avoided.
The hot key refers to a large number of requests (specific data is set according to the carrying capacity of the accessed database (such as redis, MySQL, and the like)) in a short time to access a specific key (data) in the database. At this time, the access amount is too concentrated, and reaches the upper limit of the physical network card, which often causes the system to be down.
Specifically, according to the embodiment of the present invention, the data processing apparatus 300 further includes a determining module, configured to:
responding to a data acquisition request initiated by a user based on data to be acquired, and judging whether the data to be acquired is target data;
if yes, acquiring data to be acquired from a local cache; and if not, acquiring the data to be acquired from the database.
Through the arrangement, after a data acquisition request initiated by a user based on an application is received, whether the data to be acquired indicated by the data acquisition request belongs to a hot key is firstly judged, and if yes, the data to be acquired is directly sent to a local cache for acquisition; if the data does not belong to the hot key, the data is not stored in the local cache, and then the data is acquired in the database. It should be noted that the database may be a secondary cache database (e.g., redis) or a persistent database (e.g., MySQL). Through the setting, the hot keys are directly acquired from the local cache, so that the cache breakdown phenomenon is effectively avoided, and the user experience is improved.
Further, according to the embodiment of the present invention, the key-value pair list corresponding to the target data is stored in a local cache; the judging module is further configured to:
and traversing the key value pair list according to the main key corresponding to the data to be acquired so as to judge whether the data to be acquired is the target data.
Specifically, for the stored data (whether stored in the local cache or the database), a key-value pair list corresponding to the stored data is constructed according to the key-value pairs, and a specific key-value pair list is constructed for the target data (hot key) and stored in the local cache.
An updating module 302, configured to update the target data set according to the query information; and the target data set is constructed according to the query information corresponding to the historical data.
Through the arrangement, the target data set is updated according to the query information corresponding to the data to be acquired, the target data is updated according to the real-time information, the target data can be monitored and determined in real time, the real-time performance of the target data in the target data set is guaranteed, the cache breakdown phenomenon is effectively avoided, and the applicable scene is expanded.
Further, according to the embodiment of the present invention, the data processing apparatus 300 further includes a target data set constructing module, configured to:
acquiring query information corresponding to historical data, wherein the query information comprises query quantity and query time;
and sequencing the historical data according to the query information, determining target data according to a sequencing result and a preset condition, and constructing a target data set.
Through the arrangement, the target data is determined according to the query information corresponding to the historical data, and the target data is more representative in a longer time range. Meanwhile, according to the target data determined by the query information corresponding to the historical data, a preset condition can be formulated in a targeted manner, and which type and the query quantity reaching the large quantity can be judged in advance (namely, the preset condition is specifically set differently according to time, sales promotion activities, sales promotion products and the like).
Preferably, according to an embodiment of the present invention, the update module 302 is further configured to:
monitoring the query quantity and the query rate corresponding to the data to be acquired, which are acquired from the database, and determining first data according to a first query quantity threshold and a first query rate threshold;
reporting query information corresponding to the first data;
and updating the target data set according to the reported query information corresponding to the first data and preset conditions.
Specifically, according to the embodiment of the present invention, if the data to be acquired is acquired from the database, that is, the data to be acquired is not the target data when the acquisition request is initiated, the query number and the query rate of the data to be acquired are determined according to the first query number threshold and the first query rate threshold, so as to update the target data, further ensure the real-time performance of the target data stored in the local cache, and improve the user experience.
Further, according to an embodiment of the present invention, the update module 302 is further configured to:
and acquiring an application number corresponding to the data acquisition request, updating a preset condition corresponding to the application number according to a service requirement corresponding to the application number, and updating target data in the target data set according to the updated preset condition.
Through the setting, different preset conditions are set according to the service requirements corresponding to the application numbers aiming at the application numbers, so that the target data is updated, and the applicable scene is expanded.
A storage module 303, configured to store the target data in the updated target data set to a local cache.
The data in the local cache is fast to read, and the access quantity which can be accommodated simultaneously is large. The target data (hot key) is stored in the local cache, so that the cache breakdown phenomenon can be effectively avoided, and the system maintenance cost is reduced.
Further, according to the embodiment of the present invention, the data processing apparatus 300 further includes an update information obtaining module, configured to:
and acquiring the updating information corresponding to the target data set, and storing the updating information.
The updating information of each target data in the target data set is stored, so that the subsequent data display, data analysis and operation and maintenance are facilitated; the preset conditions can be adjusted according to the updating information corresponding to the target data, so that the real-time target data can be accurately determined, the user experience is further improved, and the data processing efficiency is improved.
Preferably, according to the embodiment of the present invention, the data processing apparatus 300 further includes a eliminating module, configured to:
and setting a elimination strategy, and carrying out timed elimination on the target data stored in the local buffer according to the elimination strategy.
The elimination strategy can comprise one or more of an elimination period, the lowest limit of the query rate, the prior elimination of the data added firstly when the cache space is insufficient, and the like, and the local cache space is effectively released through the setting, so that the flexibility of the target data stored in the local cache is ensured.
According to the technical scheme of the embodiment of the invention, the query information corresponding to the data to be acquired in the query period is acquired; updating the target data set according to the query information; the target data set is constructed according to query information corresponding to historical data; the technical means of storing the target data in the updated target data set to the local cache overcomes the technical problems that the effect of solving cache breakdown is poor, the applicable scenes are few and the user experience is poor due to hysteresis of the set hot key (namely the target data) in the existing data processing method, and further the target data is updated in real time, the cache breakdown phenomenon is effectively avoided, the applicable scenes are expanded, and the technical effect of the user experience is improved.
Fig. 4 shows an exemplary system architecture 400 of a data processing method or data processing apparatus to which embodiments of the present invention may be applied.
As shown in fig. 4, the system architecture 400 may include terminal devices 401, 402, 403, a network 404, and a server 405 (this architecture is merely an example, and the components included in a particular architecture may be adapted according to application specific circumstances). The network 404 serves as a medium for providing communication links between the terminal devices 401, 402, 403 and the server 405. Network 404 may include various types of connections, such as wire, wireless communication links, or fiber optic cables, to name a few.
A user may use terminal devices 401, 402, 403 to interact with a server 405 over a network 404 to receive or send messages or the like. The terminal devices 401, 402, 403 may have installed thereon various communication client applications, such as a data processing-type application, a browser application, a search-type application, an instant messaging tool, a mailbox client, social platform software, etc. (by way of example only).
The terminal devices 401, 402, 403 may be various electronic devices having a display screen and supporting web browsing, including but not limited to smart phones, tablet computers, laptop portable computers, desktop computers, and the like.
The server 405 may be a server that provides various services, such as a server (for example only) that users use (perform testing/perform data processing on) the terminal devices 401, 402, 403. The server may analyze and perform other processing on the received data such as the query information, and feed back a processing result (e.g., an updated target data set — just an example) to the terminal device.
It should be noted that the data processing method provided by the embodiment of the present invention is generally executed by the server 405, and accordingly, the data processing apparatus is generally disposed in the server 405.
It should be understood that the number of terminal devices, networks, and servers in fig. 4 is merely illustrative. There may be any number of terminal devices, networks, and servers, as desired for implementation.
Referring now to FIG. 5, a block diagram of a computer system 500 suitable for use with a terminal device or server implementing an embodiment of the invention is shown. The terminal device or the server shown in fig. 5 is only an example, and should not bring any limitation to the functions and the scope of use of the embodiments of the present invention.
As shown in fig. 5, the computer system 500 includes a Central Processing Unit (CPU)501 that can perform various appropriate actions and processes according to a program stored in a Read Only Memory (ROM)502 or a program loaded from a storage section 508 into a Random Access Memory (RAM) 503. In the RAM 503, various programs and data necessary for the operation of the system 500 are also stored. The CPU 501, ROM 502, and RAM 503 are connected to each other via a bus 504. An input/output (I/O) interface 505 is also connected to bus 504.
The following components are connected to the I/O interface 505: an input portion 506 including a keyboard, a mouse, and the like; an output portion 507 including a display such as a Cathode Ray Tube (CRT), a Liquid Crystal Display (LCD), and the like, and a speaker; a storage portion 508 including a hard disk and the like; and a communication section 509 including a network interface card such as a LAN card, a modem, or the like. The communication section 509 performs communication processing via a network such as the internet. The driver 510 is also connected to the I/O interface 505 as necessary. A removable medium 511 such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, or the like is mounted on the drive 510 as necessary, so that a computer program read out therefrom is mounted into the storage section 508 as necessary.
In particular, according to the embodiments of the present disclosure, the processes described above with reference to the flowcharts may be implemented as computer software programs. For example, embodiments of the present disclosure include a computer program product comprising a computer program embodied on a computer readable medium, the computer program comprising program code for performing the method illustrated in the flow chart. In such an embodiment, the computer program may be downloaded and installed from a network through the communication section 509, and/or installed from the removable medium 511. The computer program performs the above-described functions defined in the system of the present invention when executed by the Central Processing Unit (CPU) 501.
It should be noted that the computer readable medium shown in the present invention can be a computer readable signal medium or a computer readable storage medium or any combination of the two. A computer readable storage medium may be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination of the foregoing. More specific examples of the computer readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer diskette, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. In the present invention, a computer readable storage medium may be any tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device. In the present invention, however, a computer readable signal medium may include a propagated data signal with computer readable program code embodied therein, for example, in baseband or as part of a carrier wave. Such a propagated data signal may take many forms, including, but not limited to, electro-magnetic, optical, or any suitable combination thereof. A computer readable signal medium may also be any computer readable medium that is not a computer readable storage medium and that can communicate, propagate, or transport a program for use by or in connection with an instruction execution system, apparatus, or device. Program code embodied on a computer readable medium may be transmitted using any appropriate medium, including but not limited to: wireless, wire, fiber optic cable, RF, etc., or any suitable combination of the foregoing.
The flowchart and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods and computer program products according to various embodiments of the present invention. In this regard, each block in the flowchart or block diagrams may represent a module, segment, or portion of code, which comprises one or more executable instructions for implementing the specified logical function(s). It should also be noted that, in some alternative implementations, the functions noted in the block may occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams or flowchart illustration, and combinations of blocks in the block diagrams or flowchart illustration, can be implemented by special purpose hardware-based systems which perform the specified functions or acts, or combinations of special purpose hardware and computer instructions.
The modules described in the embodiments of the present invention may be implemented by software or hardware. The described modules may also be provided in a processor, which may be described as: a processor includes an information acquisition module, an update module, and a storage module. The names of the modules do not form a limitation on the modules themselves in some cases, for example, the information acquisition module may also be described as a "module for acquiring query information corresponding to data to be acquired in a query period".
As another aspect, the present invention also provides a computer-readable medium that may be contained in the apparatus described in the above embodiments; or may be separate and not incorporated into the device. The computer readable medium carries one or more programs which, when executed by a device, cause the device to comprise: acquiring corresponding query information of data to be acquired in a query period; updating the target data set according to the query information; the target data set is constructed according to query information corresponding to historical data; and storing the target data in the updated target data set to a local cache.
According to the technical scheme of the embodiment of the invention, the query information corresponding to the data to be acquired in the query period is acquired; updating the target data set according to the query information; the target data set is constructed according to query information corresponding to historical data; the technical means of storing the target data in the updated target data set to the local cache overcomes the technical problems that the effect of solving cache breakdown is poor, the applicable scenes are few and the user experience is poor due to hysteresis of the set hot key (namely the target data) in the existing data processing method, and further the target data is updated in real time, the cache breakdown phenomenon is effectively avoided, the applicable scenes are expanded, and the technical effect of the user experience is improved.
The above-described embodiments should not be construed as limiting the scope of the invention. Those skilled in the art will appreciate that various modifications, combinations, sub-combinations, and substitutions can occur, depending on design requirements and other factors. Any modification, equivalent replacement, and improvement made within the spirit and principle of the present invention should be included in the protection scope of the present invention.
Claims (10)
1. A data processing method, comprising:
acquiring corresponding query information of data to be acquired in a query period;
updating a target data set according to the query information; the target data set is constructed according to query information corresponding to historical data;
and storing the target data in the updated target data set to a local cache.
2. The data processing method of claim 1, wherein the step of constructing the target data set comprises:
acquiring query information corresponding to historical data, wherein the query information comprises query quantity and query time;
and sorting the historical data according to the query information, determining target data according to a sorting result and preset conditions, and constructing the target data set.
3. The data processing method of claim 1, further comprising:
responding to a data acquisition request initiated by a user based on the data to be acquired, and judging whether the data to be acquired is target data;
if yes, acquiring the data to be acquired from a local cache; and if not, acquiring the data to be acquired from the database.
4. The data processing method of claim 3, wherein the query information includes a query number and a query rate; the updating the target data set according to the query information further comprises:
monitoring the query quantity and the query rate corresponding to the data to be acquired, which are acquired from the database, and determining first data according to a first query quantity threshold and a first query rate threshold;
reporting query information corresponding to the first data;
and updating the target data set according to the reported query information corresponding to the first data and the preset condition.
5. The data processing method according to claim 3, wherein storing the key-value pair list corresponding to the target data in a local cache, and determining whether the data to be acquired is the target data further comprises:
and traversing the key-value pair list according to the primary key corresponding to the data to be acquired so as to judge whether the data to be acquired is the target data.
6. The data processing method of claim 3, further comprising:
the method comprises the steps of obtaining an application number corresponding to a data obtaining request, updating a preset condition corresponding to the application number according to a service requirement corresponding to the application number, and updating target data in a target data set according to the updated preset condition.
7. The data processing method of claim 1, further comprising:
and acquiring the updating information corresponding to the target data set, and storing the updating information.
8. A data processing apparatus, comprising:
the information acquisition module is used for acquiring corresponding query information of the data to be acquired in a query period;
the updating module is used for updating the target data set according to the query information; the target data set is constructed according to query information corresponding to historical data;
and the storage module is used for storing the target data in the updated target data set to a local cache.
9. An electronic device, comprising:
one or more processors;
a storage device for storing one or more programs,
when executed by the one or more processors, cause the one or more processors to implement the method of any one of claims 1-7.
10. A computer-readable medium, on which a computer program is stored, which, when being executed by a processor, carries out the method according to any one of claims 1-7.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110062487.6A CN113760982B (en) | 2021-01-18 | 2021-01-18 | Data processing method and device |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110062487.6A CN113760982B (en) | 2021-01-18 | 2021-01-18 | Data processing method and device |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN113760982A true CN113760982A (en) | 2021-12-07 |
| CN113760982B CN113760982B (en) | 2024-05-17 |
Family
ID=78786473
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202110062487.6A Active CN113760982B (en) | 2021-01-18 | 2021-01-18 | Data processing method and device |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN113760982B (en) |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN114969107A (en) * | 2022-05-07 | 2022-08-30 | 广东联想懂的通信有限公司 | Data processing method, device and equipment for multi-label object and storage medium |
| CN115840770A (en) * | 2023-02-10 | 2023-03-24 | 广东徐工汉云工业互联网有限公司 | Local cache data processing method and related equipment based on distributed environment |
Citations (13)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN106202581A (en) * | 2016-08-30 | 2016-12-07 | 北京奇艺世纪科技有限公司 | A kind of data search method and device |
| CN107918617A (en) * | 2016-10-10 | 2018-04-17 | 北京京东尚科信息技术有限公司 | Data query method and apparatus |
| CN108132958A (en) * | 2016-12-01 | 2018-06-08 | 阿里巴巴集团控股有限公司 | A kind of multi-level buffer data storage, inquiry, scheduling and processing method and processing device |
| CN109101580A (en) * | 2018-07-20 | 2018-12-28 | 北京北信源信息安全技术有限公司 | A kind of hot spot data caching method and device based on Redis |
| CN109657174A (en) * | 2018-12-25 | 2019-04-19 | 北京百度网讯科技有限公司 | Method and apparatus for more new data |
| CN109656958A (en) * | 2018-12-18 | 2019-04-19 | 北京小米移动软件有限公司 | Data query method and system |
| CN109684381A (en) * | 2018-12-20 | 2019-04-26 | 恒生电子股份有限公司 | Data hot statistics method and device |
| CN109885574A (en) * | 2019-02-22 | 2019-06-14 | 广州荔支网络技术有限公司 | A kind of data query method and device |
| CN110598138A (en) * | 2018-06-12 | 2019-12-20 | 北京京东尚科信息技术有限公司 | Cache-based processing method and device |
| CN110990372A (en) * | 2019-11-06 | 2020-04-10 | 苏宁云计算有限公司 | Dimensional data processing method and device and data query method and device |
| CN111259034A (en) * | 2020-02-17 | 2020-06-09 | 腾讯科技（深圳）有限公司 | Cache data management method and related device |
| CN111324615A (en) * | 2018-12-13 | 2020-06-23 | 北京京东尚科信息技术有限公司 | Data processing method, device, medium and electronic equipment |
| CN112506973A (en) * | 2020-12-14 | 2021-03-16 | 中国银联股份有限公司 | Method and device for managing stored data |
-
2021
- 2021-01-18 CN CN202110062487.6A patent/CN113760982B/en active Active
Patent Citations (13)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN106202581A (en) * | 2016-08-30 | 2016-12-07 | 北京奇艺世纪科技有限公司 | A kind of data search method and device |
| CN107918617A (en) * | 2016-10-10 | 2018-04-17 | 北京京东尚科信息技术有限公司 | Data query method and apparatus |
| CN108132958A (en) * | 2016-12-01 | 2018-06-08 | 阿里巴巴集团控股有限公司 | A kind of multi-level buffer data storage, inquiry, scheduling and processing method and processing device |
| CN110598138A (en) * | 2018-06-12 | 2019-12-20 | 北京京东尚科信息技术有限公司 | Cache-based processing method and device |
| CN109101580A (en) * | 2018-07-20 | 2018-12-28 | 北京北信源信息安全技术有限公司 | A kind of hot spot data caching method and device based on Redis |
| CN111324615A (en) * | 2018-12-13 | 2020-06-23 | 北京京东尚科信息技术有限公司 | Data processing method, device, medium and electronic equipment |
| CN109656958A (en) * | 2018-12-18 | 2019-04-19 | 北京小米移动软件有限公司 | Data query method and system |
| CN109684381A (en) * | 2018-12-20 | 2019-04-26 | 恒生电子股份有限公司 | Data hot statistics method and device |
| CN109657174A (en) * | 2018-12-25 | 2019-04-19 | 北京百度网讯科技有限公司 | Method and apparatus for more new data |
| CN109885574A (en) * | 2019-02-22 | 2019-06-14 | 广州荔支网络技术有限公司 | A kind of data query method and device |
| CN110990372A (en) * | 2019-11-06 | 2020-04-10 | 苏宁云计算有限公司 | Dimensional data processing method and device and data query method and device |
| CN111259034A (en) * | 2020-02-17 | 2020-06-09 | 腾讯科技（深圳）有限公司 | Cache data management method and related device |
| CN112506973A (en) * | 2020-12-14 | 2021-03-16 | 中国银联股份有限公司 | Method and device for managing stored data |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN114969107A (en) * | 2022-05-07 | 2022-08-30 | 广东联想懂的通信有限公司 | Data processing method, device and equipment for multi-label object and storage medium |
| CN115840770A (en) * | 2023-02-10 | 2023-03-24 | 广东徐工汉云工业互联网有限公司 | Local cache data processing method and related equipment based on distributed environment |
Also Published As
| Publication number | Publication date |
|---|---|
| CN113760982B (en) | 2024-05-17 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN109471783B (en) | Method and device for predicting task operation parameters | |
| CN110598138A (en) | Cache-based processing method and device | |
| CN113760982B (en) | Data processing method and device | |
| CN107426336B (en) | Method and device for adjusting push message opening rate | |
| CN109428926B (en) | Method and device for scheduling task nodes | |
| CN113312553B (en) | User tag determining method and device | |
| CN108764866B (en) | Method and equipment for allocating resources and drawing resources | |
| CN112118352B (en) | Method and device for processing notification trigger message, electronic equipment and computer readable medium | |
| CN113590017B (en) | Method, electronic device and computer program product for processing data | |
| CN113778565A (en) | Interface data processing method and device | |
| CN112910855B (en) | Sample message processing method and device | |
| CN115543416A (en) | Configuration updating method and device, electronic equipment and storage medium | |
| CN113676531B (en) | E-commerce flow peak clipping method and device, electronic equipment and readable storage medium | |
| CN111831503A (en) | Monitoring method based on monitoring agent and monitoring agent device | |
| CN113238919A (en) | Statistical method, device and system for user access number | |
| CN109087097B (en) | Method and device for updating same identifier of chain code | |
| CN112866179B (en) | Current limiting method and current limiting device | |
| CN112749204B (en) | Method and device for reading data | |
| CN112688982B (en) | User request processing method and device | |
| CN109213815B (en) | Method, device, server terminal and readable medium for controlling execution times | |
| CN113282455A (en) | Monitoring processing method and device | |
| CN108984431B (en) | Method and apparatus for flushing stale caches | |
| CN110019671B (en) | Method and system for processing real-time message | |
| CN113722193A (en) | Method and device for detecting page abnormity | |
| CN111538721A (en) | Account processing method and device, electronic equipment and computer readable storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |