CN112860750B - Data processing method, device, electronic equipment and storage medium - Google Patents
Data processing method, device, electronic equipment and storage medium Download PDFInfo
- Publication number
- CN112860750B CN112860750B CN202110264709.2A CN202110264709A CN112860750B CN 112860750 B CN112860750 B CN 112860750B CN 202110264709 A CN202110264709 A CN 202110264709A CN 112860750 B CN112860750 B CN 112860750B
- Authority
- CN
- China
- Prior art keywords
- target
- database
- topic
- data
- task
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000003672 processing method Methods 0.000 title abstract description 11
- 238000012545 processing Methods 0.000 claims abstract description 105
- 238000000034 method Methods 0.000 claims abstract description 61
- 230000002776 aggregation Effects 0.000 claims description 32
- 238000004220 aggregation Methods 0.000 claims description 32
- 238000013507 mapping Methods 0.000 claims description 21
- 238000004590 computer program Methods 0.000 claims description 6
- 235000019633 pungent taste Nutrition 0.000 claims description 6
- 238000004422 calculation algorithm Methods 0.000 claims description 5
- 230000004044 response Effects 0.000 claims description 5
- 230000008569 process Effects 0.000 description 10
- 241000283965 Ochotona princeps Species 0.000 description 9
- 238000010586 diagram Methods 0.000 description 7
- 230000002085 persistent effect Effects 0.000 description 5
- 230000035515 penetration Effects 0.000 description 4
- 230000004931 aggregating effect Effects 0.000 description 3
- 230000006870 function Effects 0.000 description 3
- 238000013500 data storage Methods 0.000 description 2
- 239000002699 waste material Substances 0.000 description 2
- 238000007796 conventional method Methods 0.000 description 1
- 238000013461 design Methods 0.000 description 1
- 230000000694 effects Effects 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 230000014509 gene expression Effects 0.000 description 1
- 230000006872 improvement Effects 0.000 description 1
- 230000007246 mechanism Effects 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 230000008685 targeting Effects 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/245—Query processing
- G06F16/2455—Query execution
- G06F16/24552—Database cache management
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/245—Query processing
- G06F16/2457—Query processing with adaptation to user needs
- G06F16/24578—Query processing with adaptation to user needs using ranking
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/27—Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/28—Databases characterised by their database models, e.g. relational or object models
- G06F16/284—Relational databases
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
- Y02D10/00—Energy efficient computing, e.g. low power processors, power management or thermal management
Abstract
The application discloses a data processing method, a device and electronic equipment, wherein the method comprises the following steps: acquiring an aggregate message to be processed from a cache queue, wherein the aggregate message comprises a target identifier of a target object and a task category, and the task category represents the category of task processing matters executed on the target object; executing the task processing item on the target object according to the target identifier and the task category to obtain a processing result; and storing the processing result according to a preset storage rule. The method can reduce the data processing capacity in the system, thereby reducing the pressure of a database and improving the reading and writing speeds of data.
Description
Technical Field
The present disclosure relates to the field of internet data processing technologies, and more particularly, to a data processing method, apparatus, electronic device, and computer readable storage medium.
Background
In the internet field, users can release various objects through terminal equipment, and the objects can be videos, characters, images and the like; after the terminal equipment sends the object to be issued by the user to the server, the server can aggregate the object under the same category by performing aggregation treatment on the object; therefore, when other users browse the objects by using the terminal equipment, the server can display the objects in the categories of interest of the users for the users to view so as to promote the viscosity of the users.
In the process of realizing the application, the inventor finds that when the existing method is used for carrying out aggregation treatment, the object is usually subjected to aggregation treatment after the object is successfully released, and the obtained aggregation data is synchronously written into a database, such as a mysql database; however, during peak periods of user activity, such as late or holidays, the amount of object release increases dramatically, so the corresponding aggregate process will also increase, while the existing manner of synchronously writing aggregate data to the database after object release is successful will cause the data request amount of the database to increase suddenly during peak periods, causing greater stress on the database, and this may also have the problem of wasting database resources during off-peak periods, since the database will typically be deployed according to the maximum data request amount.
In the above, the problem of the conventional method is described by taking the aggregation processing of the object after the object is successfully issued as an example; in practice, after the object is successfully released, when related information of the object needs to be updated, the existing method also has the problems described above, so it is necessary to provide a data processing method to aggregate data according to service characteristics, so as to avoid the problems of excessive database pressure and resource waste that may be caused.
Disclosure of Invention
The embodiment of the disclosure provides a data processing method, a device, an electronic device, a storage medium and a data acquisition method, so that when the electronic device performs data aggregation processing, the data can be aggregated and stored according to service characteristics, a database is utilized efficiently, and data writing and reading speeds are improved.
In a first aspect of the present disclosure, there is provided a data processing method, the method comprising:
acquiring an aggregate message to be processed from a cache queue, wherein the aggregate message comprises a target identifier of a target object and a task category, and the task category represents the category of task processing matters executed on the target object;
executing the task processing item on the target object according to the target identifier and the task category to obtain a processing result; the method comprises the steps of,
and storing the processing result according to a preset storage rule.
In a second aspect of the present disclosure, there is also provided a data acquisition method, including:
receiving an object acquisition request sent by a terminal device, wherein the object acquisition request comprises a target identifier of the target object to be acquired, and the target object is associated with a target topic;
In the case that the target object is judged not to be stored in a third database according to the target identification and the topic identification of the target topic, acquiring the target object from a second database according to the target identification and the topic identification, wherein the second database is a non-relational database for storing data in a non-volatile memory, and the third database is a non-relational database for storing data in a volatile memory;
and providing the target object to the terminal equipment.
In a third aspect of the present disclosure, there is also provided a data processing apparatus, including:
the aggregation message acquisition module is used for acquiring an aggregation message to be processed from the cache queue, wherein the aggregation message comprises a target identifier of a target object and a task category, and the task category represents the category of task processing matters executed on the target object;
the processing result obtaining module is used for executing the task processing items on the target object according to the target identifier and the task category to obtain a processing result; the method comprises the steps of,
and the storage module is used for storing the processing result according to a preset storage rule.
According to a fourth aspect of the present disclosure, there is also provided an electronic device, comprising:
a memory for storing executable instructions;
a processor for executing the electronic device to perform the method according to the first or second aspect of the present disclosure, according to the control of the executable instructions.
According to a fifth aspect of the present disclosure there is also provided a computer readable storage medium storing a computer program readable by a computer for performing the method according to the first or second aspect of the present disclosure when the computer program is read for execution by the computer.
The method has the advantages that according to the method, the electronic device, such as a server, after obtaining the aggregate message to be processed from the cache queue, can execute corresponding task processing matters on the target object according to the task category of the message and obtain a processing result, and then store the processing result by using a preset rule. According to the method, the cache queue is set, and the aggregation information to be processed is placed in the cache queue, so that the server can asynchronously process the aggregation tasks in the service, the data request quantity in the same time, especially in the peak time is reduced, the pressure of a database is reduced, the reading and writing speeds of data are improved, and the use efficiency of database resources is improved.
Other features of the present disclosure and its advantages will become apparent from the following detailed description of exemplary embodiments of the disclosure, which proceeds with reference to the accompanying drawings.
Drawings
The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments of the disclosure and together with the description, serve to explain the principles of the disclosure.
Fig. 1 is a flow chart of a data processing method according to an embodiment of the disclosure.
FIG. 2 is a schematic diagram of a data processing framework provided by an embodiment of the present disclosure.
Fig. 3 is a flowchart of a data acquisition method according to an embodiment of the present disclosure.
Fig. 4 is a functional block diagram of a data processing apparatus provided by an embodiment of the present disclosure.
Fig. 5 is a schematic hardware structure of an electronic device according to an embodiment of the disclosure.
Detailed Description
Various exemplary embodiments of the present disclosure will now be described in detail with reference to the accompanying drawings. It should be noted that: the relative arrangement of the components and steps, numerical expressions and numerical values set forth in these embodiments do not limit the scope of the present disclosure unless it is specifically stated otherwise.
The following description of at least one exemplary embodiment is merely illustrative in nature and is in no way intended to limit the disclosure, its application, or uses.
Techniques, methods, and apparatus known to one of ordinary skill in the relevant art may not be discussed in detail, but are intended to be part of the specification where appropriate.
In all examples shown and discussed herein, any specific values should be construed as merely illustrative, and not a limitation. Thus, other examples of exemplary embodiments may have different values.
The terms first, second and the like in the description and in the claims, are used for distinguishing between similar objects and not necessarily for describing a particular sequential or chronological order. It is to be understood that the data so used may be interchanged where appropriate such that embodiments of the disclosure may be capable of being practiced otherwise than as shown or described herein.
It should be noted that: like reference numerals and letters denote like items in the following figures, and thus once an item is defined in one figure, no further discussion thereof is necessary in subsequent figures.
< method example one >
An embodiment of the disclosure provides a data processing method, please refer to fig. 1, which is a schematic flow chart of the data processing method provided by the embodiment of the disclosure. The method may be implemented by an electronic device, which may be a server, for example, a blade server, a rack-mounted server, or may be a server cluster deployed in the cloud, where the method is not limited in particular; of course, in the implementation, the method may also be applied to the terminal device according to actual needs, which is not limited herein.
As shown in fig. 1, the method of the present embodiment may include the following steps S1100-S1300, which are described in detail below.
Step S1100, obtaining an aggregate message to be processed from the cache queue, where the aggregate message includes a target identifier of a target object and a task category, and the task category represents a category of task processing items executed on the target object.
Specifically, the method aims at the problems that after the object is successfully issued, the object is generally subjected to aggregation processing in real time, the obtained aggregation data is synchronously written into a database, and when relevant information of the object needs to be updated, the processing such as database writing processing and the like is synchronously performed in real time, so that the data request amount is possibly increased suddenly, the pressure of the database is high, and the data reading and writing speeds are affected. In the application, when the electronic equipment, such as a server, processes data, aiming at the characteristics of different services, the electronic equipment can reduce the pressure of a database and improve the data reading and writing speed in the system by putting tasks which do not need to be processed in real time into a buffer queue and controlling the message processing time so as to reduce the data request amount in the system at the same time.
In this embodiment, the aggregate message is a message including task processing items to be processed corresponding to the target object, where the task processing items in this embodiment may be processing items for aggregating the target object under a certain topic, or may be processing items for aggregating relevant information of the target object, for example, attributes such as praise and recommendation values; of course, in the specific implementation, the task processing item may be another item, and is not particularly limited here.
Topics for describing common characteristics of objects or describing categories to which the objects belong in content class applications, for example, applications such as short video applications, news applications, knowledge sharing applications, etc., for example, a topic tag "#dance" may be used to indicate that the objects are dancing-related objects.
The target object may be an object such as a video, text, image, etc., for example, a short video produced by a user in a short video application; for another example, news produced by the user in a news application may be used, and is not particularly limited herein. In this embodiment, the description will be given by taking the video in the application in which the target object is a short video as an example, unless otherwise specified.
In this embodiment, the buffer queue is a queue for storing aggregate messages to be processed, for example, may be a message queue based on a Rocketmq, where Rocketmq is a distributed message publishing and subscribing system, and of course, the buffer queue may also be implemented in other manners, for example, may also be a queue based on a rabitmq, which is not limited specifically herein.
In one embodiment, the cache queue may be maintained by: receiving an object release request, wherein the object release request comprises a first object to be released and an identification of a first topic associated with the first object; issuing the first object in response to the object issuing request, and obtaining the identification of the first object; acquiring the identification of the first topic; generating an aggregate message with a task class of a first class according to the identification of the first object and the identification of the first topic, and writing the aggregate message into the cache queue, wherein the first class represents executing topic aggregation matters on the first object.
Specifically, in the short video application, when the user produces an object through the terminal device, for example, when making and uploading the short video "video a" to the server using the mobile phone, step S1101 may be performed first, and the mobile phone may send a release request for releasing the "video a" to the server, where the release request may include the short video and a topic, for example, topic "#dance", associated with the short video selected or input by the user; after receiving the release request, the server needs to timely return a response message about whether the short video is successfully released to the terminal device, so that the server needs to execute step S1102 to perform short video release processing in real time, where the release processing specifically includes generating a video identifier of the short video, allocating a storage space for the short video, acquiring a storage path of the short video, and storing related information of the short video, such as a short video name, a release user, a storage address, a praise value, and the like, in a short video list; after the short video is successfully released, the server may execute step S1103 to return a response message indicating that the release was successful to the terminal device; in this embodiment, unlike the prior art that, after the short video is successfully published, the topic aggregation process is required to be performed synchronously, that is, the short video is aggregated into its associated topics, in this embodiment, the object publishing process and the topic aggregation site may be coupled, that is, after the object is successfully published through the above step S1103, step S1104 is performed, by acquiring the video identifier of "video a", and by acquiring the topic identifier of its associated topic "#dance", an aggregated message with the task class of the first class is generated, and the aggregated message is put into the cache queue, so that the topic aggregation task of "video a" may be asynchronously processed.
As can be seen from the above description, in this embodiment, when the electronic device performs data processing, the core service, for example, the object release service, may be processed in real time, and provide a response to the user in time; for the secondary service, for example, the topic aggregation service and the object release service can be decoupled and processed asynchronously, so that the data request amount in the system, especially the data request amount in the peak period, is reduced, and the data reading and writing speed is further improved.
In order to further reduce the request amount in the system in the implementation, the request amount of data in the system can be further reduced by placing the request amount in a cache queue for other non-core services, such as a service for updating the heat value of the target object.
In particular, in one embodiment, the cache queue may also be maintained by: receiving an object heat update request, wherein the object heat update request comprises an identification of a second object of which a heat value is to be updated; and generating an aggregation message with a task class of a second class according to the identification of the second object, and writing the aggregation message into the cache queue, wherein the second class represents that the heat update item is executed on the second object. In this embodiment, the popularity value represents the degree to which the object is focused by the user, or the degree to which the object is interested by the user, and in the implementation, the popularity value may be the praise value of the object; alternatively, the update time of the object may be; or, a preset heat calculation algorithm may be used, and the calculated value is synthesized according to the weight of the related information of the object, where the related information of the object may be, for example, praise value, update time, etc. In this embodiment, for convenience of explanation, a praise value directly targeting a heat value will be explained as an example.
In implementation, when displaying objects belonging to the same topic to a user, the terminal device may first order the objects included in the objects according to their popularity values, and display the objects whose ranks satisfy a set condition, for example, the popularity value is 100 in the front, so as to be viewed by the user.
It should be noted that, in order to increase the data reading speed, in general, when the electronic device, such as a server, stores the object and the information of the object, such as a hotness value, the data may be stored in a manner of using a "cache database+a persistent storage database", that is, the data with higher user access frequency, such as the data with higher hotness value, is stored in the cache database, so as to be convenient for timely reading the data and displaying the data to the user for viewing; while the data with lower user access frequency, such as low heat value, is stored in a persistent database, wherein the cache database may be generally a volatile memory-based, such as a memory-based redis database, and the persistent storage database may be a nonvolatile memory-based, such as a disk-based mysql database.
Specifically in this embodiment, after the user endorses an object, such as "video a", through the terminal device, the terminal device will send an object hotness update request to the server; in contrast to the prior art, the server sends the write request to the database, for example, the persistent storage database and/or the cache database, in real time for each object hot update request, which causes a sudden increase in the request amount of the database, and thus may cause a problem of high database pressure. In this embodiment, after receiving the update request, the server may respond to the update request without sending a write request to the database in real time, and by acquiring the video identifier of "video a" from the request, generating an aggregate message with a task class of the second class, and placing the aggregate message in the cache queue, the server may asynchronously process the hot update task of "video a", thereby achieving the purpose of reducing the request amount of the database.
It should be noted that, in the above embodiment of maintaining the cache queue provided in this embodiment, during implementation, the aggregate message may not include the task class, but a corresponding cache queue may be set for each task processing item; of course, other types of aggregate transactions may be included in the cache queue, which is not described herein.
In one embodiment, after the buffer queue is obtained through the maintaining in the above steps, when the messages in the buffer queue are consumed, the aggregated messages may be further obtained from the buffer queue according to a preset reading algorithm, where the preset reading algorithm includes at least one of a preset time interval and a message processing time corresponding to the aggregated messages, so as to maintain the data request amount in the system within a set number range, where the preset time interval and the message processing time set for each aggregated message may be set according to needs, for example, may be finely set according to the priority of each task, which is not described herein.
After step S1100, step S1200 is executed, where the task processing item is executed on the target object according to the target identifier and the task category, so as to obtain a processing result.
After the above step S1100 is performed, after the aggregate message to be processed is obtained from the cache queue, the corresponding task processing item can be executed on the target object according to the task category in the aggregate message and the target identifier of the target object.
In one embodiment, the task category includes a first category representing topic aggregation matters performed on the target object, and the aggregate message further includes a target topic identifier, where the target topic identifier is an identifier of a target topic associated with the target object; the task processing item is executed on the target object according to the target identifier and the task category, and a processing result is obtained, including: and under the condition that the task category is the first category, according to the target identifier and the target topic identifier, mapping data reflecting the association relation between the target object and the target topic is established as the processing result.
Specifically, in the case that the task category in the aggregated message is the first category, the target topic identification of the target topic associated with the target object may also be obtained from the message, where the relevant process about how to obtain the identification of the target topic is not described herein; thereafter, aggregating the target objects under their associated target topics is represented by building list data like [ target topic identification, target identification ].
For example, for video a and its associated topic "#dance", video a may be aggregated under topic "#dance" by adding the identity of video a to the topic list to which the topic corresponds.
In this embodiment, when the object is released, the user selects the topic as the object, and the electronic device obtains the topic identifier of the corresponding topic to aggregate the object under the corresponding topic; in specific implementation, only the object can be issued, and the topic does not need to be selected by a user, but the electronic equipment automatically acquires the identification of the topic related to the target object by extracting the characteristic information of the object after acquiring the aggregate message aiming at the target object; the target ∈topic may be at least one, and is not particularly limited here.
In the above, the task category is taken as the first category, that is, the topic aggregation item, for example, how to execute the corresponding task processing item on the target object and obtain the processing result. In the specific implementation, the task category may be any other category, for example, the second category described in step S1100, that is, the hot update item, which will be described in detail below.
In one embodiment, the task categories include a second category that represents performing a hot update event on the target object; the task processing item is executed on the target object according to the target identifier and the task category, and a processing result is obtained, including: and under the condition that the task class is the second class, acquiring a heat value of the target object in a first time range as the processing result according to the target identifier, wherein the first time range comprises a time range taking the current first time as a starting time and a second time as an ending time, and the second time is obtained by adding the first time and a preset duration.
In this embodiment, the heat value may be a praise value of the target object; in a specific implementation, the obtaining, according to the target identifier, a heat value of the target object in a first time range as the processing result includes: and counting newly added praise values of the target object in the first time range as the processing result according to the target identification.
Specifically, for the hot update item corresponding to the target object, for example, the praise value update item, the server may not need to perform the database write operation in real time for each praise value; instead, after the aggregate message with the task class being the second class is obtained, counting the newly added praise value of the target object in a preset duration, for example, in a time range of one minute, from the current first moment, and taking the newly added praise value in the time range as a processing result.
In the above, for different task categories in the aggregated message, how to execute the corresponding task processing item on the target object is described in detail, so as to obtain the processing result corresponding to the target object and the task category, and after obtaining the processing result, the processing result can be stored according to the preset storage rule.
After step S1200, step S1300 is performed, and the processing result is stored according to a preset storage rule.
In particular implementations, a "redis+mysql" design may be generally employed to store data, where a redis database is used as a cache database to store data with higher user access frequency, e.g., higher heat; and mysql is used to persist the stored data. However, since rediss are based on volatile memory, e.g., memory, to store data, costs are high and capacity is limited, which also makes it impractical to cache full amounts of data in the rediss database.
For example, tens of thousands of videos may be aggregated under a certain topic, and the video objects are ordered with praise values directly as hotness values; based on capacity considerations, typically only the video with the top 100 rank of hotness values under that topic is cached in the redis database. In this case, when the user views the video using the terminal device, if only the video ranked in the top 100 under the topic is viewed, the server directly acquires data from the redis database and provides the data for the terminal device to display; when the user needs to view the video ordered after 100, the data is typically read into mysql, i.e. a cache penetration occurs.
However, since mysql is a relational database based on nonvolatile memory storing data, this results in a higher time consuming reading of data from mysql when cache penetration occurs, potentially affecting the user experience; especially in the case of large data requests, the database pressure may be further increased, resulting in a decrease in database performance, thereby affecting the stability and reliability of the entire system. In the implementation, the memory capacity of redis can be increased to cache as much data as possible, however, this method has a high cost, and may also cause a problem of resource waste when more data is cached.
In order to solve the above-mentioned problems, in the present embodiment, the data is cached by adding a non-relational database for data storage based on a non-volatile memory, and the data is stored by using a preset storage rule, which is different from a simple method for increasing the memory capacity, and the above-mentioned problems are solved as described in detail below.
In one embodiment, the storing the processing result according to a preset storage rule includes: writing the mapping data into a first database and a second database, wherein the first database is a non-relational database, the second database is a relational database, and the data of the first database and the data of the second database are both stored in a nonvolatile memory.
In this embodiment, the first database may be a database that manages data based on key values, i.e., key-value, and stores the data in a nonvolatile memory, and may be, for example, a pika database, where pika is a key-value database that can store data persistently, has higher processing performance, and can implement persistent storage of data; the second database may be, for example, an existing mysql database for persisted data.
In one embodiment, in a case that the task category is the first category, for the established mapping data representing the association relationship between the target object and the target topic, writing the mapping data into the first database includes: according to the target mark, obtaining a real-time heat value of the target object; obtaining topic records to be stored according to the mapping data and the real-time heat value; writing the topic record to be stored into a target topic data list corresponding to the target topic identification, wherein the target topic data list is an ordered list in the first database, and the target topic data list orders the stored topic records by taking the heat value of an object as a key value.
Specifically, after obtaining mapping data representing an association relationship between a target object and a target topic, when storing the mapping data, a topic record to be stored, that is, aggregate object list data corresponding to the topic, may be obtained by obtaining a real-time popularity value, for example, a praise value, of the target object; and stores the data in a first database, i.e., the ordered list data structure (SortedSet) of pika.
In this embodiment, by storing data using the above storage rule, it is possible that when cache penetration occurs, i.e. when data that the terminal device needs to present is not cached in the third database for caching data, for example, redis data, the server may not have to read data directly to the second database, for example, mysql database, but rather obtain data from the first database with relatively high processing performance; in addition, because the first database uses the ordered list to store the corresponding topic, and the list can automatically sort the data in the ordered list according to the set key value, the data can be not needed to be sorted every time the data is read, and the data reading speed can be further improved.
In this embodiment, the storing the processing result according to a preset storage rule further includes: according to the target mark, obtaining a real-time heat value of the target object; judging whether the ordering of the target object in all objects associated with the target topic meets a preset condition according to the real-time heat value, and obtaining a judging result; and writing the mapping data into a third database under the condition that the judgment result shows that the ordering of the target objects meets the preset condition, wherein the third database is a non-relational database for storing data in a volatile memory.
That is, when storing the processing result, for example, storing the above-mentioned mapping data, in order to ensure the validity of the data in the third database, for example, the redis cache, the heat value of the target object may also be obtained in real time while storing the mapping data, so as to determine whether to write the mapping data into the redis database at the same time, so as to avoid the occurrence of the cache penetration phenomenon as much as possible.
Of course, the above is only one embodiment provided in the present application, and in implementation, it is not necessary to determine whether to write the mapping data into the third database when storing the mapping data, but the existing cache update mechanism is used to update the cache data in the third database, which is not limited herein.
In one embodiment, in the case that the task class is the second class, for a heat value of the target object in the first time range, for example, an newly added praise value of the target object in the first time range, which is obtained through aggregation, the storing the processing result according to a preset storage rule includes: obtaining a second moment according to the first moment and the preset duration; and updating the heat value of the target object stored in a first database and a second database according to the target identifier and the heat value at the second moment, wherein the first database is a non-relational database, the second database is a relational database, and the data of the first database and the data of the second database are both stored in a nonvolatile memory.
Referring to FIG. 2, a schematic diagram of a data processing framework is provided according to an embodiment of the disclosure. In fig. 2, a terminal device is taken as an example of a terminal running a short video application, and as shown in fig. 2, in the implementation, the terminal device may send a request such as an object release request, an object praise request, etc. to a server; after obtaining the request, the server can process the core service, such as the release object, which needs to be processed in real time according to the service characteristics of the request, and can generate the corresponding task message according to the non-core service, such as the services of topic aggregation, praise update, and the like, which have low real-time requirements, according to the service characteristics of the request, and can reduce the data request amount in the system by placing the task message into a Rockettq queue as shown in fig. 2 and then asynchronously consuming the message in the queue, thereby reducing the database pressure. In addition, as shown in fig. 2, the storage system of the present embodiment includes a multi-level cache, for example, a third database that can be regarded as a first-level cache, that is, a redis database; a first database, i.e., pika database, which may be considered a second level cache; and a second database, mysql database, which may be considered a third level cache; when storing data, the electronic device may cache the hotspot data in the redis database, while for non-hotspot data, it may be persisted in both the pika database and the mysql database; when the read data is cached, the data does not need to be read in the mysql database, but the data is acquired from the pika database with the processing performance higher than that of mysql, and the data stored by the pika is ordered, so that the read data does not need to be reordered after being read out, and the read data can be directly provided for the terminal equipment, so that the speed of displaying the data by the terminal equipment is improved, and the user experience is further improved.
With continued reference to fig. 2, in this embodiment, after receiving an object acquisition request sent by a terminal device, an electronic device, for example, a server, may acquire, according to a target identifier included in the object acquisition request, a topic identifier of a target topic associated with the target object; then, whether the target object is stored in the first level cache, for example, a third database, redis shown in fig. 2, may be determined according to the topic identification and the target identification; if the target object does not exist, the second database with lower processing performance is not needed, the data is acquired from mysql, the target object is acquired from the full data stored in the second-level cache, and the target object is provided for the terminal equipment, so that the terminal equipment can rapidly display the object for a user to view. In addition, since the database used as the second level cache in the present embodiment is based on a nonvolatile memory, for example, a pika database of a disk, the expansion cost is far lower than that of a memory as an expansion.
In summary, in the data processing method provided in this embodiment, after obtaining the aggregate message to be processed from the cache queue, the electronic device, for example, the server may execute the corresponding task processing item on the target object according to the task class of the message and obtain the processing result, and then store the processing result by using the preset rule. According to the method, the cache queue is set, and the aggregated information to be processed is placed in the cache queue, so that the server can asynchronously process the aggregated information in the service, the data processing capacity at the same time, especially in the peak time, is reduced, the pressure of a database is reduced, and the reading and writing speeds of data are improved; in addition, when data storage is performed, the first database, for example, the pika database is added as a secondary cache in the system, so that not only can the cost be saved, but also the data acquisition speed can be further improved.
< method example two >
Corresponding to the first embodiment of the method, the present embodiment also provides a data acquisition method, which can be applied to an electronic device, for example, a server.
Please refer to fig. 3, which is a flowchart illustrating a data acquisition method according to an embodiment of the disclosure. As shown in fig. 3, the method includes the following steps S3100-S3300, which are described in detail below.
Step S3100, receiving an object acquisition request sent by a terminal device, where the object acquisition request includes a target identifier of the target object to be acquired, and the target object is associated with a target topic.
In step S3200, when it is determined that the target object is not stored in a third database according to the target identifier and the topic identifier of the target topic, the target object is acquired from a second database according to the target identifier and the topic identifier, where the second database is a non-relational database storing data in a non-volatile memory, and the third database is a non-relational database storing data in a volatile memory.
And step S3300, providing the target object to the terminal device.
< device example >
In correspondence with the above-described embodiment, in the present embodiment, there is also provided a data processing apparatus, as shown in fig. 4, the apparatus 4000 may include an aggregate message acquisition module 4100, a processing result acquisition module 4200, and a storage module 4300.
The aggregate message obtaining module 4100 is configured to obtain an aggregate message to be processed from the cache queue, where the aggregate message includes a target identifier of a target object and a task category, and the task category represents a category of task processing items executed on the target object; the processing result obtaining module 4200, configured to execute the task processing item on the target object according to the target identifier and the task category, to obtain a processing result; the storage module 4300 is configured to store the processing result according to a preset storage rule.
< device example >
In this embodiment, there is also provided an electronic device, as shown in fig. 5, the electronic device 5000 may further include a processor 5200 and a memory 5100, the memory 5100 being for storing executable instructions; the processor 5200 is configured to operate the electronic device according to control of the instructions to perform the method according to any embodiment of the present disclosure.
The electronic device 5000 may be a server, or may be another type of device, such as a terminal device, and the like, which is not limited herein.
< example of Medium >
In correspondence with the above-described method embodiments, in this embodiment, there is also provided a computer-readable storage medium storing a computer program readable and executable by a computer for performing the method according to any of the above-described embodiments of the present disclosure when the computer program is read and executed by the computer.
The flowcharts and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods and computer program products according to various embodiments of the present disclosure. In this regard, each block in the flowchart or block diagrams may represent a module, segment, or portion of instructions, which comprises one or more executable instructions for implementing the specified logical function(s). In some alternative implementations, the functions noted in the block may occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams and/or flowchart illustration, and combinations of blocks in the block diagrams and/or flowchart illustration, can be implemented by special purpose hardware-based systems which perform the specified functions or acts, or combinations of special purpose hardware and computer instructions. It is well known to those skilled in the art that implementation by hardware, implementation by software, and implementation by a combination of software and hardware are all equivalent.
The foregoing description of the embodiments of the present disclosure has been presented for purposes of illustration and description, and is not intended to be exhaustive or limited to the embodiments disclosed. Many modifications and variations will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the various embodiments described. The terminology used herein was chosen in order to best explain the principles of the embodiments, the practical application, or the technical improvement in the marketplace, or to enable others of ordinary skill in the art to understand the embodiments disclosed herein. The scope of the present disclosure is defined by the appended claims.
Claims (12)
1. A method of data processing, comprising:
acquiring an aggregate message to be processed from a cache queue, wherein the aggregate message comprises a target identifier of a target object and a task category, and the task category represents the category of task processing matters executed on the target object;
executing the task processing item on the target object according to the target identifier and the task category to obtain a processing result; the method comprises the steps of,
storing the processing result according to a preset storage rule,
Wherein the task category includes a first category representing topic aggregation matters performed on the target object, the aggregate message further including a target topic identification, the target topic identification being an identification of a target topic associated with the target object,
the task category includes a second category that indicates that a hot update item is executed on the target object, and the executing the task processing item on the target object according to the target identifier and the task category, to obtain a processing result, includes: under the condition that the task category is the second category, acquiring a heat value of the target object within a first time range as the processing result according to the target identifier,
the task processing item is executed on the target object according to the target identifier and the task category, and a processing result is obtained, including:
when the task category is the first category, according to the target identification and the target topic identification, mapping data reflecting the association relation between the target object and the target topic is established as the processing result,
the step of storing the processing result according to a preset storage rule comprises the following steps:
Writing the mapping data to a first database and a second database,
the writing of the mapping data into the first database comprises:
according to the target mark, obtaining a real-time heat value of the target object;
obtaining topic records to be stored according to the mapping data and the real-time heat value;
writing the topic record to be stored into a target topic data list corresponding to the target topic identification, wherein the target topic data list is an ordered list in the first database, and the target topic data list orders the stored topic records by taking the heat value of an object as a key value.
2. The method of claim 1, wherein the method maintains the cache queue by:
receiving an object release request, wherein the object release request comprises a first object to be released and a first topic associated with the first object;
issuing the first object in response to the object issuing request, and obtaining the identification of the first object; the method comprises the steps of,
acquiring the identification of the first topic;
generating an aggregate message with a task class of a first class according to the identification of the first object and the identification of the first topic, and writing the aggregate message into the cache queue, wherein the first class represents executing topic aggregation matters on the first object.
3. The method of claim 1, wherein the method maintains the cache queue by:
receiving an object heat update request, wherein the object heat update request comprises an identification of a second object of which a heat value is to be updated;
and generating an aggregation message with a task class of a second class according to the identification of the second object, and writing the aggregation message into the cache queue, wherein the second class represents that the heat update item is executed on the second object.
4. The method of claim 1, wherein the first database is a non-relational database and the second database is a relational database, the data of both the first database and the second database being stored in non-volatile memory.
5. The method of claim 4, wherein storing the processing results according to a preset storage rule further comprises:
according to the target mark, obtaining a real-time heat value of the target object;
judging whether the ordering of the target object in all objects associated with the target topic meets a preset condition according to the real-time heat value, and obtaining a judging result;
And writing the mapping data into a third database under the condition that the judgment result shows that the ordering of the target objects meets the preset condition, wherein the third database is a non-relational database for storing data in a volatile memory.
6. The method according to claim 1, wherein the first time range includes a time range having a current first time as a start time and a second time as an end time, the second time being obtained by adding the first time and a preset time period.
7. The method of claim 6, wherein the hotness value comprises a praise value of the target object;
the obtaining, according to the target identifier, a heat value of the target object in a first time range as the processing result includes:
and counting newly added praise values of the target object in the first time range as the processing result according to the target identification.
8. The method of claim 6, wherein storing the processing results according to a preset storage rule comprises:
obtaining a second moment according to the first moment and the preset duration;
And updating the heat value of the target object stored in a first database and a second database according to the target identifier and the heat value at the second moment, wherein the first database is a non-relational database, the second database is a relational database, and the data of the first database and the data of the second database are both stored in a nonvolatile memory.
9. The method of claim 1, wherein the obtaining the aggregate message to be processed from the cache queue comprises:
and acquiring the aggregated message from the cache queue according to a preset reading algorithm, wherein the preset reading algorithm comprises at least one of message processing time corresponding to the aggregated message according to a preset time interval.
10. A data processing apparatus, comprising:
the aggregation message acquisition module is used for acquiring an aggregation message to be processed from the cache queue, wherein the aggregation message comprises a target identifier of a target object and a task category, and the task category represents the category of task processing matters executed on the target object;
the processing result obtaining module is used for executing the task processing items on the target object according to the target identifier and the task category to obtain a processing result; the method comprises the steps of,
A storage module for storing the processing result according to a preset storage rule,
wherein the task category includes a first category representing topic aggregation matters performed on the target object, the aggregate message further including a target topic identification, the target topic identification being an identification of a target topic associated with the target object,
the task category includes a second category representing execution of a hot update item on the target object, and the processing result obtaining module is further configured to: under the condition that the task category is the second category, acquiring a heat value of the target object within a first time range as the processing result according to the target identifier,
wherein, the processing result obtaining module is further configured to: when the task category is the first category, according to the target identification and the target topic identification, mapping data reflecting the association relation between the target object and the target topic is established as the processing result,
the storage module is further configured to: writing the mapping data to a first database and a second database, comprising:
according to the target mark, obtaining a real-time heat value of the target object;
Obtaining topic records to be stored according to the mapping data and the real-time heat value;
writing the topic record to be stored into a target topic data list corresponding to the target topic identification, wherein the target topic data list is an ordered list in the first database, and the target topic data list orders the stored topic records by taking the heat value of an object as a key value.
11. An electronic device, comprising:
a memory for storing executable instructions;
a processor for executing the method according to any of claims 1-9, operating the electronic device according to the control of the instructions.
12. A computer readable storage medium, characterized in that the computer readable storage medium stores a computer program readable for execution by a computer for performing the method according to any one of claims 1-9 when being read by the computer.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110264709.2A CN112860750B (en) | 2021-03-11 | 2021-03-11 | Data processing method, device, electronic equipment and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110264709.2A CN112860750B (en) | 2021-03-11 | 2021-03-11 | Data processing method, device, electronic equipment and storage medium |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN112860750A CN112860750A (en) | 2021-05-28 |
| CN112860750B true CN112860750B (en) | 2023-11-17 |
Family
ID=75993982
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202110264709.2A Active CN112860750B (en) | 2021-03-11 | 2021-03-11 | Data processing method, device, electronic equipment and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN112860750B (en) |
Families Citing this family (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113656423A (en) * | 2021-08-18 | 2021-11-16 | 北京百度网讯科技有限公司 | Method and device for updating data, electronic equipment and storage medium |
Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103034722A (en) * | 2012-12-13 | 2013-04-10 | 合一网络技术(北京)有限公司 | Network video comment gathering device and network video comment gathering method |
| CN104244112A (en) * | 2013-06-14 | 2014-12-24 | 腾讯科技（深圳）有限公司 | Multimedia processing method and device and server |
| CN106033438A (en) * | 2015-03-13 | 2016-10-19 | 北大方正集团有限公司 | Public sentiment data storage method and server |
| CN108984675A (en) * | 2018-07-02 | 2018-12-11 | 北京百度网讯科技有限公司 | Data query method and apparatus based on evaluation |
| CN109597818A (en) * | 2018-11-28 | 2019-04-09 | 优刻得科技股份有限公司 | Data-updating method, device, storage medium and equipment |
| CN111259246A (en) * | 2020-01-17 | 2020-06-09 | 北京达佳互联信息技术有限公司 | Information pushing method and device, electronic equipment and storage medium |
| CN112307046A (en) * | 2020-11-26 | 2021-02-02 | 北京金堤征信服务有限公司 | Data acquisition method and device, computer readable storage medium and electronic equipment |
-
2021
- 2021-03-11 CN CN202110264709.2A patent/CN112860750B/en active Active
Patent Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103034722A (en) * | 2012-12-13 | 2013-04-10 | 合一网络技术(北京)有限公司 | Network video comment gathering device and network video comment gathering method |
| CN104244112A (en) * | 2013-06-14 | 2014-12-24 | 腾讯科技（深圳）有限公司 | Multimedia processing method and device and server |
| CN106033438A (en) * | 2015-03-13 | 2016-10-19 | 北大方正集团有限公司 | Public sentiment data storage method and server |
| CN108984675A (en) * | 2018-07-02 | 2018-12-11 | 北京百度网讯科技有限公司 | Data query method and apparatus based on evaluation |
| CN109597818A (en) * | 2018-11-28 | 2019-04-09 | 优刻得科技股份有限公司 | Data-updating method, device, storage medium and equipment |
| CN111259246A (en) * | 2020-01-17 | 2020-06-09 | 北京达佳互联信息技术有限公司 | Information pushing method and device, electronic equipment and storage medium |
| CN112307046A (en) * | 2020-11-26 | 2021-02-02 | 北京金堤征信服务有限公司 | Data acquisition method and device, computer readable storage medium and electronic equipment |
Also Published As
| Publication number | Publication date |
|---|---|
| CN112860750A (en) | 2021-05-28 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN105827706B (en) | Message pushing device and method | |
| KR101711456B1 (en) | System and method for improving access to search results | |
| US20180246862A1 (en) | Methods for prerendering and methods for managing and configuring prerendering operations | |
| KR101728927B1 (en) | Pre-fetching content items based on social distance | |
| US11019179B2 (en) | Access to network content | |
| US20120124038A1 (en) | Variable Length Snippet Generation | |
| TW201820903A (en) | Method and device for processing chat log of real-time chat tool | |
| CN112434015B (en) | Data storage method and device, electronic equipment and medium | |
| CN112860750B (en) | Data processing method, device, electronic equipment and storage medium | |
| JP2016519380A (en) | Information transmission based on reading speed | |
| CN111767284B (en) | Data processing method, device, storage medium and server | |
| US9176645B1 (en) | Manipulating collections of items in a user interface | |
| CN111666045A (en) | Data processing method, system, equipment and storage medium based on Git system | |
| CN112883316A (en) | Data processing method and device, electronic equipment and storage medium | |
| CN107480269B (en) | Object display method and system, medium and computing equipment | |
| CN111753141A (en) | Data management method and related equipment | |
| CN108319622B (en) | Media content recommendation method and device | |
| CN113805864A (en) | Project engineering generation method and device, electronic equipment and storage medium | |
| CN114168623A (en) | Method and device for acquiring information of object, electronic equipment and storage medium | |
| CN113742010A (en) | Page multi-language display method and device | |
| CN116662662A (en) | Task recommendation method and device | |
| CN116048468A (en) | Method, device, electronic equipment and medium for loading cache data and processing data | |
| CN113535725A (en) | Database storage space optimization method and device, electronic equipment and storage medium | |
| CN117493634A (en) | Data processing method, device, computing equipment and computer storage medium | |
| CN114861051A (en) | Information flow recommendation method and device |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |