CN108460041B - Data processing method and device - Google Patents
Data processing method and device Download PDFInfo
- Publication number
- CN108460041B CN108460041B CN201710090903.7A CN201710090903A CN108460041B CN 108460041 B CN108460041 B CN 108460041B CN 201710090903 A CN201710090903 A CN 201710090903A CN 108460041 B CN108460041 B CN 108460041B
- Authority
- CN
- China
- Prior art keywords
- identifier
- content
- attribute
- query
- value
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/245—Query processing
- G06F16/2453—Query optimisation
- G06F16/24534—Query rewriting; Transformation
- G06F16/24539—Query rewriting; Transformation using cached or materialised query results
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/242—Query formulation
- G06F16/2425—Iterative querying; Query formulation based on the results of a preceding query
Abstract
The invention provides a data processing method and a data processing device, wherein the method comprises the following steps: receiving a query operation request, wherein the query operation request comprises query parameters, and the query parameters comprise a first attribute identifier and a first attribute value; sending a first query request to a memory database; receiving a first query response returned by the memory database, wherein the first query response comprises a corresponding first content identifier queried by the memory database in an index list corresponding to the first attribute identifier according to the query parameter; sending a second query request to the key-value database, wherein the second query request comprises the first content identification; receiving a second query response returned by the key value database, wherein the second query response comprises the first data content which is searched in the key value database and corresponds to the first content identification by taking the first content identification as a key value; and sending the first data content to a requester of the query operation request. The method combines the advantages of the memory database and the key value database, and has the advantages of good performance and low cost.
Description
Technical Field
The present invention relates to the field of computer technologies, and in particular, to a method and an apparatus for processing data.
Background
A list refers to data stored in a list form. The application range of the list data is very wide, all the related list display can be abstracted into a list, and the specific meaning of the list is different according to different product scenes. For example, a list of posts for a post bar; a news list for each category of news application software, a comment list under news; a user list and a gift list in live broadcast software; a message list for instant messaging software, a buddy list, and the like.
The conventional list is usually implemented by using MySQL (relational database), records of MySQL are stored on a disk, a large amount of data needs to be read from the disk during each query, the reading performance is limited by the performance of the disk, taking a post list as an example, the more posts of one block, the larger the data table is, the more data needs to be loaded during each reading, and the reading performance is also deteriorated along with the increase of the data.
Disclosure of Invention
Therefore, a processing method and a processing device for reading data with good performance are needed.
In order to achieve the purpose, the embodiment of the invention adopts the following technical scheme:
a method of processing data, comprising:
receiving a query operation request, wherein the query operation request comprises query parameters, and the query parameters comprise a first attribute identifier and a first attribute value corresponding to the first attribute identifier;
sending a first query request to a memory database, wherein the first query request comprises the query parameters;
receiving a first query response returned by the memory database, wherein the first query response comprises a first content identifier corresponding to the first attribute value, which is queried by the memory database in an index list corresponding to the first attribute identifier according to the query parameter;
sending a second query request to a key-value database, wherein the second query request comprises the first content identification;
receiving a second query response returned by the key value database, wherein the second query response comprises first data content which is searched in a key value record and corresponds to the first content identification by using the first content identification as a key value;
and sending the first data content to a requester of the query operation request.
An apparatus for processing data, comprising: the system comprises an operation request receiving module, a request sending module, a response receiving module and a query result sending module;
the operation request receiving module is used for receiving a query operation request, wherein the query operation request comprises query parameters, and the query parameters comprise a first attribute identifier and a first attribute value corresponding to the first attribute identifier;
the request sending module is used for sending a first query request to an internal storage database, wherein the first query request comprises the query parameters; the server is further configured to send a second query request to a key-value store, where the second query request includes the first content identifier;
the response receiving module is configured to receive a first query response returned by the memory database, where the first query response includes a first content identifier corresponding to the first attribute value, where the first content identifier is queried by the memory database in an index list corresponding to the first attribute identifier according to the query parameter; the first query response is also used for receiving a second query response returned by the key value database, wherein the second query response comprises first data content which is searched in a key value record and corresponds to the first content identification by using the first content identification as a key value;
and the query result sending module is used for sending the first data content to a requester of the query operation request.
The method and the device store the index list comprising the corresponding relation of the content identification, the attribute identification and the attribute value in the memory database, store the key value records taking the content identification as the key value in the key value database, have very high reading efficiency of the memory database, large storage data volume of the key value database, high support concurrency and low cost, can quickly inquire the content identification in the memory database according to the inquiry parameters, and the key value database is a distributed storage system and gets rid of the limitation of a magnetic disk. After receiving the query operation request, the first query request is sent to the memory database to obtain the first content identifier, and the data content corresponding to the content identifier can be rapidly queried according to the key value records in the key value database by taking the first content identifier as the key value.
Drawings
FIG. 1 is a diagram illustrating an application environment of a processing method for providing data according to an embodiment;
FIG. 2 is a schematic diagram of an internal structure of a server of an embodiment;
FIG. 3 is a flow diagram of a method of processing data for one embodiment;
FIG. 4 is a flow diagram of a method of processing data according to another embodiment;
FIG. 5 is a flow chart of a method of processing data according to yet another embodiment;
FIG. 6 is a flow chart of a method of processing data of yet another embodiment;
FIG. 7 is a diagram illustrating a storage structure of a forum, according to an embodiment;
FIG. 8 is a timing diagram illustrating insertion processing of data according to one embodiment;
FIG. 9 is a timing diagram illustrating a modification process for data according to one embodiment;
FIG. 10 is a schematic diagram illustrating the timing of query processing on data according to one embodiment;
FIG. 11 is a timing diagram illustrating the deletion of data according to one embodiment;
FIG. 12 is a block diagram of a data processing apparatus according to an embodiment;
fig. 13 is a block diagram showing a configuration of a data processing device according to another embodiment.
Detailed Description
In order to make the objects, technical solutions and advantages of the present invention more apparent, the present invention is further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the detailed description and specific examples, while indicating the scope of the invention, are intended for purposes of illustration only and are not intended to limit the scope of the invention.
Fig. 1 is a schematic application environment diagram of a data processing method according to an embodiment. As shown in fig. 1, the application environment includes a user terminal 101, a server 102 and a key-value database 103, where the server 102 is provided with a memory database, an index list of data stored in the memory database, and a key-value database 1022 for storing data content of data. The key value database in this embodiment is a distributed storage system, that is, data contents may be stored in different storage servers. A query request of a user is sent to the server 102 through the user terminal 101, after receiving the query request, the server 102 finds a content identifier corresponding to the query request according to the index list through the memory database, the server 102 sends the found content identifier to the key value database 103, the key value database 103 finds data content corresponding to the content identifier according to the content identifier and returns the data content to the server 102, and the server 102 sends the data content to the user terminal 101 for display. The form of the data content in this embodiment is not limited, and corresponds to a specific application product. The data content in this embodiment may be a text content, for example, a specific content of a certain post in a post bar. The data content in this embodiment may also be picture content, for example, a specific picture of a certain expression in an expression list.
Fig. 2 is a schematic diagram of an internal configuration of a server in one embodiment. As shown in fig. 2, the server includes a processor, a storage medium, a memory, and a network interface connected by a system bus. The storage medium of the server stores an operating system and a data processing device, and the data processing device is used for realizing a data processing method. The processor is used for providing calculation and control capacity and supporting the operation of the whole server. The memory in the server provides an environment for the operation of the processing device of the data in the storage medium, the memory is provided with a memory database, and the network interface is used for carrying out network communication with the user terminal and the key value database, receiving a data query request sent by the user terminal, sending a content identifier corresponding to the query request to the key value database, and receiving data content corresponding to the content identifier returned by the key value database. It will be appreciated by those skilled in the art that the configuration shown in fig. 2 is a block diagram of only a portion of the configuration associated with the inventive arrangements and does not constitute a limitation to the terminals to which the inventive arrangements are applied, and that a particular server may include more or fewer components than shown, or some components may be combined, or have a different arrangement of components.
Referring to fig. 3, in an embodiment, a data processing method is provided, which is executed in the server 102 shown in fig. 1, and includes the following steps:
s302: a query operation request is received. The query operation request comprises query parameters, and the query parameters comprise a first attribute identifier and a first attribute value corresponding to the first attribute identifier.
A query request for data is sent to the server 102 via the user terminal 101, the query request including query parameters. In a specific embodiment, the query parameter includes a first attribute identifier and a first attribute value corresponding to the first attribute identifier.
The first, second, third, fourth, and the like in this embodiment are used to distinguish different content identifiers, attribute identifiers, and attribute values. The number of the attribute identifiers included in the first attribute identifier is related to the number of the attribute identifiers selected by the requester according to the query request, and the first attribute identifier may include one attribute identifier or a plurality of attribute identifiers. The first attribute value corresponds to the first attribute identifier, and when the first attribute identifier includes a plurality of attribute identifiers, it can be understood that the first attribute value includes attribute values corresponding to the attribute identifiers, respectively.
Taking an application scene as a forum as an example, the posts of the forum include: and attribute identifications such as user names, types, themes and the like, wherein each attribute identification of each label has a corresponding attribute value. And the user selects the attribute, inputs the attribute value according to the query condition and can quickly query the data meeting the requirement. For example, the query parameter includes a user name "zhang san" and a type "international news", where the user name and the type are attribute identifiers, and the "zhang san" and the "international news" are respectively attribute values of corresponding attribute identifiers, so that the data content with an author name "zhang san" and a type "international news" can be queried.
S304: a first query request is sent to the in-memory database, the first query request including query parameters.
The memory database refers to a non-relational database based on a memory. The memory database in this embodiment may be a Redis (open source, a log-type or Key-Value database written in ANSI C language, supporting network, and based on memory or persistent), and the memory database is used for full memory storage, which has a strong performance advantage.
S306: and receiving a first query response returned by the memory database, wherein the first query response comprises a first content identifier which is queried by the memory database in an index list corresponding to the first attribute identifier according to the query parameter and corresponds to the first attribute value.
In this embodiment, the memory database may use Redis, and the ZSet of the Redis is used to construct the index list, where each query or filter condition corresponds to one ZSet index list. In this embodiment, the index policy is designed in advance according to the function of the product and the attribute identifier of the data content, and an index list corresponding to each index policy is constructed in the memory database. When data is newly written, the memory database adds an index entry in an index list comprising the attribute identifier of the newly written data according to a pre-designed index strategy.
When the query parameters are received, the in-memory database responds to the first query request. The query parameters comprise a first attribute identification and a first attribute value corresponding to the first attribute identification. And the memory database queries the index list corresponding to the first attribute identifier according to the query parameter, and the first content identifier corresponding to the first attribute value is met. The content identifier is a unique identifier for distinguishing each data content, and in a specific embodiment, the content identifier may be a serial number of each data content. And the memory database returns a first query response to the first query request to the data processing device, wherein the first query response comprises a first content identifier queried by the memory database.
S308: and sending a second query request to the key-value database, wherein the second query request comprises the first content identification.
The Key Value database is a non-relational database (NOSQL), the data Key Value pairs are organized, indexed and stored, the values (Value) corresponding to the Key values can be inquired by inquiring through the Key values (Key), and the NOSQL database has the characteristics of high inquiry speed, large data storage quantity, high support and the like. Key value records are the correspondence between value keys and values of data. Key value records are generated when data is newly written. In this embodiment, the content identifier is a key value in a key value record, and the data content corresponding to the content identifier can be queried through the content identifier.
S310: and receiving a second query response returned by the key value database, wherein the second query response comprises the first data content which is searched in the key value record and corresponds to the first content identification by using the first content identification as a key value.
And when receiving the second query request, the key value database responds to the second query request, and searches the first data content corresponding to the first content identification in the key value record by taking the first content identification as a key value. And the key value database returns a second query response to the second query request to the data processing device, wherein the second query response comprises second data content queried by the memory database.
S312: and sending the first data content to a requester of the query operation request.
The server 102 receives the first data content returned by the key-value database 103, that is, the query result of the query operation request, and sends the query result to the requester of the query operation request, that is, the user terminal 101, and the query result is displayed on the user terminal 101.
The method stores the index list comprising the corresponding relation of the content identification, the attribute identification and the attribute value in the memory database, stores the key value record taking the content identification as the key value in the key value database, has very high reading efficiency of the memory database, large storage data volume of the key value database, high support concurrency and low cost, can quickly inquire the content identification in the memory database according to the inquiry parameters, and the key value database is a distributed storage system and gets rid of the limitation of a magnetic disk. After receiving the query operation request, the first query request is sent to the memory database to obtain the first content identifier, and the data content corresponding to the content identifier can be quickly queried in the key value database according to the key value records by taking the first content identifier as the key value.
The data query method can be applied to various application scenes, and is particularly suitable for querying list data. A list refers to data stored in the form of a list. The application range of the list data is very wide, all the data related to the list display can be abstracted into a list, and the specific meaning of the list is different according to different product scenes. For example, a list of posts for a post bar; a news listing for each category of news application, a list of reviews under news; a user list and a gift list in live broadcast software; a message list for instant messaging software, a buddy list, and the like. By using the data query method, the performance is very stable in a distributed system, the response speed is effectively improved, and the waiting time of a user is reduced. By adopting the method, the difficulty in using the non-relational database (NoSQL) is reduced, developers can realize the product requirements more quickly, the product development period is shortened, and the online time is shortened, so that the competitive advantage of the product is improved.
Step S312 is specifically to generate a list corresponding to the query operation request according to the preset list model, the query parameter, the first content identifier, and the first data content, and send the list to the requester of the query operation request.
In this embodiment, the query result is displayed in a list form, and the list model refers to a list frame designed in advance according to the display requirement of the query result. The data sources of the list are query parameters, first content identifications and first data contents, and the query parameters comprise first attribute identifications and first attribute values. And extracting corresponding fields from the query parameters, the first content identification and the first data content according to a preset list model to obtain a list, and sending the list to a requester of the query operation request for display.
In another embodiment, as shown in fig. 2, the method further comprises the steps of:
s402: an insert operation request for data is received.
The insert operation is one of write operations, and the insert operation request includes the insert operation request including the second data content, a second attribute identification associated with the second data content, and a second attribute value corresponding to the second attribute identification. The query condition or the filter condition can be set by the second attribute identification and the second attribute value.
S404: and allocating a second content identifier to the insertion operation request, wherein the second content identifier corresponds to the second data content, the second attribute identifier and the second attribute value.
The second content identifier in this embodiment is a serial number (ID) assigned to the insert operation request, and the second content identifier corresponds to the second data content, the second attribute identifier, and the second attribute value.
S406: and generating a key value record according to the inserting operation request, wherein the key value in the key value record is a second content identifier, and the value in the key value record is a second data content.
In order to establish the relationship between the inserted content and the key value database, key value records are generated according to the insertion operation request, one insertion operation request is converted into one key value record which takes the sequence number (content identification) of the request as a key value and takes the data content as a value, so that the content of each insertion operation is embodied in the form of the key value record.
S408: and sending a first storage instruction to the key value record, wherein the first storage instruction comprises the key value record, and the key value record is stored by the key value database.
The Key-Value database is a non-relational database, all data are organized, indexed and stored in a Key-Value pair mode, and the Key-Value database is large in storage data volume, high in support concurrency and low in cost.
S408: and sending a second storage instruction to the memory database, wherein the second storage instruction comprises a second content identifier, a second attribute identifier and a second attribute value, and the memory database adds an index entry containing the second content identifier and the second attribute value in an index list comprising the second attribute identifier.
The memory database is used as full memory storage and has strong performance advantages. In this embodiment, index policies are designed in advance according to the query function of the product and the attribute identification of the data content, and an index list corresponding to each index policy is constructed in the memory database. When a second storage instruction is received, an index entry including a second content identifier, a second attribute identifier and a second attribute value is added to the index list including the second attribute identifier.
In this embodiment, the list in Redis is sorted by time, and after adding an index entry, the current timestamp is used as the Score (Score).
In a specific embodiment, the application scenario of the method is forum. The attribute identification of the post of the forum includes a user name and a type.
The constructed indexing strategy may include the following: displaying all stickers by default, inquiring according to the user name, inquiring according to the sticker type and inquiring according to the user name plus the sticker type. Corresponding storage structures, namely index lists, are designed corresponding to each index strategy. The index list comprises attribute identifications of the posts, attribute values of the attribute identifications and content identifications. When an insert operation request is received, the memory database adds an index entry in the index list including the attribute identifier according to a pre-designed index strategy.
For example, if the attribute value of the type attribute identifier of the inserted content is "technical recourse", and the attribute value of the user name attribute identifier is "zhang san", an index entry related to the content identifier of the inserted data content is added to the list including the type attribute identifier, the user name attribute identifier, and the type attribute identifier + the user name attribute identifier, respectively.
According to the data processing method, when the data is inserted, the second content identification is distributed for the insertion operation request, the key value record is generated according to the insertion operation request, the key value in the key value record is the second content identification, the value in the key value record is the content of the insertion operation, the key value record is sent to the key value database to be stored, the storage data volume of the key value database is large, the support is high, and the cost is low. And sending a second content identifier, a second attribute identifier and a second attribute value corresponding to the insertion operation request to the memory database, and adding an index entry corresponding to the second content identifier in an index list of the memory database including the second attribute identifier. The method combines the advantages of the memory database and the key value database, and has the advantages of good performance and low cost.
In another embodiment, as shown in fig. 5, further comprising:
s502: a modify operation request is received, the modify operation request including a modify parameter.
S504: and when the modification parameters comprise a third content identifier and modified data content, sending a first modification instruction to the key value database, wherein the first modification instruction comprises the third content identifier and the modified data content, searching the third data content corresponding to the third content identifier in the key value record by using the third content identifier as a key in the key value database, and updating the third data content according to the modified data content.
When the modification parameter includes the modified data content, it indicates that the modification operation request is a modification for the data content. And storing the data content in the key value database, sending a first modification instruction to the key value database, finding out third data content corresponding to the third content identifier according to the first modification instruction, and updating the modified data content into third data content, thereby completing the modification of the content.
When the modification parameter includes the third content identifier, the third attribute identifier, the modified attribute value of the third attribute identifier, and the modified data content, after step S504, the method further includes:
s506: and sending a second modification instruction to the memory database, wherein the second modification instruction comprises a third content identifier, a third attribute identifier and a modified attribute value of the third attribute identifier, deleting an index entry corresponding to the third content identifier from the index list comprising the third attribute identifier by the memory database, and adding an index entry comprising the third content identifier, the third attribute identifier and the modified attribute value of the third attribute identifier into the index list comprising the third attribute identifier.
When the modification parameter includes the third content identifier, the third attribute identifier, and the modified attribute value of the third attribute identifier, step S506 is performed after step S502.
In this embodiment, when modifying an attribute value, for example, modifying a type attribute identifier of a post of a forum, it is necessary to delete an index entry including the type attribute identifier, which is related to the content identifier, in the original index entries, and add an index entry corresponding to the content identifier requested by the modification operation to an index list including the attribute identifier according to the modified attribute value of the attribute identifier.
In a specific embodiment, when data is inserted, the key-value database stores the complete attribute of the data content to facilitate subsequent operations. When the modification parameter includes the third attribute identifier and the modified attribute value of the third attribute identifier, the first modification instruction includes the modified attribute value of the third content identifier, the third attribute identifier and the third attribute identifier, the attribute value corresponding to the third content identifier and the third attribute identifier is updated by the key value database according to the modified attribute value, the key value database further uses the third content identifier as a key, the third data content corresponding to the third content identifier is searched in the key value record, and the third data content is updated according to the modified data content. The second modification instruction comprises a third content identifier, a third attribute identifier and a modified attribute value of the third attribute identifier, the index entry corresponding to the third content identifier is deleted from the index list comprising the third attribute identifier by the memory database, and the index entry comprising the modified attribute value of the third content identifier, the third attribute identifier and the third attribute identifier is added to the index list comprising the third attribute identifier. In such an embodiment, the key-value store also needs to be modified when modifications of attribute values are involved. However, if the modified attribute does not belong to the attribute used in the index, the index entry does not need to be modified.
In another embodiment, as shown in fig. 6, the method further comprises the steps of:
s602: receiving a deletion operation request, wherein the deletion operation request comprises a deletion parameter; the deletion parameter includes a fourth content identification.
S604: and sending a first deletion instruction to the memory database according to the deletion operation request, wherein the first deletion instruction comprises the fourth content identifier, searching all index entries comprising the fourth content identifier in each index list by the memory database, and deleting all searched index entries.
S606: and sending a second deletion instruction to the key value database according to the deletion operation request, wherein the second deletion instruction comprises a fourth content identifier, and deleting the key value pair taking the fourth content identifier as a key value in the key value record by the key value database.
In the data processing method, after the content is inquired, if the content needs to be deleted, the content is deleted in the key value database respectively, and all index entries corresponding to the content are deleted in the memory database.
Next, the data processing method of the present invention will be described with reference to specific application scenarios. In one embodiment, a data processing method is provided and applied to forum data processing. The storage structure of a post of a forum is shown in fig. 7, the post includes a user name, a post type and a post content. The memory database in this embodiment adopts Redis, and the index list set according to the index policy (filter condition) is stored in the Redis, and includes an index 0: default index list, index 1: index list filtered by username, index 2: index list filtered by patch type, index 3: an index list filtered by type of tile and username.
And when the posts are required to be inserted, finding the index list required to be operated according to the index strategy to operate. The working process of the data processing apparatus 801 is as shown in fig. 8, and the insertion operation request includes the second data content, the second attribute identifier associated with the second data content, and the second attribute value corresponding to the second attribute identifier. For example, the inserted data is data content of type "international news" published under the user name "zhangsan". And when an inserting operation request is received, allocating a content identifier and generating a key value record, wherein the key value is a sequence number allocated for the inserting operation, and the value is the content of the inserting operation.
The data processing apparatus 801 sends the key value record to the key value database 802 for storage, sends the sequence number, the attribute identifier, and the attribute value corresponding to the insert operation request to the memory database 803, and adds the index entry corresponding to the sequence number to the index list including the user name attribute identifier and the type attribute identifier by the memory database 803 according to the attribute and the attribute value. Since the four index lists include the user name attribute identifier or the type attribute identifier, in this embodiment, the index entries corresponding to the sequence numbers need to be added to the index 0, the index 1, the index 2, and the index 3, respectively. An index list including the username attribute and the type attribute (i.e., index 3 above, filtered by category and username) for one embodiment is shown in table 1.
TABLE 1 index List filtered by type of tile and username
| ID | Type (B) | Authors refer to |
| 4 | News | Zhang San |
| 3 | Picture paster | Li Si |
| 2 | Help seeking device | Wang Wu |
| 1 | Sharing | Zhao Liu |
When a modification operation needs to be performed on data, the working process of the modification operation is as shown in fig. 9, a data processing apparatus 901 receives a modification operation request, where the modification operation request includes a modification parameter, and the modification parameter includes modified data content and a third content identifier. The processing means 901 of the data sends a first modification instruction to the key-value store 902, the first modification instruction comprising the third content identification and the modified data content. The key-value store 902 searches for a third data content corresponding to the third content identifier, and updates the third data content according to the modified data content.
If the modified parameters further include the modified attribute values, the index entries corresponding to the attribute values before modification need to be deleted from the memory database, and then the index entries corresponding to the modified attribute values are newly added.
When a query (read) operation needs to be performed on data, the operation process of the data is as shown in fig. 10, and a data processing apparatus 1001 receives a query request, where the query request includes a query parameter, and the query parameter includes a first attribute identifier and a first attribute value corresponding to the first attribute identifier. The data processing apparatus 1001 sends a first query request to the in-memory database, where the first query request includes a query parameter. And the memory database queries the first content identifier corresponding to the first attribute value in the index list corresponding to the first attribute identifier according to the query parameter. For example, when the query parameter includes the first attribute identifier including the share and the user name, and the corresponding attribute value is "share" with the user name "zhao xi", the found content identifier (serial number) is 1, and the memory database 1002 sends the first query response to the data processing apparatus, where the first query response includes the found first content identifier 1. The data processing apparatus 1001 sends a second query request to the key-value store 1003, where the second query request includes the found first content id 1. The key value database 1003 uses the first content identifier as a key value, and finds the first data content corresponding to the first content identifier in the key value record.
When the queried content needs to be deleted, as shown in fig. 11, a data processing device 1101 receives a deletion operation request, where the deletion operation request includes a deletion parameter, where the deletion parameter includes a fourth content identifier (for example, the deletion parameter includes that the content identifier is 1, the type is sharing, and the user name is zhao, then the data processing device 1101 sends a first deletion instruction to the internal storage database 1102, where the first deletion instruction includes the fourth content identifier, and the internal storage database 1102 deletes all index entries in an index list including the content identifier 1.
In another embodiment, there is provided a data processing apparatus, as shown in fig. 12, including: an operation request receiving module 121, a request transmitting module 122, a response receiving module 123 and a query result transmitting module 124.
The operation request receiving module 121 is configured to receive a query operation request, where the query operation request includes a query parameter, and the query parameter includes a first attribute identifier and a first attribute value corresponding to the first attribute identifier.
A request sending module 122, configured to send a first query request to the in-memory database, where the first query request includes a query parameter; and further configured to send a second query request to the key-value store, the second query request including the first content identification.
A response receiving module 123, configured to receive a first query response returned by the memory database, where the first query response includes a first content identifier corresponding to a first attribute value, where the first content identifier is queried by the memory database in an index list corresponding to the first attribute identifier according to the query parameter; the first query response comprises first data content which is searched in the key value record and corresponds to the first content identification, and the first content identification is used as a key value;
and a query result sending module 124, configured to send the first data content to a requester of the query operation request.
The device stores the index list comprising the corresponding relation of the content identification, the attribute identification and the attribute value in the memory database, stores the key value record taking the content identification as the key value in the key value database, has very high reading efficiency of the memory database, large storage data volume of the key value database, high support concurrency and low cost, can quickly inquire the content identification in the memory database according to the inquiry parameters, and is a distributed storage system which gets rid of the limitation of a magnetic disk. After receiving the query operation request, the first query request is sent to the memory database to obtain the first content identifier, and the data content corresponding to the content identifier can be rapidly queried according to the key value records in the key value database by taking the first content identifier as the key value.
In another embodiment, as shown in fig. 13, the data processing apparatus further includes: an assignment module 125 and a key value record generation module 126.
The operation request receiving module 121 is further configured to receive an insert operation request, where the insert operation request includes the second data content, a second attribute identifier associated with the second data content, and a second attribute value corresponding to the second attribute identifier.
The allocating module 125 is configured to allocate a second content identifier to the insert operation request, where the second content identifier corresponds to the second data content, the second attribute identifier, and the second attribute value.
The key value record generating module 126 is configured to generate a key value record according to the insert operation request, where a key value in the key value record is a second content identifier, and a value in the key value record is a second data content.
A request sending module 122, further configured to send a first storage instruction to the key-value store, where the first storage instruction includes a key-value record, and the key-value store stores the key-value record; and the memory database is used for adding an index entry containing the second content identifier and the second attribute value in an index list comprising the second attribute identifier.
According to the data processing device, when data is inserted, a second content identifier is distributed for an insertion operation request, a key value record is generated according to the insertion operation request, a key value in the key value record is the second content identifier, a value in the key value record is the content of the insertion operation, the key value record is sent to the key value database to be stored, the key value database is large in data storage amount, high in support and low in cost. And sending a second content identifier, a second attribute identifier and a second attribute value corresponding to the insert operation request to the memory database, and adding an index entry corresponding to the second content identifier in an index list of the memory database including the second attribute identifier. The method combines the advantages of the memory database and the key value database, and has the advantages of good performance and low cost.
In another embodiment, the operation request receiving module 121 is further configured to receive a modification operation request, where the modification operation request includes a modification parameter; the modification parameter comprises a third content identifier, and further comprises any one or any combination of a third attribute identifier, a modified attribute value of the third attribute identifier, and modified data content.
The request sending module 122 is further configured to send a first modification instruction to the key-value database when the modification parameter includes the modified data content, where the first modification instruction includes the third content identifier and the modified data content, and the key-value database uses the third content identifier as a key to search for the third data content corresponding to the third content identifier in the key-value record, and update the third data content according to the modified data content.
In another embodiment, the request sending module 122 is further configured to send, to the in-memory database, a second modification instruction when the modification parameter includes the third attribute identifier, the modified attribute value of the third attribute identifier, and the modified data content, or when the modification parameter includes the third attribute identifier and the modified attribute value of the third attribute identifier, where the second modification instruction includes the third attribute identifier and the modified attribute value of the third attribute identifier, delete, by the in-memory database, the index entry corresponding to the third content identifier in the index list that includes the third attribute identifier, and add, to the index list that includes the third attribute identifier, an index entry that includes the modified attribute value of the third content identifier, the third attribute identifier, and the third attribute identifier.
In this embodiment, when modifying an attribute value, for example, modifying a type attribute identifier of a post of a forum, it is necessary to delete an index entry including the type attribute identifier, which is related to the content identifier, in the original index entries, and add an index entry corresponding to the content identifier requested by the modification operation to an index list including the attribute identifier according to the modified attribute value of the attribute identifier.
That is, when only modifying the attribute value is involved, the key value database does not need to be processed, only the index entry including the attribute is deleted from the memory database, and the corresponding index entry is added according to the modified attribute, so that the operation is convenient.
In another embodiment, the operation request receiving module 121 is further configured to receive a deletion operation request, where the deletion operation request includes a deletion parameter, and the deletion parameter includes the fourth content identifier.
The request sending module 122 is further configured to send a first deletion instruction to the memory database according to the deletion operation request, where the first deletion instruction includes the fourth content identifier, search, by the memory database, all index entries including the fourth content identifier in each index list, and delete all searched index entries; and the key value database is used for deleting the key value pairs with the fourth content identifications as key values in the key value records according to the deletion operation request.
After the content is inquired, if the content needs to be deleted, the data processing device deletes the content in the key value database and deletes all index entries corresponding to the content in the memory database.
It will be understood by those skilled in the art that all or part of the processes in the methods of the embodiments described above may be implemented by a computer program, which is stored in a non-volatile computer readable storage medium, and in the embodiments of the present invention, the program may be stored in the storage medium of a computer system and executed by at least one processor in the computer system to implement the processes of the embodiments including the methods described above. The storage medium may be a magnetic disk, an optical disk, a Read-Only Memory (ROM), a Random Access Memory (RAM), or the like.
The technical features of the embodiments described above may be arbitrarily combined, and for the sake of brevity, all possible combinations of the technical features in the embodiments described above are not described, but should be considered as being within the scope of the present specification as long as there is no contradiction between the combinations of the technical features.
The above-mentioned embodiments only express several embodiments of the present invention, and the description thereof is more specific and detailed, but not construed as limiting the scope of the invention. It should be noted that, for a person skilled in the art, several variations and modifications can be made without departing from the inventive concept, which falls within the scope of the present invention. Therefore, the protection scope of the present patent should be subject to the appended claims.
Claims (11)
1. A method of processing data, comprising:
receiving a query operation request for list data, wherein the query operation request comprises query parameters, and the query parameters comprise a first attribute identifier and a first attribute value corresponding to the first attribute identifier; the list data comprises at least one of a post, a news list, a user list, a message list and a friend list;
sending a first query request to Redis, the first query request comprising the query parameters;
receiving a first query response returned by the Redis, wherein the first query response comprises a first content identifier corresponding to the first attribute value, which is queried by the Redis in an index list corresponding to the first attribute identifier according to the query parameter; the Redis stores an index list set according to an index strategy; each index strategy corresponds to an index list; when data is newly written, inserting new data into the key value database, and adding an index entry in an index list comprising the attribute identifier of the newly written data by Redis according to the index strategy;
sending a second query request to a key-value database, wherein the second query request comprises the first content identification; the key-value store comprises a distributed storage system of a plurality of storage servers;
receiving a second query response returned by the key value database, wherein the second query response comprises first data content which is searched in a key value record and corresponds to the first content identification by using the first content identification as a key value;
generating and extracting corresponding fields from the query parameters, the first content identification and the first data content according to a preset list model to obtain a list, and sending the list to a requester of a query operation request;
the method further comprises the following steps:
receiving a modification operation request, wherein the modification operation request comprises modification parameters; the modification parameter comprises a third content identifier, a third attribute identifier and a modified attribute value of the third attribute identifier;
sending a second modification instruction to the Redis, where the second modification instruction includes the third content identifier, the third attribute identifier, and a modified attribute value of the third attribute identifier, deleting, by the Redis, an index entry corresponding to the third content identifier in an index list including the third attribute identifier, and adding, by the Redis, an index entry including the third content identifier, the third attribute identifier, and the modified attribute value of the third attribute identifier in the index list including the third attribute identifier.
2. The method of claim 1, further comprising:
receiving an insertion operation request, wherein the insertion operation request comprises a second data content, a second attribute identifier associated with the second data content and a second attribute value corresponding to the second attribute identifier;
distributing a second content identifier to the insert operation request, wherein the second content identifier corresponds to the second data content, the second attribute identifier and the second attribute value;
generating a key value record according to the insertion operation request, wherein a key value in the key value record is the second content identifier, and a value in the key value record is the second data content;
sending a first storage instruction to the key-value database, the first storage instruction including the key-value record, the key-value database storing the key-value record;
sending a second storage instruction to the Redis, the second storage instruction including the second content identifier, the second attribute identifier, and a second attribute value, and adding, by the Redis, an index entry including the second content identifier and the second attribute value in an index list including the second attribute identifier.
3. The method of claim 1, further comprising:
receiving a modification operation request, wherein the modification operation request comprises modification parameters; the modification parameters comprise a third content identifier and modified data content;
and sending a first modification instruction to the key value database, wherein the first modification instruction comprises the third content identifier and modified data content, the key value database searches for third data content corresponding to the third content identifier in the key value record by taking the third content identifier as a key, and updates the third data content according to the modified data content.
4. The method of claim 1, wherein: the method further comprises the following steps:
receiving a modification operation request, wherein the modification operation request comprises modification parameters; the modification parameters comprise a third content identifier, a third attribute identifier, a modified attribute value of the third attribute identifier and modified data content;
sending a first modification instruction to the key-value database, wherein the first modification instruction comprises the third content identifier and modified data content, the key-value database searches for third data content corresponding to the third content identifier in the key-value record by taking the third content identifier as a key, and updates the third data content corresponding to the third content identifier according to the modified data content;
sending a second modification instruction to the Redis, the second modification instruction including the third content identifier, the third attribute identifier, and a modified attribute value of the third attribute identifier, deleting, by the Redis, an index entry corresponding to the third content identifier in an index list including the third attribute identifier, and adding, by the Redis, an index entry including the third content identifier, the third attribute identifier, and a modified attribute value of the third attribute identifier in an index list including the third attribute identifier.
5. The method of claim 1, further comprising:
receiving a deletion operation request, wherein the deletion operation request comprises a deletion parameter, and the deletion parameter comprises a fourth content identifier;
sending a first deletion instruction to the Redis according to the deletion operation request, wherein the first deletion instruction comprises the fourth content identifier, searching all index entries comprising the fourth content identifier in each index list by the Redis, and deleting all searched index entries;
and sending a second deletion instruction to the key value database according to the deletion operation request, wherein the second deletion instruction comprises the fourth content identifier, and the key value database deletes the key value pair taking the fourth content identifier as the key value in the key value record.
6. An apparatus for processing data, comprising: the device comprises an operation request receiving module, a request sending module, a response receiving module and a query result sending module;
the operation request receiving module is used for receiving an inquiry operation request for list data, wherein the inquiry operation request comprises inquiry parameters, and the inquiry parameters comprise a first attribute identifier and a first attribute value corresponding to the first attribute identifier; the list data comprises at least one of a post, a news list, a user list, a message list and a friend list;
the request sending module is configured to send a first query request to the Redis, where the first query request includes the query parameter; the system is also used for sending a second query request to the key value database, wherein the second query request comprises the first content identification; the key value database is a distributed storage system comprising a plurality of storage servers;
the response receiving module is configured to receive a first query response returned by the Redis, where the first query response includes a first content identifier corresponding to the first attribute value, which is queried by the Redis in an index list corresponding to the first attribute identifier according to the query parameter, and the Redis stores an index list set according to an index policy; each index strategy corresponds to an index list; when data is newly written, inserting new data into the key value database, and adding an index entry in an index list comprising the attribute identifier of the newly written data by Redis according to the index strategy; the first query response is also used for receiving a second query response returned by the key value database, wherein the second query response comprises first data content which is searched in a key value record and corresponds to the first content identification by using the first content identification as a key value;
the query result sending module is used for generating and extracting corresponding fields from the query parameters, the first content identification and the first data content according to a preset list model to obtain a list, and sending the list to a requester of a query operation request;
the operation request receiving module is further configured to receive a modification operation request, where the modification operation request includes a modification parameter; the modification parameters comprise a third content identifier, a third attribute identifier and a modified attribute value of the third attribute identifier;
the request sending module is further configured to send a second modification instruction to the Redis, where the second modification instruction includes the third content identifier, the third attribute identifier, and a modified attribute value of the third attribute identifier, delete, by the Redis, an index entry corresponding to the third content identifier in an index list including the third attribute identifier, and add, in the index list including the third attribute identifier, an index entry including the modified attribute value of the third content identifier, the third attribute identifier, and the third attribute identifier.
7. The apparatus of claim 6, further comprising: distribution module and key value record generation module
The operation request receiving module is further configured to receive an insertion operation request, where the insertion operation request includes a second data content, a second attribute identifier associated with the second data content, and a second attribute value corresponding to the second attribute identifier;
the distribution module is used for distributing a second content identifier to the insertion operation request, wherein the second content identifier corresponds to the second data content, the second attribute identifier and the second attribute value;
the key value record generating module is configured to generate a key value record according to the insert operation request, where a key value in the key value record is the second content identifier, and a value in the key value record is the second data content;
the request sending module is further configured to send a first storage instruction to the key value database, where the first storage instruction includes the key value record, and the key value database stores the key value record; and the Redis is further configured to send a second storage instruction to the Redis, where the second storage instruction includes the second content identifier, the second attribute identifier, and a second attribute value, and an index entry including the second content identifier and the second attribute value is added by the Redis to an index list including the second attribute identifier.
8. The apparatus of claim 6, wherein:
the operation request receiving module is further configured to receive a modification operation request, where the modification operation request includes a modification parameter; the modification parameters comprise a third content identifier and modified data content;
the request sending module is further configured to send a first modification instruction to the key-value database, where the first modification instruction includes the third content identifier and modified data content, and the key-value database uses the third content identifier as a key to search for the third data content corresponding to the third content identifier in the key-value record, and update the third data content according to the modified data content.
9. The apparatus of claim 6, wherein:
the operation request receiving module is used for receiving a modification operation request, wherein the modification operation request comprises modification parameters; the modification parameters comprise a third content identifier, a third attribute identifier, a modified attribute value of the third attribute identifier and modified data content;
the request sending module is further configured to send a first modification instruction to the key-value database, where the first modification instruction includes the third content identifier and modified data content, and the key-value database uses the third content identifier as a key to search for the third data content corresponding to the third content identifier in the key-value record, and update the third data content corresponding to the third content identifier according to the modified data content; the Redis is further configured to send a second modification instruction to the Redis, where the second modification instruction includes the third content identifier, the third attribute identifier, and a modified attribute value of the third attribute identifier, delete, by the Redis, an index entry corresponding to the third content identifier in an index list including the third attribute identifier, and add, by the Redis, an index entry including the third content identifier, the third attribute identifier, and the modified attribute value of the third attribute identifier in the index list including the third attribute identifier.
10. The apparatus of claim 6, wherein:
the operation request receiving module is further configured to receive a deletion operation request, where the deletion operation request includes a deletion parameter, and the deletion parameter includes a fourth content identifier;
the request sending module is further configured to send a first deletion instruction to the Redis according to the deletion operation request, where the first deletion instruction includes the fourth content identifier, all index entries including the fourth content identifier are searched by the Redis in each index list, and all searched index entries are deleted; and the key value database is further configured to send a second deletion instruction to the key value database according to the deletion operation request, where the second deletion instruction includes the fourth content identifier, and the key value database deletes, in the key value record, the key value pair with the fourth content identifier as a key value.
11. A computer-readable storage medium, storing a computer program which, when executed by a processor, causes the processor to carry out the steps of the method according to any one of claims 1 to 5.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201710090903.7A CN108460041B (en) | 2017-02-20 | 2017-02-20 | Data processing method and device |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201710090903.7A CN108460041B (en) | 2017-02-20 | 2017-02-20 | Data processing method and device |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN108460041A CN108460041A (en) | 2018-08-28 |
| CN108460041B true CN108460041B (en) | 2022-12-23 |
Family
ID=63229196
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201710090903.7A Active CN108460041B (en) | 2017-02-20 | 2017-02-20 | Data processing method and device |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN108460041B (en) |
Families Citing this family (10)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109325063B (en) * | 2018-09-25 | 2022-03-15 | 奇瑞汽车股份有限公司 | Data management method and device |
| CN111159213A (en) * | 2018-11-07 | 2020-05-15 | 航天信息股份有限公司 | Data query method, device, system and storage medium |
| CN109542907B (en) * | 2018-11-21 | 2021-09-03 | 万兴科技股份有限公司 | Database cache construction method and device, computer equipment and storage medium |
| CN111291018B (en) * | 2018-12-07 | 2023-06-23 | 北京沃东天骏信息技术有限公司 | Data management method, device, equipment and storage medium |
| CN109670975B (en) * | 2018-12-17 | 2021-02-05 | 泰康保险集团股份有限公司 | Method, medium, and electronic device for generating a single number in a computer system |
| CN111506582A (en) * | 2019-01-30 | 2020-08-07 | 普天信息技术有限公司 | Data storage method and device |
| CN109800336B (en) * | 2019-01-30 | 2020-12-25 | 百度在线网络技术（北京）有限公司 | Table implementation method, device, equipment and medium based on key value pair storage system |
| CN110910654B (en) * | 2019-12-03 | 2021-07-27 | 上海眼控科技股份有限公司 | Illegal information processing method and device, electronic equipment and readable storage medium |
| CN113986942B (en) * | 2021-12-28 | 2022-04-26 | 零犀(北京)科技有限公司 | Message queue management method and device based on man-machine conversation |
| CN114385663B (en) * | 2022-03-23 | 2022-06-28 | 北京国联政信科技有限公司 | Data processing method and device |
Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103853727A (en) * | 2012-11-29 | 2014-06-11 | 深圳中兴力维技术有限公司 | Method and system for improving large data volume query performance |
| CN106250443A (en) * | 2016-07-27 | 2016-12-21 | 福建富士通信息软件有限公司 | The method and system of data base's complex text inquiry are solved based on internal memory full-text search |
| CN106294595A (en) * | 2016-07-29 | 2017-01-04 | 海尔优家智能科技（北京）有限公司 | A kind of document storage, search method and device |
Family Cites Families (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US8527546B2 (en) * | 2010-11-25 | 2013-09-03 | International Business Machines Corporation | Generating a checkpoint image for use with an in-memory database |
-
2017
- 2017-02-20 CN CN201710090903.7A patent/CN108460041B/en active Active
Patent Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103853727A (en) * | 2012-11-29 | 2014-06-11 | 深圳中兴力维技术有限公司 | Method and system for improving large data volume query performance |
| CN106250443A (en) * | 2016-07-27 | 2016-12-21 | 福建富士通信息软件有限公司 | The method and system of data base's complex text inquiry are solved based on internal memory full-text search |
| CN106294595A (en) * | 2016-07-29 | 2017-01-04 | 海尔优家智能科技（北京）有限公司 | A kind of document storage, search method and device |
Also Published As
| Publication number | Publication date |
|---|---|
| CN108460041A (en) | 2018-08-28 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN108460041B (en) | Data processing method and device | |
| CN106959963B (en) | Data query method, device and system | |
| US20170031948A1 (en) | File synchronization method, server, and terminal | |
| CN109299157B (en) | Data export method and device for distributed big single table | |
| CN107911461B (en) | Object processing method in cloud storage system, storage server and cloud storage system | |
| CN111008521B (en) | Method, device and computer storage medium for generating wide table | |
| JP5356657B2 (en) | Content distribution apparatus and content distribution method | |
| CN108228799B (en) | Object index information storage method and device | |
| CN110427386B (en) | Data processing method, device and computer storage medium | |
| CN108614837B (en) | File storage and retrieval method and device | |
| CN110597852A (en) | Data processing method, device, terminal and storage medium | |
| US10051552B2 (en) | Method for realizing resource attribute notification, and common service entity | |
| CN110807028B (en) | Method, apparatus and computer program product for managing a storage system | |
| CN113094370A (en) | Data index construction method and device, storage medium and electronic equipment | |
| CN106161193B (en) | Mail processing method, device and system | |
| CN110019980B (en) | Index processing method and device, storage medium and computer equipment | |
| CN105843809B (en) | Data processing method and device | |
| CN108345699B (en) | Method, device and storage medium for acquiring multimedia data | |
| CN110555020B (en) | Mapping relation establishing and data querying methods, devices and equipment | |
| CN113934742B (en) | Data updating method, node information storage method, electronic device and medium | |
| CN105095283A (en) | Quasi-friend recommending method in social networking system and quasi-friend recommending system in social networking system | |
| CN106372121B (en) | Server and data processing method | |
| CN111309932B (en) | Comment data query method, comment data query device, comment data query equipment and storage medium | |
| CN107181715B (en) | Service checking method and device | |
| CN109726254B (en) | Method and device for constructing triple knowledge base |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant | ||
| TR01 | Transfer of patent right |
Effective date of registration: 20230927 Address after: 518000 Tencent Building, No. 1 High-tech Zone, Nanshan District, Shenzhen City, Guangdong Province, 35 Floors Patentee after: TENCENT TECHNOLOGY (SHENZHEN) Co.,Ltd. Patentee after: TENCENT CLOUD COMPUTING (BEIJING) Co.,Ltd. Address before: 518000 Tencent Building, No. 1 High-tech Zone, Nanshan District, Shenzhen City, Guangdong Province, 35 Floors Patentee before: TENCENT TECHNOLOGY (SHENZHEN) Co.,Ltd. |
|
| TR01 | Transfer of patent right |