CN116431716A

CN116431716A - Data processing method, device, network equipment and storage medium

Info

Publication number: CN116431716A
Application number: CN202310182179.6A
Authority: CN
Inventors: 胡东旭; 陈存利; 赵鹏; 胡兆运
Original assignee: Du Xiaoman Technology Beijing Co Ltd
Current assignee: Du Xiaoman Technology Beijing Co Ltd
Priority date: 2023-02-21
Filing date: 2023-02-21
Publication date: 2023-07-14

Abstract

The invention discloses a data processing method, a device, network equipment and a storable medium. The method comprises the following steps: and receiving a command sent by a user side under the condition that a database architecture corresponding to the database comprises X database fragments and proxy middleware, wherein the command comprises the following components: a read command, or a write command, X is an integer greater than 1; splitting the command according to the command content of the command to generate Y subcommands, wherein the Y subcommands correspond to Y databases in the X database partitions, data information related to the Y subcommands is stored in the Y databases, Y is smaller than or equal to X, and Y is an integer; the Y subcommands are executed in the Y database partitions.

Description

Data processing method, device, network equipment and storage medium

Technical Field

The present disclosure relates to the field of communications, and in particular, to a data processing method, apparatus, network device, and storable medium.

Background

With the rise of internet technology, non-relational (NoSQL) databases are also continuously developing. Based on the independence of data in NoSQL, noSQL has very high read-write capability and has the characteristics of large data volume and high performance, so that a plurality of problems of Internet with large data volume are solved.

In the related art, a database architecture is generally built according to a cluster model, so that various commands can be conveniently executed. Typically, a database architecture consists of proxy middleware (twomproxy) and multiple database shards of the NoSQL type. In the process of using the database fragments through the database architecture, the network device can input a command to the proxy middleware, and then operate one of the database fragments to execute the command through the proxy middleware according to the address corresponding to the command.

However, since it is often performed on some of the plurality of database tiles, such a manner of performing the command may result in an excessive amount of data for one or several database tiles, which in turn may result in data tilting, with data stacking occurring in individual database tiles.

Disclosure of Invention

The application provides a data processing method, a data processing device, a network device and a storage medium, which are used for solving the problem that data inclination easily occurs in a database of the NoSQL type at present and data accumulation occurs in individual database fragments.

In order to solve the technical problems, one technical scheme adopted by the application is as follows: and receiving a command sent by a user side under the condition that a database architecture corresponding to the database comprises X database fragments and proxy middleware, wherein the command comprises the following components: a read command, or a write command, X is an integer greater than 1; splitting the command according to the command content of the command to generate Y subcommands, wherein the Y subcommands correspond to Y databases in the X database partitions, data information related to the Y subcommands is stored in the Y databases, Y is smaller than or equal to X, and Y is an integer; the Y subcommands are executed in the Y database partitions.

The beneficial effects of the embodiment of the application are that: when the database architecture corresponding to the database comprises X database fragments and proxy middleware, receiving a read command or a write command, and then splitting the command according to the command content of the command to generate Y subcommands corresponding to Y databases in the X database fragments (the Y databases store data information related to the Y subcommands); finally, the Y subcommands are executed in the Y database partitions. Therefore, by splitting the command, the command is not required to be concentrated in a single database partition to be executed, but is executed in a plurality of database partitions, and for the write command, the data can be stored in different databases in a distributed manner, so that the problem of data inclination caused by data storage in a single database partition is avoided, and the use efficiency of the database partitions is improved.

Optionally, the database includes an index table, the index table includes index information, and the index information includes: space address information in each database partition in the X database partitions and data volume information of the space address information in each database partition; the data format of the spatial address information corresponds to the data format in the X database partitions.

Therefore, compared with the related art, the space address information in each database partition is stored in a mode that the character strings are not used by the space address information, and under the condition that the same space address information is contained in a plurality of database partitions in X database partitions, excessive occupied memory is not caused by the increase of the space address information, and meanwhile, the space address information and the corresponding data volume information can be accurately represented through the index table, so that the subsequent execution of a read command and a write command is convenient.

Optionally, before the receiving the command sent by the user side, the method further includes: traversing the X database fragments; generating index information of the X database fragments; and generating an index table of the X database fragments according to the index information.

Therefore, under the condition that the plurality of database partitions in the X database partitions all contain the same space address information, excessive occupied memory can not be caused by the increase of the space address information, and meanwhile, the space address information and the corresponding data quantity information can be accurately represented through the index table, so that the subsequent execution of a read command and a write command is convenient.

Optionally, in the case that the command is a read command, splitting the command to generate Y subcommands includes: splitting the command into X sub-read commands, wherein the X sub-read commands correspond to X database partitions; after executing the Y subcommands in the Y database partitions, the method further includes: obtaining X result data of the X database partitions; and determining the command result of the command according to the X result data.

Therefore, the read command is directly split into the X sub-read commands corresponding to the X database fragments, so that the information required to be read by the read command can be quickly and comprehensively queried, omission is avoided, and the information reading efficiency is improved.

Optionally, when the command is a first write command, the first write command is a command for writing data in at least two database partitions, the data written by the first write command is a character string or an integer, and after receiving the command sent by the user side, the method further includes: querying data quantity information of each database partition in the X database partitions through the index table, wherein Y database partitions in the X database partitions contain first space address information of the first write command; sequencing the X database fragments according to the data quantity information in the X database fragments; acquiring the sorting sequence of the X database fragments; splitting the command according to the command content of the command to generate Y subcommands, wherein the Y subcommands comprise: splitting the first write command according to the sorting order to generate Y sub first write commands, wherein each word first write command in the Y sub first write commands comprises first space address information and at least one data value, and the number of the data values of the Y sub first write commands is inversely related to the sorting order of the Y database fragments; executing the Y subcommands in the Y database shards comprises: and executing Y sub first write commands according to the data quantity information in the Y database partitions.

In this way, before executing the first write command, the index table is used to query the data volume information in each database partition, then the first write command is split according to the data volume information, and data is written into the plurality of database partitions, so that the data volume in the X database partitions is always in a relatively balanced state, and the problems of data inclination and the like are avoided.

Optionally, executing the Y sub-first write commands according to the data amount information in the Y database partitions includes: under the condition that the P database fragments of the Y database fragments comprise first space address information, replacing an original data value corresponding to the first space address information in the P database fragments with the at least one data value of the P write commands, wherein P is a positive integer smaller than or equal to Y; and writing first space address information and at least one data value corresponding to the Y-P sub first write commands into the rest database fragments except the P database fragments.

Optionally, after executing the Y sub first write commands according to the data amount information in the Y database partitions, the method further includes: calculating first index information of the Y database fragments in an index table according to a first preset algorithm; updating index information of the Y database fragments in an index table according to the first index information; the first preset algorithm corresponds to the data structure of the database shard.

Therefore, after new data is written in through the first writing command, index information (namely, space address information and data amount information corresponding to the space address information) in each database partition is possibly changed, so that the index table can be changed in time through the method, and the subsequent command can be executed in the database partition more accurately and quickly.

Optionally, in the case that the command is a second write command, the second write command includes second space address information and an integer data value, splitting the command according to command content of the command, and generating Y sub-commands includes: generating X query commands according to the second write commands, wherein the X query commands correspond to the X database partitions, and the X query commands are used for querying second space address information in the X database partitions; querying a first database partition in the X database partitions by using the X query commands, wherein the first database partition comprises second space address information of the second write command; executing the Y subcommands in the Y database shards comprises: and executing the second write command in the first database shard when the first database shard is included in the X database shards.

Thus, before executing the second write command, the first database including the second space address information in the X database partitions is split and generated according to the second write command, so as to execute the second write command in the first database. Therefore, by splitting and generating X query commands according to the second write command, the first database capable of executing the second write command can be queried in time, and the accuracy and reliability of executing the second write command are ensured.

Optionally, in the case that the second spatial address information is not included in the X database slices, the method further includes: inquiring data quantity information in X database partitions through the index table; sequencing the X database fragments according to the data quantity information in the X database fragments; acquiring the sorting sequence of the X database fragments; screening out the second database fragments with the minimum data quantity information in the X database fragments; and writing the second space address information and the integer data value in the second database partition.

Therefore, under the condition that the X database partitions do not comprise the second space address information, the data are written into the database partitions with the smallest data quantity in the X database partitions by acquiring the ordering sequence, so that the data balance can be ensured, and the data inclination problem can be avoided.

Optionally, after writing the second spatial address information and the integer data value in the second database partition, the method further includes: calculating second index information of the second database fragments in an index table according to a second preset algorithm; updating the index information of the second database fragments in an index table according to the second index information; the second preset algorithm corresponds to the data structure of the database shard.

Therefore, the index table can be changed in time by the method, so that the command can be executed in the database partition more accurately and quickly later.

Drawings

The application is further described below with reference to the accompanying drawings:

FIG. 1 is a schematic diagram of a basic flow of a data processing method according to an embodiment of the present application;

FIG. 2 is a schematic diagram of a data processing method according to an embodiment of the present application;

FIG. 3 is a second schematic diagram of a data processing method according to an embodiment of the present application;

FIG. 4 is a schematic diagram of a basic structure of a data processing apparatus according to an embodiment of the present application;

fig. 5 is a basic structural block diagram of a network device according to an embodiment of the present application.

Detailed Description

Technical solutions in the embodiments of the present application will be clearly described below with reference to the drawings in the embodiments of the present application, and it is apparent that the described embodiments are some embodiments of the present application, but not all embodiments. All other embodiments obtained by a person of ordinary skill in the art based on the embodiments in the present application are within the scope of the protection of the present application.

The terms first, second and the like in the description and in the claims, are used for distinguishing between similar objects and not necessarily for describing a particular sequential or chronological order. It is to be understood that the data so used may be interchanged, as appropriate, such that embodiments of the present application may be implemented in sequences other than those illustrated or described herein, and that the objects identified by "first," "second," etc. are generally of a type and not limited to the number of objects, e.g., the first object may be one or more. Furthermore, in the description and claims, "and/or" means at least one of the connected objects, and the character "/", generally means that the associated object is an "or" relationship.

The following terms appearing in the present application are explained as follows:

1. non-relational database (Not Only SQL, non-relational SQL, noSQL)

NoSQL can be used to solve the problem of large-scale data collection heterogeneous data in traditional databases. NoSQL has the following advantages: the NoSQL database is easy to expand, and has a plurality of types, but the common characteristic of NoSQL of any type is to remove the relational characteristic of the relational database. That is, there is no relation between data in NoSQL, which is very easy to expand. Intangible space also brings about scalability at the architectural level. NoSQL has very high read-write performance, and is excellent especially under the condition of large data volume. Thanks to its independence, the structure of the database is simple

NoSQL includes the following types: key-Value (Key-Value) stores a database, and column stores a database.

2. Remote dictionary service cluster (Remote Dictionary Server, redis)

The remote dictionary service Redis (Remote Dictionary Server), i.e., the remote dictionary service, belongs to a database of the NoSQL type. Redis is a key-value store system that supports relatively many types of values stored, including strings (strings), linked lists (lists), sets (sets), ordered sets (zset), and hashes (hashes).

Redis Cluster (Redis Cluster)

Redis cluster in this patent refers not to a dis-cluster mode of an open source in a general sense, but to an architecture formed by a plurality of fragments (sharding), each fragment stores a part of data, and all fragments are overlapped to store the whole data of the dis cluster.

4. Proxy middleware

The proxy middleware is arranged between the client and the redis cluster, processes the request sent by the client (such as shrting) to a certain extent, and forwards the request to the real redis cluster at the back end.

The proxy middleware and the redis cluster may be deployed on the same server or may be deployed on different servers, which is not limited in this embodiment of the present application.

The proxy middleware is more in type, for example, a proxy middleware twomproxy with a twitter open source on the gate, which acts as a proxy middleware for memcache and redis. The client is connected with and accesses the twomproxy, the twomproxy uniformly forwards the read-write request, and the redis cluster or memcache is distributed.

The data transmission method provided by the embodiment of the application is described in detail below through specific embodiments and application scenarios thereof with reference to the accompanying drawings.

The present embodiment provides a data processing method, as shown in fig. 1, which is applied to a network device, and includes the following steps 301 to 303:

Step 301: and receiving a command sent by the user side under the condition that the database architecture corresponding to the database comprises X database fragments and proxy middleware.

In an embodiment of the present application, the command includes: read commands, or write commands, X is an integer greater than 1.

In the embodiment of the present application, as shown in fig. 2, the database architecture 100 includes a database shard cluster (X database shards) 101 and a proxy middleware 102.

It should be noted that;

first, in the embodiment of the present application, the database shards include 2 or more database shards.

Second, in the database architecture of the present application, the X database slices do not directly receive the command (write command or read command) sent by the user side, and are received by the proxy middleware, and then the command is executed in at least one database slice. It will be appreciated that in the case where the X database slices are not deployed in the same server as the proxy middleware, the proxy middleware needs to first send the command to at least one database slice, and then execute the command in the at least one database slice. In the case where X database slices are deployed in the same server as the proxy middleware, the proxy middleware may execute the command in the at least one database slice without retransmitting the command to other servers.

In the embodiment of the present application, the data structure supported by the database architecture may be one or more.

In one possible embodiment, the data structure supported by the database architecture includes at least one of: hash, set, zset and list, to which the embodiments of the present application are not limited.

In the embodiment of the present application, the read command is used to read data from the X database partitions.

In a possible embodiment, in the case that the data structure is a hash structure, the read command may include at least one of the following: command hgeltal for acquiring all key/value pairs under a specific address (base key) in the X database slices, command hlen for acquiring data length in the specific address in the X database slices, command hget for acquiring value corresponding to the specific address in the X database slices, command hmget for acquiring value corresponding to a plurality of specific addresses in the X database slices.

In an embodiment of the present application, the write command is used to write data in at least two database partitions.

In a possible embodiment, in the case that the data structure is a hash structure, the above-mentioned write command may include at least one of the following: a command hset for writing data in one of the X database slices, a command hmset for writing data in at least two of the X database slices, a command hinyby for increasing the value of the specific address in a specific database of the X database slices, a command hdecrby for decreasing the value of the specific address in a specific database of the X database slices.

Step 302: and splitting the command according to the command content of the command to generate Y subcommands.

In this embodiment of the present application, the Y subcommands correspond to Y databases in the X database partitions, and data information related to the Y subcommands is stored in the Y databases, where Y is smaller than or equal to X, and Y is an integer.

In the embodiment of the present application, the manner of splitting the command is different according to the difference of the command content of the command.

In one embodiment, when the command is a read command, the proxy middleware of the network device splits the read command into X sub-read commands, each of which may be used to read data in the database partition corresponding to the address of its command.

Further, the command contents of the above X sub-read commands may be the same, and each sub-read command is used to read data in one database partition.

In another embodiment, when the command is a write command, the proxy middleware of the network device first queries data amount information in the X database partitions or a situation corresponding to the write command (the corresponding situation refers to whether the write command can be executed in the X database partitions), obtains a query result, and then splits the write command into at least one sub-write command according to the query result. It will be appreciated that after splitting the write command, the number of sub-write commands is related to the specific command content of the write command, and the following details will be described in detail herein.

Step 303: the Y subcommands are executed in the Y database partitions.

In the data processing method provided in the embodiment of the present application, when a database architecture corresponding to a database includes X database slices and proxy middleware, a read command or a write command is received, and then the command is split according to a command content of the command, so as to generate Y subcommands corresponding to Y databases in the X database slices (data information related to the Y subcommands is stored in the Y databases); finally, the Y subcommands are executed in the Y database partitions. Therefore, by splitting the command, the command is not required to be concentrated in a single database partition to be executed, but is executed in a plurality of database partitions, and for the write command, the data can be stored in different databases in a distributed manner, so that the problem of data inclination caused by data storage in a single database partition is avoided, and the use efficiency of the database partitions is improved.

Optionally, in an embodiment of the present application, the database includes an index table, where the index table includes index information, and the index information includes: the data volume information of each space address information in each database partition in the X database partitions; the data format of the spatial address information corresponds to the data format in the X database partitions.

Illustratively, the above data amount information refers to: how much data is for each spatial address in each database tile. For example, for any database shard, there are 3 pieces of data corresponding to the first space address in the database shard 1, and the 3 pieces are data amount information.

It will be appreciated that:

first, the index table is used for indicating the space address information in each database partition in the X database partitions and the data amount information corresponding to each space address information.

Second, the spatial address information in different database partitions may be different, and the data amount information corresponding to the spatial address information in different database partitions may also be different.

For example, the index table includes 2 database slices, namely a database slice 1 and a database slice 2, and the database slice 1 includes 3 pieces of space address information, namely space address information 1, space address information 2 and space address information 3, and the database slice 2 includes 2 pieces of space address information, namely space address information 3 and space address information 4. The data volume information corresponding to the space address information 1 in the database shard 1 is 3, the data volume information corresponding to the space address information 2 in the database shard 1 is 2, and the data volume information corresponding to the space address information 3 in the database shard 1 is 2; the data amount information corresponding to the space address information 3 in the database fragment 2 is 4, and the data amount information corresponding to the space address information 4 in the database fragment 2 is 2.

In one embodiment, when the data structure is a hash structure, the spatial address information may be a base key.

The spatial address information is stored in a data format corresponding to the data structure of the database. For example, when the data structure of the database is a hash structure, then the spatial address information in each database fragment is stored using the hash structure.

Optionally, in the embodiment of the present application, before the step 301 receives the command sent by the user side, the data transmission method provided in the embodiment of the present application may include the following steps A1 to A3:

step A1: traversing the X database partitions.

Step A2: and generating index information of the X database partitions.

Step A3: and generating an index table of the X database fragments according to the index information.

It will be appreciated that the above steps A1 to A3 may be used to generate a new index table, and thus, the above steps A1 to A3 may be used in a scenario without an index table or in a scenario requiring updating the index table again, or may be used in other scenarios, which is not limited in this embodiment of the present application.

Illustratively, traversing the X database partitions refers to: and traversing and reading all the database related information of the database fragments once.

Illustratively, generating the index information for the X database partitions described above refers to: the space address information in each database fragment and the data volume information corresponding to the space address information in each database fragment are calculated according to a calculation mode corresponding to the data structure of the database, and index information is generated.

It can be understood that the data structure of the native space address information in each database partition directly acquired by the network device is generally of a character string type, and the memory space occupied by the character string type is larger, so that the native space address information can be converted into other data types occupying smaller memory space by calculating the native space address information. In general, the network device calculates the native spatial address information in a manner corresponding to the data structure of the database, and changes the native spatial address information data type.

Example 1: assuming that the data included in the database slice 1 is a base key1 value1 key2 value2, where base is the name of its native spatial address information, the subsequent key1 value1 is one key value pair, and key2 value2 is another key value pair, that is, the database slice 1 contains the native spatial address information base and 2 key value pairs, the data size information of the base in the database slice 1 is 2. Assuming that index information needs to be generated according to the data stored in the database shard 1, and the data structure of the database is a hash structure, referring to fig. 3, the generation process is as follows: first, spatial address information in index information is generated: splitting the original space address information base into 'ba' and 'se', performing hash operation on the first half character string by using a murmur hash3 algorithm to obtain 0xFF3e, performing hash operation on the second half character string by using a cityhash algorithm to obtain 0xe340, and performing exclusive-or operation on the two parts of results to obtain a result 0xFF35, wherein the result is used as the space address information in the index information; second, data amount information of the space address information in the index information is generated: according to the key value pair number, 2 data amount information can be stored.

It should be noted that, the data amount information corresponding to the index table is an array. In order to save the memory usage of the proxy middleware in the network device, static array allocation can be performed on numbers from 0 to 100000, that is, as long as the number of key value pairs is below 100000, pointers in the static array are actually stored, and when the number of key value pairs is only greater than 100000, the real number of key value pairs information (namely, the data quantity information) needs to be stored in the allocated memory. After obtaining the result 0xff35, the data size information of the corresponding space address information is then indicated by a stored pointer, as shown in fig. 2, which points to a value in the static array.

The network device gathers the spatial address information in each database partition and the data amount information corresponding to the spatial address information in each database partition, and finally generates an index table.

Optionally, in the embodiment of the present application, in the case where the command is a read command, splitting the command in the step 302 to generate Y subcommands may include the following steps B1 to B3:

step B1: the above command is split into X sub-read commands.

On the basis of the step B1, after the step 303, the data transmission method provided in the embodiment of the present application further includes the following steps B2 and B3:

step B2: and obtaining X result data of the X database partitions.

Step B3: and determining the command result of the command according to the X result data.

Illustratively, the X sub-read commands described above correspond to X database partitions.

Illustratively, the splitting into X sub-read commands described above may include: the command content of the read command is copied into X sub-read command contents, then X database block addresses are respectively added to each sub-read command content, and finally the X database block addresses and the X sub-read command contents are synthesized to generate X sub-read commands.

Illustratively, the above X sub-read commands corresponding to X database partitions refer to: the command addresses of the X sub-read commands are in one-to-one correspondence with the database shard addresses in the X database shards.

For example, after the X database fragments execute the X sub-read commands, X result data will be generated, and the network device may determine a final command result according to the X result data.

It can be appreciated that in the embodiment of the present application, since the same spatial address information and corresponding specific key value pairs may occur in multiple, even all, database partitions, during the process of executing the read command, the read command is directly split into X sub-read commands corresponding to the X database partitions.

Taking the foregoing different read commands as an example, assume that the X database partitions are 3 database partitions, redis1, redis2, and redis3, respectively, and the proxy middleware is twomproxy, to name 4 examples of executing read commands.

Example 2: assuming the read command is Hlen, twomproxy of the network device splits Hlen into 3 sub-read commands, which correspond one-to-one with redis1, redis2, and redis3, respectively. Then, 3 sub-read commands are sent to redis1, redis2 and redis3 one by one, and then the lengths of the corresponding space address information (i.e., the above X pieces of result data) are acquired in redis1, redis2 and redis3, wherein the result data of redis1 is 5, the result data of redis2 is 6, and the result data of redis3 is 6. Finally, according to the 3 result data, the result data is combined and calculated in the twomproxy, the twomproxy performs the combined and accumulated operation 5+6+6=17 on the three results, and finally sends 17 to the user side.

Example 3: assuming that the read command is Hget, the twomproxy of the network device splits the Hget into 3 sub-read commands, which are in one-to-one correspondence with redis1, redis2, and redis3, respectively. Then, 3 sub-read commands are sent to redis1, redis2 and redis3 one by one, and then a value (i.e., the above X pieces of result data) corresponding to one of the space address information is acquired in redis1, redis2 and redis3, wherein the result data of redis1 is3, the result data of redis2 is 0 (i.e., null), and the result data of redis3 is 2. Finally, according to the 3 result data, the summary is merged in the twomproxy, the twomproxy filters out the empty result data in the redis2, and sends the result data 3 of the redis1 and the result data 2 of the redis3 to the client.

Example 4: assuming that the read command is Hmget, twomproxy of the network device splits Hmget into 3 sub-read commands, which are in one-to-one correspondence with redis1, redis2, and redis3, respectively. Then, 3 sub-read commands are sent to redis1, redis2 and redis3 one by one, and then values (namely the above X pieces of result data) corresponding to Z pieces of space address information are acquired in redis1, redis2 and redis3, wherein the result data of redis1 is value1, the result data of redis2 is value2, and the result data of redis3 is value3. Finally, according to the 3 result data, merging and summarizing in the twomproxy, and sending the result data value1 of redis1, the result data value3 of redis2 and the result data value3 of redis3 to the user side.

Example 5: assuming that the read command is Hgetal, the twomproxy of the network device splits the Hgetal into 3 sub-read commands, which are in one-to-one correspondence with redis1, redis2, and redis3, respectively. Then, 3 sub-read commands are sent to redis1, redis2 and redis3 one by one, and then in redis1, the corresponding key/value of the space address information (namely the above X pieces of result data) is acquired in redis2 and redis3, wherein the result data of redis1 is key1/value1, the result data of redis2 is key2/value2, and the result data of redis3 is key3/value3. Finally, according to the 3 result data, merging and summarizing in the twomproxy, and sending the result data key1/value1 of redis1, the result data key2/value2 of redis2 and the result data key3/value3 of redis3 to the user side.

Optionally, in the embodiment of the present application, in the case where the command is a first write command, the first write command is a command for writing data in at least two database slices, the data written by the first write command is a character string or an integer, and after the command sent by the receiving user side in step 301, the data transmission method provided in the embodiment of the present application may include the following steps C1 to C5:

Step C1: and querying data quantity information of each database partition in the X database partitions and Y database partitions containing first space address information of a first write command in the X database partitions through the index table.

Step C2: and sequencing the X database fragments according to the data volume information in the X database fragments.

Step C3: and acquiring the ordering sequence of the X database partitions.

Based on the steps C1-C3, in the step 302, the data transmission method provided in the embodiment of the present application may include the following step C4:

step C4: splitting the first write command according to the sorting order to generate Y sub first write commands.

Illustratively, each of the Y sub-first write commands includes first space address information and at least one data value therein, the number of data values of the Y sub-first write commands being inversely related to the order of ordering of the Y database partitions.

Based on the foregoing C1-C4, in the foregoing step 303, the data transmission method provided in the embodiment of the present application may include the following step C5:

step C5: and executing Y sub first write commands according to the data quantity information in the Y database partitions.

The data writing manner of the first writing command is, for example, alternative writing. The above-mentioned replacement writing means: and replacing the numerical content (for example, value) except the space address information carried by the first write command with the numerical content in the original space address information.

It should be noted that if the database fragment written by the first write command does not include the space address information carried by the first write command, the database fragment is directly written.

In one example, when the data structure is a hash structure, the first write command may be hmset.

For example, the foregoing data amount information may refer to the foregoing description, and will not be repeated herein.

The first space address information refers to space address information corresponding to a first write command. It can be appreciated that the first spatial address information may be spatial address information existing in all of the X database partitions; the method can also be space address information which does not exist in the database in the X database partitions; and the space address information of partial database fragments (Y database fragments) and the rest database fragments which do not exist in the X database fragments can also exist in the partial database fragments.

Illustratively, the sorting of the X database partitions may be sorting the X database partitions from large to small according to the size of the data volume information; the X database partitions may also be ordered from small to large according to the size of the data size information, which is not limited in the embodiment of the present application.

It will be appreciated that after the information of the data amount and the ordering order are obtained, when executing the first write command, in order to equalize the number of data values of each of the X database partitions, it is necessary to write the most data in the database partition with the least number of data values, write the least data, or even not write the most data in the database partition with the most number of data values, split the first write command into Y sub-first write commands (Y is less than or equal to X, and when Y is less than X, it indicates that some database partitions have too much data to be written), and the number of data values of each sub-first write command is inversely related to the ordering thereof.

Taking a write command as an example of hmset, assuming that X database slices are 3 database slices, which are redis1, redis2 and redis3 respectively, the proxy middleware is twomproxy, the following 4 examples of executing hmset commands are listed, and specifically, the hmset commands are: hmset base key1/value1, key2/value2, key3/value3, key4/value4, key5/value5.

Example 6: and inquiring the data quantity information of the redis1, the redis2 and the redis3 and the first space address information base corresponding to hmset in the index table. And if the data quantity information of the redis1 is3, the data quantity information of the redis2 is2, and the data quantity information of the redis3 is3, and the first space address information base is not included in the redis1, the redis2 and the redis3, the ordering sequence of the redis1, the redis2 and the redis3 is the redis1, the redis3 and the redis2 from large to small according to the data quantity information. According to the above ordering order, the hmset command is split into 3 sub-hmset commands, hmset base key1/value1, key2/value2, and hmset base key3/value3, key4/value4, and hmset base key5/value5, respectively. And executing the 3 pieces of sub hmset command in redis1, redis2 and redis3, respectively, and dividing data volume information of redis1, redis2 and redis3 into 5,4,4 after execution.

It should be noted that, generally, after the split Y sub-write commands are executed, the difference between the data amount information in each database partition will not exceed 1.

Optionally, in the embodiment of the present application, in the step C5, the data transmission method provided in the embodiment of the present application may include the following step D1 and step D2:

step D1: and under the condition that the P database fragments of the Y database fragments comprise the first space address information, replacing the original data value corresponding to the first space address information in the P database fragments with the at least one data value of the P write commands, wherein P is a positive integer less than or equal to Y.

Step D2: and writing first space address information and at least one data value corresponding to the Y-P sub first write commands into the rest database fragments except the P database fragments.

Illustratively, according to the foregoing, since the first write command is executed in an alternative manner, that is, in a case where the first space address information is included in the database partition in which the sub-write command corresponding to the first write command is executed, it is necessary to replace the original data value with the data value of the above-described sub-write command.

Example 6: and inquiring the data quantity information of the redis1, the redis2 and the redis3 and the first space address information base corresponding to hmset in the index table. The data volume information of the redis1 is3, the data volume information of the redis2 is2, the data volume information of the redis3 is3, and the first space address information base is included in the redis1, the first space address information base is not included in the redis2 and the redis3, and then the ordering sequences of the redis1, the redis2 and the redis3 are the redis1, the redis3 and the redis2 from large to small according to the data volume information. According to the above ordering order, the hmset command is split into 3 sub-hmset commands, namely hmset base key1/value1, key2/value2, key3/value3, and hmset base key4/value4, and hmset base key5/value5, respectively. And executing the 3 pieces of sub hmset command in redis1, redis2 and redis3 respectively, and dividing the data volume information after execution into 3,3 and 4.

Optionally, in the embodiment of the present application, after step C5, the data transmission method provided in the embodiment of the present application may include the following steps E1 and E2:

step E1: and calculating first index information of the Y database fragments in an index table according to a first preset algorithm.

Step E2: and updating the index information of the Y database fragments in an index table according to the first index information.

The first preset algorithm is illustratively corresponding to the data structure of the database shard.

In one example, in a case where the data structure of the database shard is a hash structure, the first preset algorithm is a hash algorithm.

It can be understood that after new data is written in the Y database partitions by the first write command, the index information of the Y databases is changed, so that the first index information in the Y database partitions can be obtained by calculating the first index information, and the index table can be updated in time.

Further, the manner of calculating the first index information in the step E1 and the step E2 may refer to the manner of generating the index information in the step A1 to the step A2, which is not described herein again.

Optionally, in the embodiment of the present application, in the case where the command is a second write command, the second write command includes second space address information and an integer data value, and in step 302, the data transmission method provided in the embodiment of the present application may include the following steps F1 and F2:

step F1: and generating X query commands according to the second write command.

Step F2: and querying a first database fragment in the X database fragments by using the X query commands.

Based on the step F1 and the step F2, in the step 303, the data transmission method provided in the embodiment of the present application may include the following step F3:

step F3: and executing the second write command in the first database shard when the first database shard is included in the X database shards.

Illustratively, the X query commands correspond to the X database partitions.

Illustratively, the first database fragment includes second spatial address information of the second write command.

Illustratively, the X query commands are used to query the second spatial address information in the X database partitions. That is, the X query commands are used to query whether the second spatial address information exists in the X database partitions.

In one example, the X query commands may be X query commands generated by splitting the network device according to the second write command, where each query command may be used to query the second spatial address information in the X database partitions.

The writing data of the second writing command is written in a superposition manner or a decremental manner. The above superimposed writing means: numerical content (e.g., integer numerical value) other than the space address information carried by the second write command is superimposed on the numerical content in the original space address information. The decremental write refers to: the value content (e.g., integer value) outside the space address information carried by the second write command is decremented based on the value content in the original space address information.

It should be noted that if the database fragment written by the second write command does not include the second space address information carried by the second write command, the direct writing may not continue to use the second write command for writing, and the following steps G1-G5 are referred to in detail.

In one example, when the data structure is a hash structure, the second write command may be a hi ncrby or hdecrby.

It can be understood that, since the writing data manner of the second writing command is a superposition manner or a decrementing manner, in general, whether the X database slices include the second space address corresponding to the second writing command is first queried, and if the second space address exists, the second writing command is directly executed in the database slice corresponding to the second space address (i.e., the first database slice).

Taking a write command as a hit as an example, assuming that the X database slices are 3 database slices, which are redis1, redis2 and redis3 respectively, the proxy middleware is twomproxy, the following 4 examples of executing hit commands are listed, and specifically, the hit commands are: hincrby base key value. Where value is an integer type and may not be a string type.

Example 7: after receiving the hit command, the network device generates 3 hit commands (i.e. the query command described above), where the 3 hit commands are used to query keys in the hit command (i.e. the second spatial address information described above), and send 3 hit commands to corresponding redis1, redis2, and redis3, respectively, where redis3 includes keys in the hit command, and redis is the first database slice, and then superimpose values corresponding to the keys in redis3 (i.e. execute the second write command described above).

Inquiring the data volume information in each database partition, splitting a first writing command according to the data volume information, and writing data in a plurality of database partitions, so that the data volume in X database partitions is always in a relatively balanced state, and the problems of data inclination and the like are avoided.

Optionally, in the embodiment of the present application, in a case where the X database slices do not include the second spatial address information, the data transmission method provided in the embodiment of the present application may further include the following steps G1 to G5:

step G1: and inquiring the data quantity information in the X database partitions through the index table.

Step G2: and sequencing the X database fragments according to the data volume information in the X database fragments.

Step G3: and acquiring the ordering sequence of the X database partitions.

Step G4: and screening out the second database fragments with the smallest data quantity information in the X database fragments.

Step G5: and writing the second space address information and the integer data value in the second database partition.

Illustratively, the above ordering sequence refers to the foregoing description, and is not repeated herein.

It will be appreciated that, since the above second spatial address information is not included in all of the X database slices, it is necessary to write the second spatial address information and an integer data value corresponding to the second spatial address information in one of the X database slices, and in order to ensure the data size uniformity of the database slices, it is necessary to directly write data in the database slice with the smallest data size among the X database slices using the first write command.

In one embodiment, in steps G1-G5 above, where the data structure is a hash structure, the data may be written using an hset command.

Example 8: after receiving the hit command, the network device generates 3 hit commands (i.e. the query command described above), where the 3 hit commands are used to query keys in the hit command (i.e. the second spatial address information described above), and send 3 hit commands to corresponding redis1, redis2, and redis3, where each of redis1, redis2, and redis3 does not include a key in the hit command, and queries data amount information in 3 database fragments through the index table, and ranks redis1, redis2, and redis3 from large to small according to the data amount information, where the ranking order of redis1, redis3, and redis2 is redis1, redis3, and redis2. The network device screens out that redis2 is the database fragment with the smallest data amount information in the 3 database fragments, so that the hset command, namely hset base key value, is executed in redis2, and the base key value corresponding to the above-mentioned high command is written.

Optionally, in the embodiment of the present application, after the step G5, the data transmission method provided in the embodiment of the present application may further include the following step H1 and step H2:

step H1: and calculating second index information of the second database fragments in the index table according to a second preset algorithm.

Step H2: updating the index information of the second database fragments in an index table according to the second index information;

the second preset algorithm is illustratively corresponding to the data information of the database shard.

In one example, in a case where the data structure of the database shard is a hash structure, the second preset algorithm is a hash algorithm.

It can be understood that after new data is written in the Y database partitions by the second write command, the index information of the Y databases is changed, so that the second index information can be obtained by calculating the second index information in the Y database partitions, and the index table can be updated in time.

Further, the manner of calculating the second index information in the step H1 and the step H2 may refer to the manner of generating the index information in the step A1 to the step A2, which is not described herein again.

Fig. 4 is a schematic structural diagram of a first possible implementation of the data transmission device according to the embodiment of the present application. As shown in fig. 4, the data transmission apparatus 600 includes: the receiving module 601 is configured to receive a command sent by a user side, where a database architecture corresponding to a database includes X database fragments and proxy middleware, where the command includes: a read command, or a write command, X is an integer greater than 1; the execution module 602 is configured to split the command according to the command content of the command received by the receiving module 602, generate Y subcommands, where the Y subcommands correspond to Y databases in the X database partitions, the Y databases store data information related to the Y subcommands, and Y is less than or equal to X, where Y is an integer; the execution module is further configured to execute the Y subcommands in the Y database partitions.

The data processing device receives a read command or a write command under the condition that a database architecture corresponding to the database comprises X database fragments and proxy middleware, then splits the command according to the command content of the command to generate Y subcommands corresponding to Y databases in the X database fragments (the Y databases store data information related to the Y subcommands); finally, the Y subcommands are executed in the Y database partitions. Therefore, by splitting the command, the command is not required to be concentrated in a single database partition to be executed, but is executed in a plurality of database partitions, and for the write command, the data can be stored in different databases in a distributed manner, so that the problem of data inclination caused by data storage in a single database partition is avoided, and the use efficiency of the database partitions is improved.

Optionally, the data processing apparatus 600 further includes: the traversing module is used for traversing the X database fragments; the generation module is used for generating index information of the X database fragments; the generation module is further configured to generate an index table of the X database partitions according to the index information.

Optionally, the data processing apparatus 600 further includes: in the case that the command is a read command, the execution module 602 is specifically configured to split the command into X sub-read commands, where the X sub-read commands correspond to X database partitions; the acquisition module is used for acquiring X result data of the X database fragments; and the determining module is used for determining the command result of the command according to the X result data.

Optionally, in the case where the command is a first write command, the first write command is a command for writing data in at least two database partitions, the data written by the first write command is a character string or an integer, and the data processing apparatus 600 further includes: the query module is used for querying data volume information of each database fragment in the X database fragments and Y database fragments containing first space address information of the first write command in the X database fragments through the index table; the sorting module is used for sorting the X database fragments according to the data quantity information in the X database fragments; the acquisition module is used for acquiring the sorting sequence of the X database fragments; the execution module 602 is specifically configured to split the first write command according to the sorting order to generate Y sub-first write commands, where each word first write command in the Y sub-first write commands includes first space address information and at least one data value, and the number of data values of the Y sub-first write commands is inversely related to the sorting order of the Y database partitions; the execution module is further specifically configured to execute the Y sub-first write commands according to the data amount information in the Y database partitions.

Optionally, the executing module 602 is specifically configured to, when the P database slices of the Y database slices include the first spatial address information, replace the original data value corresponding to the first spatial address information in the P database slices with the at least one data value of the P write commands, where P is a positive integer less than or equal to Y; the execution module is further specifically configured to write first space address information and at least one data value corresponding to the Y-P sub first write commands into the rest of the database partitions except the P database partitions.

Optionally, the data processing apparatus 600 further includes: the computing module is used for computing first index information of the Y database fragments in the index table according to a first preset algorithm; the updating module is used for updating the index information of the Y database fragments in the index table according to the first index information; the first preset algorithm corresponds to the data structure of the database shard.

Optionally, in the case that the command is a second write command, the second write command includes second space address information and an integer data value, and the data processing apparatus 600 further includes: the execution module 602 is specifically configured to generate X query commands according to the second write command, where the X query commands correspond to the X database partitions, and the X query commands are used for querying second spatial address information in the X database partitions; the query module is used for querying a first database fragment in the X database fragments by using the X query commands, wherein the first database fragment comprises second space address information of the second write command; the execution module is specifically configured to execute the second write command in the first database shard when the first database shard is included in the X database shards.

Optionally, in a case where the second spatial address information is not included in the X database slices, the data processing apparatus 600 further includes: the query module is used for querying data quantity information in the X database partitions through the index table; sequencing the X database fragments according to the data quantity information in the X database fragments; the acquisition module is used for acquiring the sorting sequence of the X database fragments; screening out the second database fragments with the minimum data quantity information in the X database fragments; the execution module 600 is specifically configured to write the second spatial address information and the integer data value in the second database partition.

Optionally, the data processing apparatus 600 further includes: the calculation module is used for calculating second index information of the second database fragments in the index table according to a second preset algorithm; the updating module is used for updating the index information of the second database fragments in the index table according to the second index information; the second preset algorithm corresponds to the data structure of the database shard.

The embodiment of the application also provides a network device, which comprises a processor and a communication interface, wherein the processor is used for receiving a command sent by a user terminal under the condition that a database architecture corresponding to a database comprises X database fragments and proxy middleware, splitting the command according to the command content of the command, and executing the Y subcommands in the Y database fragments. The network device embodiment corresponds to the network device method embodiment, and each implementation process and implementation manner of the method embodiment can be applied to the network device embodiment, and the same technical effects can be achieved.

Specifically, the embodiment of the application also provides a network device. As shown in fig. 5, the network device 700 includes: an antenna 71, a radio frequency device 72, a baseband device 73, a processor 74 and a memory 75. The antenna 71 is connected to a radio frequency device 72. In the uplink direction, the radio frequency device 72 receives information via the antenna 71, and transmits the received information to the baseband device 73 for processing. In the downlink direction, the baseband device 73 processes information to be transmitted, and transmits the processed information to the radio frequency device 72, and the radio frequency device 72 processes the received information and transmits the processed information through the antenna 71.

The method performed by the network device in the above embodiment may be implemented in a baseband apparatus 73, the baseband apparatus 73 comprising a baseband processor.

The baseband device 73 may, for example, comprise at least one baseband board, on which a plurality of chips are disposed, as shown in fig. 5, where one chip, for example, a baseband processor, is connected to the memory 75 through a bus interface, so as to invoke a program in the memory 75 to perform the network device operation shown in the above method embodiment.

The network device may also include a network interface 76, such as a common public radio interface (common public radio interface, CPRI).

Specifically, the network device 700 of the embodiment of the present invention further includes: instructions or programs stored in the memory 75 and executable on the processor 74, the processor 74 invokes the instructions or programs in the memory 75 to perform the methods performed by the modules shown in fig. 5 and achieve the same technical effects, and are not repeated here.

The embodiment of the present application further provides a readable storage medium, where a program or an instruction is stored on the readable storage medium, and when the program or the instruction is executed by a processor, the program or the instruction implements each process of the method embodiment of the data processing method, and the same technical effects can be achieved, so that repetition is avoided, and no further description is given here.

Wherein the processor is a processor in the terminal described in the above embodiment. The readable storage medium includes computer readable storage medium such as computer readable memory ROM, random access memory RAM, magnetic or optical disk, etc.

While the invention has been described with reference to several particular embodiments, it will be understood by those skilled in the art that various changes may be made and equivalents may be substituted without departing from the scope of the invention. In addition, many modifications may be made to adapt a particular situation or material to the teachings of the invention without departing from its scope. Therefore, it is intended that the invention not be limited to the particular embodiment disclosed, but that the invention will include all embodiments falling within the scope of the appended claims.

Claims

1. A data processing method, for application to a network device, the method comprising:

and receiving a command sent by a user side under the condition that a database architecture corresponding to the database comprises X database fragments and proxy middleware, wherein the command comprises the following components: a read command, or a write command, X is an integer greater than 1;

splitting the command according to the command content of the command to generate Y subcommands, wherein the Y subcommands correspond to Y databases in X database partitions, data information related to the Y subcommands is stored in the Y databases, Y is smaller than or equal to X, and Y is an integer;

the Y subcommands are executed in the Y database tiles.

2. The method of claim 1, wherein the database comprises an index table, the index table comprising index information, the index information comprising: space address information in each database partition in the X database partitions and data volume information of the space address information in each database partition; the data format of the spatial address information corresponds to the data format in the X database partitions.

3. The method according to claim 1 or 2, wherein before receiving the command sent by the user terminal, the method further comprises:

Traversing the X database partitions;

generating index information of the X database partitions;

and generating an index table of the X database fragments according to the index information.

4. A method according to any one of claims 1 to 3, wherein, in the case where the command is a read command, the splitting the command generates Y subcommands, comprising:

splitting the command into X sub-read commands, wherein the X sub-read commands correspond to X database partitions;

after said executing said Y subcommands in said Y database tiles, the method further comprises:

obtaining X result data of the X database partitions;

and determining a command result of the command according to the X result data.

5. The method according to claim 2, wherein in the case that the command is a first write command, the first write command is a command for writing data in at least two database partitions, the data written by the first write command is a character string or an integer, and after the command sent by the receiving user side, the method further comprises:

querying data quantity information of each database fragment in the X database fragments and Y database fragments containing first space address information of the first write command in the X database fragments through the index table;

Sequencing the X database partitions according to the data quantity information in the X database partitions;

acquiring the sorting sequence of the X database partitions;

splitting the command according to the command content of the command to generate Y subcommands, wherein the Y subcommands comprise:

splitting the first write command according to the sorting order to generate Y sub first write commands, wherein each word first write command in the Y sub first write commands comprises first space address information and at least one data value, and the number of the data values of the Y sub first write commands is inversely related to the sorting order of the Y database fragments;

the executing the Y subcommands in the Y database tiles comprises:

and executing Y sub first write commands according to the data volume information in the Y database partitions.

6. The method of claim 5, wherein said executing Y sub-first write commands based on data volume information in said Y database partitions comprises:

under the condition that P database fragments of the Y database fragments comprise first space address information, replacing original data values corresponding to the first space address information in the P database fragments with the at least one data value of the P write commands, wherein P is a positive integer smaller than or equal to Y;

And writing first space address information and at least one data value corresponding to the Y-P sub first write commands into the rest database fragments except the P database fragments.

7. The method of claim 5, wherein after said executing Y sub-first write commands according to data volume information in said Y database tiles, the method further comprises:

calculating first index information of the Y database fragments in an index table according to a first preset algorithm;

updating index information of the Y database fragments in an index table according to the first index information;

the first preset algorithm corresponds to a data structure of the database shard.

8. The method of claim 2, wherein in the case where the command is a second write command, the second write command includes second space address information and an integer data value, the splitting the command according to command contents of the command, generating Y subcommands includes:

generating X query commands according to the second write commands, wherein the X query commands correspond to the X database partitions, and the X query commands are used for querying second space address information in the X database partitions;

Querying a first database fragment in the X database fragments by using the X query commands, wherein the first database fragment comprises second space address information of the second write command;

the executing the Y subcommands in the Y database tiles comprises:

and executing the second write command in the first database shard when the first database shard is included in the X database shards.

9. The method of claim 8, wherein in the event that the second spatial address information is not included in the X database slices, the method further comprises:

inquiring data quantity information in X database partitions through the index table;

sequencing the X database fragments according to the data volume information in the X database fragments;

acquiring the sorting sequence of the X database partitions;

screening out a second database fragment with the minimum data quantity information in the X database fragments;

writing the second spatial address information and the integer data value in the second database partition.

10. The method of claim 8, wherein after writing the second spatial address information and the integer data value in the second database tile, the method further comprises:

Calculating second index information of the second database fragments in an index table according to a second preset algorithm;

updating the index information of the second database fragment in the index table according to the second index information;

the second preset algorithm corresponds to a data structure of the database shard.

11. A data processing apparatus, the apparatus comprising:

the receiving module is configured to receive a command sent by a user side when a database architecture corresponding to a database includes X database fragments and proxy middleware, where the command includes: a read command, or a write command, X is an integer greater than 1;

the execution module is used for splitting the command according to the command content of the command received by the receiving module to generate Y subcommands, the Y subcommands correspond to Y databases in the X database partitions, the Y databases store data information related to the Y subcommands, and Y is smaller than or equal to X and is an integer;

the execution module is further configured to execute the Y subcommands in the Y database partitions.

12. A network device comprising a processor and a memory storing a program or instructions executable on the processor, which when executed by the processor, implement the steps of the data processing method of any one of claims 1 to 10.

13. A readable storage medium, characterized in that the readable storage medium has stored thereon a program or instructions which, when executed by a processor, implement the steps of the data processing method according to any of claims 1-10.