WO2020199760A1

WO2020199760A1 - Data storage method, memory and server

Info

Publication number: WO2020199760A1
Application number: PCT/CN2020/075170
Authority: WO
Inventors: 閤先军; 尹文波; 王伶俐; 邱云辉; 任予琛; 陈华; 余博伟
Original assignee: 华为技术有限公司
Priority date: 2019-03-29
Filing date: 2020-02-14
Publication date: 2020-10-08
Also published as: CN110083307A

Abstract

A data storage method, a memory and a server. The specific method comprises: a key-value memory receiving a key-value storage request, the key-value storage request comprising a key and a first value; when the first value is large-size data, the key-value memory segmenting the first value into a plurality of second values, and generating a plurality of secondary key-value storage requests, each secondary key-value storage request carrying the key and one of the second values; then, according to the position sequence, in the first value, of the second value carried in each secondary key-value storage request, storing in a first storage space the second value carried in each secondary key-value storage request, thereby solving the problem of low data storage efficiency of large-size data in a key-value storage system.

Description

Data storage method, storage and server

Technical field

This application relates to the field of storage technology, and in particular to a data storage method, storage and server.

Background technique

Key-value storage technology is a data access method that uses key pairs to store data. Each key corresponds to a value. Key-value storage technology is widely used in data centers. With the development of semiconductor technology, the cost of storage has dropped rapidly, and more and more high-performance storage has been applied to data centers, forming different types of key-value storage systems.

The key-value storage system has gradually developed from a system based on mechanical hard disks to a cache-type system that uses dynamic random access memory (DRAM) as a cache, and has developed to store all data including keys and values in DRAM A memory-based system as memory.

However, when the size of the value corresponding to the key is large, the data storage efficiency is low in the process of writing the value. Take the writing process of a value as an example. First of all, the value needs to be cached. Because the size of the value is large, a large cache space is required. In order to store the value completely in DRAM, the value needs to be stored The transmission process from the cache to the DRAM is started after the complete cache. The cache time and transmission time of this value are relatively long during the entire process, resulting in a large transmission delay and low data storage efficiency.

Summary of the invention

This application provides a data storage method, memory and server to solve the problem of low data storage efficiency of large-size data in the key-value storage system in the traditional technology.

In a first aspect, the present application provides a data storage method. The method includes: first, the key-value storage receives a key-value storage request, the key-value storage request includes a key and a first value, and the key and the first value are corresponding; In order to store the first value, the key-value memory may divide the first storage space for storing the first value in the first memory, where the first memory may be a memory other than the key-value memory; if the first value is a large size Data; Exemplarily, the capacity of the data contained in the first value is greater than or equal to the first threshold, the key-value storage can perform data segmentation, divide the first value into multiple second values, and generate multiple secondary key-value stores Request, each secondary key value storage request can carry a key and one of the second values; the number of secondary key value storage requests is the same as the number of multiple second values obtained by dividing the first value. When storing the first value, the key-value storage can separately store the second value carried in each secondary key-value storage request according to the position order of the second value carried in each secondary key-value storage request in the first value. The value is stored in the first storage space. Among them, the capacity of the data contained in the first value can also be referred to as the size of the first value, which refers to the total size of the data contained in the first value, or the minimum storage space required to store the first value. the size of. For example, if the content contained in the first value is "0123456789", the capacity of the data contained in the first value is 10 bytes.

Through the above method, for large-size data, the key-value memory can be divided into multiple smaller second values, and the value of the smaller size can effectively shorten the time of data buffering and transmission, reduce the delay, and be more effective Improve the data storage efficiency of large-size data.

In a possible design, when the key-value memory divides the first value into multiple second values, it can be divided while caching; for example, the key-value memory can cache the first value (such as in the key-value memory). In the process of caching the first value in the second memory), whenever the size of the cached data is less than or equal to the second threshold, the cached data is taken as the second value and carried in a secondary key value storage request until The first value cache is completed. When the final stage of the first value of the cache, the size of the last cached data is less than the second threshold, the cached data can be directly used as the second value and carried in a secondary key value storage request; in this case, each time The size of the second value carried in the level key value storage request is not necessarily the same.

Through the above method, the method of partitioning while caching is adopted, and the operation of data partitioning can be completed at the same time during the caching process, data partitioning can be performed faster, data storage can be performed as soon as possible, and data storage efficiency can be improved.

In a possible design, when the key-value memory divides the first value into a plurality of second values, the data division method may also be adopted after the cache is completed. Exemplarily, after the first value cache is completed, the key value storage may divide the first value into multiple second values, and the key value storage may divide the first value into N+1 second values, in which the order of division is Among the multiple second values arranged, the size of each of the first N second values can be equal to the second threshold, and the size of the N+1th second value can be less than or equal to the second threshold, and N is A positive integer greater than or equal to 1. Exemplarily, the key value storage may also set the second value to multiple second values.

Through the above method, the method of caching first and then segmentation is adopted, and the key value memory can set the second threshold value according to the size of the first value, which can segment the first value more flexibly.

In a possible design, a field may be set in the secondary key value storage request to indicate the relevant information of the second value carried, such as the size and offset of the second value. Exemplarily, a value offset may be included, and the value offset may indicate the position of the second value carried in the secondary key value storage request in the first value.

Through the above method, by setting the corresponding fields, the key value storage can more conveniently determine the related information of the second value, which can make it convenient to determine that the second value carried in each secondary key value storage request is in the first value. The order of positions in.

In a possible design, when the key-value storage stores the second value carried in each secondary key-value storage request to the first storage space, it needs to first determine the second value carried in each secondary key-value storage request. The position order of the binary value in the first value; since the secondary key value storage request may include a value offset; the key value storage may determine the position order according to the value offset in each secondary key value storage request. For example, the position order is determined by comparing the magnitude of the value offset, and then the key value memory stores the second value in the first storage space according to the position order.

Through the above method, the key-value storage can determine the position sequence more conveniently by comparing the value offset in each secondary key-value storage request, improve data storage efficiency, and further ensure an efficient data storage process.

In a possible design, when the key-value storage allocates the first storage space for the first value, the second storage for the first value carried in one or more key-value storage requests can be configured in the first storage in advance. Space. The key-value memory selects the first storage space from the second storage space, and the size of the first storage space may be equal to or greater than the size of the first value. The key-value storage can pre-configure the second storage space before receiving the key-value storage request; it can also pre-configure the second storage space after receiving the key-value storage request and before storing the first value. In this case, the first 2. The storage space can be equal to the size of the first value or greater than the size of the first value.

Through the above method, the key-value memory can select the first storage space from the pre-configured second storage space, and the first storage space can be determined more quickly, so that the first value can be stored in the first storage space faster, which can be realized Efficient data storage process.

In a possible design, when the key-value memory allocates the first storage space for the first value, if there is not enough space in the second storage space to store the first value, that is, the amount of free space in the second storage space The size is smaller than the size of the first value, and the key value storage may allocate the first storage space for the first value from the first storage.

Through the above method, the key value storage can select the first storage space from the first storage to store the first value, which can ensure that the first value can be stored in the first storage, and the efficiency of data storage can be guaranteed.

In a possible design, the key-value storage may further include a second storage, and after receiving the key-value storage request, the key-value storage may store the keys in the key-value storage request in the second storage; exemplary, the first 2. A hash table can be stored in the storage, and the key value storage can update the hash table and add a hash table entry including the key; wherein the added hash table entry also includes the address of the first storage space.

Through the above method, the key value storage can update the hash table in time, and record the key and the corresponding first storage space address.

In a possible design, when the total size of the second value stored in the first storage space is equal to the size of the first value, the key-value storage device determines that the storage of the first value has been completed, and stops the data storage operation ; The secondary key value storage request includes the value size, which is used to indicate the size of the second value carried in the secondary key value storage request. The key value storage device can accumulate the stored second value corresponding to the secondary key value storage request The value size, until the accumulated value is equal to the size of the first value, indicating that the first value has been stored.

Through the above method, the key value storage can determine that the first value has been stored in different ways, and can stop the data storage operation in time.

In the second aspect, the present application provides a key-value memory, which has the function of the key-value memory in the first aspect and any one of the possible designs of the first aspect. The function can be realized by hardware, or by hardware executing corresponding software. The hardware or software includes one or more modules corresponding to the above-mentioned functions. In a possible design, the structure of the device includes a receiving unit, a distributing unit, a dividing unit, and a storage unit. These units can perform the corresponding functions in the method example of the first aspect. For details, please refer to the detailed description in the method example. Do not repeat it.

In the third aspect, the present application also provides a key value memory. For beneficial effects, please refer to the description of the first aspect and any possible design of the first aspect, which will not be repeated here. The structure of the key-value memory includes a processor and a memory, and the processor is configured to support the key-value memory to perform corresponding functions in the first aspect and any one of the possible design methods of the first aspect. The memory is coupled with the processor, and it stores the program instructions and data necessary for the key-value memory. The structure of the key-value memory also includes a transceiver for communicating with other devices.

In a fourth aspect, this application provides a key-value storage server. The key-value storage server includes a first storage and a key-value storage. The first storage is a storage other than the key-value storage. The key-value storage server is used to execute the first aspect. And the operation steps of the method provided in any one of the possible designs in the first aspect.

In a fifth aspect, the present application also provides a computer-readable storage medium that stores instructions in the computer-readable storage medium, which when run on a computer, causes the computer to execute the methods of the above aspects.

In a sixth aspect, the present application also provides a computer program product containing instructions, which when run on a computer, causes the computer to execute the methods of the above aspects.

In a seventh aspect, the present application also provides a computer chip, which is connected to a memory, and the chip is used to read and execute a software program stored in the memory, and execute the methods of the above aspects.

Description of the drawings

Figure 1 is a schematic diagram of the architecture of a key-value storage system provided by this application;

Figure 2 is a schematic structural diagram of a key-value storage server provided by this application;

FIG. 3 is a schematic diagram of a data storage method provided by this application;

FIG. 4 is a schematic diagram of a flow of data fragmentation and splicing provided by this application;

FIG. 5 is a schematic diagram of the format of a key-value storage request provided by this application;

Figure 6 is a schematic diagram of another key-value storage request format provided by this application;

FIG. 7 is a schematic diagram of a first value segmentation provided by this application;

FIG. 8 is a schematic diagram of state switching of a state machine provided by this application;

FIG. 9 is a schematic structural diagram of a key value storage provided by this application;

FIG. 10 is a schematic structural diagram of another key value storage provided by this application.

detailed description

As shown in FIG. 1, a schematic diagram of a network architecture provided by an embodiment of this application, in which a plurality of application server key-value clients 101 and key-value storage servers (key-value servers) 102 are provided. The application server key-value client 101 and the key-value storage server 102 interact through the network to send and receive information. The embodiment of the present application does not limit the type of the network. For example, the transmission control protocol/Internet interconnection protocol (transmission control protocol/ The wired transmission method of internet protocol (TCP/IP) can also be the wireless transmission method using wireless fidelity (Wifi). As a possible embodiment, the key-value storage server 102 may also be a key-value storage device composed of other hardware.

Each key-value storage server 102 can be provided with a memory 103, and the memory 103 can be used to store values. The memory 103 can be a cache, flash memory (flash), or hard disk drive (HDD) in the key-value storage server. Or storage media such as solid-state drives (SSD). Exemplarily, the memory 103 may also be used to store a hash table for recording the correspondence between keys and values, and the storage location of the value, so that the hash table can be used in data processing to query the storage location of the specified value. There are multiple hash table entries stored in the hash table. The hash table entries are the correspondence between the key and the address of the storage space where the value corresponding to the key is stored. The storage of the value corresponding to the key can be retrieved through the hash table Address (the storage address is the address of the storage space in the memory 103). In the process of reading the value, the storage address of the value is determined by the hash table, and then the value is obtained from the memory 103; in the process of storing the value, the corresponding hash table entry needs to be added to the hash table to record the value The storage address.

The multiple key-value storage servers 102 shown in FIG. 1 can respectively receive key-value storage requests sent by different application server key-value clients 101 via the network in parallel. If the key-value storage request is a read request, the key-value storage server 102 may write the value carried in the key-value storage request into the memory 103. In addition, the key-value storage server 102 may also update the hash table and insert new hash table entries according to the key carried in the key-value storage request and the storage space allocated for the value in the memory 103. The inserted hash table entry includes a key and a storage address; by hashing the key, the generated hash value is the index of the inserted hash table entry. If the key-value storage request is a write request, the key-value storage server 102 can read the corresponding value from the storage according to the key carried in the key-value storage request. Specifically, the key-value storage server 102 may perform a hash operation on the key carried in the key-value storage request to generate a hash value, query the corresponding hash table entry in the hash table through the hash value, and then compare the queried Whether the key in the hash table entry is consistent with the key carried in the key value storage request, if they are the same, read the value from the storage according to the storage address in the hash table entry and feed it back to the application server key value client 101.

As a possible embodiment, in order to improve the storage efficiency of the key-value storage system, multiple processing logics may be called in parallel to store different data in the same key-value storage server 102. For example, call two hash functions to perform hash operations at the same time. Each hash function corresponds to a set of processing logic, and the address of the storage space mapped by each processing logic corresponds to 4 buckets to form a set of buckets. , That is, each key can correspond to 8 storage buckets for storing hash table entries; when updating the hash table entries, you can search the hash table entries stored in 8 storage buckets at the same time (keys pass a hash function The hash table items stored in the mapped 4 buckets are a group of hash table items), find out the unoccupied buckets, and then use the polling mode to insert new hash table items; for each group of storage Buckets are inserted into the previous empty bucket first. When inserting new hash table entries, two groups can be inserted alternately. That is to say, when inserting a hash table entry for the first time, one of the buckets in one group is selected, and when inserting a hash table entry for the second time, one of the storage buckets in another group is selected. When a hash collision occurs, that is, when all 8 buckets corresponding to a key are occupied, a bucket is selected through preset rules, and some of the hash table entries are deleted to insert new hash table entries; for example, , Delete the earliest stored 10 hash table entries. When deleting an existing hash table entry, a polling mode can also be used to alternately delete the hash table entries in the two groups of buckets. In other words, when deleting a hash table entry for the first time, delete the hash table entry in one bucket of one group of buckets, and delete the hash table entry for the second time, and delete one of the other group of buckets. The hash table entry in the bucket.

In the network architecture shown in FIG. 1, there are multiple application server key value clients 101 and multiple key value storage servers 102, and any application server key value client 101 can send a key to any key value storage server 102. Value storage request. As a possible embodiment, multiple key-value storage servers 102 may also communicate with the application server key-value client 101 through a key-value proxy (not shown in FIG. 1), and the key-value proxy is used to key the application server. The key-value storage requests of the value client 101 are managed and scheduled, and distributed to different key-value storage servers. For example, the key-value agent can allocate key-value storage requests according to the current load of the key-value storage server. The key-value proxy and the key-value storage server 102 can be deployed in one server, or can be deployed using a single server, or deployed in a cluster using multiple servers. When a key-value proxy is deployed in the form of a cluster, an Internet protocol (IP) address can be presented to the outside. The application server key-value client 101 can send a key-value storage request to the key-value proxy, and the key-value proxy is based on load balancing The algorithm distributes the key-value storage request to the key-value storage server 102.

It should be noted that in the above description, only the fixed-length key value storage method is used for the description, which only involves the key hash operation, the hash table query, and the process of determining the hash table entry. The embodiments of the present application are also applicable to variable-length key value storage methods. For variable-length keys, additional fields are allowed in the key field. For example, variable-length keys are used to indicate different version numbers in scenarios such as data upgrades and snapshots. The value corresponding to the key will change to a certain extent. Taking the key-value storage request as a write request as an example, the variable-length key is processed as follows: first perform a hash operation on the variable-length key, generate a hash value, query the hash table, and confirm Hash table entry. After that, the corresponding block address and the tree structure information stored in the block address are determined according to the hash table entry. The parent node and the leaf node in the tree structure are determined according to the tree structure information, and the tree node may indicate the address of the storage space of the value corresponding to the variable-length key. The tree structure may be a B-tree structure that identifies the size of the node, that is, in the tree structure, the value of the left leaf node is greater than the value of the parent node, and the value of the parent node is less than the value of the right leaf node. For different types of keys, the data storage methods provided in the embodiments of this application are all applicable. For ease of description, the following embodiments of this application only take fixed-length keys as an example for description.

The embodiment of the present application does not limit the structure of the key-value storage server. For example, the processor in the key-value storage server may be a central processing unit (CPU), an application specific integrated circuit (ASIC), and an on-site Field-programmable gate array (FPGA), artificial intelligence (AI) chip, system on chip (SoC) or complex programmable logic device (CPLD), of which, The data required by the key-value storage server, for example, the hash table and value, can be stored in the same memory or in different memories. For example, the hash table and value can be stored in the memory of the host. For another example, the hash table can be stored in the memory of the host, and the value is stored in the flash memory. The foregoing is only an example, and the embodiment of the present application does not limit the storage location of the hash table and the value.

As shown in FIG. 2, a schematic structural diagram of a key-value storage server provided by an embodiment of this application. The key-value storage server 201 is a key-value storage device constructed based on FPGA. The key-value storage server 201 includes a key-value storage 202, The first storage 203; optionally, the key-value storage server 201 may also include a central processing unit 204.

The key value storage 202 includes a key value storage engine 205 and a second storage 206, and optionally, may also include a network offload engine (NOE) 207. The second memory 206 may be a double data rate (DDR) memory, which may be specifically used to store a hash table. In the key-value storage server shown in FIG. 2, the value is stored in the first memory 203. The first memory 203 may be a memory in a host, and may be specifically implemented by a dynamic random access memory.

It is worth noting that FPGA is a processing logic that uses hardware such as capacitors and resistors to implement specified functions. For ease of description, the processing logic that implements specific functions in Figure 2 is named after the function name, for example, network offloading engine and key-value storage engine. In addition, as shown in FIG. 2, the key-value storage server 201 includes a central processing unit 204 and an FPGA-based key-value storage. For the same server including two or more types of processors, it can also be called a heterogeneous server.

The network offloading engine 207 is used to receive the key-value storage request from the key-value client of the application server, process the key-value storage request based on the user datagram protocol (UDP), and send the processed key-value storage request to the key The value storage engine 205 reduces the overhead incurred in the process of processing user datagram protocol messages by the CPU 204. Optionally, when the processing capability of the CPU 204 is strong, the CPU 204 can continue to complete the processing of the user datagram protocol. At this time, the CPU 204 receives the key value storage request from the key value client of the application server, and based on the user data packet The protocol processes the key-value storage request, and then sends the processed key-value storage request to the key-value storage engine 205. For ease of description, in the following description of the embodiments of the present application, the processing of the key-value storage request by the network offloading engine 207 in the key-value storage is taken as an example for description.

It is worth noting that whether the user datagram protocol message is processed by the network offloading engine 207 or the CPU204, the specific processing process is to parse the message data, that is, to remove the header content of the protocol and obtain the payload data in the message. The embodiment of the present application does not limit the process of the network offloading engine 207 or the CPU 204 processing the user data protocol message.

The key-value storage engine 205 is used to process the key-value storage request, for example, read the value from the first memory 203 according to the key in the key-value storage request or the value carried in the key-value storage request (corresponding to the value in the embodiment of this application). The first key value of) is written into the first memory 203. In this embodiment of the present application, before the key-value storage engine 205 writes the first value into the first memory 203, if the first value is greater than or equal to the first threshold, the key-value storage engine 205 may add the value carried in the key-value storage request The first value is divided into multiple second values to generate multiple secondary key storage requests; where each secondary key value storage request carries a key and a second value; then, each secondary key value is stored separately The second value carried in the request is stored in the first memory 203.

As shown in FIG. 2, the key-value memory 202 may also be connected to other processors, such as the CPU 204. In the embodiment of the present application, the key-value memory 202 may transfer the operation of data storage to the CPU 204. Exemplarily, after receiving the key-value storage request, the key-value storage 202 may directly forward the key-value storage request to the CUP 204; or after determining that the first value carried in the key-value storage request is greater than or equal to the first threshold, The key-value storage request is forwarded to CUP 204, and CUP 204 executes the data storage method provided in the embodiment of this application.

The structure of the key-value storage server shown in FIG. 2 is only an example, in which only one CPU and one key-value storage are shown. In fact, the embodiment of the present application does not limit the number of CPUs and key-value storages. One CPU or multiple key-value memories. The key-value storage request can be forwarded between multiple key-value storages to cooperate with the data storage method provided in the embodiment of this application, and the data storage operation can also be selectively transferred to any CUP connected to the key-value storage device . The embodiments of the present application do not limit the modules included in the key-value storage, the names of the modules, and the functions of each module. Any device that can implement the data storage method provided in the embodiments of the present application can be used as the key-value storage.

The following describes a data storage method proposed in an embodiment of the present application based on the key-value storage shown in FIG. 2 with reference to the accompanying drawings. As shown in FIG. 3, the method includes:

Step 301: The key-value storage receives a key-value storage request. The key-value storage request carries a key and a first value, where the first value is a value corresponding to the key.

Step 302: The key value storage allocates a first storage space for the first value in the first storage.

As a possible implementation manner, a second storage space may be pre-configured in the first memory in the key-value storage for storing values. When the key-value storage allocates the storage space in the first storage for the first value, it may Priority is given to allocating storage space for the first value from the second storage space. The size of the second storage space may be equal to or greater than the first value. Exemplarily, the second storage space may store the first value carried in one or more key-value storage requests.

For the second storage space, the second storage space may be the storage space configured by the key-value storage from the first storage in advance after receiving the key-value storage request and before storing the first value carried in the key-value storage request, The size of the second storage space may be equal to the first value or greater than the first value, so that the key value storage can flexibly allocate the first storage space for the first value from the second storage space.

As a possible implementation manner, the second storage space may be a storage space pre-configured by the key value storage from the first storage before the key value storage request is received. Since it is pre-configured before receiving the key-value storage request, the key-value storage can be pre-configured with a larger storage space as the second storage space, so that the key-value storage can store the first carried in multiple key-value storage key values. value.

After receiving the key-value storage request, the key-value memory can determine whether there is free space in the second storage space, and whether the free space can store the first value; if there is enough free space in the second storage space to store the first value , Select the free space from the second storage space as the first storage space.

If there is not enough free space in the second storage space to store the first value, that is, the free space in the second storage space is less than the storage space required by the first value, the key value storage can be the first value from the first storage. Value allocation storage space.

In order to effectively use the storage space in the second storage space, the storage space allocated by the key-value storage for the first value from the first storage and the free space in the second storage space can be used to store the first value (such as storing one or Multiple second values). That is, the sum of the allocated storage space and the free space in the second storage space may be equal to the size of the first value. Of course, the key value memory may also allocate a first storage space with a size equal to the size of the first value for the first value from the first storage.

As another possible implementation manner, when there is not enough space in the second storage space to store the first value key value storage, the key value storage may also configure a third storage space from the first storage, and the third storage space is larger than The storage space required for the first value. Exemplarily, the third storage space may be a relatively large storage space such as 1G or 500M. In addition to storing the first value, the third storage space can also be used to store the first value carried in the key-value storage request subsequently received by the key-value storage. The key-value storage allocates the first value in the third storage space. The first storage space.

It should be understood that the key-value storage may not be configured with the second storage space in the first storage in advance, and when data storage is required, the first storage space is dynamically allocated in the first storage for the first value. Exemplarily, the key value storage may allocate the first storage space in the first storage according to the size of the first value.

In the above description, the size of the first storage space is equal to the size of the storage space required for the first value as an example. The embodiment of the present application does not limit the size of the first storage space. Exemplarily, the maximum threshold of the first value carried in the key-value storage request may be set. In other words, the maximum threshold is used to represent the maximum size of the value carried in the key-value storage request. When the key value storage allocates the first storage space in the first storage for the first value, the allocated first storage space may be equal to the maximum threshold. When the key-value memory stores the second value, it can store the second value in the allocated first storage space, and the storage location of the second value in the first storage space can be flexibly set, so that the way of storing the second value is more flexible . In addition, the first storage space may be a continuous storage space or a discontinuous storage space, which is not limited in the embodiment of the present application.

Step 303: When the capacity of the data contained in the first value is greater than or equal to the first threshold, the key value store divides the first value into multiple second values, and generates multiple secondary key value storage requests, each secondary The key-value storage request carries the key and a second value. The number of secondary key value storage requests is the same as the number of multiple second values obtained by dividing the first value.

The embodiment of the present application does not limit the sequence of step 302 and step 303, and can be performed simultaneously or sequentially.

When the capacity of the data contained in the first value is large, if the data is stored directly, the first value needs to be cached in the key value storage first. Exemplarily, the key value storage first caches the first value in the second storage. After that, the first value is transferred to the first memory, which will lead to a longer time for data storage. Among them, the capacity of the data contained in the first value can also be referred to as the size of the first value, which refers to the total size of the data contained in the first value (size), or it can be said to refer to the storage space required to store the first value. The size of the minimum storage space. For example, if the content contained in the first value is "0123456789", the capacity of the data contained in the first value is 10 bytes.

The first threshold can be set here. When the first value is greater than or equal to the first threshold, the first value is considered to be larger and belongs to large-size data; otherwise, the first value belongs to small-size data. For small-size data, you can follow the traditional The data is stored in the key-value storage mode of the first threshold; the specific value of the first threshold can be an empirical value, such as 1 mega (M), or it can be set according to specific scenarios and key-value storage performance. For example, in some key-value storage scenarios, the size of the value that needs to be stored in a period of time can be counted, and the first threshold can be determined according to the size of the value that needs to be stored in the statistics. Exemplarily, when the size of the stored value is concentrated in 3M to 10M, and the value is found to be greater than 5M, the storage efficiency of the key value memory is significantly slowed down, and 5M may be used as the first threshold. The foregoing method of setting the threshold is only an example, and the embodiment of the present application does not limit the specific value and setting method of the threshold.

In order to improve the storage efficiency of large-size data, the embodiment of the present application proposes a method of data partitioning. The first value is divided into multiple second values. The second value is stored in the first memory in stages, and the key value memory is Complete the positive sequence, storage and merging process of multiple second values. The embodiment of the present application does not limit the specific implementation of data segmentation, and any method that can finally divide the first value into multiple smaller second values is applicable to the embodiment of the present application.

As shown in FIG. 4, it is a schematic diagram of generating multiple secondary key value storage requests and merging after dividing the first value, which is divided into two different processes, the slicing process and the splicing process. During the sharding process, the key-value storage can divide the first value carried in the key-value storage request into multiple second values, and generate multiple secondary key-value storage requests. Each secondary key-value storage request includes a message Header and a second value (the format of the secondary key value storage request will be explained later); the splicing process is the process in which the key value storage reads the third value from the first storage, which can be seen as the inverse process of the sharding process. If the third value is greater than the third threshold (the third threshold can be equal to the second threshold, or other values), for example, the third threshold is 10M, the key value memory reads multiple fourth values from the first memory, and generates After multiple secondary key-value storage requests, multiple fourth values are combined into a third value, and the third value is carried in the key-value storage response that needs to be fed back to the key-value client of the application server.

In order to better explain the technical solutions of the embodiments of the present application, first, the format of the key-value storage request is described. As shown in FIG. 5, a schematic diagram of the format of a key-value storage request provided by an embodiment of the present application.

The key-value storage request includes two parts, one is a header and the other is a packet. Among them, the message header carries some attribute information of keys and values. Specifically, the message header includes key identifier (k ID), type (type), key size (key size), value total size, value size, and value offset (value offset) ; The data packet includes keys and values.

The key identifier is used to indicate the serial number of the key. The type is used to indicate the operation type of the key-value storage request, for example, it can be insert (put), search (get), and delete (delete). Wherein, the insertion instruction key value storage request is a write request, and the value carried in the key value storage request (such as the first value in the embodiment of this application) needs to be stored in the first memory; the search instruction key value storage request is read To fetch the request, the value corresponding to the key carried in the key-value storage request needs to be read from the first storage; the deletion indicates that the value corresponding to the key carried in the key-value storage request needs to be deleted from the first storage. The above three operation types are only examples, and the embodiments of the present application are not limited, and other operation types, such as update, may also exist.

The key size is used to indicate the size of the key carried in the key-value storage request. The total value size is used to indicate the total size of the value carried in the key-value storage request (the corresponding key-value storage request is a write request, and the type is put), or the total size of the value obtained by the key-value storage request (the corresponding key-value storage request is Read request, the type is get), or the total size of the value deleted by the key value storage request (corresponding to the key storage request of type delete).

The value offset is used to indicate the offset of the value carried in the key value storage request (such as the second value in the embodiment of the present application) relative to the first position of the total value (corresponding to the first value in the embodiment of the present application). In the embodiment of the present application, the value offset may indicate the position of the second value in the first value. The larger the value offset, the lower the position of the second value in the first value.

The value size is used to indicate the size of the value (corresponding to the first value or the second value in the embodiment of the present application) carried in the key value storage request. Exemplarily, if the total length of a value is 2024, it is divided according to 1024. After the division, for the first value (such as the second value in the embodiment of this application), the value offset is 0 and the value size is 1024; The second value, the value offset is 1024, and the value size is 1000.

The format of a secondary key value storage request provided by the embodiment of the present application is similar to the key value storage request shown in FIG. 5, which can be referred to FIG. 5. The secondary key value storage request can also be divided into two parts, one is the message header, and the other is the data packet. The message header carries some attribute information of keys and values. Specifically, the message header includes key identifiers, types, key sizes, total value sizes, value sizes, and value offsets; data packets include keys and values.

In the embodiment of the present application, the type of the key-value storage request is insert as an example, and correspondingly, the type in the secondary key-value storage request is also insert. The total value size indicates the size of the value before data segmentation, that is, the size of the first value carried in the key value storage request. The value size indicates the size of the value carried in the secondary key value storage request (corresponding to the second value in the embodiment of the present application). The value offset indicates the offset position of the value carried in the secondary key value storage request (such as the second value in the embodiment of this application) in the value (corresponding to the first value in the embodiment of this application) in the key value storage request. In the embodiment of this application, usually the value offset in the secondary key value storage request may be 0, indicating that the second value carried in the secondary key value storage request is divided from the starting position of the second value in the first value of. The value offsets in the secondary key value storage requests that carry different second values are different, and the value offsets can reflect the position order of the second value in the first value.

It should be noted that the type may also indicate whether the key value storage request is a key value storage request before data segmentation or a secondary key value storage request formed after data segmentation. Exemplarily, an identification bit can be added to the type to indicate whether the key-value storage request is a key-value storage request before data segmentation or a secondary key-value storage request.

As another possible implementation, as shown in Figure 6, a field can be added to the key-value storage request as a slice, which is used to indicate whether the request is a key-value storage request before data segmentation or a secondary key. Value storage request. When the fragment has the first value (such as null), it indicates that the request is a key-value storage request before data segmentation, and when the fragment has the second value (such as 1), it indicates that the request is a secondary key-value storage request.

As shown in FIG. 7, taking the first value of 5M as an example, the key value memory can divide the first value into five second values, and the size of each second value is 1M. In this way, five secondary key value storage requests can be generated. Each secondary key value storage request carries a second value. For the secondary key value storage request carrying the first second value, the value size is 1M, Indicates that the size of the second value carried is 1M, and the value offset can be set to 0, indicating that the second value is divided from the starting position of the first value; for the secondary key value storage that carries the second second value The size of the value in the request is 1M, the size of the second value carried in the instruction is 1M, and the value offset can be set to 1M, indicating that the second value is segmented from the 1M position of the first value.

As a possible implementation manner, the secondary key value storage request may set a field to indicate the order of the carried second value in the first value more intuitively. Exemplarily, a fragment field can be used. For example, 1 indicates that the key-value storage request is a secondary key-value storage request, and the second value carried in the secondary key-value storage request is the first value after the first value is split. Binary value; Use 2 to indicate that the key value storage request is a secondary key value storage request, and the second value carried in the secondary key value storage request is a second value whose position order is second after the first value is divided. The above indication of using the fragment field is only an example, for example, other fields may be added, which is not limited in the embodiment of the present application.

The value in the data packet in the secondary key-value storage request is part of the value in the key-value storage request. In this embodiment of the application, the value in the data packet in the secondary key-value storage request is the second value, and the key-value storage The value in the data packet in the request is the first value.

The key identifier, type, key size, and total value of the message header in the secondary key-value storage request are the same as the content of the corresponding field in the key-value storage request. The key and key value in the data packet in the secondary key-value storage request The keys in the storage request are the same.

The embodiments of this application provide two different data segmentation methods, which will be introduced separately as follows:

Method 1: In the process of caching the first value, partition while caching.

After receiving the first value, the key value storage may first buffer the first value. In the caching process, when the size of the cached data reaches the second threshold, the cached data is taken as a second value, and a corresponding secondary key value storage request is generated. When generating a secondary key value storage request, it is necessary to carry the cached data (that is, the second value) in the secondary key value storage request, and set various fields in the header of the secondary key value storage request. Exemplarily, the value size field can be set to the size of the second value, and the value offset is the position of the second value in the first value, indicating the offset of the second value relative to the first value of the first value. The setting method can be seen in the foregoing Description, not repeat them here.

After the secondary key value storage request is generated, the second value carried in the secondary key value storage request can be stored in the first storage space allocated for the first value, because the second value is only a part of the first value , The second value only takes up part of the storage space.

Then, the key-value storage will continue to cache the remaining first value, when the size of the cached data reaches the second threshold (the previously cached data has been carried in the secondary key-value storage request, in this case, The size of the cached data starts from 0), the cached data is used as a second value; the corresponding secondary key request is generated, and the same operation as before is performed, including setting the secondary storage key request, and setting the secondary key The second value carried in the value storage request is stored in the first storage space allocated for the first value; when the second value is stored, because a second value is already stored in the first storage space allocated for the first value. Value, the second value carried in the secondary key value storage request is stored in the storage space after the stored second value to achieve sequential storage.

The above process is looped until the first value is cached. It should be noted that when the first value is cached in the final stage, the first value may be cached, but the cached data does not reach the second threshold. In this case, the key-value storage may directly use the cached data as a second value, generate a corresponding secondary key request, and store the second value in the secondary storage key request into the first storage. In this case, it can be found that the sizes of the divided second values are not completely the same, and there may be a second value whose size is smaller than the second threshold.

In the above description, the process of receiving the first value and dividing the first value by the key-value memory is performed in parallel, that is, the key-value memory performs data division during the process of caching the first value and completes the storage process of the second value at the same time, that is In other words, step 303 and step 304 are performed alternately, and this method is only an example. In the specific implementation process, the operation of storing the second value (step 304) can be executed after the first value is cached and all secondary key value storage requests (step 303) are generated, which is not limited in this embodiment of the application.

As a possible implementation manner, in order to implement a cache-side partition mode, the key-value memory may adopt a state machine control mode.

Fig. 8 is a schematic diagram of a state machine provided by an embodiment of the application. As shown in the figure, the finite state machine can be set in 6 states, namely, idle state (IDLE), waiting command (WAIT_CMD), and waiting descriptor (WAIT_DSC) , Wait for data (WAIT_DATA), generate descriptor (GEN_DSC), complete (FINISH) state, the state transition diagram is shown in Figure 8. The description and switching of each state can be seen as follows:

IDLE state: When the first value is received, jump to WAIT_CMD state; otherwise, keep in IDLE state.

WAIT_CMD state: Waiting to receive insert command. In this state, if the type of the key-value storage request is parsed by the key-value store, it will jump to the IDLE state, and the first value received is invalid and discarded; if the type of the key-value storage request is parsed, it will jump to WAIT_DSC State; otherwise, keep in WAIT_CMD state.

WAIT_DSC state: waiting for receiving card to host (card to host, C2H) descriptor. The C2H descriptor is used to indicate the size of the first value and the first storage space allocated by the key value storage for the first value. After the key value storage receives the key value storage request and allocates the first storage space for the first value, it may generate the C2H descriptor.

When the key value memory determines the C2H descriptor, it jumps to the WAIT_DATA state; otherwise, it remains in the WAIT_DSC state.

WAIT_DATA state: Waiting for the first value to be buffered to one frame (taking the second threshold as one frame as an example). When the first value buffer reaches one frame, the buffered data is used as the second value. When the first value is cached for one frame, and the first value has not been cached, that is, it is not the last frame, jump to the GEN_DSC state; when the last frame of the first value is cached, jump to the FINISH state; otherwise , Remain in the WAIT_DATA state; here, taking the size of the second value as one frame as an example, the embodiment of the present application does not limit the size of the second value.

GEN_DSC state: Generate C2H fragment descriptor. The key value memory can start a data transfer and transfer the second value to the first storage space.

The C2H fragment descriptor is used to indicate the size of the second value, the storage address of the second value, or the source address of the second value, etc., where the source address of the second value is the address of the key value memory cache second value, such as the key The value memory caches the address of the second value in the second memory; it should be noted that caching the second value here means that when the key value memory caches the first value, the cached data is used as the second value. The key value memory may store the second value carried in the corresponding secondary key value storage request in the first storage space according to the information indicated by the C2H fragment descriptor.

It should be noted that the storage address indication of the second value is a storage space for storing the second value. The key value memory may determine the location of the storage space in the first storage space according to the size of the second value and the position of the second value in the first value, and determine the storage address of the second value. Exemplarily, if the second value currently cached in the key-value memory is the first frame of the first value, the storage space for storing the second value is one frame from the starting position in the first storage space The storage space can then determine the storage address of the second value; if the second value currently cached in the key-value memory is the third frame of the first value, the storage space used to store the second value is 3M in the first storage space The storage space starting from the position and the size is one frame, and then the storage address of the second value is determined.

In this state, the corresponding secondary key value storage request is generated, and then it jumps to the WAIT_DATA state; otherwise, it remains in the GEN_DSC state.

FINISH state: Generate the last piece of C2H fragment descriptor (that is, cache the last frame of the first value, and generate the last second value), marking the completion of the cache operation of the first value. When the first value is completely transmitted, it jumps to the IDLE state; otherwise, it remains in the FINISH state.

It should be noted that the embodiment of the present application is not a specific value of the second threshold, but may be an empirical value, or may be set according to a specific scenario, and the embodiment of the present application is not limited.

Method 2: After the first value is cached, segmentation is performed.

The key value memory can also divide the first value into multiple second values after fully buffering the first value. Exemplarily, the key-value storage may evenly divide the first value into a plurality of second values with the same size. You can also set a second threshold to divide the first value into a second value of the second threshold. In this case, since the size of the first value is not necessarily an integer multiple of the second threshold, one or more of them may exist. The second value is less than the second threshold.

The foregoing division method is only an example, and the embodiment of the present application does not limit the division method. For example, the key-value memory may also randomly divide the first value, and the size of each second value after division is different.

After the division of the first value is completed, multiple secondary key value storage requests can be generated, where each secondary key value storage request carries one of the second values, and the message in each secondary key value storage request The value offset in the header indicates the position of the second value carried in the first value.

Step 304: The key-value storage stores the second value carried in each secondary key-value storage request to the first storage according to the order in which the second value carried in each secondary key-value storage request is located in the first value space.

As a possible implementation manner, when storing the second value, the key value storage may store the second value carried in the secondary key value storage request in the order of the position of the second value in the first value, in the order from front to back in the order of position. Exemplarily, the position of the second value carried in the secondary key value storage request in the first value can be determined according to the value offset in the header of each secondary key value storage request, and each secondary key value can be The value offsets in the message header in the value storage request are sorted in order of magnitude, from small to large, and the second value carried in the corresponding secondary key value storage request is sequentially stored in the first storage space.

As another possible implementation manner, when the key value storage stores the second value carried in the secondary key value storage request. The storage position of the second value in the first storage space may be determined according to the position of the second value in the first value. The second value is stored in the storage location in the first storage space, and the location of the second value in the first value can be determined by the value offset in the message header of the secondary key value storage request.

Exemplarily, if the value offset in the header of the secondary key value storage request is 0, it indicates that the second value carried in the secondary key value storage request is located at the beginning of the first value, and the second value is in the first value. The storage location of a storage space is a storage space with a size of a second value starting from the starting point of the storage space.

The value offset in the message header of the secondary key value storage request is 1M, which indicates that the second value carried in the secondary key value storage request starts at the 1M position of the first value, and the second value is in the first storage space The storage location starts from the location corresponding to the 1M data that is stored empty, and the size is the second value size storage space.

After storing the second value in all secondary key value storage requests in the first storage space, the key value storage ends the data storage operation.

The key value memory judges whether the total size of the second value stored in the first storage space is consistent with the size of the first value, and determines whether the first value is stored through the judgment result; if the key value memory can query the first storage space For the stored second value, if the total size of the stored second value is consistent with the size of the first value, it is determined that the storage of the first value is completed.

As another possible implementation manner, the key value storage may accumulate the key value size in the header of each stored secondary key value storage request to generate an accumulated value until the accumulated value is equal to the total size of the key value. Determine whether the storage of the first value is complete, and end the data storage operation.

Based on the same inventive concept as the method embodiment, the embodiment of the application also provides a key-value memory for executing the method of the key-value memory in the above-mentioned method embodiment. For related features, please refer to the above-mentioned method embodiment. To repeat, as shown in FIG. 9, the device includes a receiving unit 901, a distributing unit 902, a dividing unit 903, and a storage unit 904:

The receiving unit 901 is configured to receive a key-value storage request. The key-value storage request includes a key and a first value, where the key corresponds to the first value.

The allocating unit 902 is configured to allocate a first storage space for the first value in the first storage, and the first storage is a storage other than the key value storage.

The dividing unit 903 is configured to divide the first value into multiple second values when the total length of the first value is greater than or equal to the first threshold, and generate multiple secondary key value storage requests, each secondary key value The storage request carries the key and a second value, and the number of secondary key value storage requests is the same as the number of multiple second values obtained by dividing the first value.

The storage unit 904 is configured to store the second value carried in each secondary key value storage request in the first storage according to the position order of the second value carried in each secondary key value storage request in the first value space.

Optionally, when the dividing unit 903 divides the first value into multiple second values, it may divide while buffering. Exemplarily, in the process of caching the first value, the dividing unit 903 may use the cached data as the second value whenever the size of the cached data is equal to the second threshold, and carry the second value in a secondary key. In the value storage request, until the first value cache is completed. When in the final stage of the first value cache, the remaining data size of the last first value is less than the second threshold, you can directly use the cached data as the second value and carry it in a secondary key value storage request; in this case , The size of the second value carried in each secondary key value storage request may not be exactly the same.

Optionally, when the dividing unit 903 divides the first value into multiple second values, it may also use a method of data division after the buffering is completed; for example, the dividing unit 903 may perform data division after the first value is buffered. The first value is divided into multiple second values; for example, the dividing unit 903 may divide the first value into N+1 second values, and among the multiple second values arranged in the order of division, the first N second values The size of can be equal to the second threshold, the last second value, that is, the size of the N+1th second value can be less than or equal to the second threshold, and N is a positive integer greater than or equal to 1. Exemplarily, the dividing unit 903 may also set the second values to be multiple second values.

Optionally, the secondary key value storage request may include some fields for indicating related information of the second value carried, such as the size and offset of the second value. Exemplarily, a value offset may be included, and the value offset may indicate the position of the second value carried in the secondary key value storage request in the first value.

Optionally, the storage unit 904 stores the second value carried in each secondary key value storage request to the first value in the order of the second value carried in each secondary key value storage request. For a storage space, the position order needs to be determined first, and then the second value is stored; for example, the storage unit 904 may determine each secondary key value storage request according to the value offset in each secondary key value storage request The position order of the second value carried in the first value in the first value; after that, the second value is stored in the first storage space according to the position order.

Optionally, when the storage unit 904 allocates the first storage space for the first value in the first storage, the second storage for the first value carried in one or more key value storage requests may be pre-configured in the first storage. Storage space. The storage unit 904 may allocate the first storage space for the first value from the second storage space. The size of the first storage space may be equal to or greater than the first value. The storage unit 904 may pre-configure the second storage space before the receiving unit 901 receives the key-value storage request; it may also pre-configure the second storage space after the receiving unit 901 receives the key-value storage request and before the storage unit 904 stores the first value. Storage space. In this case, the size of the second storage space can be equal to or greater than the size of the first value.

Optionally, when the storage unit 904 allocates storage space for the first value in the first storage, if the size of the free space in the second storage space is less than the size of the first value, the storage unit 904 may not store the first value. The first storage space is allocated for the first value from the first storage.

Optionally, the storage unit 904 may also determine whether the first value has been stored; for example, when the total size of the second value stored in the first storage space is equal to the size of the first value, the storage unit 904 may determine The storage of the first value has been completed.

Optionally, if the secondary key value storage request includes a value size, it is used to indicate the size of the second value carried in the secondary key value storage request; the storage unit 904 may accumulate the secondary key value corresponding to the stored second value Store the size of the requested value until the accumulated value is equal to the size of the first value, and it is determined that the first value has been stored.

It should be noted that the division of units in the embodiments of the present application is illustrative, and is only a logical function division, and there may be other division methods in actual implementation. The functional units in the embodiments of the present application may be integrated into one processing unit, or each unit may exist alone physically, or two or more units may be integrated into one unit. The above-mentioned integrated unit can be implemented in the form of hardware or software functional unit.

The foregoing embodiments can be implemented in whole or in part by software, hardware, firmware or any other combination. When implemented by software, the above-mentioned embodiments may be implemented in the form of a computer program product in whole or in part. The computer program product includes one or more computer instructions. When the computer program instructions are loaded or executed on a computer, the processes or functions described in the embodiments of the present invention are generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions may be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions may be transmitted from a website, computer, server, or data center. Transmission to another website site, computer, server or data center via wired (such as coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (such as infrared, wireless, microwave, etc.). The computer-readable storage medium may be any available medium that can be accessed by a computer or a data storage device such as a server or a data center that includes one or more sets of available media. The usable medium may be a magnetic medium (for example, a floppy disk, a hard disk, a magnetic tape), an optical medium (for example, a DVD), or a semiconductor medium. The semiconductor medium may be a solid state drive (SSD).

FIG. 10 is a schematic diagram of a key value storage 1000 according to an embodiment of the application. As shown in the figure, the key value storage 1000 includes a processor 1001 and a memory 1002. Optionally, the key value storage 1000 may further include a communication interface 1003. Among them, the number of the memory 1002 and the communication interface 1003 does not constitute a limitation to the embodiment of the present application, and can be configured arbitrarily according to business requirements during specific implementation.

The memory 1002 may be a volatile memory, such as a random access memory; the memory may also be a nonvolatile memory, such as a read-only memory, flash memory, hard disk drive (HDD) or solid-state drive (SSD) Or the memory 1002 is any other medium that can be used to carry or store desired program codes in the form of instructions or data structures and that can be accessed by a computer, but is not limited thereto. The memory 1002 may be a combination of the above-mentioned memories. The memory 1002 may be the second memory in the foregoing method embodiment.

The specific connection medium between the foregoing processor 1001 and the memory 1002 is not limited in the embodiment of the present application.

The processor 1001 may be a CPU, and the processor 1001 may also be other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field programmable gate array (FPGA) or other programmable logic devices, discrete Gate or transistor logic devices, discrete hardware components, artificial intelligence chips, chips on a chip, etc. The general-purpose processor may be a microprocessor or any conventional processor. It has a data transceiver function and can communicate with other devices. In the device shown in Figure 10, an independent data transceiver module, such as a communication interface 1003, can be used to send and receive data; when the processor 1001 communicates with other devices, it can Data transmission is performed through the communication interface 1003, such as receiving a key-value storage request from an application server key-value client.

When the key-value memory adopts the form shown in FIG. 10, the processor 1001 in FIG. 10 can call the computer-executed instructions stored in the memory 1002, so that the key-value memory can execute the key-value memory execution in any of the foregoing method embodiments. The steps of the method.

Optionally, the processor 1001 of the key-value storage in FIG. 10 may implement the functions of the key-value storage engine 205 and the network offloading engine 207 in FIG. 2.

The functions/implementation processes of the receiving unit, the allocation unit, the dividing unit, and the storage unit in FIG. 9 can all be implemented by the processor 1001 in FIG. 10 calling the computer execution instructions stored in the memory 1002. Alternatively, the function/implementation process of the allocation unit, the division unit, and the storage unit in FIG. 9 can be implemented by the processor 1001 in FIG. 10 calling the computer execution instructions stored in the memory 1002, and the function/implementation of the receiving unit in FIG. 9 The process can be implemented through the communication interface 1003 in FIG. 10.

Based on the same inventive concept as the method embodiment, the embodiment of the present application also provides a key-value storage server. The structure of the key-value storage server can be seen in FIG. 2. The key-value storage server may include a key-value storage and a first storage; The memory is a memory other than the key value memory; the key value memory is used to execute the method executed by the key value memory in the foregoing method embodiment, and the first memory is used to store the first value; optionally, it may also include a central processing unit.

In the several embodiments provided in this application, it should be understood that the disclosed system, device, and method may be implemented in other ways. For example, the device embodiments described above are only illustrative. For example, the division of the units is only a logical function division, and there may be other divisions in actual implementation, for example, multiple units or components can be combined or It can be integrated into another system, or some features can be ignored or not implemented. In addition, the displayed or discussed mutual coupling or direct coupling or communication connection may be indirect coupling or communication connection through some interfaces, devices or units, and may be in electrical, mechanical or other forms.

The units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the objectives of the solutions of the embodiments.

The above are only specific embodiments of the present invention. Those skilled in the art can think of changes or substitutions according to the specific implementation manners provided by the present invention, which should all fall within the protection scope of the present invention.

Claims

A data storage method, characterized in that the method includes:

The key value storage receives a key value storage request, where the key value storage request carries a key and a first value, wherein the key corresponds to the first value;

The key value storage allocates a first storage space for the first value in the first storage, and the first storage is a storage other than the key value storage;

When the capacity of the data contained in the first value is greater than or equal to the first threshold, the key value storage divides the first value into multiple second values, and generates multiple secondary key value storage requests, each A secondary key value storage request carries the key and one of the second values, and the number of secondary key value storage requests is the same as the number of the multiple second values obtained by dividing the first value ；

The key-value storage device stores the second value carried in each secondary key-value storage request in the order of the position in the first value of each secondary key-value storage request to The first storage space.
8. The method of claim 1, wherein the key-value storage partitioning the first value into a plurality of second values comprises:

When the key value memory is in the process of caching the first value, whenever the size of the cached data is equal to the second threshold, it uses the cached data as the second value, and carries the second value in In one of the secondary key value storage requests, until the first value cache is completed.
8. The method of claim 1, wherein the key-value storage partitioning the first value into a plurality of second values comprises:

After the first value cache is completed, the key value memory divides the first value into a plurality of the second values, wherein the plurality of second values of the first value are arranged in the order of division. Among the values, the magnitude of the first N second values is equal to the second threshold, the magnitude of the N+1th second value is less than or equal to the second threshold, and N is a positive integer greater than or equal to 1.
The method according to claim 1, wherein the secondary key value storage request includes a value offset, and the value offset is used to indicate that the second value carried in the secondary key value storage request is The position in the first value.
The method according to claim 1 or 4, characterized in that the key-value storage stores the order of the second value carried in each secondary key-value storage request in the first value. The storage of the second value carried in a secondary key value storage request in the first storage space includes:

The key value storage determines the position sequence of the second value carried in each secondary key value storage request in the first value according to the value offset in each secondary key value storage request;

The key value storage stores the second value in the first storage space according to the position sequence.
The method according to claim 1, wherein the key value storage allocating a first storage space for the first value in the first storage, comprising:

The key value storage allocates the first storage space for the first value from a second storage space, and the second storage space is pre-configured in the first storage for storing a plurality of the key values For storing the first value carried in the request, the size of the first storage space is equal to the size of the first value.
The method according to claim 6, wherein the key value storage allocating a first storage space for the first value in the first storage, comprising:

When the size of the free space in the second storage space of the key value storage is smaller than the size of the first value, the first storage space is allocated from the first storage for the first value .
The method of claim 1 or 6, wherein the method further comprises:

When the total size of the second value stored in the first storage space is equal to the size of the first value, the key value storage device determines that the storage of the first value has been completed.
A key-value storage is characterized in that the key-value storage includes a receiving unit, a distribution unit, a dividing unit, and a storage unit:

The receiving unit is configured to receive a key-value storage request, where the key-value storage request carries a key and a first value, wherein the key corresponds to the first value;

The allocating unit is configured to allocate a first storage space for the first value in a first storage, where the first storage is a storage other than the key value storage;

The dividing unit is configured to divide the first value into multiple second values when the capacity of the data contained in the first value is greater than or equal to the first threshold value, and generate multiple secondary key value storage requests Each secondary key value storage request carries the key and one of the second values, and the number of secondary key value storage requests is divided by the number of the second values obtained by dividing the first value. The same number

The storage unit is configured to store the second value carried in each secondary key value storage request in the order of the position of the second value carried in each secondary key value storage request in the first value Stored in the first storage space.
9. The key-value storage of claim 9, wherein when the dividing unit divides the first value into a plurality of second values, it is specifically configured to:

In the process of caching the first value, whenever the size of the cached data is equal to the second threshold, the cached data is used as the second value, and the second value is carried in one of the secondary In the key-value storage request, until the first value cache is completed.
9. The key-value storage of claim 9, wherein when the dividing unit divides the first value into a plurality of second values, it is specifically configured to:

After the first value is cached, the first value is divided into a plurality of the second values, wherein, among the plurality of second values of the first value arranged in the order of division, the first N The size of the second value is equal to the second threshold, the size of the N+1th second value is less than or equal to the second threshold, and N is a positive integer greater than or equal to 1.
The key value storage according to claim 9, wherein the secondary key value storage request includes a value offset, and the value offset is used to indicate the second value carried in the secondary key value storage request The position in the first value.
The key-value storage according to claim 9 or 12, wherein the storage unit stores the second value in the first value according to the order of the second value carried in each secondary key-value storage request. When the second value carried in each secondary key value storage request is stored in the first storage space, it is specifically used for:

According to the value offset in each secondary key value storage request, determine the position sequence of the second value carried in each secondary key value storage request in the first value;

According to the position sequence, the second value is stored in the first storage space.
9. The key-value storage of claim 9, wherein when the storage unit allocates the first storage space for the first value in the first storage, it is specifically used for:

Allocate the first storage space for the first value from the second storage space, where the first storage space is pre-configured in the first storage for storing multiple key-value storage requests carried The first value, the size of the first storage space is equal to the size of the first value.
The key-value storage according to claim 14, wherein when the storage unit allocates storage space for the first value in the first storage, it is specifically used for:

When the size of the free space in the second storage space is smaller than the size of the first value, the first storage space is allocated for the first value from the first memory.
The key-value storage according to claim 9 or 14, wherein the storage unit is further used for:

When the total size of the second value stored in the first storage space is equal to the size of the first value, it is determined that the storage of the first value has been completed.
A key-value storage server, wherein the key-value storage server includes a first storage and a key-value storage, the first storage is a storage other than the key-value storage, and the key-value storage is used to execute The method according to any one of claims 1-8.