WO2023109864A1 - Système de données, et procédé d'accès à des données et dispositif - Google Patents

Système de données, et procédé d'accès à des données et dispositif Download PDF

Info

Publication number
WO2023109864A1
WO2023109864A1 PCT/CN2022/139000 CN2022139000W WO2023109864A1 WO 2023109864 A1 WO2023109864 A1 WO 2023109864A1 CN 2022139000 W CN2022139000 W CN 2022139000W WO 2023109864 A1 WO2023109864 A1 WO 2023109864A1
Authority
WO
WIPO (PCT)
Prior art keywords
state
data
metadata
address
target data
Prior art date
Application number
PCT/CN2022/139000
Other languages
English (en)
Chinese (zh)
Inventor
任仁
马超
申勇
刘宏伟
赵博南
Original Assignee
华为技术有限公司
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 华为技术有限公司 filed Critical 华为技术有限公司
Publication of WO2023109864A1 publication Critical patent/WO2023109864A1/fr

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F12/00—Accessing, addressing or allocating within memory systems or architectures
    • G06F12/02—Addressing or allocation; Relocation
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/10—File systems; File servers
    • G06F16/13—File access structures, e.g. distributed indices
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
    • G06F16/23—Updating
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L67/00—Network arrangements or protocols for supporting network services or applications
    • H04L67/01—Protocols
    • H04L67/10—Protocols in which an application is distributed across nodes in the network
    • H04L67/1097—Protocols in which an application is distributed across nodes in the network for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS]
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L67/00—Network arrangements or protocols for supporting network services or applications
    • H04L67/14—Session management
    • H04L67/141—Setup of application sessions
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L67/00—Network arrangements or protocols for supporting network services or applications
    • H04L67/14—Session management
    • H04L67/142—Managing session states for stateless protocols; Signalling session states; State transitions; Keeping-state mechanisms

Definitions

  • the present application relates to the field of storage technologies, and in particular to a data system, a data access method and equipment.
  • Remote direct memory access can bypass the processor in the remote host and directly access the data in the remote host through the network card of the remote host. Because RDMA bypasses the processor, it can save a lot of processor resources and effectively improve the efficiency of data interaction.
  • the storage system includes multiple nodes, and changes in node status (such as node expansion, node failure, data migration between nodes, etc.) will affect the storage location of data in the storage system.
  • changes in node status such as node expansion, node failure, data migration between nodes, etc.
  • the client accesses the data stored by the nodes in the distributed storage system through unilateral RDMA, since the node's processor is bypassed, it cannot perceive the change of the node status in the distributed storage system, which will lead to data access failure.
  • the application provides a data system, a data access system and equipment to ensure accurate reading of data in a server.
  • an embodiment of the present application provides a data system, and the data system includes a client and a server.
  • the client needs to read data from the server, it can send a data request to the server.
  • the data request includes a sequence, and the sequence includes: the address of the first state, the address of the metadata, and the address of the second state, where the first state and the second state is the state of the metadata, the first state and the second state are used to describe whether the metadata changes, and the address of the second state is the same as the address of the first state.
  • the server may acquire a plurality of data according to a sequence, the plurality of data including metadata, a first state before obtaining the metadata, and a second state after obtaining the metadata. After acquiring the plurality of data, the server may send a data response to the client, where the data response includes metadata, the first state, and the second state.
  • the client when the client needs to read the data in the server, when the client requests metadata from the server, it can request metadata from the server through a data request and obtain the metadata status before the metadata (section state) and the state of the metadata after the metadata is acquired (the second state). That is to say, the client can obtain multiple data from the server through a data request, which can effectively improve the interaction efficiency between the server and the client.
  • the client after the client obtains the first state and the second state, the client can determine the change of the metadata when obtaining the metadata through the first state and the second state, so as to determine the integrity and validity of the metadata. To ensure that the client can accurately access the data in the server.
  • the sequence includes the address of the first state, the address of the metadata, the address of the second state, and the address of the home state of the target data. That is to say, after the address of the first state, the address of metadata, and the address of the second state, the sequence further includes the address of the belonging state of the target data.
  • the target data is the data described by the metadata.
  • the data response fed back from the server to the client also includes the attribution status of the target data. After the server obtains the metadata, the first status, and the second status, it can also The address gets the attribution status of the target data.
  • the server can obtain the first state, the metadata, the second state and the attribution state of the target data sequentially according to the order of each address in the sequence.
  • the server can carry multiple acquired data in the data response, and feed the data response back to the client, so that the client can obtain the first state, metadata, second state, and the attribution of the target data from the data response state, the client can further determine the integrity of the metadata to ensure that the target data can be accessed accurately in the future.
  • the client may acquire metadata, the first state, and the second state therefrom.
  • the client may compare the first state with the second state to determine whether the first state is consistent with the second state.
  • the client can determine the integrity of the metadata by using the comparison result between the first state and the second state, so as to ensure that the client can accurately obtain the target data based on the complete metadata.
  • the attribution status of the target data is recorded in the metadata, and after receiving the data response, the client can obtain the first status, the metadata, the second status, and the attribution status of the target data. In addition to comparing the first state with the second state, the client can also compare the attribution state of the target data with the attribution state of the target data recorded in the metadata, and determine that the attribution state of the target data is consistent with the attribution state recorded in the metadata. Whether the attribution status of the target data is consistent.
  • the client can determine the integrity of the metadata by using the comparison result between the first state and the second state, and the comparison result between the attribution state of the target data and the attribution state of the target data recorded in the metadata, so as to ensure The client can accurately obtain target data based on complete metadata.
  • the client obtains the target data from the server according to the metadata after determining that the first state is consistent with the second state .
  • the client determines that the first state is consistent with the second state, and that the attribution state of the target data is consistent with the target data recorded in the metadata.
  • the target data is obtained from the server according to the metadata.
  • the client determines that the first state is inconsistent with the second state, or that the attribution state of the target data is different from the target data recorded in the metadata. If the attribution status is inconsistent, the client can determine that the metadata is invalid, and the client can request the metadata from the server again, or interact with the server to determine the server to which the target data belongs.
  • the client can determine the integrity of the metadata according to the data carried in the data response, so that the target data can be accurately read from the server later.
  • the sequence includes the address of the first state, the address of the metadata, the address of the second state, the address of the state to which the target data belongs, and the address of the target data. That is to say, in this sequence, the address of the first state, the address of the metadata, and the address of the second state also include the address of the belonging state and the address of the target data, and the target data is the data described by the metadata; the data response Including the attribution status of the target data and the target data; after the server obtains the metadata, the first state, and the second state, it can also obtain the attribution status of the target data according to the address of the attribution status of the target data, and obtain the attribution status of the target data according to the address of the target data target data.
  • the address of the target data is pre-saved by the client. For example, when the client uses the address of the target data to access the target data, the metadata of the target data can be saved.
  • the metadata of the target data includes the address of the target data.
  • the client may also only save the address of the target data.
  • the client when the client needs to read the data in the server, when the client requests metadata from the server, it can request metadata, the first state, the second state, and the target from the server through a data request data.
  • the client can request multiple data through one data request, which can effectively reduce the number of interactions between the client and the server, and improve the interaction efficiency between the server and the client.
  • the client may store metadata locally, and if the client determines that the locally stored metadata is inconsistent with the metadata obtained from the server, the terminal to obtain target data.
  • the client can directly compare the consistency of the metadata to determine whether the metadata has been changed. If the inconsistency is determined, the target data can be read by using the metadata obtained from the server to ensure the final target data accuracy.
  • the client when sending the data request, may send the data request to the network card of the server based on RDMA.
  • the client sends a data request to the network card of the server based on RDMA, which can bypass the processor of the server and reduce the occupation of the processor of the server.
  • the data request when the server internally processes the data request, can be processed by a network card in the server, and the network card can respectively obtain the metadata, the first state before obtaining the metadata, the And the second state after obtaining the metadata; sending a data response to the client.
  • the network card in the server can process the data request more efficiently, and can also reduce the resource occupation of the processor in the server.
  • the server can receive a data request from the client, and the data request includes a sequence, and the sequence includes: the address of the first state, the address of the metadata, and the address of the second state, wherein the first state and the second state use To describe whether the metadata has changed, the address of the second state is the same as the address of the first state; after receiving the data request, the server can obtain the metadata, the first state before obtaining the metadata, and obtain The second state after the metadata; and sending a data response to the client, the data response includes the metadata, the first state, and the second state.
  • the sequence further includes the address of the attribution state of the target data, the address of the attribution state of the target data is located after the address of the second state, the target data is the data described by the metadata, and the data response also includes the target For the attribution status of the data, after the server obtains the metadata, the first status, and the second status, the attribution status can be obtained according to the address of the attribution status of the target data.
  • the sequence further includes the address of the home state of the target data and the address of the target data, the address of the home state of the target data is located after the address of the second state, and the address of the target data is located in the home state of the target data
  • the target data is the data described by the metadata
  • the data response also includes the attribution status of the target data and the target data.
  • the server can also obtain the attribution state according to the address of the attribution state, and obtain the target data according to the address of the target data.
  • the network card of the server may process the data request and feed back a data response. That is to say, the network card at the server end can obtain the metadata, the first state before obtaining the metadata, and the second state after obtaining the metadata respectively according to the sequence; and send a data response to the client.
  • the embodiment of the present application provides a data access method, which can be executed by the client.
  • the client can send a data request to the server, and the data request includes a sequence, and the sequence includes: the address of the first state, the address of the metadata, and the address of the second state, wherein the first state and the second state are used for Describes whether the metadata has changed, and the address of the second state is the same as the address of the first state.
  • the client may receive the data response sent by the server, where the data response includes metadata, the first state, and the second state.
  • the sequence further includes the address of the attribution state of the target data, the address of the attribution state of the target data is located after the address of the second state, the target data is the data described by the metadata, and the data response also includes the target The attribution status of the data.
  • the client may compare the first state with the second state to determine whether the first state is consistent with the second state.
  • the attribution status of the target data is recorded in the metadata.
  • the client can check the attribution status of the target data and the attribution of the target data recorded in the metadata. The status is compared to determine whether the attribution status of the target data is consistent with the attribution status of the target data recorded in the metadata.
  • the client determines the integrity of the metadata
  • the client determines that the first state is consistent with the second state
  • the target data is obtained from the server according to the metadata.
  • the client determines that the first state is consistent with the second state, and that the attribution state of the target data is consistent with the target data recorded in the metadata.
  • the target data is obtained from the server according to the metadata.
  • the client determines that the first state is inconsistent with the second state, or that the attribution state of the target data is different from the target data recorded in the metadata. If the attribution status is inconsistent, the client can determine that the metadata is invalid, and the client can request the metadata from the server again, or interact with the server to determine the server to which the target data belongs.
  • the sequence further includes the address of the home state of the target data and the address of the target data, the address of the home state of the target data is located after the address of the second state, and the address of the target data is located in the home state of the target data
  • the target data is the data described by the metadata; the data response includes the attribution status of the target data and the target data.
  • the data request may be sent to the network card of the server based on RDMA.
  • the embodiment of the present application also provides a data access device, the data access device has the function of realizing the behavior in the method example of the second aspect above, and the beneficial effects can be referred to the description of the first aspect, which will not be repeated here.
  • the functions may be implemented by hardware, or may be implemented by executing corresponding software through hardware.
  • Hardware or software includes one or more units corresponding to the functions described above.
  • the structure of the device includes a receiving unit, an acquiring unit, and a sending unit, and these units can perform the corresponding functions in the method example of the second aspect above. For details, refer to the detailed description in the method example, which will not be repeated here. .
  • the embodiment of the present application also provides a data access device, the data access device has the function of implementing the behavior in the method example of the third aspect above, and the beneficial effects can be referred to the description of the first aspect, which will not be repeated here.
  • the functions may be implemented by hardware, or may be implemented by executing corresponding software through hardware.
  • Hardware or software includes one or more units corresponding to the functions described above.
  • the structure of the device includes a receiving unit, a sending unit, and optionally a determining unit. These units can perform the corresponding functions in the method example of the third aspect above. For details, refer to the detailed description in the method example. description, and will not be repeated here.
  • the embodiment of the present application further provides a computing device, where the computing device includes a processor, a network card, and a memory.
  • the network card has the function of realizing the above-mentioned second aspect and the behaviors in the method examples in each possible implementation manner of the second aspect.
  • the beneficial effects please refer to the description of the first aspect and will not repeat them here. .
  • computer program instructions are stored in the memory, and the processor is coupled to the memory, and the processor can call the computer-executed instructions stored in the memory to execute the above third aspect and various possible implementations of the third aspect methods in methods.
  • the present application also provides a computer-readable storage medium, where instructions are stored in the computer-readable storage medium, and when it is run on a computer, the computer executes the above-mentioned second aspect and various possible implementations of the second aspect methods in methods. When it runs on a computer, the computer is made to execute the third aspect and the method in each possible implementation manner of the third aspect.
  • the present application further provides a computer program product including instructions, which, when run on a computer, cause the computer to execute the above second aspect and the method in each possible implementation manner of the second aspect.
  • the computer When it runs on a computer, the computer is made to execute the third aspect and the method in each possible implementation manner of the third aspect.
  • the present application also provides a computer chip, the chip is connected to the memory, and the chip is used to read and execute the software program stored in the memory, and execute the method in the above-mentioned second aspect and each possible implementation manner of the second aspect Or execute the third aspect and the method in each possible implementation manner of the third aspect.
  • FIG. 1 is a schematic structural diagram of a system provided by the present application.
  • FIG. 2 is a schematic diagram of a data access method provided by the present application.
  • 3-4 are schematic structural diagrams of a data access device provided in the present application.
  • Metadata is data describing data (data about data). Metadata can indicate the attributes of data. For example, metadata can record the storage address of data, the attribution status of data, etc.
  • the storage address is usually the physical address of the data in the storage system, and the attribution status of the data describes the node in the storage system where the data described by the metadata is located.
  • the embodiment of the present application does not limit the manner in which the metadata records the attribution status of the data, and a manner for recording the attribution status of the data is introduced below.
  • data is stored in units of shards, and data migration between nodes in the storage system can also be performed in units of shards.
  • a shard includes multiple pieces of data.
  • the version number of the shard will be updated every time it is migrated between nodes in the storage system. For example, if the shard is migrated from node A to node B, the shard version number of the shard recorded in node A will remain unchanged, and the version number of the shard on node B will be increased by one.
  • the metadata of the data in the slice may include a version number of the slice. Still take the migration of a fragment from node A to node B as an example.
  • the version number of the fragment in the metadata of the data in the fragment is consistent with the version number of the fragment recorded by node A.
  • the version number of the shard in the metadata of the new data is consistent with the version number of the shard recorded by Node B. It should be noted that the embodiment of the present application does not limit the type of the storage system, which may be a distributed storage system or a centralized storage system.
  • Metadata itself can also be regarded as a kind of data.
  • Metadata itself can also be regarded as a kind of data.
  • metadata description header which can record the status of the metadata.
  • metadata status is updated accordingly.
  • the embodiment of the present application does not limit the way of recording the metadata state.
  • the metadata status can be represented by a variable value, which is incremented by one every time the metadata changes.
  • RDMA is a technology that bypasses the operating system kernel of a remote device (such as a node in a storage system) to access data in its memory. Because it does not go through the operating system, it not only saves a lot of processor resources, but also improves system throughput and reduces It reduces the network communication delay of the system, and is especially suitable for wide application in large-scale parallel computer clusters.
  • RDMA has several major characteristics, (1) data is transmitted between the network and the remote device; (2) without the participation of the operating system kernel, all content related to sending and transmitting is offloaded to the network card; (3) virtual memory in user space The direct data transmission between the NIC does not involve the operating system kernel, and there is no additional data movement and copying.
  • client the two ends that need to exchange information
  • server the two ends that need to exchange information
  • the client is deployed on the user side, and the user can initiate a request to the server through the client.
  • the server can be deployed at the remote end.
  • the server generally refers to the storage system, and can be specifically understood as a node in the storage system.
  • Unilateral RDMA includes RDMA read (READ) operations and RDMA write (WRITE) operations.
  • the RDMA read operation is used for the client to read data in the storage space of the server.
  • the server Before the client performs the RDMA read operation, the server first allows the client to obtain the address of the data to be read in advance, that is, the address and length of the data to be read by the RDMA read operation are sent to the client.
  • the address of the data includes the address of some or all of the following metadata, the address of the metadata state, the address of the data attribution state, the address of the target data, and the like.
  • the client After the client obtains the address of the data to be read, it can directly read the data from the storage space of the server through the network card of the server, and the whole process does not require the participation of the processor.
  • the client device may send a data request in a unilateral RDMA manner, and read the data requested by the data request from the server.
  • Bilateral RDMA includes an RDMA send (SEND) operation and an RDMA receive (RECEIVE) operation, which requires the participation of the server processor.
  • the data system includes a client 120 and a server 110.
  • the server 110 may be a node in the storage system.
  • the embodiment of the present application does not The type of the storage system is not limited, and may be a distributed storage system or a centralized storage system.
  • Client 120 may be a physical machine or a virtual machine. Physical machines include, but are not limited to, desktop computers, servers, laptops, and mobile devices.
  • Users can access data in the server 110 through the client 120 , such as writing data in the server 110 or reading data from the server 110 .
  • the client 120 When the client 120 detects the user's operation on the application program and determines that the user needs to access the data in the server 110 , the client 120 can interact with the server 110 to access the data in the server 110 .
  • the client 120 may first obtain the metadata of the data, and obtain the data according to the metadata of the data.
  • the client 120 may request the metadata of the data from the server 110 through a data request, as well as the state of the server 110 before and after reading the metadata. After the client 120 obtains the metadata of the data and the status of the server 110 before and after reading the metadata, it can determine the metadata of the data by comparing the status of the server 110 before and after reading the metadata. The integrity of the data further determines the validity of the data obtained using this metadata. In this way, it can avoid the situation that the metadata obtained according to the metadata is invalid due to the change of the metadata during the process of reading the metadata. In addition, multiple data can be requested from the server 110 through one data request, which can effectively reduce the number of interactions between the server 110 and the client 120 and improve data access efficiency.
  • the client 120 may also request the metadata of the data, the status of the server 110 before and after reading the metadata, and the attribution status of the data from the server 110 through a data request.
  • the client 120 After the client 120 obtains the metadata of the data, the status of the server 110 before and after reading the metadata, and the attribution status of the data, it can compare the status of the server 110 before and after reading the metadata , or compare the attribution status of the acquired data with the attribution status of the data recorded in the metadata, determine the integrity of the metadata of the data, and further determine the validity of the data acquired by using the metadata.
  • the client 120 can acquire the data by using the previously saved data address of the data while acquiring the metadata of the data.
  • the client 120 may request the server 110 for the metadata of the data, the state of the server 110 before and after reading the metadata, and the data (the The data request carries the data address).
  • the client 120 After the client 120 obtains the metadata of the data and the status of the server 110 before and after reading the metadata, it can determine the metadata of the data by comparing the status of the server 110 before and after reading the metadata. The integrity of the data further determines the validity of obtaining the data by using the previously saved data address of the data.
  • the obtained data is also valid, and there is no need to use the metadata to obtain the data, which can reduce the number of interactions between the server 110 and the client 120 .
  • multiple data can be requested from the server 110 through one data request, which can effectively reduce the number of interactions between the server 110 and the client 120 and improve data access efficiency.
  • the server 110 includes a bus 111 , a processor 112 , a memory 113 , a network card 114 and a hard disk 115 .
  • a hard disk is used as the persistent storage of the server 110 as an example for illustration, but mechanical hard disks or other types of hard disks are also applicable to the embodiments of the present application.
  • the processor 112 may be a central processing unit (central processing unit, CPU), and the processor 112 may also be other general-purpose processors or modules with computing functions.
  • CPU central processing unit
  • the memory 113 may include a volatile memory (volatile memory), such as a random access memory (random access memory, RAM), a dynamic random access memory (dynamic random access memory, DRAM), and the like. It may also be a non-volatile memory (non-volatile memory), such as a storage-class memory (storage-class memory, SCM), or a combination of a volatile memory and a non-volatile memory.
  • volatile memory such as a random access memory (random access memory, RAM), a dynamic random access memory (dynamic random access memory, DRAM), and the like. It may also be a non-volatile memory (non-volatile memory), such as a storage-class memory (storage-class memory, SCM), or a combination of a volatile memory and a non-volatile memory.
  • the server 110 may also include one or more hard disks 115 .
  • the hard disk 115 can be used to permanently store data.
  • the hard disk 115 may also include a hard disk cache and a persistent storage medium.
  • the network card 114 is used for communicating with the client 120 and receiving a data request sent by the client 120 .
  • the network card can process the data request, and feed back to the client 120 according to the data request
  • the data request requests the data.
  • the network card 114 may process the data access request by invoking the computer-executed instructions stored in the memory 113, that is, execute the data access method provided in the embodiment of the present application.
  • the network card 114 may also invoke computer-executed instructions stored inside the network card 114 to execute the data access method provided in the embodiment of the present application.
  • computer storage instructions may also be programmed on the network card 114, and the network card 114 may execute the data access method provided in the embodiment of the present application.
  • the client 120 includes a bus 121 , a processor 122 , a memory 123 , and a network card 124 .
  • the functions and functions of the bus 121, processor 122, memory 123, and network card 124 are similar to those of the bus 111, processor 112, memory 113, network card 114, and hard disk 115.
  • the bus 121, processor 122, memory 123, and network card 124 are similar to those of the bus 111, processor 112, memory 113, network card 114, and hard disk 115.
  • the processor 122 can call the computer execution instructions stored in the memory 123 to execute the data access method provided by the embodiment of the present application.
  • the following describes the data access method provided by the embodiment of the present application with reference to FIG. 2 .
  • the data described by the metadata is called target data.
  • the method includes:
  • Step 201 the client 120 sends a data request to the server 110 when it needs to access target data.
  • the data request is used to request data from the server 110, and the data request carries an address sequence (the address sequence may also be referred to as a sequence for short), and the address sequence includes multiple addresses.
  • the multiple addresses are the addresses of the data that needs to be requested from the server 110 .
  • the user can trigger the client 120 to access the target data stored in the server by operating the application program running on the client 120 .
  • the client 120 accesses the target data, it needs to obtain the metadata of the target data first, and determine the data address of the target data according to the metadata, and then the client 120 can request the target from the server 110 according to the metadata of the target data. data.
  • the client 120 needs to obtain metadata first, and the data requested by the data request includes metadata.
  • the data requested by the data request may also include other data.
  • the address types included in the address sequence in the request and the data requested by the data request are explained:
  • the address sequence includes the address of the metadata state, the address of the metadata, and the address of the metadata state in order.
  • the data requested in the data request is metadata, a state of the metadata before reading the metadata, and a state of the metadata after reading the state of the metadata.
  • the metadata state before reading the metadata and the metadata state after reading the metadata state in this embodiment of the application, the metadata state before reading the metadata is referred to as the first state .
  • the state of the metadata after reading the metadata is referred to as the second state.
  • the address of the metadata here refers to the address of the metadata of the target data.
  • the address of the metadata state refers to the address of the metadata state of the target data.
  • the address of the metadata state may be the address of the description header of the metadata of the target data.
  • the address sequence includes in order the address of the first state, the address of the metadata, the address of the second state, and the address of the attribution state of the target data.
  • the data requested in the data request are metadata, the first state and the second state.
  • the address of the data attribution status refers to the address of the attribution status of the target data.
  • the data attribution status may be the version number of the segment to which the target data belongs in the server 110
  • the address of the data attribution status is the address in the server 110 that records the allocated version number.
  • the address sequence includes the address of the first state, the address of the metadata, the address of the second state, the address of the belonging state of the target data, and the address of the target data in order.
  • the data requested in the data request are metadata, the first state, the second state, the attribution state of the target data, and the target data.
  • the address of the target data is the address used by the client 120 to read the target data before. If the client 120 has also read the target data before, the client 120 can save the target locally when reading the target data.
  • the metadata of the data or the address of that target data are examples of the address of that target data.
  • the address sequence includes the address of the first state, the address of the metadata, the address of the second state, and the address of the target data in sequence.
  • the data requested by the data request are metadata, first state, second state, and target data.
  • the addresses that may be included in the above address sequence can be pre-acquired by the client 120, and this embodiment of this application does not limit the way that the client 120 pre-acquires the addresses included in the address sequence.
  • the server 110 can centrally store each metadata in the memory, the server 110 can provide the client 120 with the starting address of the metadata in the memory in advance, and the client 120 device calculates based on a preset algorithm The starting address of the metadata is processed to obtain the address of the metadata of the target data.
  • metadata state addresses when storing each metadata state, the server 110 may establish an association relationship between the address of the metadata state and the address of the corresponding metadata.
  • the association relationship may be notified to the client 120 by the server 110 in advance, or may be pre-scheduled between the server 110 and the client 120 .
  • the client 120 may determine the address of the metadata state based on the association relationship and the address of the metadata.
  • the server 110 may notify the client 120 of the address for recording the version number of each fragment in advance, and when the client 120 needs to determine the attribution status of the target data, it may first determine The segment to which the target data belongs is further determined to record the address of the version of the segment.
  • Step 202 the server 110 receives the data request, and obtains the data requested by the data according to the data request.
  • the server 110 can obtain the address sequence in the data request; the server 110 obtains the data stored at the address according to the order of multiple addresses in the address sequence.
  • the manner in which the server 110 processes different address sequences is described below:
  • the address sequence includes the address of the metadata state, the address of the metadata, and the address of the metadata state in order.
  • the server 110 first reads the first state according to the address of the metadata state, then reads the metadata according to the address of the metadata, and then reads the second state according to the address of the metadata state after obtaining the metadata.
  • the server 110 when processing the address sequence, the server 110 reads the metadata status once before and after reading the metadata, and the read metadata status is the first status and the second status respectively.
  • the processing of the data by the server 110 itself or the interaction between the servers 110 will cause the metadata of the data to change.
  • the server 110 performs data migration (the data address of the metadata record will change), modification, and deletion (the metadata of the data may be deleted) within the server 110 .
  • the processing of the data by the server 110 may lead to changes in the metadata of the data, that is, the server 110 may update the metadata of the data, resulting in a change in the status of the metadata. If the server 110 updates the metadata while the client 120 sends a data request to obtain the metadata, the metadata obtained by the client 120 from the server may be incomplete.
  • the server 110 reads the metadata at the request of the client 120 .
  • the address of the metadata state is also placed before and after the address of the metadata.
  • the client 120 may also obtain the state of the metadata before and after reading the metadata from the server.
  • the address sequence includes in order the address of the metadata state, the address of the metadata, the address of the metadata state, and the address of the attribution state of the target data.
  • the server 110 first reads the first state according to the address of the metadata state, then reads the metadata according to the address of the metadata, and after obtaining the metadata, reads the second state according to the address of the metadata state, and then reads the second state according to the address of the metadata state.
  • the address of the attribution status of the target data reads the attribution status of the target data.
  • the server 110 when processing the address sequence, not only reads the metadata, the first status, and the second status, but also reads the attribution status of the target data.
  • the processing of the data by the server 110 itself or the interaction between the servers 110 will cause the server 110 where the data is located to change, for example, the shard to which the target data belongs is migrated from the server 110 to another
  • the server 110 that is, the data ownership status changes.
  • data migration can be performed between the server 110 and other server 110, which will cause the ownership of the data to change from the server 110 to another server 110, and the server 110 itself can delete data, which will cause the data to no longer exist.
  • Attributed server 110 If the attribution status of the target data changes when the client 120 sends a data request to obtain the data request, the metadata obtained by the client 120 from the server may be incomplete.
  • the attribution status of the target data is in the Variety.
  • the address sequence carried in the data request also includes the address of the data attribution status.
  • the client 120 can also acquire the attribution status of the read target data from the server.
  • the address sequence includes the address of the first state, the address of the metadata, the address of the second state, the address of the belonging state of the target data, and the address of the target data in sequence.
  • the data requested in the data request are metadata, the first state, the second state, the attribution state of the target data, and the target data.
  • the server 110 first reads the first state according to the address of the metadata state, then reads the metadata according to the address of the metadata, and after obtaining the metadata, reads the second state according to the address of the metadata state, and then reads the second state according to the address of the metadata state.
  • the data attribution status reads the data attribution status, and then reads the target data according to the address of the target data.
  • the server 110 when processing the address sequence, the server 110 will not only read the metadata, the first state, the second state, and the data attribution state, but also read the target data.
  • the address sequence also carries the address of the target data, which is the address used by the client 120 to access the target data from the server 110 before.
  • the client 120 may save the address of the target data when accessing the target data before. Since the address sequence carries the address of the target data, that is to say, the server 110 will also read the target data and feed it back to the client 120 device. After the client 120 determines the integrity of the metadata, It also determines the validity of the target data fed back by the server 110. If the target data is valid, the client 120 does not need to obtain the target data from the server 110 according to the metadata, but directly obtains the target data from the server 110 this time. Feedback target data is enough.
  • the address sequence includes the address of the first state, the address of the metadata, the address of the second state, and the address of the target data in order.
  • the data requested by the data request are metadata, first state, second state, and target data.
  • the server 110 first reads the first state according to the address of the metadata state, then reads the metadata according to the address of the metadata, and after obtaining the metadata, reads the second state according to the address of the metadata state, and then reads the second state according to the address of the metadata state.
  • the address of the target data reads the target data.
  • the server 110 when processing the address sequence, the server 110 will not only read the metadata, the first state, and the second state, but also read the target data.
  • Step 203 The server 110 feeds back a data response to the client 120, and the data response carries the data read by the server 110.
  • the server 110 After the server 110 reads the data according to the address sequence, it can feed back the read data to the client 120 .
  • the address sequence includes the address of the first state, the address of the metadata, and the address of the second state in order.
  • the data response includes the first state, metadata, and the second state.
  • the address sequence includes the address of the first state, the address of the metadata, the address of the second state, and the address of the belonging state of the target data in order.
  • the data response includes the first status, metadata and the second status, the attribution status of the target data.
  • the address sequence includes the address of the first state, the address of the metadata, the address of the second state, the address of the belonging state of the target data, and the address of the target data in order.
  • the data response includes a first status, metadata, a second status, an attribution status of the target data, and the target data.
  • the address sequence includes the address of the first state, the address of the metadata, the address of the second state, and the address of the target data in order.
  • the data response includes a first state, metadata, a second state, and target data.
  • Step 204 the client 120 receives the data response, and checks the integrity of the metadata according to the data response.
  • the client 120 may parse the data carried in the data response to determine the integrity of the metadata.
  • the client 120 can compare the first state with the second state. When it is determined that the first state and the second state are consistent, it means that the metadata has not changed before and after reading, and the metadata carried in the data response is complete. , the client 120 may request the server 110 to read the target data based on the metadata. When it is determined that the first state and the second state are inconsistent, it means that the metadata has changed before and after reading, and the metadata carried in the data response is incomplete, the client 120 can request the metadata from the server 110 again, and the client The terminal 120 may also use bilateral RDMA to interact with the server 110 to obtain the target data.
  • the data response includes the first status, metadata and the second status, the attribution status of the target data.
  • the client 120 may compare the first status with the second status, and compare the attribution status of the target data recorded in the metadata with the attribution status of the target data carried in the data response. When the results of the above two comparisons are consistent, it means that the metadata obtained this time is complete and the attribution status of the target data has not changed.
  • the client 120 can request the server 110 to read the target data based on the metadata. When at least one result of the above two comparisons indicates inconsistency, it means that the metadata obtained this time is incomplete. Afterwards, the client 120 may request the metadata from the server 110 again, and the client 120 may also use bilateral RDMA to interact with the server 110 to obtain the target data. When the attribution status of the target data is inconsistent, the client 120 can interact with the server 110 to determine the server 110 to which the target data belongs, and then the client 120 can interact with the server 110 to which the target data belongs to obtain the target data.
  • the data response includes the first status, the metadata, the second status, the attribution status of the target data, and the target data.
  • the client 120 may compare the first status with the second status, and compare the data attribution status recorded in the metadata with the data attribution status carried in the data response. When the results of the above two comparisons are consistent, it means that the metadata obtained this time is complete, and the attribution status of the data has not changed, and the target data carried in the data response is valid, and the client 120 can directly obtain the data from the data response. Read target data. When at least one result of the above two comparisons indicates inconsistency, it means that the metadata obtained this time is incomplete. Afterwards, the client 120 may request the metadata from the server 110 again, and the client 120 may also use bilateral RDMA to interact with the server 110 to obtain the target data. When the attribution status of the target data is inconsistent, the client 120 can interact with the server 110 to determine the server 110 to which the target data belongs, and then the client 120 can interact with the server 110 to which the target data belongs to obtain the target data.
  • the client 120 will not only compare the first state with the second state, but also compare the data attribution state recorded in the metadata and the data carried in the data response.
  • the data attribution status comparison can also compare the metadata of the target data stored locally with the metadata obtained this time. That is to say, three kinds of comparisons need to be performed in this case, and when the results of these three kinds of comparisons all indicate the same, it means that the metadata obtained this time is complete, and the client 120 can directly read the target data from the data response. When at least one of the three comparisons indicates an inconsistency, it means that the target data read this time is invalid, and the client 120 cannot directly use the target data in the data response. 110 reacquires the target data (for example, among the three types of comparisons, only the locally stored metadata of the target data is inconsistent with the metadata obtained this time), and the client 120 can also acquire the target data in the aforementioned manner.
  • the data response includes the first state, metadata, second state, and target data.
  • the client 120 can compare the first state with the second state. When the comparison results indicate that they are consistent, it means that the metadata acquired this time is complete, and the target data carried in the data response is valid, and the client 120 can directly read the target data from the data response. When the result of the comparison indicates inconsistency, it means that the metadata obtained this time is incomplete. After that, the client 120 can request the metadata from the server 110 again.
  • the client 120 may also use bilateral RDMA to interact with the server 110 to acquire the target data.
  • the client 120 can compare the metadata of the target data stored locally and the obtained metadata for comparison. That is to say, in this case, two comparisons need to be performed. When the results of the two comparisons indicate the same, it means that the metadata acquired this time is complete, and the client 120 can directly read the target data from the data response. When at least one of the results of these two comparisons indicates an inconsistency, it means that the target data read this time is invalid, and the client 120 cannot directly use the target data in the data response. 110 reacquires the target data (for example, only the locally stored metadata of the target data in the two comparisons is inconsistent with the metadata obtained this time), and the client 120 can also acquire the target data in the aforementioned manner.
  • the interaction between the client 120 and the server 110 in step 201 and step 203 may be based on unilateral RDMA.
  • the embodiment of the present application does not limit the way in which the server 110 interacts with the client 120, any method that can bypass the processor of the server 110, and directly interacts between the client 120 and the network card of the server 110 is applicable In the embodiment of this application.
  • the embodiment of the present application also provides a data access device, which is used to execute the method executed by the server in the method embodiment shown in Figure 2 above, and related features can be found in The foregoing method embodiments will not be repeated here.
  • the data access device 300 includes a receiving unit 301, an acquiring unit 302, and a sending unit 303;
  • the receiving unit 301 is configured to receive a data request from a client, the data request includes a sequence, and the sequence includes: an address of a first state, an address of metadata, and an address of a second state, wherein the first state and the second state are used to describe Whether the metadata changes, the address of the second state is the same as the address of the first state; the receiving unit 301 can execute the method executed by the server in step 201 as shown in FIG. 2 .
  • the obtaining unit 302 is configured to respectively obtain metadata, a first state before obtaining the metadata, and a second state after obtaining the metadata according to the sequence.
  • the acquiring unit 302 may execute step 202 as shown in FIG. 2 .
  • the sending unit 303 is configured to send a data response to the client, where the data response includes metadata, a first state, and a second state.
  • the sending unit 303 may execute step 203 shown in FIG. 2 .
  • the sequence further includes the address of the attribution state of the target data, the address of the attribution state of the target data is located after the address of the second state, the target data is the data described by the metadata, and the data response also includes the target For the attribution status of the data, after acquiring the metadata, the first status, and the second status, the acquisition unit 302 may also acquire the attribution status of the target data according to the address of the attribution status of the target data.
  • the sequence further includes the address of the attribution state of the target data and the address of the target data, the address of the attribution state of the target data is located after the address of the second state, and the address of the target data is at the attribution state of the target data
  • the target data is the data described by the metadata; after acquiring the metadata, the first state, and the second state, the acquisition unit 302 can also acquire the attribution status of the target data according to the address of the attribution status of the target data, according to The address of the target data acquires the target data.
  • the acquiring unit 302 is deployed on a network card of the server.
  • the embodiment of the present application also provides a data access device, which is used to execute the method executed by the client in the method embodiment shown in Figure 2 above, and related features can be found in The foregoing method embodiments will not be repeated here.
  • the data access device 400 includes a receiving unit 401 and a sending unit 403 .
  • a determining unit 402 may also be included.
  • the sending unit 403 is configured to send a data request to the server, the data request includes a sequence, and the sequence includes: an address of a first state, an address of metadata, and an address of a second state, wherein the first state and the second state are used to describe Whether the metadata changes, the address of the second state is the same as the address of the first state.
  • the sending unit 403 may execute step 201 as shown in FIG. 2 .
  • the receiving unit 401 is configured to receive a data response sent by the server, where the data response includes metadata, a first state, and a second state.
  • the receiving unit 401 may execute step 203 shown in FIG. 2 .
  • the sequence further includes the address of the attribution state of the target data, the address of the attribution state of the target data is located after the address of the second state, the target data is the data described by the metadata, and the data response also includes the target The attribution status of the data.
  • the determining unit 402 may receive the data access response, and determine whether the obtained first state is consistent with the second state.
  • the attribution status of the target data is recorded in the metadata, and the determining unit 402 may also determine whether the attribution status of the target data is consistent with the attribution status of the target data recorded in the metadata.
  • the determining unit 402 obtains the target from the server according to the metadata when it is determined that the first state is consistent with the second state. data.
  • the determination unit 402 determines that the first status is consistent with the second status, and the attribution status of the target data is the same as the target data recorded in the metadata.
  • the target data is obtained from the server according to the metadata.
  • the determining unit 402 determines that the first status is inconsistent with the second status, or the attribution status of the target data is different from the target data recorded in the metadata.
  • inconsistent attribution status the client can determine that the metadata is invalid, and the client can request the metadata from the server again, or interact with the server to determine the server to which the target data belongs.
  • the sequence further includes the address of the attribution state of the target data and the address of the target data, the address of the attribution state of the target data is located after the address of the second state, and the address of the target data is at the attribution state of the target data
  • the target data is the data described by the metadata; the data response includes the attribution status of the target data and the target data.
  • the determining unit 402 obtains the target data from the server according to the metadata when determining that the metadata stored locally by the client is inconsistent with the metadata.
  • the sending unit 403 may send the data request to the network card of the server based on RDMA.
  • each functional unit in the embodiment of the present application may be integrated into one processing unit, each unit may exist separately physically, or two or more units may be integrated into one unit.
  • the above-mentioned integrated units can be implemented in the form of hardware or in the form of software functional units.
  • the above-mentioned embodiments may be implemented in whole or in part by software, hardware, firmware or other arbitrary combinations.
  • the above-described embodiments may be implemented in whole or in part in the form of computer program products.
  • the computer program product includes one or more computer instructions. When the computer program instructions are loaded or executed on the computer, the processes or functions according to the embodiments of the present invention will be generated in whole or in part.
  • the computer may be a general purpose computer, a special purpose computer, a computer network, or other programmable devices.
  • the computer instructions may be stored in or transmitted from one computer-readable storage medium to another computer-readable storage medium, for example, the computer instructions may be transmitted from a website, computer, server or data center Transmission to another website site, computer, server, or data center by wired (eg, coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (eg, infrared, wireless, microwave, etc.).
  • the computer-readable storage medium may be any available medium that can be accessed by a computer, or a data storage device such as a server or a data center that includes one or more sets of available media.
  • the available media may be magnetic media (eg, floppy disk, hard disk, magnetic tape), optical media (eg, DVD), or semiconductor media.
  • the semiconductor medium may be a solid state drive (SSD).
  • the embodiments of the present application may be provided as methods, systems, or computer program products. Accordingly, the present application may take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present application may take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) having computer-usable program code embodied therein.
  • computer-usable storage media including but not limited to disk storage, CD-ROM, optical storage, etc.
  • These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing apparatus to operate in a specific manner, such that the instructions stored in the computer-readable memory produce an article of manufacture comprising instruction means, the instructions
  • the device realizes the function specified in one or more procedures of the flowchart and/or one or more blocks of the block diagram.

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Computer Networks & Wireless Communication (AREA)
  • Signal Processing (AREA)
  • Physics & Mathematics (AREA)
  • General Engineering & Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Data Mining & Analysis (AREA)
  • Databases & Information Systems (AREA)
  • Information Retrieval, Db Structures And Fs Structures Therefor (AREA)

Abstract

L'invention concerne un système de données ainsi qu'un procédé d'accès à des données et un dispositif. Dans la présente demande, lors de l'accès à des données sur un serveur, un client envoie une demande de données au serveur, la demande de données comprenant une séquence et la séquence comprenant une adresse d'un état de métadonnées, une adresse de métadonnées et une adresse d'un état de métadonnées. Le serveur acquiert une pluralité d'éléments de données selon la séquence, la pluralité d'éléments de données comprenant des métadonnées, un premier état avant l'acquisition des métadonnées et un second état après l'acquisition des métadonnées. Après l'acquisition de la pluralité d'éléments de données, le serveur envoie une réponse de données à un client, la réponse de données comprenant les métadonnées, le premier état et le second état. Lorsque le client demande les métadonnées au serveur, il acquiert la pluralité d'éléments de données auprès du serveur au moyen de la demande de données, de telle sorte que l'efficacité d'interaction entre le serveur et le client peut être efficacement améliorée. Après l'acquisition du premier état et du second état, le client détermine l'intégrité des métadonnées au moyen du premier état et du second état, de telle sorte qu'il est garanti que le client peut accéder avec précision aux données sur le serveur.
PCT/CN2022/139000 2021-12-16 2022-12-14 Système de données, et procédé d'accès à des données et dispositif WO2023109864A1 (fr)

Applications Claiming Priority (4)

Application Number Priority Date Filing Date Title
CN202111542089 2021-12-16
CN202111542089.0 2021-12-16
CN202210142866.0A CN116266842A (zh) 2021-12-16 2022-02-16 一种数据系统、数据访问方法及设备
CN202210142866.0 2022-02-16

Publications (1)

Publication Number Publication Date
WO2023109864A1 true WO2023109864A1 (fr) 2023-06-22

Family

ID=86744080

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2022/139000 WO2023109864A1 (fr) 2021-12-16 2022-12-14 Système de données, et procédé d'accès à des données et dispositif

Country Status (2)

Country Link
CN (1) CN116266842A (fr)
WO (1) WO2023109864A1 (fr)

Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN104618482A (zh) * 2015-02-02 2015-05-13 浙江宇视科技有限公司 访问云数据的方法、服务器、传统存储设备、架构
CN104754021A (zh) * 2013-12-31 2015-07-01 伊姆西公司 用于促进对分布式存储系统中的数据的访问的装置和方法
CN110191194A (zh) * 2019-06-13 2019-08-30 华中科技大学 一种基于rdma网络的分布式文件系统数据传输方法和系统
CN110888889A (zh) * 2018-08-17 2020-03-17 阿里巴巴集团控股有限公司 一种数据信息更新方法、装置及设备
CN111708734A (zh) * 2020-05-29 2020-09-25 苏州浪潮智能科技有限公司 一种分布式系统的元数据访问方法、系统、设备以及介质

Patent Citations (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN104754021A (zh) * 2013-12-31 2015-07-01 伊姆西公司 用于促进对分布式存储系统中的数据的访问的装置和方法
US20150186520A1 (en) * 2013-12-31 2015-07-02 Emc Corporation Facilitating access to data in distributed storage system
CN104618482A (zh) * 2015-02-02 2015-05-13 浙江宇视科技有限公司 访问云数据的方法、服务器、传统存储设备、架构
CN110888889A (zh) * 2018-08-17 2020-03-17 阿里巴巴集团控股有限公司 一种数据信息更新方法、装置及设备
CN110191194A (zh) * 2019-06-13 2019-08-30 华中科技大学 一种基于rdma网络的分布式文件系统数据传输方法和系统
CN111708734A (zh) * 2020-05-29 2020-09-25 苏州浪潮智能科技有限公司 一种分布式系统的元数据访问方法、系统、设备以及介质

Also Published As

Publication number Publication date
CN116266842A (zh) 2023-06-20

Similar Documents

Publication Publication Date Title
US20200356277A1 (en) De-duplication of client-side data cache for virtual disks
US20220382685A1 (en) Method and Apparatus for Accessing Storage System
CN104239575A (zh) 一种虚拟机镜像文件存储、分发方法及装置
TW201220197A (en) for improving the safety and reliability of data storage in a virtual machine based on cloud calculation and distributed storage environment
CN110908589B (zh) 数据文件的处理方法、装置、系统和存储介质
US11314454B2 (en) Method and apparatus for managing storage device in storage system
WO2022218160A1 (fr) Système et procédé d'accès à des données et dispositif et carte réseau
WO2021213281A1 (fr) Procédé et système de lecture de données
CN108540510B (zh) 一种云主机创建方法、装置及云服务系统
WO2023185770A1 (fr) Procédé et appareil de mise en cache de données en nuage, dispositif et support de stockage
US11321021B2 (en) Method and apparatus of managing mapping relationship between storage identifier and start address of queue of storage device corresponding to the storage identifier
CN110018879B (zh) 应用于分布式系统的延迟加载方法及装置
US10545667B1 (en) Dynamic data partitioning for stateless request routing
US11157456B2 (en) Replication of data in a distributed file system using an arbiter
CN111221826A (zh) 一种处理共享缓存同步消息的方法、系统、设备及介质
CN116560562A (zh) 一种读、写数据方法及装置
US20240012800A1 (en) Data processing method, server, and system
WO2016206070A1 (fr) Procédé de mise à jour de fichier et dispositif de mémorisation
WO2021226822A1 (fr) Procédé et appareil d'écriture de journal, dispositif électronique et support de stockage
WO2023109864A1 (fr) Système de données, et procédé d'accès à des données et dispositif
US10503409B2 (en) Low-latency lightweight distributed storage system
WO2023050856A1 (fr) Procédé de traitement de données et système de stockage
WO2022257685A1 (fr) Système de stockage, carte d'interface réseau, processeur et procédé, appareil et système d'accès à des données
WO2023273803A1 (fr) Procédé et appareil d'authentification et système de stockage
CN116594551A (zh) 一种数据存储方法及装置

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 22906609

Country of ref document: EP

Kind code of ref document: A1