WO2021082868A1 - 分布式存储系统的数据处理方法、装置及电子设备 - Google Patents

分布式存储系统的数据处理方法、装置及电子设备 Download PDF

Info

Publication number
WO2021082868A1
WO2021082868A1 PCT/CN2020/119346 CN2020119346W WO2021082868A1 WO 2021082868 A1 WO2021082868 A1 WO 2021082868A1 CN 2020119346 W CN2020119346 W CN 2020119346W WO 2021082868 A1 WO2021082868 A1 WO 2021082868A1
Authority
WO
WIPO (PCT)
Prior art keywords
data
server
control server
request
copy information
Prior art date
Application number
PCT/CN2020/119346
Other languages
English (en)
French (fr)
Inventor
黎海兵
Original Assignee
北京金山云网络技术有限公司
北京金山云科技有限公司
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 北京金山云网络技术有限公司, 北京金山云科技有限公司 filed Critical 北京金山云网络技术有限公司
Priority to US17/772,939 priority Critical patent/US11966305B2/en
Publication of WO2021082868A1 publication Critical patent/WO2021082868A1/zh

Links

Images

Classifications

    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L41/00—Arrangements for maintenance, administration or management of data switching networks, e.g. of packet switching networks
    • H04L41/06—Management of faults, events, alarms or notifications
    • H04L41/0654—Management of faults, events, alarms or notifications using network fault recovery
    • H04L41/0663—Performing the actions predefined by failover planning, e.g. switching to standby network elements
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F11/00—Error detection; Error correction; Monitoring
    • G06F11/07—Responding to the occurrence of a fault, e.g. fault tolerance
    • G06F11/16—Error detection or correction of the data by redundancy in hardware
    • G06F11/20—Error detection or correction of the data by redundancy in hardware using active fault-masking, e.g. by switching out faulty elements or by switching in spare elements
    • G06F11/202—Error detection or correction of the data by redundancy in hardware using active fault-masking, e.g. by switching out faulty elements or by switching in spare elements where processing functionality is redundant
    • G06F11/2023—Failover techniques
    • G06F11/2028—Failover techniques eliminating a faulty processor or activating a spare
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F11/00—Error detection; Error correction; Monitoring
    • G06F11/07—Responding to the occurrence of a fault, e.g. fault tolerance
    • G06F11/0703—Error or fault processing not based on redundancy, i.e. by taking additional measures to deal with the error or fault not making use of redundancy in operation, in hardware, or in data representation
    • G06F11/0751—Error or fault detection not based on redundancy
    • G06F11/0754—Error or fault detection not based on redundancy by exceeding limits
    • G06F11/0757—Error or fault detection not based on redundancy by exceeding limits by exceeding a time limit, i.e. time-out, e.g. watchdogs
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F11/00—Error detection; Error correction; Monitoring
    • G06F11/07—Responding to the occurrence of a fault, e.g. fault tolerance
    • G06F11/16—Error detection or correction of the data by redundancy in hardware
    • G06F11/20—Error detection or correction of the data by redundancy in hardware using active fault-masking, e.g. by switching out faulty elements or by switching in spare elements
    • G06F11/202—Error detection or correction of the data by redundancy in hardware using active fault-masking, e.g. by switching out faulty elements or by switching in spare elements where processing functionality is redundant
    • G06F11/2038—Error detection or correction of the data by redundancy in hardware using active fault-masking, e.g. by switching out faulty elements or by switching in spare elements where processing functionality is redundant with a single idle spare processing component
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F11/00—Error detection; Error correction; Monitoring
    • G06F11/07—Responding to the occurrence of a fault, e.g. fault tolerance
    • G06F11/16—Error detection or correction of the data by redundancy in hardware
    • G06F11/20—Error detection or correction of the data by redundancy in hardware using active fault-masking, e.g. by switching out faulty elements or by switching in spare elements
    • G06F11/202—Error detection or correction of the data by redundancy in hardware using active fault-masking, e.g. by switching out faulty elements or by switching in spare elements where processing functionality is redundant
    • G06F11/2048—Error detection or correction of the data by redundancy in hardware using active fault-masking, e.g. by switching out faulty elements or by switching in spare elements where processing functionality is redundant where the redundant components share neither address space nor persistent storage
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F11/00—Error detection; Error correction; Monitoring
    • G06F11/07—Responding to the occurrence of a fault, e.g. fault tolerance
    • G06F11/16—Error detection or correction of the data by redundancy in hardware
    • G06F11/20—Error detection or correction of the data by redundancy in hardware using active fault-masking, e.g. by switching out faulty elements or by switching in spare elements
    • G06F11/2053—Error detection or correction of the data by redundancy in hardware using active fault-masking, e.g. by switching out faulty elements or by switching in spare elements where persistent mass storage functionality or persistent mass storage control functionality is redundant
    • G06F11/2056—Error detection or correction of the data by redundancy in hardware using active fault-masking, e.g. by switching out faulty elements or by switching in spare elements where persistent mass storage functionality or persistent mass storage control functionality is redundant by mirroring
    • G06F11/2082—Data synchronisation
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F11/00—Error detection; Error correction; Monitoring
    • G06F11/07—Responding to the occurrence of a fault, e.g. fault tolerance
    • G06F11/16—Error detection or correction of the data by redundancy in hardware
    • G06F11/20—Error detection or correction of the data by redundancy in hardware using active fault-masking, e.g. by switching out faulty elements or by switching in spare elements
    • G06F11/2097—Error detection or correction of the data by redundancy in hardware using active fault-masking, e.g. by switching out faulty elements or by switching in spare elements maintaining the standby controller/processing unit updated
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
    • G06F16/27—Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L67/00—Network arrangements or protocols for supporting network services or applications
    • H04L67/01—Protocols
    • H04L67/10—Protocols in which an application is distributed across nodes in the network
    • H04L67/1097—Protocols in which an application is distributed across nodes in the network for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS]

Definitions

  • the present disclosure relates to the field of distributed storage technology, and more specifically, to a data processing method of a distributed storage system, a data processing device of a distributed storage system, an electronic device, and a distributed storage system.
  • Distributed storage is a storage solution that distributes data to multiple independent devices.
  • the distributed network storage system adopts an expandable system structure and utilizes multiple storage servers to share the storage load. It not only improves the reliability, availability, and access efficiency of the system, it is also easy to expand.
  • a dedicated control server usually coordinates the storage of data on multiple data servers. Metadata used to describe data attributes is stored in the control server, which can realize functions such as storage location records, historical data records, and resource search. In order to improve the reliability of the system, the number of control servers is usually more than one.
  • the master control server provides control services, and the slave control server serves as a backup.
  • the master control server and the slave control server can maintain metadata consistency through data synchronization. However, if the main control server crashes during the data synchronization process, some metadata update information will be lost, and the operation of the distributed storage system will be affected.
  • One purpose of the embodiments of the present disclosure is to provide a new technical solution for data processing in a distributed storage system.
  • a data processing method of a distributed storage system implemented by a slave control server of the distributed storage system, including:
  • mapping data used to reflect the correspondence between the data copy information and the storage server is updated.
  • the method further includes: detecting whether the main control server is in an unavailable state;
  • the sending the first request for obtaining data copy information to each storage server respectively includes:
  • the method further includes:
  • mapping data Based on the updated mapping data, replacing the main control server to provide control services for the distributed storage system.
  • the sending the first request for obtaining data copy information to each storage server separately further includes:
  • the first request for obtaining data copy information is periodically sent to each storage server.
  • the updating the mapping data for reflecting the correspondence between the data copy information and the storage server according to the data copy information returned by each storage server in response to the first request includes:
  • a data processing method of a distributed storage system implemented by a storage server in the distributed storage system, including:
  • a data processing device of a distributed storage system which is applied to a slave control server of the distributed storage system, and includes:
  • the request sending module is configured to send the first request for obtaining data copy information to each storage server respectively;
  • the data update module is configured to update the mapping data reflecting the correspondence between the data copy information and the storage server according to the data copy information returned by each storage server in response to the first request.
  • the device may further include:
  • Detection module set to detect whether the main control server is in an unavailable state
  • the request sending module is set to:
  • the device further includes a service providing module, and the service providing module is configured to:
  • mapping data Based on the updated mapping data, replacing the main control server to provide control services for the distributed storage system.
  • the request sending module is further configured to:
  • the first request for obtaining data copy information is periodically sent to each storage server.
  • the data update module is configured as:
  • a data processing device of a distributed storage system includes:
  • the request receiving module is configured to receive the first request for obtaining data copy information sent from the control server;
  • the data sending module is configured to send the stored data copy information to the slave control server in response to the first request.
  • an electronic device including a processor and a memory, the memory storing machine executable instructions that can be executed by the processor, and the processor executing the machine executable instructions.
  • the instructions are executed to implement the data processing method of the distributed storage system described in the first aspect or the second aspect of the embodiments of the present disclosure.
  • a distributed storage system including a master control server, a slave control server that implements the method described in the first aspect of the embodiments of the present disclosure, and a secondary control server that implements the second aspect of the embodiments of the present disclosure.
  • the storage server of the method wherein the storage server is respectively connected in communication with the master control server and the slave control server.
  • the data processing method of a distributed storage system in an embodiment of the present disclosure does not require metadata synchronization between the master and slave control servers, and avoids metadata loss due to the failure of the master control server during the synchronization process and the inability to synchronize the master and slave control servers
  • the metadata in the database helps improve the reliability of distributed storage systems.
  • FIG. 1 shows a schematic diagram of a hardware configuration of a distributed storage system that can be used to implement an embodiment of the present disclosure
  • Figure 2 shows a schematic structural diagram of a server that can be used to implement embodiments of the present disclosure
  • Fig. 3 shows a flowchart of a data processing method of a distributed storage system according to an embodiment of the present disclosure
  • Fig. 4 shows a flowchart of a specific example of the implementation of the data processing method of the distributed storage system according to an embodiment of the present disclosure.
  • Figure 1 shows a schematic structural diagram of a distributed storage system that can be used to implement embodiments of the present disclosure.
  • the distributed storage system 100 includes a user agent server 1000, a storage server 2001, a storage server 2002, a storage server 2003, a control server 3001, and a control server 3002.
  • the number of storage servers and control servers can both be multiple (two or more).
  • the numbers of storage servers, control servers, and user proxy servers are all exemplary, and the embodiments of the present disclosure do not limit this.
  • the distributed storage system uses a data block as a basic unit to store target data.
  • the target file is decomposed into multiple data blocks, and multiple copies of each data block are stored in multiple independent storage servers.
  • each data block corresponds to three data block copies, one of the data block copies is the master copy, and the other two data block copies are slave copies.
  • the storage server where the master copy is located gives priority to processing read and write requests for the target file, and the storage server where the copy is located plays a backup role.
  • the storage server 2001, the storage server 2002, and the storage server 2003 are used for storing target data, for example, storing a copy of a data block corresponding to the target data.
  • the control server 3001 and the control server 3002 are used to manage metadata, for example, to manage mapping data reflecting the correspondence between a copy of a data block corresponding to a target file and a storage server.
  • One of the control servers serves as the main control server and provides control services for the distributed storage system.
  • Another control server is used as a slave control server to back up the data in the master control server.
  • the control server 3001 is the master control server
  • the control server 3002 is the slave control server.
  • the roles of the master and slave control servers can be interchanged. When the master control server fails, the slave control server can replace the master control server to provide control services for the distributed storage system.
  • the user proxy server 1000 is configured to receive a data read and write request for target data sent by a user terminal, and forward the data read and write request to the control server.
  • the main control server 3001 After the main control server 3001 receives the data read and write request sent by the user agent server 1000, it can query the storage server where the copy of the data block corresponding to the target data is located according to its stored metadata, and return to the user agent server 1000 to process the data read The identification information of the storage server for the write request.
  • the user agent server 1000 interacts with the corresponding storage server according to the identification information to complete the read and write operations on the target data.
  • the user agent server 1000, the storage server 2001, the storage server 2002, the storage server 2003, the control server 3001, and the control server 3002 may communicate through a wired network or a wireless network.
  • the server 1100 may include a processor 1110, a memory 1120, an interface device 1130, a communication device 1140, a display device 1150, and an input device 1160.
  • the processor 1110 may be, for example, a central processing unit CPU or the like.
  • the memory 1120 includes, for example, ROM (Read-Only Memory), RAM (Random Access Memory, random access memory), non-volatile memory such as a hard disk, and the like.
  • the interface device 1130 includes, for example, a USB interface, a serial interface, and the like.
  • the communication device 1140 can perform wired or wireless communication, for example.
  • the display device 1150 is, for example, a liquid crystal display.
  • the input device 1160 may include, for example, a touch screen, a keyboard, and the like.
  • the memory 1120 of the server 1100 is used to store instructions, which are used to control the processor 1110 to operate to support the realization of the data processing method according to any embodiment of the specification.
  • Technicians can design instructions according to the scheme disclosed in this specification. How the instruction controls the processor to operate is well known in the art, so it will not be described in detail here.
  • server 1100 in the embodiment of this specification may only involve some of the devices, for example, only the processor 1110, the memory 1120, and the Communication device 1140.
  • the distributed storage system 100 shown in FIG. 1 is only explanatory, and is by no means intended to limit this specification, its application or usage.
  • the embodiment of the present disclosure provides a data processing method of a distributed storage system, and the method is implemented by, for example, the slave control server 3002 in FIG. 1. As shown in Figure 3, the method includes the following steps S1100-S1200:
  • Step S1100 Send a first request for obtaining data copy information to each storage server.
  • control server is used to manage metadata, for example, to manage mapping data that reflects the correspondence between the data block copy corresponding to the target file and the storage server.
  • main control server provides control services for the distributed storage system, and the control server plays a backup role.
  • the distributed storage system uses data blocks as basic units to store target data.
  • the target file is decomposed into multiple data blocks, and multiple copies of each data block are stored in multiple independent storage servers.
  • each data block corresponds to three data block copies, one of the data block copies is the master copy, and the other two data block copies are slave copies.
  • the three copies are A, B, and C, and they are stored in the storage servers numbered 1, 2, and 3.
  • a first request for obtaining data copy information is sent from the control server to each storage server to obtain information about the data block copy stored in each storage server.
  • the storage server where the master copy is located can preferentially respond to the first request.
  • the method of the embodiment of the present disclosure may further include the step of detecting whether the main control server is in an unavailable state.
  • the slave control server can send a status inquiry request to the master control server (for example, through heartbeat interaction). If the master control server does not respond, it can be determined that the master control server is currently in an unavailable state.
  • step S1100 may include: when the main control server is in an unavailable state, respectively sending a first request for obtaining data copy information to each storage server.
  • the main control server is in an unavailable state, which includes, for example, the main control server has a crash or the communication between the main control server and the outside is interrupted. It is easy to understand that the main control server is in an unavailable state, which means that the main control server cannot normally provide control services for the distributed storage system.
  • the step of detecting from the control server whether the main control server is in an unavailable state may include: receiving a notification sent by a monitoring device (for example, zookeeper) about whether the main control server is in an unavailable state; and determining according to the notification Whether the main control server is in an unavailable state.
  • the monitoring device obtains the active status of the main control server by receiving the heartbeat packet periodically sent by the main control server.
  • the monitoring device for example, zookeeper
  • the master control server If the heartbeat signal of the master control server is not received within a preset time period, it can be determined The master control server is in an unavailable state, and then a notification is sent to the slave control server.
  • the slave control server can determine that the master control server is in an unavailable state according to the notification from the monitoring device.
  • the monitoring device is used to monitor the active state of the main control server.
  • the monitoring device may be an additional physical device or the slave control server itself, for example, the slave control server monitors the active state of the master control server through its own special process.
  • the monitoring device may send a corresponding notification to the slave control server when detecting that the master control server changes from an available state to an unavailable state.
  • the slave control server determines that the master control server is in an unavailable state according to the notification.
  • the step of detecting whether the master control server is in an unavailable state from the slave control server includes: the slave control server can make a heartbeat signal connection with the master control server, if not within a preset time period (for example, 1 minute) After receiving the heartbeat signal of the main control server, it can be determined that the main control server is in an unavailable state.
  • a preset time period for example, 1 minute
  • Step S1200 according to the data copy information returned by each storage server in response to the first request, update the mapping data used to reflect the correspondence between the data copy information and the storage server.
  • mapping data reflecting the correspondence between the data copy and the storage server is stored in the control server.
  • each storage server sends its own stored data copy information to the slave control server in response to the first request sent from the control server.
  • Receive from the control server the data copy information sent by each storage server to its own storage.
  • a correspondence relationship (mapping relationship) between the data copy information and the storage server where it is located is established, and the mapping data (or correspondence relationship) table is updated.
  • the mapping data table or the corresponding relationship table may include more than one corresponding relationship (or mapping data) between the data copy information and the storage server where it is located.
  • step S1200 the latest correspondence between the data copy and the storage server can be obtained from the control server.
  • the data processing method of the distributed storage system in this embodiment does not require metadata synchronization between the master and slave control servers, avoids the loss of data update information due to the failure of the master control server during the synchronization process, and is beneficial to improve the performance of the distributed storage system. reliability.
  • the data processing method of the distributed storage system further includes: when it is determined that the master control server is in an unavailable state, the slave control server may replace the master control server to provide control for the distributed storage system based on the updated mapping data. service.
  • the main control server may be in an unavailable state due to power failure, downtime, or unexpected restart.
  • the slave control server detects that the master control server is unavailable, after updating its stored mapping data, it replaces the master control server in time to provide control services for the distributed storage system. Ensure the normal operation of the distributed storage system and provide guarantee for the data storage security of the distributed storage system.
  • the process of the main control server providing control services for the distributed storage system is, for example: the main control server receives a data read request for the target data sent by the user agent server, and queries the target data corresponding to the mapping data stored by itself.
  • the storage server where the copy of the data block is located and returns the identification information of the storage server that processes the data read and write request to the user agent server, so that the user agent server can interact with the corresponding storage server according to the identification information to complete the reading of the target data Write operation.
  • the roles of the master and slave control servers can be interchanged.
  • the slave control server can replace the master control server to provide control services for the distributed storage system.
  • the slave control server can automatically replace the master control server to provide control services for the distributed storage system (that is, automatic switching), or in response to manual operations instead of the master control server to provide control services for the distributed storage system (that is, manual switching). ).
  • automatic switching is a preferred embodiment, so as to ensure the security and timeliness of data storage and avoid data loss.
  • the slave control server provides control services for the distributed storage system according to the updated mapping data, which can ensure the normal operation of the distributed storage system.
  • the foregoing step S1100 may be implemented as periodically sending a first request for obtaining data copy information (hereinafter referred to as a periodic request) to each storage server according to a set time.
  • a periodic request a first request for obtaining data copy information
  • the slave control server periodically sends a first request for obtaining data copy information to each storage server at a set time.
  • the slave control server when the master control server is unavailable, the slave control server periodically sends the first request to each storage server at a set time.
  • the set time can be any time from 1 second to 1 minute, for example, 5 seconds, 10 seconds, 15 seconds, 20 seconds, 25 seconds, etc., which are not limited in the embodiments of the present disclosure, and can be based on Pre-set the experience value of the technician
  • mapping data By regularly obtaining data copy information and regularly updating the mapping data, the amount of data that needs to be processed for each update can be reduced, which is especially beneficial for the slave control server to quickly update the mapping data when the master control server is in an unavailable state, and improve the master-slave switching speed.
  • the foregoing step S1100 may be implemented as sending a first request (hereinafter referred to as an initial request) to obtain data copy information to each storage server during the initial startup phase.
  • a first request hereinafter referred to as an initial request
  • the control server sends a first request for obtaining data copy information to each storage server during the initial startup phase (in the following description, it may be referred to as an initial request). It is easy to understand that the initial request is different from the request sent by the slave control server when the master control server is in an unavailable state, and the request sent periodically by the slave control server.
  • step S1200 is executed, that is, based on the data copy information returned by each storage server in response to the initial request, mapping data reflecting the correspondence between the data copy and the storage server is established.
  • each storage server sends its own stored data copy information to the slave control server in response to the initial request. Receive the data copy information sent by each storage server from the control server, and establish mapping data reflecting the correspondence between the data copy and the storage server.
  • the new mapping data reflecting the corresponding relationship between the data copy and the storage server from the control server during the initial startup stage can establish a basis for subsequent updating of the mapping data.
  • the process of updating the mapping data stored in the control server includes: modifying the current mapping data stored in the control server to obtain updated mapping data.
  • the slave control service performs data updates based on the current mapping data stored by itself, which can reduce the amount of data that needs to be processed for each update, which is beneficial to increase the update speed.
  • mapping data reflecting the correspondence between the data copy and the storage server is stored in the memory of the slave control server. In this way, it is beneficial to increase the speed of reading and writing the mapped data, thereby increasing the data update speed.
  • the slave control server for the slave control server to actively send the first request to the storage server, to send the first request to the storage server when the master control server is in an unavailable state, and to send the first request to the storage server during the initial startup phase of the slave control server (that is, when it starts running) Sending the first request to the storage server for the implementation methods and corresponding technical effects in the three states.
  • the full cycle may include the initial startup phase of the slave control server, the normal working phase of the slave control server, and the detection that the master control server is in an unavailable state. And, under different states in the whole cycle, the role of the slave control server in processing data may be different.
  • Stage (1) In the initial startup stage of the slave control server, a third request is sent to each storage server to obtain data copy information, and the initial mapping data is created based on the data copy information obtained in this stage.
  • Stage (2) In the normal working stage of the slave control server, a fourth request can be sent periodically to obtain data copy information, which is used to periodically update the mapping data.
  • Stage (3) When the main control server is in an unavailable state, a fourth request is sent from the control server to obtain the current data copy information and update the mapping data, and then the main control server can be replaced based on the latest mapping data to provide services.
  • the first request, the second request, the third request, and the fourth request in the embodiment of the present disclosure are to distinguish read requests issued from the control server at different stages (or in different states), but the first request,
  • the operation methods and implementation principles of the second request, the third request, and the fourth request are all consistent.
  • mapping data can be continuously updated through interaction with the storage server during its entire life cycle.
  • the foregoing embodiment is adopted in stages. Compared with the embodiment in which the mapping data is updated when, for example, it is in an unavailable state, the data to be updated is relatively small, so the data update speed is faster.
  • FIG. 4 shows a specific example of the implementation of the data processing method of the distributed storage system in the embodiment of the present disclosure.
  • the slave control server monitors the heartbeat of the master control server through its own monitoring process, thereby detecting the active state of the master control server, that is, step S101 is executed.
  • the slave control server periodically sends a second request for obtaining data copy information to the storage server, and the storage server sends the data copy information stored by itself to the slave control server in response to the second request, that is, step S102 is executed.
  • the slave control server According to the data copy information sent by the storage server, the slave control server periodically updates the mapping data stored in its memory that reflects the correspondence between the data copy and the storage server, that is, step S103 is executed.
  • step S104 the control server sends the first request for obtaining the copy information to each storage server respectively, that is, step S105 is executed.
  • step S106 each storage server sends the information of its own stored data copy to the slave control server, that is, step S106 is executed.
  • step S107 the slave control server revises the data obtained in the last periodic update to obtain the updated mapping data, that is, step S107 is executed.
  • the slave control server automatically replaces the master control server to provide control services for the distributed storage system, that is, step S108 is executed.
  • the embodiments of the present disclosure also provide another data processing method of a distributed storage system, which is implemented by, for example, any storage server shown in FIG. 1.
  • the method includes: receiving a request for obtaining data copy information sent from a control server; and in response to the request for obtaining data copy information, sending its own corresponding data copy information to the slave control server.
  • the embodiment of the present disclosure provides a data processing device of a distributed storage system, which is applied to a slave control server of a distributed storage system, and includes a request sending module and a data update module.
  • the request sending module is configured to send the first request for obtaining data copy information to each storage server respectively.
  • the data update module is configured to update the mapping data reflecting the correspondence between the data copy information and the storage server according to the data copy information returned by each storage server in response to the first request.
  • the data processing device of the distributed storage system of the embodiment of the present disclosure may further include: a detection module: configured to detect whether the main control server is in an unavailable state.
  • the request sending module is configured to send the first request for obtaining data copy information to each storage server separately when the main control server is in an unavailable state.
  • the device further includes a service providing module, and the service providing module is configured to replace the main control server to provide control services for the distributed storage system based on the updated mapping data.
  • the request sending module may also be configured to periodically send a first request for obtaining data copy information to each storage server according to a set time.
  • the data update module may be configured to modify the existing mapping data according to the data copy information returned by each storage server in response to the first request to obtain updated mapping data.
  • the embodiment of the present disclosure also provides another data processing device of a distributed storage system, which is applied to a storage server in the distributed storage system, and includes a request receiving module and a data sending module.
  • the request receiving module is configured to receive the first request for obtaining data copy information sent from the control server.
  • the data sending module is configured to send the stored data copy information to the slave control server in response to the first request.
  • An embodiment of the present disclosure provides an electronic device including a processor and a memory, the memory stores machine executable instructions that can be executed by the processor, and the processor executes the machine executable instructions to implement the distributed storage system described in the method embodiments of the present disclosure Data processing method.
  • the embodiments of the present disclosure provide a distributed storage system, including a master control server, a slave control server that implements the first method described in the embodiments of the present disclosure, and a storage server that implements the second method described in the method embodiments of the present disclosure, Among them, the storage server communicates with the master control server and the slave control server respectively.
  • the embodiments of the present disclosure provide a machine-readable storage medium, and the machine-readable storage medium stores machine-executable instructions.
  • the machine-executable instructions When the machine-executable instructions are called and executed by a processor, the machine-executable instructions prompt the processor to implement the method of the present disclosure.
  • the example describes the service coordination method of the distributed storage system.
  • the embodiments of the present disclosure may be systems, methods, and/or computer program products.
  • the computer program product may include a computer-readable storage medium loaded with computer-readable program instructions for enabling a processor to implement various aspects of the embodiments of the present disclosure.
  • the computer-readable storage medium may be a tangible device that can hold and store instructions used by the instruction execution device.
  • the computer-readable storage medium may be, for example, but not limited to, an electrical storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the foregoing.
  • Non-exhaustive list of computer-readable storage media include: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM) Or flash memory), static random access memory (SRAM), portable compact disk read-only memory (CD-ROM), digital versatile disk (DVD), memory stick, floppy disk, mechanical encoding device, such as a printer with instructions stored thereon
  • RAM random access memory
  • ROM read-only memory
  • EPROM erasable programmable read-only memory
  • flash memory flash memory
  • SRAM static random access memory
  • CD-ROM compact disk read-only memory
  • DVD digital versatile disk
  • memory stick floppy disk
  • mechanical encoding device such as a printer with instructions stored thereon
  • the computer-readable storage medium used here is not interpreted as the instantaneous signal itself, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through waveguides or other transmission media (for example, light pulses through fiber optic cables), or through wires Transmission of electrical signals.
  • the computer-readable program instructions described herein can be downloaded from a computer-readable storage medium to various computing/processing devices, or downloaded to an external computer or external storage device via a network, such as the Internet, a local area network, a wide area network, and/or a wireless network.
  • the network may include copper transmission cables, optical fiber transmission, wireless transmission, routers, firewalls, switches, gateway computers, and/or edge servers.
  • the network adapter card or network interface in each computing/processing device receives computer-readable program instructions from the network, and forwards the computer-readable program instructions for storage in the computer-readable storage medium in each computing/processing device .
  • the computer program instructions used to perform the operations of the embodiments of the present disclosure may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-related instructions, microcode, firmware instructions, state setting data, or one or more programming instructions.
  • Programming languages include object-oriented programming languages-such as Smalltalk, C++, etc., and conventional procedural programming languages-such as "C" language or similar programming languages.
  • Computer-readable program instructions can be executed entirely on the user's computer, partly on the user's computer, executed as a stand-alone software package, partly on the user's computer and partly executed on a remote computer, or entirely on the remote computer or server carried out.
  • the remote computer can be connected to the user's computer through any kind of network-including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computer (for example, using an Internet service provider to connect to the user's computer) connection).
  • LAN local area network
  • WAN wide area network
  • an electronic circuit such as a programmable logic circuit, a field programmable gate array (FPGA), or a programmable logic array (PLA), can be customized by using the status information of the computer-readable program instructions.
  • the computer-readable program instructions are executed to implement various aspects of the embodiments of the present disclosure.
  • These computer-readable program instructions can be provided to the processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, thereby producing a machine that makes these instructions when executed by the processor of the computer or other programmable data processing device , A device that implements the functions/actions specified in one or more blocks in the flowcharts and/or block diagrams is produced. It is also possible to store these computer-readable program instructions in a computer-readable storage medium. These instructions make computers, programmable data processing apparatuses, and/or other devices work in a specific manner. Thus, the computer-readable medium storing the instructions includes An article of manufacture, which includes instructions for implementing various aspects of the functions/actions specified in one or more blocks in the flowcharts and/or block diagrams.
  • each block in the flowchart or block diagram can represent a module, program segment, or part of an instruction, and a module, program segment, or part of an instruction contains one or more executables for realizing the specified logic function. instruction.
  • the functions marked in the block may also occur in a different order than the order marked in the drawings. For example, two consecutive blocks can actually be executed substantially in parallel, or they can sometimes be executed in the reverse order, depending on the functions involved.
  • each block in the block diagram and/or flowchart, and the combination of the blocks in the block diagram and/or flowchart can be implemented by a dedicated hardware-based system that performs the specified functions or actions Or it can be realized by a combination of dedicated hardware and computer instructions. It is well known to those skilled in the art that implementation through hardware, implementation through software, and implementation through a combination of software and hardware are all equivalent.

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Engineering & Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Quality & Reliability (AREA)
  • Databases & Information Systems (AREA)
  • Signal Processing (AREA)
  • Computing Systems (AREA)
  • Computer Networks & Wireless Communication (AREA)
  • Data Mining & Analysis (AREA)
  • Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
  • Multi Processors (AREA)
  • Hardware Redundancy (AREA)

Abstract

本发明提供一种分布式存储系统的数据处理方法、装置及电子设备。该方法包括：分别向每一存储服务器发送获取数据副本信息的第一请求；根据每一所述存储服务器响应于所述第一请求返回的所述数据副本信息，更新用于反映所述数据副本信息和存储服务器对应关系的映射数据。

Description

分布式存储系统的数据处理方法、装置及电子设备
本公开要求于2019年10月31日提交中国国家知识产权局、申请号为201911051737.5、发明名称为“分布式存储系统的数据处理方法、装置及电子设备”的中国专利申请的优先权，其全部内容通过引用结合在本公开中。
技术领域
本公开涉及分布式存储技术领域，更具体地，涉及一种分布式存储系统的数据处理方法、一种分布式存储系统的数据处理装置、一种电子设备及一种分布式存储系统。
背景技术
分布式存储是将数据分散存储在多台独立的设备上的存储方案。分布式网络存储系统采用可扩展的系统结构，利用多台存储服务器分担存储负荷，它不但提高了系统的可靠性、可用性和存取效率，还易于扩展。
在分布式存储系统中，通常由专门的控制服务器来协调数据在多台数据服务器上的存储。控制服务器中存储有用于描述数据属性的元数据，可以实现存储位置记录、历史数据记录、资源查找等功能。为了提高系统的可靠性，控制服务器的数目通常为多个，由其中的主控制服务器提供控制服务，从控制服务器作为备份。
主控制服务器和从控制服务器可以通过数据同步的方式保持元数据的一致性。但是，如果在数据同步过程中出现主控制服务器死机等情况，会导致部分元数据更新信息丢失，影响分布式存储系统的运行。
发明内容
本公开实施例的一个目的是提供一种分布式存储系统数据处理的新技术方案。
根据本公开实施例的第一方面，提供了一种分布式存储系统的数据处理方法，由所述分布式存储系统的从控制服务器实施，包括：
分别向每一存储服务器发送获取数据副本信息的第一请求；
根据每一所述存储服务器响应于所述第一请求返回的所述数据副本信息，更新用于反映所述数据副本信息和存储服务器对应关系的映射数据。
在一实施方式中，所述方法还包括：检测所述主控制服务器是否处于不可用状态；
所述分别向每一存储服务器发送获取数据副本信息的第一请求，包括：
在所述主控制服务器处于不可用状态的情况下，分别向每一存储服务器发送获取数据副本信息的请求。
在一实施方式中，所述方法还包括：
基于更新后的所述映射数据，取代所述主控制服务器为所述分布式存储系统提供控制服务。
在一实施方式中，所述分别向每一存储服务器发送获取数据副本信息的第一请求，还包括：
按照设定的时间，定期向每一存储服务器发送获取数据副本信息的第一请求。
在一实施方式中，所述根据每一所述存储服务器响应于所述第一请求返回的所述数据副本信息，更新用于反映所述数据副本信息和存储服务器对应关系的映射数据，包括：
根据每一所述存储服务器响应于所述第一请求返回的所述数据副本信息，对现有的所述映射数据进行修改，获得更新后的所述映射数据。
根据本公开实施例的第二方面，提供了一种分布式存储系统的数据处理方法，由所述分布式存储系统中的存储服务器实施，包括：
接收从控制服务器发送的获取数据副本信息的请求；
响应于所述获取数据副本信息的请求，向所述从控制服务器发送自身对应的数据副本信息。
根据本公开实施例的第三方面，提供了一种分布式存储系统的数据处理装置，应用于所述分布式存储系统的从控制服务器，包括：
请求发送模块，设置为分别向每一存储服务器发送获取数据副本信息的第一请求；
数据更新模块，设置为根据每一所述存储服务器响应于所述第一请求返回的所述数据副本信息，更新用于反映所述数据副本信息和存储服务器对应关系的映射数据。
在一实施方式中，所述装置还可以包括：
检测模块：设置为检测主控制服务器是否处于不可用状态；
所述请求发送模块设置为：
在所述主控制服务器处于不可用状态的情况下，分别向每一存储服务器发送获取数据副本信息的第一请求。
在一实施方式中，所述装置还包括服务提供模块，所述服务提供模块设置为：
基于更新后的所述映射数据，取代所述主控制服务器为所述分布式存储系统提供控制服务。
在一实施方式中，所述请求发送模块还设置为：
按照设定的时间，定期向每一存储服务器发送获取数据副本信息的第一请求。
在一实施方式中，所述数据更新模块设置为：
根据每一所述存储服务器响应于所述第一请求返回的所述数据副本信息，对现有的所述映射数据进行修改，获得更新后的所述映射数据。
根据本公开实施例的第四方面，提供了一种分布式存储系统的数据处理装置，设置于所述分布式存储系统中的存储服务器，包括：
请求接收模块，用于接收从控制服务器发送的获取数据副本信息的第一请求；
数据发送模块，用于响应于所述第一请求，向所述从控制服务器发送自身存储的数据副本信息。
根据本公开实施例的第五方面，提供了一种电子设备，包括处理器和存储器，所述存储器存储有能够被所述处理器执行的机器可执行指令，所述处理器执行所述机器可执行指令以实现本公开实施例第一方面或者第二方面所述的分布式存储系统的数据处理方法。
根据本公开实施例的第六方面，提供了一种分布式存储系统，包括主 控制服务器、实施本公开实施例第一方面所述方法的从控制服务器和实施本公开实施例第二方面所述方法的存储服务器，其中，所述存储服务器分别与所述主控制服务器和所述从控制服务器通信连接。
本公开一个实施例中的分布式存储系统的数据处理方法，无需主从控制服务器之间进行元数据同步，避免了因同步过程中主控制服务器故障导致元数据丢失而无法同步主、从控制服务器中的元数据，有利于提高分布式存储系统的可靠性。
通过以下参照附图对本公开的示例性实施例的详细描述，本公开的其它特征及其优点将会变得清楚。
附图说明
被结合在说明书中并构成说明书的一部分的附图示出了本公开的实施例，并且连同其说明一起用于解释本公开实施例的原理。
图1示出了可用于实现本公开的实施例的分布式存储系统的硬件配置示意图；
图2示出了可用于实现本公开实施例的服务器的结构示意图；
图3示出了根据本公开实施例的分布式存储系统的数据处理方法的流程图；
图4示出了根据本公开实施例的分布存储系统的数据处理方法实施的具体例子的流程图。
具体实施方式
现在将参照附图来详细描述本公开的各种示例性实施例。应注意到：除非另外具体说明，否则在这些实施例中阐述的部件和步骤的相对布置、数字表达式和数值不限制本公开实施例的范围。
以下对至少一个示例性实施例的描述实际上仅仅是说明性的，决不作为对本公开实施例及其应用或使用的任何限制。
对于相关领域普通技术人员已知的技术、方法和设备可能不作详细讨论，但在适当情况下，技术、方法和设备应当被视为说明书的一部分。
在这里示出和讨论的所有例子中，任何具体值应被解释为仅仅是示例性的，而不是作为限制。因此，示例性实施例的其它例子可以具有不同的值。
应注意到：相似的标号和字母在下面的附图中表示类似项，因此，一旦某一项在一个附图中被定义，则在随后的附图中不需要对其进行进一步讨论。
<硬件配置>
图1示出了可用于实现本公开实施例的分布式存储系统的结构示意图。
如图1所示，分布式存储系统100包括用户代理服务器1000、存储服务器2001、存储服务器2002、存储服务器2003、以及控制服务器3001、控制服务器3002。其中，存储服务器和控制服务器的数目均可以为多个(两个以上)。在本公开实施例中，存储服务器、控制服务器、用户代理服务器的数量均为示例性的，本公开实施例对此不做限制。
本实施例中，分布式存储系统以数据块(Block)为基本单元存储目标数据。例如，将目标文件分解为多个数据块，并将每一数据块的多个副本分别存储在相互独立的多个存储服务器中。在一个例子中，每个数据块对应于三个数据块副本，其中一个数据块副本为主副本，其他两个数据块副本为从副本。主副本所在的存储服务器优先处理针对目标文件的读写请求，从副本所在的存储服务器起到备份作用。
存储服务器2001、存储服务器2002和存储服务器2003用于存储目标数据，例如存储目标数据对应的数据块副本。
控制服务器3001和控制服务器3002用于管理元数据，例如管理反映目标文件对应的数据块副本和存储服务器对应关系的映射数据。其中一个控制服务器作为主控制服务器，为分布式存储系统提供控制服务。另外一个控制服务器作为从控制服务器，用于备份主控制服务器中的数据。本实施例中，控制服务器3001为主控制服务器，控制服务器3002为从控制服务器。主从控制服务器的角色可以相互转换，在主控制服务器出现故障时，从控制服务器可以取代主控制服务器为分布式存储系统提供控制服务。
用户代理服务器1000用于接收用户端发送的针对目标数据的数据读写请求，并将该数据读写请求转发至控制服务器。
主控制服务器3001接收到用户代理服务器1000发送的数据读写请求后，可以根据自身存储的元数据，查询目标数据对应的数据块副本所在的存储服务器，并向用户代理服务器1000返回处理该数据读写请求的存储服务器的标识信息。用户代理服务器1000根据该标识信息与对应的存储服务器交互，完成对目标数据的读写操作。
用户代理服务器1000、存储服务器2001、存储服务器2002、存储服务器2003、控制服务器3001、控制服务器3002之间可以通过有线网络或者无线网络进行通信。
用户代理服务器1000、存储服务器2001、存储服务器2002、存储服务器2003、控制服务器3001、控制服务器3002都具有如图2所示的服务器1100的硬件配置。如图2所示，服务器1100可以包括处理器1110、存储器1120、接口装置1130、通信装置1140、显示装置1150和输入装置1160。处理器1110例如可以是中央处理器CPU等。存储器1120例如包括ROM(Read-Only Memory，只读存储器)、RAM(Random Access Memory，随机存取存储器)、诸如硬盘的非易失性存储器等。接口装置1130例如包括USB接口、串行接口等。通信装置1140例如能够进行有线或无线通信。显示装置1150例如是液晶显示屏。输入装置1160例如可以包括触摸屏、键盘等。
应用于本说明书的实施例中，服务器1100的存储器1120用于存储指令，该指令用于控制处理器1110进行操作以支持实现根据本说明书任意实施例的数据处理方法。技术人员可以根据本说明书所公开方案设计指令。指令如何控制处理器进行操作，这是本领域公知，故在此不再详细描述。
本领域技术人员应当理解，尽管在图2中示出了服务器1100的多个装置，但是，本说明书实施例的服务器1100可以仅涉及其中的部分装置，例如，只涉及处理器1110、存储器1120和通信装置1140。
图1所示的分布式存储系统100仅是解释性的，并且决不是为了要限制本说明书、其应用或用途。
<方法实施例>
本公开实施例提供了一种分布式存储系统的数据处理方法，该方法例如由图1中的从控制服务器3002实施。如图3所示，该方法包括以下步骤S1100-S1200：
步骤S1100，分别向每一存储服务器发送获取数据副本信息的第一请求。
本实施例中，控制服务器用于管理元数据，例如管理反映目标文件对应的数据块副本和存储服务器对应关系的映射数据。其中，主控制服务器为分布式存储系统提供控制服务，从控制服务器起到备份作用。
本实施例中，分布式存储系统以数据块为基本单元存储目标数据。例如，将目标文件分解为多个数据块，并将每一数据块的多个副本分别存储在相互独立的多个存储服务器中。在一个例子中，每个数据块对应于三个数据块副本，其中一个数据块副本为主副本，其他两个数据块副本为从副本。例如，三个副本分别为A、B、C，分别存储到编号为1、2、3的存储服务器中。
本实施例中，从控制服务器向每一存储服务器发送获取数据副本信息的第一请求，以获取每一存储服务器中所存储的数据块副本的信息。在本步骤一实施方式中，主副本所在的存储服务器可以优先响应第一请求。
在一个实施例中，本公开实施例的方法还可以包括检测主控制服务器是否处于不可用状态的步骤。示例性地，从控制服务器可以向主控制服务器发送状态询问请求(例如，通过心跳交互的方式)，如主控制服务器无响应，则可以确定主控制服务器当前为不可用状态。在此基础上，步骤S1100可以包括：在主控制服务器处于不可用状态的情况下，分别向每一存储服务器发送获取数据副本信息的第一请求。
本实施例中，主控制服务器处于不可用状态，例如包括主控制服务器出现死机故障、主控制服务器与外界通信中断等情况。容易理解，主控制服务器处于不可用状态，意味着主控制服务器不能正常为分布式存储系统提供控制服务。
在一个实施例中，从控制服务器检测主控制服务器是否处于不可用状 态的步骤可以包括：接收监听设备(例如，zookeeper)发送的关于主控制服务器是否处于不可用状态的通知；根据该通知，确定主控制服务器是否处于不可用状态。在一个例子中，监听设备通过接收主控制服务器定期发送的心跳包来获取主控制服务器的活动状态。示例性地，监听设备(例如，zookeeper)可以定期与控制服务器(主控制服务器、从控制服务器)进行心跳信号连接，如果在预设时间段内没有收到主控制服务器的心跳信号，则可以确定主控制服务器处于不可用状态，进而向从控制服务器发送通知。从控制服务器可以根据监听设备的通知确定主控制服务器处于不可用状态。
本实施例中，监听设备用于对主控制服务器的活动状态进行监听。监听设备可以是额外的实体设备，也可以是从控制服务器自身，例如，从控制服务器中通过自身的专门进程来监听主控制服务器的活动状态。
本实施例中，监听设备可以在检测到主控制服务器由可用状态变为不可用状态的情况下，向从控制服务器发送相应的通知。从控制服务器根据该通知确定主控制服务器处于不可用状态。
在另一个实施例中，从控制服务器检测主控制服务器是否处于不可用状态的步骤包括：从控制服务器可以与主控制服务器进行心跳信号连接，如果在预设时间段内(例如，1分钟)未收到主控制服务器的心跳信号，则可以确定主控制服务器是处于不可用状态的。
步骤S1200，根据每一存储服务器响应于第一请求返回的数据副本信息，更新用于反映数据副本信息和存储服务器对应关系的映射数据。
本实施例中，从控制服务器中存储有反映数据副本与存储服务器对应关系的映射数据。
本实施例中，每一存储服务器响应于从控制服务器发送的第一请求，向从控制服务器发送自身存储的数据副本信息。从控制服务器接收每一存储服务器发送自身存储的数据副本信息。并根据接收到的数据副本信息，建立数据副本信息与其所在的存储服务器之间的对应关系(映射关系)，对映射数据(或者对应关系)表进行更新。其中，映射数据表或者对应关系表中可以包括一条以上的数据副本信息与其所在的存储服务器之间的对 应关系(或映射数据)。
通过执行步骤S1200，从控制服务器可获得数据副本与存储服务器之间最新的对应关系。
本实施例中的分布式存储系统的数据处理方法，无需主从控制服务器之间进行元数据同步，避免了因同步过程中主控制服务器故障导致数据更新信息丢失，有利于提高分布式存储系统的可靠性。
在一个实施例中，分布式存储系统的数据处理方法还包括：在确定主控制服务器处于不可用状态时，从控制服务器可以基于更新后的映射数据，取代主控制服务器为分布式存储系统提供控制服务。主控制服务器可能因为掉电、宕机或者意外重启等而处于不可用状态。从控制服务器检测到主控制服务器不可用时，在更新其存储的映射数据后，及时取代主控制服务器为分布式存储系统提供控制服务。保证分布式存储系统的正常运行，为分布式存储系统的数据存储安全提供保证。
本实施例中，主控制服务器为分布式存储系统提供控制服务的过程例如是：主控制服务器接收到用户代理服务器发送的针对目标数据的数据读取请求，根据自身存储的映射数据查询目标数据对应的数据块副本所在的存储服务器，并向用户代理服务器返回处理该数据读写请求的存储服务器的标识信息，以供用户代理服务器根据该标识信息与对应的存储服务器交互，完成对目标数据的读写操作。
本实施例中，主从控制服务器的角色可以相互转换，在主控制服务器处于不可用状态的情况下，从控制服务器可以取代主控制服务器为分布式存储系统提供控制服务。
本实施例中，从控制服务器可自动取代主控制服务器为分布式存储系统提供控制服务(即自动切换)，也可以响应于人工操作取代主控制服务器为分布式存储系统提供控制服务(即手动切换)。在本公开实施例中，以自动切换为优选实施例，从而保证数据存储的安全性和及时性，避免数据丢失。
本实施例中，从控制服务器根据更新后的映射数据为分布式存储系统提供控制服务，能够保证分布式存储系统的正常运行。
根据本公开实施例，前述步骤S1100可以实现为根据设定的时间，定期分别向每一存储服务器发送获取数据副本信息的第一请求(以下称为定期请求)。
在一实施例中，从控制服务器以设定的时间定期向每一存储服务器发送获取数据副本信息的第一请求。在另一些实施例中，可以为在主控制服务器不可用的状态下，从控制服务器按照设定的时间定期向每个存储服务器发送第一请求。示例性地，设定的时间1秒至1分钟中的任意时长，例如可以为5秒、10秒、15秒、20秒、25秒等，在本公开实施例中不对此进行限制，可以根据技术人员的经验值进行预设置
通过定期获取数据副本信息以及定期更新映射数据，可以减少每次更新需要处理的数据量，尤其有利于从控制服务器在主控制服务器处于不可用状态时快速更新映射数据，提高主从切换速度。
在一个实施例中，前述步骤S1100可以实现为在初始启动阶段，分别向每一存储服务器发送获取数据副本信息的第一请求(以下称为初始请求)。
本实施例中，从控制服务器在初始启动阶段向每一存储服务器发送获取数据副本信息的第一请求(在之后的叙述中，可以称之为初始请求)。容易理解，初始请求区别于从控制服务器在主控制服务器处于不可用状态的情况下发送的请求，以及从控制服务器定期发送的请求。
进而执行步骤S1200，即根据每一存储服务器响应于初始请求返回的数据副本信息，建立反映数据副本和存储服务器对应关系的映射数据。
本实施例中，每一存储服务器响应于初始请求，向从控制服务器发送自身存储的数据副本信息。从控制服务器接收每一存储服务器发送自身存储的数据副本信息，建立反映数据副本和存储服务器对应关系的映射数据。
从控制服务器在初始启动阶段新建反映数据副本和存储服务器对应关系的映射数据，可以为后续更新映射数据建立基础。
在一个实施例中，从控制服务器更新自身存储的映射数据的过程包括：对自身存储的当前的映射数据进行修改，获得更新后的映射数据。
本实施例中，从控制服务基于自身存储的当前的映射数据进行数据更 新，可以降低每次更新需要处理的数据量，有利于提高更新速度。
在一个实施例中，反映数据副本和存储服务器对应关系的映射数据存储于从控制服务器的内存中。如此，有利于提高对映射数据的读写速度，进而提高数据更新速度。
前述方法实施例中，针对从控制服务器主动向存储服务器发送第一请求、在主控制服务器处于不可用状态下，向存储服务器发送第一请求、在从控制服务器初始启动阶段(即启动运行时)向存储服务器发送第一请求三种状态下的实施方法和对应的技术效果。在分布式存储系统的实际运行过程中，全周期可以包括从控制服务器初始启动阶段、从控制服务器正常工作阶段以及检测到主控制服务器处于不可用状态等。以及，全周期中不同的状态下，从控制服务器对数据的处理的作用可能有一定差异。
下面将基于分布式存储系统的实际运行过程可能出现的系统状态，对本公开提供的另一实施例进行描述：
阶段(1)：在从控制服务器的初始启动阶段，向每个存储服务器发送第三请求获取数据副本信息，并基于此阶段得到的数据副本信息来创建初始映射数据。
阶段(2)：在从控制服务器的正常工作阶段，可以定期发送第四请求获取数据副本信息，用来定期更新映射数据。
阶段(3)：在主控制服务器处于不可用状态的情况下，从控制服务器发送第四请求，获取当前的数据副本信息并更新映射数据，进而可以基于最新的映射数据替代主控制服务器提供服务。
其中，本公开实施例中的第一请求、第二请求、第三请求和第四请求，是为了区别从控制服务器在不同阶段(或者不同状态下)发出的读取请求，但第一请求、第二请求、第三请求和第四请求操作方法和实现原理都是一致的。
总体而言，从控制服务器从启动时起，其全生命周期中均可以通过与存储服务器的交互，不断更新其存储的映射数据。且采用前述实施例分阶段进行，与在例如处于不可用状态时再去更新映射数据的实施例相比较，其须更新的数据相对较少，因此数据更新速度更快。
图4示出了本公开实施例中分布式存储系统的数据处理方法实施的具体例子。如图4所示，从控制服务器通过自身的监听进程监听主控制服务器的心跳，从而检测主控服务器的活动状态，即执行步骤S101。此外，从控制服务器还定期向存储服务器发送获取数据副本信息的第二请求，存储服务器响应于该第二请求向从控制服务器发送自身存储的数据副本的信息，即执行步骤S102。从控制服务器根据存储服务器发送的数据副本信息，定期更新自身内存中存储的反映数据副本和存储服务器对应关系的映射数据，即执行步骤S103。假设主控制服务器在运行过程中出现了死机故障，这时从控制服务器可能无法在预设时间内监听到主控制服务器的，可以确定主控制服务器处于不可用状态，即执行步骤S104。之后，从控制服务器分别向每个存储服务器发送获取副本信息的第一请求，即执行步骤S105。每个存储服务器响应于该第一请求向从控制服务器发送自身存储的数据副本的信息，即执行步骤S106。从控制服务器根据存储服务器发送的数据副本信息，对最近一次定期更新所获得数据进行修正，获得更新后的映射数据，即执行步骤S107。最后，从控制服务器自动取代主控制服务器为分布式存储系统提供控制服务，即执行步骤S108。
本公开实施例还提供另一种分布式存储系统的数据处理方法，该方法例如由图1所示的任意的存储服务器实施。该方法包括：接收从控制服务器发送的获取数据副本信息的请求；响应于获取数据副本信息的请求，向从控制服务器发送自身对应的数据副本信息。
该方法的具体实施方式可参见前文对分布式存储系统的数据处理方法的描述，此处不再赘述。
<装置实施例>
本公开实施例提供一种分布式存储系统的数据处理装置，应用于分布式存储系统的从控制服务器，包括请求发送模块和数据更新模块。
请求发送模块，设置为分别向每一存储服务器发送获取数据副本信息的第一请求。
数据更新模块，设置为根据每一存储服务器响应于第一请求返回的数据副本信息，更新用于反映数据副本信息和存储服务器对应关系的映射数 据。
在一实施方式中，本公开实施例的分布式存储系统的数据处理装置还可以包括：检测模块：设置为检测主控制服务器是否处于不可用状态。在一实施方式中，请求发送模块设置为在主控制服务器处于不可用状态的情况下，分别向每一存储服务器发送获取数据副本信息的第一请求。
在一实施方式中，装置还包括服务提供模块，服务提供模块设置为：基于更新后的映射数据，取代主控制服务器为分布式存储系统提供控制服务。
在一实施方式中，请求发送模块还可以设置为：按照设定的时间，定期向每一存储服务器发送获取数据副本信息的第一请求。
在一实施方式中，数据更新模块可以设置为根据每一存储服务器响应于第一请求返回的数据副本信息，对现有的映射数据进行修改，获得更新后的映射数据。
本公开实施例还提供另外一种分布式存储系统的数据处理装置，应用于分布式存储系统中的存储服务器，包括请求接收模块和数据发送模块。
请求接收模块，设置为接收从控制服务器发送的获取数据副本信息的第一请求。
数据发送模块，设置为响应于第一请求，向从控制服务器发送自身存储的数据副本信息。
<电子设备实施例>
本公开实施例提供一种电子设备，包括处理器和存储器，存储器存储有能够被处理器执行的机器可执行指令，处理器执行机器可执行指令以实现本公开方法实施例描述的分布式存储系统的数据处理方法。
<系统实施例>
本公开实施例提供一种分布式存储系统，包括主控制服务器、实施本公开方式实施例描述的第一种方法的从控制服务器和实施本公开方法实施例描述的第二种方法的存储服务器，其中，存储服务器分别与主控制服务器和从控制服务器通信连接。
<计算机可读存储介质实施例>
本公开实施例提供一种机器可读存储介质，机器可读存储介质存储有机器可执行指令，机器可执行指令在被处理器调用和执行时，机器可执行指令促使处理器实现本公开方法实施例描述的分布式存储系统的服务协调方法。
本公开实施例可以是系统、方法和/或计算机程序产品。计算机程序产品可以包括计算机可读存储介质，其上载有用于使处理器实现本公开实施例的各个方面的计算机可读程序指令。
计算机可读存储介质可以是可以保持和存储由指令执行设备使用的指令的有形设备。计算机可读存储介质例如可以是――但不限于――电存储设备、磁存储设备、光存储设备、电磁存储设备、半导体存储设备或者上述的任意合适的组合。计算机可读存储介质的更具体的例子(非穷举的列表)包括：便携式计算机盘、硬盘、随机存取存储器(RAM)、只读存储器(ROM)、可擦式可编程只读存储器(EPROM或闪存)、静态随机存取存储器(SRAM)、便携式压缩盘只读存储器(CD-ROM)、数字多功能盘(DVD)、记忆棒、软盘、机械编码设备、例如其上存储有指令的打孔卡或凹槽内凸起结构、以及上述的任意合适的组合。这里所使用的计算机可读存储介质不被解释为瞬时信号本身，诸如无线电波或者其他自由传播的电磁波、通过波导或其他传输媒介传播的电磁波(例如，通过光纤电缆的光脉冲)、或者通过电线传输的电信号。
这里所描述的计算机可读程序指令可以从计算机可读存储介质下载到各个计算/处理设备，或者通过网络、例如因特网、局域网、广域网和/或无线网下载到外部计算机或外部存储设备。网络可以包括铜传输电缆、光纤传输、无线传输、路由器、防火墙、交换机、网关计算机和/或边缘服务器。每个计算/处理设备中的网络适配卡或者网络接口从网络接收计算机可读程序指令，并转发该计算机可读程序指令，以供存储在各个计算/处理设备中的计算机可读存储介质中。
用于执行本公开实施例操作的计算机程序指令可以是汇编指令、指令集架构(ISA)指令、机器指令、机器相关指令、微代码、固件指令、状态设置数据、或者以一种或多种编程语言的任意组合编写的源代码或目标代 码，编程语言包括面向对象的编程语言—诸如Smalltalk、C++等，以及常规的过程式编程语言—诸如“C”语言或类似的编程语言。计算机可读程序指令可以完全地在用户计算机上执行、部分地在用户计算机上执行、作为一个独立的软件包执行、部分在用户计算机上部分在远程计算机上执行、或者完全在远程计算机或服务器上执行。在涉及远程计算机的情形中，远程计算机可以通过任意种类的网络—包括局域网(LAN)或广域网(WAN)—连接到用户计算机，或者，可以连接到外部计算机(例如利用因特网服务提供商来通过因特网连接)。在一些实施例中，通过利用计算机可读程序指令的状态信息来个性化定制电子电路，例如可编程逻辑电路、现场可编程门阵列(FPGA)或可编程逻辑阵列(PLA)，该电子电路可以执行计算机可读程序指令，从而实现本公开实施例的各个方面。
这里参照根据本公开实施例的方法、装置(系统)和计算机程序产品的流程图和/或框图描述了本公开实施例的各个方面。应当理解，流程图和/或框图的每个方框以及流程图和/或框图中各方框的组合，都可以由计算机可读程序指令实现。
这些计算机可读程序指令可以提供给通用计算机、专用计算机或其它可编程数据处理装置的处理器，从而生产出一种机器，使得这些指令在通过计算机或其它可编程数据处理装置的处理器执行时，产生了实现流程图和/或框图中的一个或多个方框中规定的功能/动作的装置。也可以把这些计算机可读程序指令存储在计算机可读存储介质中，这些指令使得计算机、可编程数据处理装置和/或其他设备以特定方式工作，从而，存储有指令的计算机可读介质则包括一个制造品，其包括实现流程图和/或框图中的一个或多个方框中规定的功能/动作的各个方面的指令。
也可以把计算机可读程序指令加载到计算机、其它可编程数据处理装置、或其它设备上，使得在计算机、其它可编程数据处理装置或其它设备上执行一系列操作步骤，以产生计算机实现的过程，从而使得在计算机、其它可编程数据处理装置、或其它设备上执行的指令实现流程图和/或框图中的一个或多个方框中规定的功能/动作。
附图中的流程图和框图显示了根据本公开的多个实施例的系统、方法 和计算机程序产品的可能实现的体系架构、功能和操作。在这点上，流程图或框图中的每个方框可以代表一个模块、程序段或指令的一部分，模块、程序段或指令的一部分包含一个或多个用于实现规定的逻辑功能的可执行指令。在有些作为替换的实现中，方框中所标注的功能也可以以不同于附图中所标注的顺序发生。例如，两个连续的方框实际上可以基本并行地执行，它们有时也可以按相反的顺序执行，这依所涉及的功能而定。也要注意的是，框图和/或流程图中的每个方框、以及框图和/或流程图中的方框的组合，可以用执行规定的功能或动作的专用的基于硬件的系统来实现，或者可以用专用硬件与计算机指令的组合来实现。对于本领域技术人员来说公知的是，通过硬件方式实现、通过软件方式实现以及通过软件和硬件结合的方式实现都是等价的。
以上已经描述了本公开的各实施例，上述说明是示例性的，并非穷尽性的，并且也不限于所披露的各实施例。在不偏离所说明的各实施例的范围和精神的情况下，对于本技术领域的普通技术人员来说许多修改和变更都是显而易见的。本文中所用术语的选择，旨在最好地解释各实施例的原理、实际应用或对市场中的技术改进，或者使本技术领域的其它普通技术人员能理解本文披露的各实施例。本公开实施例的范围由所附权利要求来限定。
工业实用性
基于本公开实施例提供的分布式存储系统的数据处理方法、装置及电子设备，无需主从控制服务器之间进行元数据同步，避免了因同步过程中主控制服务器故障导致元数据丢失而无法同步主、从控制服务器中的元数据，有利于提高分布式存储系统的可靠性。

Claims (10)

  1. 一种分布式存储系统的数据处理方法，由所述分布式存储系统的从控制服务器实施，包括：
    分别向每一存储服务器发送获取数据副本信息的第一请求；
    根据每一所述存储服务器响应于所述第一请求返回的所述数据副本信息，更新用于反映所述数据副本信息和存储服务器对应关系的映射数据。
  2. 根据权利要求1所述的方法，其中，所述方法还包括：
    检测所述主控制服务器是否处于不可用状态；
    所述分别向每一存储服务器发送获取数据副本信息的第一请求，包括：
    在所述主控制服务器处于不可用状态的情况下，分别向每一存储服务器发送获取数据副本信息的第一请求。
  3. 根据权利要求2所述的方法，其中，所述方法还包括：
    基于更新后的所述映射数据，取代所述主控制服务器为所述分布式存储系统提供控制服务。
  4. 根据权利要求1-3任一项所述的方法，其中，所述分别向每一存储服务器发送获取数据副本信息的第一请求，还包括：
    按照设定的时间，定期向每一存储服务器发送获取数据副本信息的第一请求。
  5. 根据权利要求1-4任一项所述的方法，其中，所述根据每一所述存储服务器响应于所述第一请求返回的所述数据副本信息，更新用于反映所述数据副本信息和存储服务器对应关系的映射数据，包括：
    根据每一所述存储服务器响应于所述第一请求返回的所述数据副本信息，对当前的所述映射数据进行修改，以得到更新后的所述映射数据。
  6. 一种分布式存储系统的数据处理方法，由所述分布式存储系统中的存储服务器实施，包括：
    接收从控制服务器发送的获取数据副本信息的第一请求；
    响应于所述第一请求，向所述从控制服务器发送自身存储的数据副本信息。
  7. 一种分布式存储系统的数据处理装置，设置于所述分布式存储系统 的从控制服务器，包括：
    请求发送模块，设置为分别向每一存储服务器发送获取数据副本信息的第一请求；
    数据更新模块，设置为根据每一所述存储服务器响应于所述第一请求返回的所述数据副本信息，更新用于反映所述数据副本信息和存储服务器对应关系的映射数据。
  8. 一种分布式存储系统的数据处理装置，设置于所述分布式存储系统中的存储服务器，包括：
    请求接收模块，设置为接收从控制服务器发送的获取数据副本信息的第一请求；
    数据发送模块，设置为响应于所述第一请求，向所述从控制服务器发送自身存储的数据副本信息。
  9. 一种电子设备，包括处理器和存储器，所述存储器存储有能够被所述处理器执行的机器可执行指令，所述处理器执行所述机器可执行指令以实现权利要求1至5中任一项所述的分布式存储系统的数据处理方法，或者实现权利要求6中所述的分布式存储系统的数据处理方法。
  10. 一种分布式存储系统，包括主控制服务器、实施权利要求1-5中任一项所述方法的从控制服务器和实施权利要求6所述方法的存储服务器，其中，所述存储服务器分别与所述主控制服务器、所述从控制服务器通信连接。
PCT/CN2020/119346 2019-10-31 2020-09-30 分布式存储系统的数据处理方法、装置及电子设备 WO2021082868A1 (zh)

Priority Applications (1)

Application Number Priority Date Filing Date Title
US17/772,939 US11966305B2 (en) 2019-10-31 2020-09-30 Data processing method for distributed storage system, apparatus, and electronic device

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
CN201911051737.5A CN112751693B (zh) 2019-10-31 2019-10-31 分布式存储系统的数据处理方法、装置及电子设备
CN201911051737.5 2019-10-31

Publications (1)

Publication Number Publication Date
WO2021082868A1 true WO2021082868A1 (zh) 2021-05-06

Family

ID=75641530

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2020/119346 WO2021082868A1 (zh) 2019-10-31 2020-09-30 分布式存储系统的数据处理方法、装置及电子设备

Country Status (3)

Country Link
US (1) US11966305B2 (zh)
CN (1) CN112751693B (zh)
WO (1) WO2021082868A1 (zh)

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN115373896A (zh) * 2022-06-23 2022-11-22 北京志凌海纳科技有限公司 一种基于分布式块存储的副本数据恢复方法和系统

Families Citing this family (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US8935630B2 (en) 2005-05-04 2015-01-13 Hillcrest Laboratories, Inc. Methods and systems for scrolling and pointing in user interfaces

Citations (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN101697494A (zh) * 2009-10-30 2010-04-21 福建星网锐捷网络有限公司 备份方法、设备和系统
US20130173553A1 (en) * 2011-12-29 2013-07-04 Anand Apte Distributed Scalable Deduplicated Data Backup System
CN103577500A (zh) * 2012-08-10 2014-02-12 腾讯科技（深圳）有限公司 分布式文件系统进行数据处理的方法及该分布式文件系统
US20150046394A1 (en) * 2013-08-07 2015-02-12 Fujitsu Limited Storage system, storage control device, and storage medium storing control program
CN108173971A (zh) * 2018-02-05 2018-06-15 江苏物联网研究发展中心 一种基于主备切换的MooseFS高可用方法及系统
CN109284069A (zh) * 2018-08-23 2019-01-29 郑州云海信息技术有限公司 一种存放备份数据的分布式存储系统及方法

Family Cites Families (17)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US6732289B1 (en) * 2000-08-31 2004-05-04 Sun Microsystems, Inc. Fault tolerant data storage system
US7437608B2 (en) * 2004-11-15 2008-10-14 International Business Machines Corporation Reassigning storage volumes from a failed processing system to a surviving processing system
US8812900B2 (en) * 2011-06-30 2014-08-19 International Business Machines Corporation Managing storage providers in a clustered appliance environment
US8793467B2 (en) * 2011-09-30 2014-07-29 Pure Storage, Inc. Variable length encoding in a storage system
US11086898B2 (en) * 2013-03-13 2021-08-10 Amazon Technologies, Inc. Token-based admission control for replicated writes
CN103384211B (zh) * 2013-06-28 2017-02-08 百度在线网络技术（北京）有限公司 一种具有容错性的数据操作方法及分布式的数据存储系统
US9336134B2 (en) * 2013-11-12 2016-05-10 Skyera, Llc Apparatus and method for accessing a non-volatile memory blade using multiple controllers in a non-volatile memory based storage device
US9274901B2 (en) * 2013-11-20 2016-03-01 Avago Technologies General Ip (Singapore) Pte. Ltd. I/O request mirroring in a clustered storage system
US9933946B2 (en) * 2014-09-15 2018-04-03 Hewlett Packard Enterprise Development Lp Fibre channel storage array methods for port management
US10552058B1 (en) * 2015-07-17 2020-02-04 Radian Memory Systems, Inc. Techniques for delegating data processing to a cooperative memory controller
US10069688B2 (en) * 2016-03-07 2018-09-04 International Business Machines Corporation Dynamically assigning, by functional domain, separate pairs of servers to primary and backup service processor modes within a grouping of servers
US10372561B1 (en) * 2017-06-12 2019-08-06 Amazon Technologies, Inc. Block storage relocation on failure
US10642526B2 (en) * 2017-08-28 2020-05-05 Vmware, Inc. Seamless fault tolerance via block remapping and efficient reconciliation
CN109672544B (zh) * 2017-10-13 2020-12-11 杭州海康威视系统技术有限公司 一种数据处理方法、装置及分布式存储系统
US10621059B2 (en) * 2017-12-19 2020-04-14 International Business Machines Corporation Site recovery solution in a multi-tier storage environment
CN109684412A (zh) * 2018-12-25 2019-04-26 成都虚谷伟业科技有限公司 一种分布式数据库系统
KR20210121503A (ko) * 2020-03-30 2021-10-08 에스케이하이닉스 주식회사 전자 장치와, 이를 위한 데이터 저장 장치 및 동작 방법

Patent Citations (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN101697494A (zh) * 2009-10-30 2010-04-21 福建星网锐捷网络有限公司 备份方法、设备和系统
US20130173553A1 (en) * 2011-12-29 2013-07-04 Anand Apte Distributed Scalable Deduplicated Data Backup System
CN103577500A (zh) * 2012-08-10 2014-02-12 腾讯科技（深圳）有限公司 分布式文件系统进行数据处理的方法及该分布式文件系统
US20150046394A1 (en) * 2013-08-07 2015-02-12 Fujitsu Limited Storage system, storage control device, and storage medium storing control program
CN108173971A (zh) * 2018-02-05 2018-06-15 江苏物联网研究发展中心 一种基于主备切换的MooseFS高可用方法及系统
CN109284069A (zh) * 2018-08-23 2019-01-29 郑州云海信息技术有限公司 一种存放备份数据的分布式存储系统及方法

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN115373896A (zh) * 2022-06-23 2022-11-22 北京志凌海纳科技有限公司 一种基于分布式块存储的副本数据恢复方法和系统

Also Published As

Publication number Publication date
CN112751693B (zh) 2022-09-06
US11966305B2 (en) 2024-04-23
CN112751693A (zh) 2021-05-04
US20230033754A1 (en) 2023-02-02

Similar Documents

Publication Publication Date Title
US10049119B2 (en) Synchronization of client machines with a content management system repository
JP6865219B2 (ja) 連続クエリ処理におけるイベントバッチ処理、出力シーケンス化、およびログベースの状態記憶
CN109976667B (zh) 一种镜像管理方法、装置及系统
US11526470B2 (en) Data validation for data record migrations
US9037677B2 (en) Update protocol for client-side routing information
US10956374B2 (en) Data recovery method, apparatus, and system
US10097659B1 (en) High performance geographically distributed data storage, retrieval and update
US10019452B2 (en) Topology aware distributed storage system
US20180373767A1 (en) Database table conversion
US20200019543A1 (en) Method, apparatus and device for updating data, and medium
US10628380B2 (en) Enabling data replication processes between heterogeneous storage systems
US11544232B2 (en) Efficient transaction log and database processing
WO2012045245A1 (zh) 一种保持数据一致性的方法及系统
WO2021082868A1 (zh) 分布式存储系统的数据处理方法、装置及电子设备
JP2020506456A (ja) スタビングによるコンテンツ記憶の最適化
CN113452617B (zh) 动态网关路由管理方法、装置及存储介质
EP3817338A2 (en) Method and apparatus for acquiring rpc member information, electronic device and storage medium
KR20210042865A (ko) 데이터 처리 방법 및 장치
CN111031126B (zh) 集群缓存共享方法、系统、设备及存储介质
US20230244579A1 (en) Event-driven system failover and failback
WO2021078294A1 (zh) 分布式存储系统的服务协调方法、装置及电子设备
US11531596B2 (en) Hyperscaler independent versioning of cloud storage objects
CN113190532A (zh) 数据处理方法、装置、设备及计算机可读存储介质
US11281683B1 (en) Distributed computation system for servicing queries using revisions maps
US11379146B1 (en) Method and system for performing data protection management using an add-on application programming interface

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 20880823

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 20880823

Country of ref document: EP

Kind code of ref document: A1