WO2021003935A1 - Procédé et appareil de stockage de groupes de données et dispositif informatique - Google Patents

Procédé et appareil de stockage de groupes de données et dispositif informatique Download PDF

Info

Publication number
WO2021003935A1
WO2021003935A1 PCT/CN2019/118232 CN2019118232W WO2021003935A1 WO 2021003935 A1 WO2021003935 A1 WO 2021003935A1 CN 2019118232 W CN2019118232 W CN 2019118232W WO 2021003935 A1 WO2021003935 A1 WO 2021003935A1
Authority
WO
WIPO (PCT)
Prior art keywords
physical
physical cluster
cluster
stored
file
Prior art date
Application number
PCT/CN2019/118232
Other languages
English (en)
Chinese (zh)
Inventor
兰东平
Original Assignee
平安科技（深圳）有限公司
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 平安科技（深圳）有限公司 filed Critical 平安科技（深圳）有限公司
Publication of WO2021003935A1 publication Critical patent/WO2021003935A1/fr

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
    • G06F3/0601—Interfaces specially adapted for storage systems
    • G06F3/0602—Interfaces specially adapted for storage systems specifically adapted to achieve a particular effect
    • G06F3/0626—Reducing size or complexity of storage systems
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
    • G06F3/0601—Interfaces specially adapted for storage systems
    • G06F3/0628—Interfaces specially adapted for storage systems making use of a particular technique
    • G06F3/0638—Organizing or formatting or addressing of data
    • G06F3/0643—Management of files
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
    • G06F3/0601—Interfaces specially adapted for storage systems
    • G06F3/0668—Interfaces specially adapted for storage systems adopting a particular infrastructure
    • G06F3/067—Distributed or networked storage systems, e.g. storage area networks [SAN], network attached storage [NAS]

Definitions

  • cluster storage aggregates the storage space of multiple storage devices into one that can provide unified access to application servers.
  • the storage pool of the interface and management interface the application can transparently access and utilize the disks on all storage devices through the access interface, and can give full play to the performance and disk utilization of the storage device.
  • index libraries need to be established to record the correspondence between data and clusters. Query the clusters stored in historical files in the index database.
  • the aforementioned storage method cannot guarantee the regular storage of data in the cluster, and cannot quickly locate the location of the data storage, resulting in poor storage performance.
  • this application discloses a method, device and computer equipment for data cluster storage.
  • the main purpose is to solve the problem that when data cluster storage is performed, the regular storage of data in the cluster cannot be guaranteed, and the location of data storage cannot be quickly located. , Resulting in poor storage performance.
  • a data cluster storage method including:
  • the uniformly mapping the physical clusters to the physical nodes of the consistent hash ring specifically includes: obtaining the storage space of the physical cluster; and the first physical cluster with the storage space greater than or equal to a preset threshold according to the preset Suppose the proportion is divided into multiple sub-physical clusters with equal space; according to the naming rule, the second physical cluster whose storage space is less than the preset threshold and each of the sub-physical clusters are configured with an identity code; according to the identity code Determine the hash value of the second physical cluster and each of the sub-physical clusters; use the hash value to calculate the physical node positions of the second physical cluster and the sub-physical cluster on the consistent hash ring;
  • a data cluster storage device which includes:
  • the acquisition module is used to acquire all physical clusters used to store data
  • a mapping module which is used to map the physical cluster to a physical node of a consistent hash ring
  • the mapping unit is specifically configured to: obtain the storage space of the physical cluster; divide the first physical cluster with the storage space greater than or equal to a preset threshold into a plurality of sub-physical clusters with equal space according to a preset ratio;
  • the naming rule is that the second physical cluster whose storage space is less than the preset threshold and each of the sub-physical clusters are configured with an identification code; the second physical cluster and each of the sub-physical clusters are determined according to the identification code Use the hash value to calculate the physical node positions of the second physical cluster and the sub-physical cluster on the consistent hash ring;
  • the determining module is used to determine the optimal storage target physical cluster according to the hash value of the file to be stored;
  • the storage module is used to store the file to be stored in the target physical cluster.
  • a non-volatile readable storage medium having computer readable instructions stored thereon, and the computer readable instructions are executed by a processor to implement the above-mentioned data cluster storage method.
  • a computer device including a non-volatile readable storage medium, a processor, and a computer-readable storage medium that is stored on the non-volatile readable storage medium and can run on the processor. Instructions, when the processor executes the computer-readable instructions, the method for data cluster storage is implemented.
  • this application can evenly map physical clusters to consistent data.
  • the logical node position of the file to be stored in the consistent hash ring is determined according to the hash value of the file to be stored, and the optimal storage target physical cluster is filtered based on the logical node position, and then the file to be stored is selected Stored in the target physical cluster.
  • This application can quickly locate the cluster where the data file should be stored through calculation. Because the hash value of the data file is fixed, it can ensure the regular storage of the data in the cluster.
  • each physical cluster is evenly mapped to the physical nodes of the consistent hash ring, so that each physical cluster can store data, avoiding the centralized storage of data in a physical cluster, resulting in increased storage pressure and data avalanche The problem.
  • integrating the consistent hash ring into the data cluster storage of the present application can effectively reduce the complexity of data storage, thereby reducing costs, and can achieve efficient positioning of the physical cluster to meet the needs of massive storage expansion.
  • FIG. 1 shows a schematic flowchart of a data cluster storage method provided by an embodiment of the present application
  • FIG. 2 shows a schematic flowchart of another data cluster storage method provided by an embodiment of the present application
  • FIG. 3 shows an example schematic diagram of a data cluster storage method provided by an embodiment of the present application
  • FIG. 4 shows an example schematic diagram of another data cluster storage method provided by an embodiment of the present application
  • FIG. 5 shows an example schematic diagram of yet another data cluster storage method provided by an embodiment of the present application
  • FIG. 6 shows a schematic structural diagram of a data cluster storage device provided by an embodiment of the present application.
  • FIG. 7 shows a schematic structural diagram of another data cluster storage device provided by an embodiment of the present application.
  • the embodiment of the present application provides a method for data cluster storage. As shown in FIG. 1, the method includes:
  • the purpose of obtaining all physical clusters is to configure all physical clusters equally in a consistent hash ring, so as to realize uniform configuration distribution of physical clusters.
  • a unified namespace can be used to name each physical cluster, thereby mapping each physical cluster to a consistent hash ring.
  • the identification code or host name of the physical cluster can be selected as the key to calculate the hash value , So that each machine can determine its position on the hash ring, so as to achieve targeted storage of data files based on a consistent hash algorithm.
  • the consistent hash ring can be imagined as a ring composed of 2 ⁇ 32 points, the point directly above the ring represents 0, the first point to the right of the 0 point represents 1, and so on, 2. 3, 4, 5, 6... until 2 ⁇ 32-1, which means that the first point to the left of 0 points represents 2 ⁇ 32-1.
  • the consistent hash ring has two layers of nodes: the first layer is a logical node, the number is 2 ⁇ 32; the second layer is a physical node, which is the actual storage cluster.
  • the target physical cluster is the physical cluster that is most suitable for the storage of the file to be stored determined according to the consistent hash algorithm.
  • the method of using the consistent hash algorithm to determine the target physical cluster is: starting from the logical node location where the file to be stored is located, The first physical cluster with normal storage status encountered in the clockwise direction is determined as the target physical cluster.
  • the files to be stored can be stored in the target physical cluster, and queries and data acquisitions of the data to be stored can be received.
  • the physical cluster can be evenly mapped to the physical nodes of the consistent hash ring, and the logical node of the file to be stored in the consistent hash ring is determined according to the hash value of the file to be stored Location, the optimal storage target physical cluster is filtered out based on the logical node location, and then the files to be stored are stored in the target physical cluster.
  • This application can quickly locate the cluster where the data file should be stored through calculation. Because the hash value of the data file is fixed, it can ensure the regular storage of the data in the cluster.
  • each physical cluster is evenly mapped to the physical nodes of the consistent hash ring, so that each physical cluster can store data, avoiding the centralized storage of data in a physical cluster, resulting in increased storage pressure and data avalanche The problem.
  • integrating the consistent hash ring into the data cluster storage of the present application can effectively reduce the complexity of data storage, thereby reducing costs, and can achieve efficient positioning of the physical cluster to meet the needs of massive storage expansion.
  • the method includes:
  • all physical clusters used to store data can be obtained from the data storage system. For example, if the data storage system contains four physical clusters A, B, C, and D, the basic information of the four clusters A, B, C, and D needs to be extracted.
  • the preset threshold value is the minimum storage space for judging to divide the physical cluster into multiple sub-physical clusters.
  • the preset ratio is the number of divisions that divide the physical cluster into sub-physical clusters, and the value of the preset ratio can be preset according to actual needs.
  • set the preset ratio of the unit capacity of the physical cluster and the sub-physical cluster is 30TB. If the storage space of physical cluster A is 200TB, the storage space of physical cluster B is 100TB, and the storage space of physical cluster C is 100TB. The storage space is 20TB. Because the storage space of physical cluster A and physical cluster B is greater than the preset threshold, physical cluster A and physical cluster B are defined as the first physical cluster to be divided, and physical cluster A is divided into 10 20TB pieces according to the preset ratio Sub-physical cluster, which divides physical cluster B into 10 sub-physical clusters of 10TB. Since the storage space of the physical cluster C is less than the preset threshold, it can be determined that the storage space is small and does not need to be divided into multiple sub-physical clusters, and it is defined as the second physical cluster.
  • the naming rule configure an identification code for the second physical cluster and each sub-physical cluster whose storage space is less than the preset threshold.
  • a unified namespace needs to be used to configure identities that comply with the naming rules for each sub-physical cluster and the second physical cluster to facilitate the physical Unified management of storage space.
  • the naming rule can be uniformly set as cluster[cluster number]-[physical node number]. Before naming the physical cluster, it is necessary to obtain the cluster number of the physical cluster, and determine that the physical cluster corresponding to the cluster number is the first physical cluster It is the second physical cluster. If it is determined to be the first physical cluster, it is necessary to further obtain the sequence number of the sub-physical cluster in the first physical cluster, that is, the physical node number in the corresponding naming rule. When it is determined that the physical cluster corresponding to the cluster number is the second physical cluster, the physical node number can be directly set to 1.
  • the two sub-physical nodes can be named cluster1-1 and cluster1-2 in sequence; if it is determined that the storage cluster 2 has four sub-physical nodes, the four sub-physical nodes can be named in sequence As, cluster2-1, cluster2-2, cluster2-3, cluster2-4; if it is determined that the storage cluster 3 is the second physical cluster, it can be named cluster3-1.
  • the method for determining the mapping of the second physical cluster and sub-physical clusters to the consistent hash ring may be: using the MD5 message digest algorithm to generate a 128-bit (16 byte) Hash value is used to ensure complete and consistent information transmission.
  • the specific implementation method is: MD5 processes the input identification code in 512-bit groups, and each group is divided into 16 32-bit sub-groups. After a series of processing, the output of the algorithm is composed of four 32-bit groups. , Cascading these four 32-bit packets will generate a 128-bit hash value.
  • the result calculated by the above formula must be an integer between 0 and 2 ⁇ 32-1, so the calculated integer represents the physical cluster. Since this integer must be between 0 and 2 ⁇ 32-1, then, The location of the physical node must be determined on the consistent hash ring, that is, to map each physical cluster to the consistent hash ring.
  • the calculation method in step 204 is the same. After the identification code of the file to be stored is obtained, the identification code is converted into a hash value.
  • the result calculated by the above formula must be an integer between 0 and 2 ⁇ 32-1, so the calculated integer represents the file to be stored. Since this integer must be between 0 and 2 ⁇ 32-1, then , The location of the logical node corresponding to the file to be stored must be determined on the consistent hash ring.
  • a consistent hash algorithm can be used to determine the target physical cluster corresponding to the file to be stored.
  • the principle of the consistent hash algorithm is: After mapping the physical cluster and the file to be stored on the hash ring, starting from the location of the file to be stored, the first physical cluster encountered in the clockwise direction is the current The physical cluster where the object will be cached. Since the hashed value of the file to be stored and the physical cluster is fixed, the file to be stored must be cached on the fixed physical cluster when the physical cluster remains unchanged. Then, When you want to access the file to be stored next time, just use the same algorithm again to calculate the location where the file to be stored is cached, and go directly to the corresponding physical cluster to find it.
  • the physical cluster is searched clockwise using key1 as the starting point.
  • the physical node key2 on the ring is the first second physical cluster or the first sub-physical cluster retrieved, and the physical cluster corresponding to the key2 point can be determined as the target physical cluster of the file to be stored at the key1 point.
  • the physical cluster has fault tolerance and scalability.
  • the failed physical cluster needs to be removed.
  • physical clusters can be added to the consistent hash ring according to the actual situation.
  • the physical node where the physical cluster is located can be cleared; after it is determined that a new physical cluster needs to be added, an identity identifier needs to be configured for the physical cluster according to the naming rules Code, and determine the physical node position of the physical cluster on the consistent hash ring based on the identification code.
  • step 211 of the embodiment specifically includes: acquiring a newly added second physical cluster or each sub-physical cluster;
  • the naming rule is to configure the identity code for the newly added second physical cluster or each sub-physical cluster; based on the identity code, determine the position of the newly added physical node of the newly added second physical cluster or each sub-physical cluster on the consistent hash ring ; Extract the data to be migrated between the location of the newly added physical node and the location of the previous cluster physical node in the ring space, where the location of the previous cluster physical node is the first and second one that is checked counterclockwise from the location of the newly added physical node as the starting point The physical node location corresponding to the physical cluster or the first sub-physical cluster; the data to be migrated is migrated and stored in the sub-physical cluster or the second physical cluster corresponding to the location of the newly added physical node.
  • step 211 of the embodiment specifically includes: determining the second physical cluster or sub-physical cluster to be deleted; and setting the second physical cluster to be deleted Or all storage files in the sub-physical cluster are migrated and stored in the first second physical cluster or the first sub-physical cluster in the clockwise direction; after the storage files are migrated and stored, the second physical cluster to be deleted Or delete the sub-physical cluster.
  • the affected data is only the data stored in the physical cluster to be deleted, and other data will not be affected.
  • the affected data is only between this physical cluster and the previous physical cluster in its ring space (that is, the first physical cluster encountered when walking in a counterclockwise direction) Data, other data will not be affected.
  • the data to be stored in the physical cluster may specifically include: receiving a query request for the file to be stored; The hash value of the file determines the target physical cluster; the data file is retrieved in the target physical cluster.
  • the file to be stored when the physical cluster remains unchanged, the file to be stored must be cached on a fixed physical cluster. Then, when you want to access the file to be stored next time, you only need to use the same algorithm for calculation again. You can calculate where the file to be stored is cached, and go directly to the corresponding physical cluster to find it.
  • a physical cluster with a larger storage space can be divided into multiple sub-physical clusters with equal space to achieve uniform distribution of physical clusters, so that different data can be evenly stored in corresponding locations, thereby balancing each
  • the storage pressure of the physical cluster avoids the avalanche of stored data caused by centralized data storage.
  • the second physical cluster and each of the sub-physical clusters whose storage space is less than the preset threshold is configured with an identification code, and the hash value is calculated according to the identification code, and then the second physical cluster and each of the sub-physical clusters
  • the physical clusters are evenly mapped into the consistent hash ring, and then the consistent hash algorithm is used to determine the target storage cluster corresponding to the optimal storage of the file to be stored, and the file to be stored is stored in the target storage cluster.
  • it can also receive the addition and deletion instructions of the physical cluster to meet the expansion needs of mass storage.
  • the queryability of data storage can be guaranteed, that is, when querying the file to be stored, the original storage location of the data is not considered, and the current storage physical cluster location can be accurately located according to the hash value of the file to be stored. And then realize the efficient positioning of data storage.
  • an embodiment of the present application provides a data cluster storage device.
  • the device includes: an acquisition module 31, a mapping module 32, and a determination module 33.
  • Storage module 34 is provided.
  • the obtaining module 31 can be used to obtain all physical clusters used to store data
  • the mapping module 32 can be used to map the physical cluster to the physical nodes of the consistent hash ring;
  • the determining module 33 can be used to determine the optimal storage target physical cluster according to the hash value of the file to be stored;
  • the storage module 34 can be used to store the files to be stored in the target physical cluster.
  • the mapping module 32 can be specifically used to obtain the storage space of the physical cluster; the storage space is greater than or equal to the preset threshold of the first physical
  • the cluster is divided into multiple sub-physical clusters with equal space according to the preset ratio; according to the naming rules, the second physical cluster with storage space less than the preset threshold and each sub-physical cluster are configured with identification codes; the second physical cluster is determined according to the identification codes And the hash value of each sub-physical cluster; the hash value is used to calculate the physical node positions of the second physical cluster and the sub-physical cluster on the consistent hash ring.
  • the determining module 33 can be specifically used to calculate the hash value of the file to be stored according to the identification code of the file to be stored; the hash value is used to determine the location of the file to be stored The location of the logical node on the consistent hash ring; the first second physical cluster or the first sub-physical cluster that is checked out clockwise from the logical node location on the consistent hash ring is determined as the target physical cluster.
  • the device further includes: a receiving module 35, an update module 36, and an adjustment module 37.
  • the receiving module 35 can be used to receive addition and deletion instructions to the physical cluster
  • the update module 36 can be used to update the second physical cluster and/or sub-physical cluster on the physical node according to the addition and deletion instructions;
  • the adjustment module 37 can be used to adjust the storage location of the files to be stored that meet the preset conditions.
  • the adjustment module 37 can be specifically used to obtain the newly added second physical cluster or each sub-physical cluster; according to the naming rule, the newly added physical cluster Configure the identity code for the second physical cluster or each sub-physical cluster; determine the position of the newly added physical node of the second physical cluster or each sub-physical cluster on the consistent hash ring based on the identity code; extract the new physical node
  • the data to be migrated between the location and the location of the physical node of the previous cluster in the ring space, where the location of the physical node of the previous cluster is the first second physical cluster or the first sub-physical that was checked counterclockwise from the location of the new physical node as the starting point The location of the physical node corresponding to the cluster; the data to be migrated is migrated and stored in the sub-physical cluster or the second physical cluster corresponding to the location of the newly added physical node.
  • the adjustment module 37 can be specifically used to determine the second physical cluster or sub-physical cluster to be deleted; and the second physical cluster or sub-physical cluster to be deleted All storage files in the sub-physical cluster are migrated and stored to the first second physical cluster or the first sub-physical cluster in the clockwise direction; after the storage files are migrated and stored, the second physical cluster or The child physical cluster is deleted.
  • the device in order to provide a query on the storage location of the file to be stored, as shown in FIG. 7, the device further includes: a query module 38.
  • the receiving module 35 can also be used to receive query requests for files to be stored
  • the determining module 33 may also be used to determine the target physical cluster according to the hash value of the file to be stored;
  • the retrieval module 38 can be used to retrieve data files in the target physical cluster.
  • embodiments of the present application also provide a storage medium on which computer-readable instructions are stored.
  • the foregoing 1 and Figure 2 show the method of data cluster storage.
  • the technical solution of this application can be embodied in the form of a software product.
  • the software product can be stored in a non-volatile storage medium (which can be a CD-ROM, U disk, mobile hard disk, etc.), including several
  • the instructions are used to make a computer device (which may be a personal computer, a server, or a network device, etc.) execute the methods in each implementation scenario of the present application.
  • an embodiment of the present application also provides a computer device, which may be a personal computer, Servers, network devices, etc.
  • the physical device includes a storage medium and a processor; the storage medium is used to store a computer program; the processor is used to execute the computer program to implement the data cluster storage method shown in FIG. 1 and FIG. 2 .
  • the computer device may also include a user interface, a network interface, a camera, a radio frequency (RF) circuit, a sensor, an audio circuit, a WI-FI module, and so on.
  • the user interface may include a display screen (Display), an input unit such as a keyboard (Keyboard), etc., and the optional user interface may also include a USB interface, a card reader interface, and the like.
  • the network interface can optionally include a standard wired interface, a wireless interface (such as a Bluetooth interface, a WI-FI interface), etc.
  • the computer device structure provided in this embodiment does not constitute a limitation on the physical device, and may include more or fewer components, or combine certain components, or arrange different components.
  • the non-volatile readable storage medium may also include an operating system and a network communication module.
  • the operating system is the computer readable instructions of the physical device hardware and software resources stored in the data cluster, and supports the operation of the information processing computer readable instructions and other software and/or computer readable instructions.
  • the network communication module is used to implement communication between various components in the non-volatile readable storage medium and communication with other hardware and software in the physical device.
  • the present application can be implemented by means of software plus a necessary general hardware platform, or by hardware.
  • the present application can divide a physical cluster with a larger storage space into multiple sub-physical clusters with equal space, realize the uniform distribution of the physical clusters, and enable different data Evenly store to the corresponding location, thereby balancing the storage pressure of each physical cluster, and avoiding stored data avalanches caused by centralized data storage.
  • the second physical cluster and each of the sub-physical clusters whose storage space is less than the preset threshold is configured with an identification code, and the hash value is calculated according to the identification code, and then the second physical cluster and each of the sub-physical clusters
  • the physical clusters are evenly mapped into the consistent hash ring, and then the consistent hash algorithm is used to determine the target storage cluster corresponding to the optimal storage of the file to be stored, and the file to be stored is stored in the target storage cluster.
  • it can also receive the addition and deletion instructions of the physical cluster to meet the expansion needs of mass storage.
  • the queryability of data storage can be guaranteed, that is, when querying the file to be stored, the original storage location of the data is not considered, and the current storage physical cluster location can be accurately located according to the hash value of the file to be stored. And then realize the efficient positioning of data storage.

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Human Computer Interaction (AREA)
  • Physics & Mathematics (AREA)
  • General Engineering & Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
  • Storage Device Security (AREA)

Abstract

L'invention concerne un procédé et un appareil de stockage de groupes de données et un dispositif informatique, qui se rapportent au domaine du traitement de données, et peuvent résoudre le problème des mauvaises performances de stockage provoquées par l'incapacité de garantir le stockage régulier de données dans un groupe et de localiser rapidement la position de stockage des données lorsque le groupe est utilisé pour le stockage des données. Le procédé consiste à : obtenir tous les groupes physiques pour stocker des données (101) ; mapper de manière uniforme les groupes physiques sur un nœud physique d'une bague de hachage constante (102) ; déterminer, en fonction d'une valeur de hachage d'un fichier à stocker, le groupe physique cible optimal pour le stockage (103) ; et stocker ledit fichier dans le groupe physique cible (104). Le procédé peut être appliqué au stockage de groupes de données.
PCT/CN2019/118232 2019-07-11 2019-11-13 Procédé et appareil de stockage de groupes de données et dispositif informatique WO2021003935A1 (fr)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
CN201910625543.5A CN110489059B (zh) 2019-07-11 2019-07-11 数据集群存储的方法、装置及计算机设备
CN201910625543.5 2019-07-11

Publications (1)

Publication Number Publication Date
WO2021003935A1 true WO2021003935A1 (fr) 2021-01-14

Family

ID=68547014

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2019/118232 WO2021003935A1 (fr) 2019-07-11 2019-11-13 Procédé et appareil de stockage de groupes de données et dispositif informatique

Country Status (2)

Country Link
CN (1) CN110489059B (fr)
WO (1) WO2021003935A1 (fr)

Cited By (7)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN113672524A (zh) * 2021-08-20 2021-11-19 上海哔哩哔哩科技有限公司 基于多级缓存的数据处理方法及系统
CN113689103A (zh) * 2021-08-18 2021-11-23 国电南瑞南京控制系统有限公司 自适应负载均衡用采分流智能调度管理方法、装置及系统
CN113708937A (zh) * 2021-10-28 2021-11-26 湖南天河国云科技有限公司 区块链交易的处理方法和系统
CN113934377A (zh) * 2021-10-28 2022-01-14 山东英信计算机技术有限公司 一种元数据集群部署方法、装置、设备及可读存储介质
CN114666338A (zh) * 2022-05-19 2022-06-24 杭州指令集智能科技有限公司 一种基于消息的多实例负载均衡方法和系统
CN115002131A (zh) * 2022-05-24 2022-09-02 中欣链证数字科技有限公司 一种用户请求分配方法、装置、设备及系统
CN115001969A (zh) * 2022-05-24 2022-09-02 中欣链证数字科技有限公司 一种数据存储节点部署方法、数据存储方法、装置及设备

Families Citing this family (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN111258508B (zh) * 2020-02-16 2020-11-10 西安奥卡云数据科技有限公司 一种分布式对象存储中的元数据管理方法
CN111756828B (zh) * 2020-06-19 2023-07-14 广东浪潮大数据研究有限公司 一种数据存储方法、装置及设备
CN113778341A (zh) * 2021-09-17 2021-12-10 北京航天泰坦科技股份有限公司 遥感数据分布式存储方法和装置及遥感数据读取方法
CN114489483A (zh) * 2021-12-24 2022-05-13 深圳市捷顺科技实业股份有限公司 一种基于对象储存的磁盘管理方法及对象存储模组

Citations (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20130179481A1 (en) * 2012-01-11 2013-07-11 Tonian Inc. Managing objects stored in storage devices having a concurrent retrieval configuration
CN103929500A (zh) * 2014-05-06 2014-07-16 刘跃 一种分布式存储系统的数据分片方法
CN106572153A (zh) * 2016-10-21 2017-04-19 乐视控股（北京）有限公司 集群的数据存储方法及装置
CN106909557A (zh) * 2015-12-23 2017-06-30 中国电信股份有限公司 内存集群的存储方法及装置、内存集群的读取方法及装置
CN107844269A (zh) * 2017-10-17 2018-03-27 华中科技大学 一种基于一致性哈希的分层混合存储系统及方法
CN109639777A (zh) * 2018-11-28 2019-04-16 优刻得科技股份有限公司 数据同步的方法、装置、系统和非易失性存储介质

Family Cites Families (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN104754000B (zh) * 2013-12-30 2018-08-24 国家电网公司 一种负载均衡方法及系统
US9860316B2 (en) * 2014-09-19 2018-01-02 Facebook, Inc. Routing network traffic based on social information
CN109271391B (zh) * 2018-09-29 2021-05-28 武汉极意网络科技有限公司 数据存储方法、服务器、存储介质及装置

Patent Citations (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20130179481A1 (en) * 2012-01-11 2013-07-11 Tonian Inc. Managing objects stored in storage devices having a concurrent retrieval configuration
CN103929500A (zh) * 2014-05-06 2014-07-16 刘跃 一种分布式存储系统的数据分片方法
CN106909557A (zh) * 2015-12-23 2017-06-30 中国电信股份有限公司 内存集群的存储方法及装置、内存集群的读取方法及装置
CN106572153A (zh) * 2016-10-21 2017-04-19 乐视控股（北京）有限公司 集群的数据存储方法及装置
CN107844269A (zh) * 2017-10-17 2018-03-27 华中科技大学 一种基于一致性哈希的分层混合存储系统及方法
CN109639777A (zh) * 2018-11-28 2019-04-16 优刻得科技股份有限公司 数据同步的方法、装置、系统和非易失性存储介质

Cited By (10)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN113689103A (zh) * 2021-08-18 2021-11-23 国电南瑞南京控制系统有限公司 自适应负载均衡用采分流智能调度管理方法、装置及系统
CN113689103B (zh) * 2021-08-18 2023-11-24 国电南瑞南京控制系统有限公司 自适应负载均衡用采分流智能调度管理方法、装置及系统
CN113672524A (zh) * 2021-08-20 2021-11-19 上海哔哩哔哩科技有限公司 基于多级缓存的数据处理方法及系统
CN113708937A (zh) * 2021-10-28 2021-11-26 湖南天河国云科技有限公司 区块链交易的处理方法和系统
CN113934377A (zh) * 2021-10-28 2022-01-14 山东英信计算机技术有限公司 一种元数据集群部署方法、装置、设备及可读存储介质
CN114666338A (zh) * 2022-05-19 2022-06-24 杭州指令集智能科技有限公司 一种基于消息的多实例负载均衡方法和系统
CN114666338B (zh) * 2022-05-19 2022-08-26 杭州指令集智能科技有限公司 一种基于消息的多实例负载均衡方法和系统
CN115002131A (zh) * 2022-05-24 2022-09-02 中欣链证数字科技有限公司 一种用户请求分配方法、装置、设备及系统
CN115001969A (zh) * 2022-05-24 2022-09-02 中欣链证数字科技有限公司 一种数据存储节点部署方法、数据存储方法、装置及设备
CN115002131B (zh) * 2022-05-24 2024-03-01 中欣链证数字科技有限公司 一种用户请求分配方法、装置、设备及系统

Also Published As

Publication number Publication date
CN110489059B (zh) 2022-04-12
CN110489059A (zh) 2019-11-22

Similar Documents

Publication Publication Date Title
WO2021003935A1 (fr) Procédé et appareil de stockage de groupes de données et dispositif informatique
US10977277B2 (en) Systems and methods for database zone sharding and API integration
US10997211B2 (en) Systems and methods for database zone sharding and API integration
US11144651B2 (en) Secure cloud-based storage of data shared across file system objects and clients
US10467245B2 (en) System and methods for mapping and searching objects in multidimensional space
US10191914B2 (en) De-duplicating distributed file system using cloud-based object store
US20200125758A1 (en) File System Permission Setting Method and Apparatus
RU2475988C2 (ru) Способ и система использования локального поддерживаемого хост-узлом кэша и криптографических хэш-функций для того, чтобы уменьшать сетевой трафик
US20160217167A1 (en) Hash Database Configuration Method and Apparatus
US20190340171A1 (en) Data Redistribution Method and Apparatus, and Database Cluster
US20220329591A1 (en) Method, apparatus and device for generating device fingerprint and storage medium
TWI734744B (zh) 路由表的同步方法、裝置及系統
US20200065306A1 (en) Bloom filter partitioning
EP4231167A1 (fr) Procédé et appareil de stockage de données basés sur un réseau à chaîne de blocs
CN110737663B (zh) 一种数据存储方法、装置、设备及存储介质
US20180225048A1 (en) Data Processing Method and Apparatus
US20170212939A1 (en) Method and mechanism for efficient re-distribution of in-memory columnar units in a clustered rdbms on topology change
EP3093789B1 (fr) Stockage d'informations structurées
US10664349B2 (en) Method and device for file storage
US10924452B1 (en) Auditing IP address assignments
US20150278543A1 (en) System and Method for Optimizing Storage of File System Access Control Lists
WO2019174558A1 (fr) Procédé et dispositif d'indexation de données
CN115563073A (zh) 分布式元数据的数据处理的方法、装置及电子设备
CN106527960B (zh) 多存储盘负载管理方法、装置、文件系统及存储网络系统
CN117075823B (zh) 对象查找方法、系统、电子设备及存储介质

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 19937110

Country of ref document: EP

Kind code of ref document: A1

122 Ep: pct application non-entry in european phase

Ref document number: 19937110

Country of ref document: EP

Kind code of ref document: A1