WO2015067004A9 - Procédé et dispositif de traitement de requête d'accès simultanés - Google Patents

Procédé et dispositif de traitement de requête d'accès simultanés Download PDF

Info

Publication number
WO2015067004A9
WO2015067004A9 PCT/CN2014/075558 CN2014075558W WO2015067004A9 WO 2015067004 A9 WO2015067004 A9 WO 2015067004A9 CN 2014075558 W CN2014075558 W CN 2014075558W WO 2015067004 A9 WO2015067004 A9 WO 2015067004A9
Authority
WO
WIPO (PCT)
Prior art keywords
access
engine
concurrent
hash
access requests
Prior art date
Application number
PCT/CN2014/075558
Other languages
English (en)
Chinese (zh)
Other versions
WO2015067004A1 (fr
Inventor
童燕群
李成林
Original Assignee
华为技术有限公司
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 华为技术有限公司 filed Critical 华为技术有限公司
Publication of WO2015067004A1 publication Critical patent/WO2015067004A1/fr
Publication of WO2015067004A9 publication Critical patent/WO2015067004A9/fr

Links

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/10—File systems; File servers
    • G06F16/17—Details of further file system functions
    • G06F16/176—Support for shared access to files; File sharing support
    • G06F16/1767—Concurrency control, e.g. optimistic or pessimistic approaches

Definitions

  • the present invention relates to the field of computer data storage technologies, and in particular, to a method and apparatus for processing concurrent access requests. Background technique
  • the object storage technology based on HTTP is rapidly developing, and the object storage technology based on the two-layer business model of container Container and object Object is more and more widely used.
  • a container can be understood as a special top-level directory.
  • An object can be a file or a directory, and the object belongs to the container.
  • user data is stored in a container in an object mode, using a technical architecture in which an upper layer application establishes object storage on the underlying distributed storage engine.
  • the number of objects in the container is unrestricted. When there are many objects in the container, there will be a very large index table between the container and the object. Ordinary stand-alone databases can no longer meet the storage requirements. Therefore, the storage engine based on weak consistency is generally maintained in a B-tree structure.
  • Figure 1 shows a simple B-tree structure
  • Figure 2 shows an object storage system based on the underlying distributed storage engine architecture.
  • the sub-blocks N1, N2, N3, ... of the B-tree structure in Fig. 1 can be stored in the figure.
  • the child data block N1 may be stored on the child node 2, the child node 4, and the child node 6.
  • the process of "read-modify-write" is generally employed.
  • Figure 2 based on the object storage system set up by the underlying distributed storage engine, it is reflected that the client initiates an access request to the data resource when accessing After the data resource is obtained, the data resource is modified and then written back to the access storage engine.
  • the initiating access request process is: the client based on the HTTP protocol sends an access request for accessing the data resource, and the upper layer application analyzes the metadata of the related container and the object and the data resource to be accessed from the received access request, and then The corresponding engine access proxy of the upper application requests data resources from the underlying distributed storage engine.
  • the sub-blocks in the B-tree structure become hot spots. For example, there may be multiple clients that need to write the sub-block N1 into the B-tree at the same time, which will cause the sub-block N1 in the B-tree structure to become a hot spot.
  • a concurrent access request is initiated for multiple clients, so that multiple engine access agents simultaneously request access to the child nodes of the storage engine that store the sub-block N1. , causing concurrent access violations of the underlying storage engine.
  • the upper-layer applications APP1, APP2, and APP3 will simultaneously request access to the sub-node 6 through their respective engine access agents.
  • the storage engine based on the weak consistency uses the B-tree structure to maintain the index list
  • the underlying distributed storage engine returns a data conflict response to the upper layer application, and the upper layer application according to the specific service. Choose to rewrite.
  • it will affect the write performance of the B-tree structure, and even cause the data to be rewritten, eventually resulting in the loss of objects in the container.
  • Embodiments of the present invention provide a method and apparatus for processing concurrent access requests to avoid concurrent access conflicts.
  • a method for processing a concurrent access request including:
  • the concurrent access request is sorted, including:
  • the at least two concurrent access requests are ordered in an engine access proxy.
  • the sorting the at least two concurrent access requests includes:
  • An access request that is not on the sorting engine access agent is routed to the sorting engine access agent, and the sorting engine accesses the agent to sort the at least two concurrent access requests.
  • the method further includes:
  • the same data resource is cached in an engine access proxy that sorts the concurrent access requests.
  • a processing apparatus for concurrent access request including a receiving unit, a sorting unit, and an access unit, where
  • the receiving unit is configured to receive at least two concurrent access requests for the same data resource, and transmit the at least two concurrent access requests to the sorting unit;
  • the sorting unit is configured to receive the at least two concurrent access requests transmitted by the receiving unit, and sort the at least two concurrent access requests, and transmit the sorted concurrent access request to the access unit;
  • the access unit is configured to receive the sorted concurrent access request transmitted by the sorting unit, and sequentially access the same data resource according to the sorted concurrent access request.
  • the sorting unit is specifically configured to: in an engine access proxy, sort the at least two concurrent access requests.
  • the sorting unit is specifically configured to: associate each data resource on the storage engine with a hash key value to form a hash space;
  • An access request that is not on the sorting engine access agent is routed to the sorting engine access agent, and the sorting engine accesses the agent to sort the at least two concurrent access requests.
  • the sorting unit is further configured to:
  • a cache unit is further included, where
  • the cache unit is configured to cache the same data resource in an engine access proxy that sorts the concurrent access requests.
  • the concurrent access request processing method provided by the first aspect of the present invention and the concurrent access request processing apparatus provided by the second aspect after receiving at least two concurrent access requests for the same data resource, sorting the at least two concurrent access requests According to the sorted concurrent access request, sequentially accessing the same data resource can ensure that only one request accesses the corresponding data resource at the same time, thereby avoiding concurrent access conflicts.
  • FIG. 1 is a schematic diagram of an organization structure of an index table based on a B-tree structure in the prior art
  • FIG. 2 is a schematic diagram of an architecture of an object storage system based on an underlying distributed storage engine in the prior art
  • FIG. 3 is a schematic diagram of an access violation occurring in a concurrent access request in the prior art
  • FIG. 4 is a schematic flowchart of a method for processing a concurrent access request according to an embodiment of the present invention
  • FIG. 5 is a flowchart of a method for scheduling concurrent access requests according to an embodiment of the present invention.
  • FIG. 6 is a schematic diagram of hash space division in an embodiment of the present invention.
  • FIG. 7 is a schematic diagram of a process of sorting concurrent access requests in an embodiment of the present invention.
  • FIG. 8 is a schematic structural diagram of a concurrent access request processing apparatus according to an embodiment of the present invention
  • FIG. 9 is another schematic structural diagram of a concurrent access request processing apparatus according to an embodiment of the present invention.
  • FIG. 11 is still another schematic diagram of a controller configuration according to an embodiment of the present invention.
  • FIG. 12 is still another schematic diagram of a controller configuration according to an embodiment of the present invention. detailed description
  • different clients may initiate concurrent access requests to the same data resource, and the upper layer application may receive at least two concurrent data resources.
  • Access request the at least two concurrent access requests are processed by different ⁇ (upper application) and engine access agent.
  • the at least two concurrent access requests are processed in the upper layer application to ensure that only one request accesses the corresponding data resource at the same time, thereby preventing concurrent access conflicts.
  • a first embodiment of the present invention provides a method for processing a concurrent access request, as shown in FIG. 4, including:
  • 5101 Receive at least two concurrent access requests for the same data resource.
  • an access request for the data resource is initiated by different clients based on the HTTP protocol, that is, at least the same data resource exists.
  • the upper application receives at least two concurrent access requests for the same data resource.
  • 5102 Sort the received at least two concurrent access requests.
  • At least two concurrent access requests of the same data resource received by the upper layer application are sorted, and may be directly sorted on different engine access agents that request data resources of the underlying storage engine, or may be different.
  • the access request on the engine access proxy is routed to an engine access proxy for sorting.
  • the embodiment of the present invention preferably routes access requests on different engine access agents to an engine access agent for sorting, so that when the concurrent access requests are sorted, no state query is required between different engine access agents.
  • At least two concurrent access requests for accessing the same data resource are sorted. According to the sorted concurrent access request, accessing the data resource in turn can ensure that only one request accesses the data resource at a time, thereby avoiding concurrent access conflicts.
  • a client adds an object to a container at the same time, it causes the sub-block data of the index relation table to become a hot spot, that is, there is more than one access request.
  • the access table between the container and the object is requested to be accessed.
  • the concurrent access requests of the same index table are sorted to ensure that the same index table has only one read and write operation at the same time, thereby avoiding concurrent access conflicts.
  • an engine access proxy in an engine access proxy, at least two concurrent access requests of the same data resource are sorted, and access requests on different engine access agents are routed to an engine access proxy for sorting, so that concurrent access is performed.
  • access requests on different engine access agents are routed to an engine access proxy for sorting, so that concurrent access is performed.
  • the process of sorting at least two concurrent access requests is as shown in FIG. 5, including:
  • each data resource on the storage engine is hashed according to a unified hash algorithm between the upper layer applications.
  • Each data resource corresponds to a hash key value and constitutes a hash space.
  • the hash space is a one-dimensional space that is sufficiently large for the number of APPs applied to the upper layer.
  • the hash space in the partition S201 is N sub-hash spaces.
  • the number of divided sub-hash spaces is the same as the number of engine access agents corresponding to the upper layer application. If the number of the upper layer application is N, the hash space is divided into N parts, and N sub-hash spaces are obtained.
  • the upper layer application includes APP1, APP2, and APP3, and the hash space partitioning process is shown in FIG. 6. Shown.
  • the partitioning may be performed by using an equal division method, or may be performed by using an unequal division method, as long as the number of divided sub-hash spaces is corresponding to the engine accessing agent corresponding to the upper layer application.
  • the number is the same.
  • the number of upper layer applications is N
  • the hash space can be equally divided into N parts, in the embodiment of the present invention, the hash may be equally divided.
  • the space is divided into N parts, and a sub-hash space with the same hash space is obtained. If the hash space cannot be equally divided, the hash space can be divided into N by using the unequal division method in the embodiment of the present invention.
  • - 1 sub-hash space with equal hash space divide the remaining hash space into 1 sub-hash space, and finally get a sub-hash space.
  • a simple cluster can be established between the engine access agents of the upper layer application, the agent number is accessed for each engine, and a sub-hash space is allocated for each engine access agent's own sequence number.
  • the sub-hash space divided in S202 is allocated to one engine access agent, and each engine access agent allocates a sub-hash space, and different sub-hash spaces allocated by different engine access agents are different.
  • 5204 Determine a hash value of at least two data resources requested to access, and determine an engine access proxy that sorts at least two concurrent access requests according to a sub-hash space allocated to the sub-engine access proxy, and obtain a sorting engine access proxy. .
  • each data resource corresponds to a hash value
  • the hash value belongs to a sub-hash space obtained by dividing in S202.
  • each hash space is assigned to an engine access agent. Therefore, the engine access agent that sorts the at least two concurrent access requests may be determined according to the sub-hash space to which the hash value of the same data resource accessed by the at least two concurrent access requests belongs.
  • the determined engine access agent is referred to as a sorting engine access agent through which the at least two concurrent access requests are sorted.
  • 5205 Forward concurrent access requests that are not on the sort engine access agent to the sorting engine access agent.
  • Each access request in the embodiment of the present invention is sent to the upper application , so different access requests will be on different engine access agents.
  • the access request that is not on the determined sorting engine accessing proxy is routed to the sorting engine accessing proxy, and the sorting engine accesses the proxy, and routes to the proxy All concurrent access requests are sorted. For example, there are currently 3 concurrent access requests that need to be accessed at the storage engine.
  • the hash value obtained by hashing the data resource on the node 6 and storing the data resource on the engine sub-node 6 belongs to the engine access proxy corresponding to APP1.
  • the access request on the engine access proxy corresponding to APP2 and APP3 is respectively routed to the engine access proxy corresponding to APP1.
  • the three concurrent access requests are sorted by the engine access agent corresponding to APP1, as shown in FIG.
  • the request for the index table of the same container is routed to the engine access proxy of the same application for sorting processing, and sequentially accessed according to the sorted concurrent access request.
  • Index tables ensure that there are no access violations, thus avoiding data conflicts with the underlying distributed storage engine.
  • the number of accessing agents of the engine is monitored.
  • the hash space is re-divided to accommodate the case where the upper application node exits or newly joins.
  • the read/write cache may be added to the engine access proxy that sorts the concurrent access requests, and the data resources are cached to improve the data resources. Access speed.
  • the embodiment of the present invention provides a concurrent access request processing device.
  • the concurrent access request processing device provided by the embodiment of the present invention includes a receiving unit. 801, a sorting unit 802 and an access unit 803, wherein
  • the receiving unit 801 is configured to receive at least two concurrent access requests for the same data resource, and transmit at least two concurrent access requests to the sorting unit 802.
  • the sorting unit 802 is configured to receive at least two concurrent access requests transmitted by the receiving unit 801, and sort the at least two concurrent access requests, and transmit the sorted concurrent access requests to the access unit 803;
  • the access unit 803 is configured to receive the sorted concurrent access request transmitted by the sorting unit 802, and sequentially access the same data resource according to the sorted concurrent access request.
  • the sorting unit 802 is configured to sort at least two concurrent access requests in an engine access proxy.
  • the sorting unit 802 is specifically configured to:
  • an engine access proxy that sorts at least two concurrent access requests according to a sub-hash space to which the hash value of the same data resource accessed by the at least two concurrent access requests belongs, to obtain a sorting engine accessing the proxy;
  • An access request that is not on the sort engine access proxy is routed to the sort engine access proxy, and the sort engine accesses the proxy to sort at least two concurrent access requests.
  • sorting unit 802 in the embodiment of the present invention is further configured to:
  • the concurrent access request processing apparatus further includes a cache unit 804, as shown in FIG. 9, wherein the cache unit 804 is configured to cache the same data resource in an engine access proxy that sorts concurrent access requests.
  • the concurrent access request processing apparatus when there are at least two concurrent access requests for the same data resource, sorts the at least two concurrent access requests, and sequentially accesses the same data according to the sorted concurrent access request.
  • the resource can ensure that only one request accesses the corresponding data resource at a time, thereby avoiding concurrent access conflicts.
  • the above-mentioned concurrent access request processing device provided by the embodiment of the present invention may be an independent component, or may be integrated into other components.
  • the concurrent access request processing device provided by the embodiment of the present invention may be an engine access proxy, or may be New components integrated into the engine access agent. It should be noted that the function implementation and the interaction manner of each module/unit of the concurrent access request processing apparatus in the embodiment of the present invention may be further referred to the description of the related method embodiments.
  • the embodiment of the present invention provides a controller, which can be applied to an object storage service based on a container and an object two-layer service model, as shown in FIG. 10, in accordance with an embodiment of the present invention.
  • the controller includes: a processor 1001 and an I/O interface 1002, wherein the processor 1001 is configured to receive at least two concurrent access requests for the same data resource, and receive the received at least two concurrent access requests. Sorting, and transmitting the sorted concurrent access request to the I/O interface 1002;
  • the I/O interface 1002 is configured to receive the sorted concurrent access request transmitted by the processor 1001, and output the sorted concurrent access request.
  • processor 1001 is configured to sort at least two concurrent access requests in an engine access agent.
  • the processor 1001 is specifically configured to: associate each data resource on the storage engine with a hash key value to form a hash space; divide the hash space into a sub-hash space, where ⁇ is the number of engine access agents; A sub-hash space is allocated to each of the engine access agents, so that each engine access agent is assigned a sub-hash space, and different sub-hash spaces are assigned by different engine access agents; accesses according to at least two concurrent access requests The sub-hash space to which the hash value of the same data resource belongs, determine the engine access proxy that sorts at least two concurrent access requests, obtain the sorting engine access proxy, and route the access request that is not in the sorting engine access proxy to the sorting On the engine access agent, the sorting engine accesses the agent and sorts at least two concurrent access requests.
  • the controller in the embodiment of the present invention further includes a monitor 1003. As shown in FIG. 11, the monitor 1003 monitors the number of engine access agents, and sends a re-division to the processor 1001 when the number of engine access agents changes. Hash space instructions.
  • the controller further includes a buffer 1004.
  • the buffer 1004 is used in an engine access proxy for sorting concurrent access requests by the processor 1001. Cache the same data resource.
  • the controller provided by the embodiment of the present invention, when there are at least two concurrent access requests for the same data resource, sort the at least two concurrent access requests, and sequentially access the same data resource according to the sorted concurrent access request. It is guaranteed that only one request for accessing the corresponding data resource at the same time can avoid concurrent access conflicts.

Abstract

La présente invention concerne un procédé et un dispositif de traitement de requête d'accès simultanés pour éviter un conflit d'accès simultanés. Le procédé consiste : à recevoir au moins deux requêtes d'accès simultanés pour la même ressource de données et trier lesdites deux requêtes d'accès simultanés ; et accéder à la même ressource de données séquentiellement selon les requêtes d'accès simultanés triées. La présente invention garantit qu'il y a uniquement la ressource de données correspondant à uns requête d'accès, permettant ainsi d'éviter un conflit d'accès simultanés.
PCT/CN2014/075558 2013-11-07 2014-04-17 Procédé et dispositif de traitement de requête d'accès simultanés WO2015067004A1 (fr)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
CN201310549721.3 2013-11-07
CN201310549721.3A CN103634374B (zh) 2013-11-07 2013-11-07 一种并发访问请求的处理方法及装置

Publications (2)

Publication Number Publication Date
WO2015067004A1 WO2015067004A1 (fr) 2015-05-14
WO2015067004A9 true WO2015067004A9 (fr) 2015-09-03

Family

ID=50214990

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2014/075558 WO2015067004A1 (fr) 2013-11-07 2014-04-17 Procédé et dispositif de traitement de requête d'accès simultanés

Country Status (2)

Country Link
CN (1) CN103634374B (fr)
WO (1) WO2015067004A1 (fr)

Families Citing this family (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN103634374B (zh) * 2013-11-07 2017-04-12 华为技术有限公司 一种并发访问请求的处理方法及装置
CN105354328B (zh) * 2015-11-25 2019-03-26 南京莱斯信息技术股份有限公司 一种解决NoSQL数据库并发访问冲突的系统及方法
CN106649141B (zh) * 2016-11-02 2019-10-18 郑州云海信息技术有限公司 一种基于ceph的存储交互装置及存储系统
CN113253933B (zh) 2017-04-17 2024-02-09 伊姆西Ip控股有限责任公司 用于管理存储系统的方法、设备和计算机可读存储介质
CN111600940B (zh) * 2020-05-06 2022-11-11 中国银行股份有限公司 一种分布式会话管理方法及系统

Family Cites Families (7)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN101064604B (zh) * 2006-04-29 2012-04-18 西门子公司 远程访问方法、系统及设备
CN103299271B (zh) * 2011-01-11 2016-04-13 惠普发展公司，有限责任合伙企业 并发请求调度
WO2011113390A2 (fr) * 2011-04-27 2011-09-22 华为技术有限公司 Procédé et dispositif pour améliorer la vitesse d'accès utilisateur au réseau internet mobile à bande passante
CN103297456B (zh) * 2012-02-24 2016-09-28 阿里巴巴集团控股有限公司 一种分布式系统下共享资源的访问方法及分布式系统
CN102739440A (zh) * 2012-05-24 2012-10-17 大唐移动通信设备有限公司 一种硬件设备的访问方法及装置
CN102999377B (zh) * 2012-11-30 2015-06-10 北京东方通科技股份有限公司 服务并发访问控制方法及装置
CN103634374B (zh) * 2013-11-07 2017-04-12 华为技术有限公司 一种并发访问请求的处理方法及装置

Also Published As

Publication number Publication date
CN103634374B (zh) 2017-04-12
WO2015067004A1 (fr) 2015-05-14
CN103634374A (zh) 2014-03-12

Similar Documents

Publication Publication Date Title
EP3361387B1 (fr) Système, équipement et procédé de transmission de données
CN105144121B (zh) 高速缓存内容可寻址数据块以供存储虚拟化
US9590915B2 (en) Transmission of Map/Reduce data in a data center
US20160132541A1 (en) Efficient implementations for mapreduce systems
US9998531B2 (en) Computer-based, balanced provisioning and optimization of data transfer resources for products and services
CN103155524B (zh) 用于在多核系统中的多个核之间共享iip地址的系统和方法
JP6275119B2 (ja) メモリ要素の割当てのために一方向リンク付けリストを区分化するシステム及び方法
JP2019139759A (ja) ソリッドステートドライブ（ｓｓｄ）及び分散データストレージシステム並びにその方法
WO2015067004A1 (fr) Procédé et dispositif de traitement de requête d'accès simultanés
WO2010072083A1 (fr) Système de base de données fondé sur des applications web et méthode associée de gestion des données
KR20210075845A (ko) 네이티브 키-밸류 분산 스토리지 시스템
CN103312624A (zh) 一种消息队列服务系统和方法
US11080207B2 (en) Caching framework for big-data engines in the cloud
JP2005056077A (ja) データベース制御方法
US9483523B2 (en) Information processing apparatus, distributed processing system, and distributed processing method
WO2016101662A1 (fr) Procédé de traitement de données et serveur approprié
CN107493309B (zh) 一种分布式系统中的文件写入方法及装置
US20200394077A1 (en) Map reduce using coordination namespace hardware acceleration
JP4271967B2 (ja) 分散ファイルシステム及び分散ファイルシステムの運用方法
WO2017113277A1 (fr) Procédé, dispositif et système de traitement de données
Ren et al. Design, implementation, and evaluation of a NUMA-aware cache for iSCSI storage servers
US11714573B1 (en) Storage optimization in a distributed object store
US10824640B1 (en) Framework for scheduling concurrent replication cycles
WO2016197607A1 (fr) Procédé et appareil pour réaliser une recherche d'itinéraire
KR101512647B1 (ko) 질의처리엔진을 선택하는 방법

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 14860733

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase in:

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 14860733

Country of ref document: EP

Kind code of ref document: A1