WO2022105048A1 - 一种分布式共享文件系统及其数据处理方法 - Google Patents

一种分布式共享文件系统及其数据处理方法 Download PDF

Info

Publication number
WO2022105048A1
WO2022105048A1 PCT/CN2021/071098 CN2021071098W WO2022105048A1 WO 2022105048 A1 WO2022105048 A1 WO 2022105048A1 CN 2021071098 W CN2021071098 W CN 2021071098W WO 2022105048 A1 WO2022105048 A1 WO 2022105048A1
Authority
WO
WIPO (PCT)
Prior art keywords
load balancer
storage container
client
storage
access request
Prior art date
Application number
PCT/CN2021/071098
Other languages
English (en)
French (fr)
Inventor
杜若峰
Original Assignee
网宿科技股份有限公司
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 网宿科技股份有限公司 filed Critical 网宿科技股份有限公司
Publication of WO2022105048A1 publication Critical patent/WO2022105048A1/zh

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/10—File systems; File servers
    • G06F16/17—Details of further file system functions
    • G06F16/176—Support for shared access to files; File sharing support
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/10—File systems; File servers
    • G06F16/18—File system types
    • G06F16/182—Distributed file systems
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F9/00—Arrangements for program control, e.g. control units
    • G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
    • G06F9/44—Arrangements for executing specific programs
    • G06F9/455—Emulation; Interpretation; Software simulation, e.g. virtualisation or emulation of application or operating system execution engines
    • G06F9/45533—Hypervisors; Virtual machine monitors
    • G06F9/45558—Hypervisor-specific management and integration aspects
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F9/00—Arrangements for program control, e.g. control units
    • G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
    • G06F9/46—Multiprogramming arrangements
    • G06F9/50—Allocation of resources, e.g. of the central processing unit [CPU]
    • G06F9/5083—Techniques for rebalancing the load in a distributed system
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F9/00—Arrangements for program control, e.g. control units
    • G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
    • G06F9/44—Arrangements for executing specific programs
    • G06F9/455—Emulation; Interpretation; Software simulation, e.g. virtualisation or emulation of application or operating system execution engines
    • G06F9/45533—Hypervisors; Virtual machine monitors
    • G06F9/45558—Hypervisor-specific management and integration aspects
    • G06F2009/4557—Distribution of virtual machine instances; Migration and load balancing

Definitions

  • the present application relates to the field of Internet technologies, and in particular, to a distributed shared file system and a data processing method thereof.
  • the purpose of this application is to provide a distributed shared file system and a data processing method thereof, which can improve the stability of the distributed shared file system.
  • an embodiment of the present application provides a distributed shared file system, where the distributed shared file system includes a Domain Name System (Domain Name System, DNS for short) server, a load balancer, a storage container, and a distributed storage cluster,
  • the DNS server is configured to, when receiving a domain name resolution request initiated by a client, allocate a target load balancer to the client according to the current load status of each of the load balancers, so that the client Mounted on the target load balancer
  • the load balancer is configured to receive the access request from the client, and identify the current weight coefficient of each of the storage containers, so as to schedule the access request to At the target storage container
  • the storage container is configured to parse the access request scheduled by the load balancer, so as to process the data pointed to by the access request in the distributed storage cluster.
  • an embodiment of the present application also provides a data processing method for a distributed shared file system, the method comprising: when the DNS server receives a domain name resolution request initiated by a client, according to the current load of each load balancer state, assign a target load balancer to the client to mount the client on the target load balancer; the target load balancer receives the client's access request and identifies the current to schedule the access request to the target storage container according to the weight coefficient; the target storage container parses the access request scheduled by the target load balancer, so that the access request is processed in the distributed storage cluster The data pointed to by the request is processed.
  • the technical solution provided by this application can distribute the user's data traffic to multiple storage containers for processing by means of multi-level load balancing. While utilizing idle resources, it reduces the overload of a certain node. possibility, thereby improving the overall stability of the system.
  • the DNS server can mount the client to the corresponding target load balancer through the first-level load balancing by judging the current load state of the load balancer.
  • the load balancer analyzes the weight coefficient of the back-end storage container, so as to schedule the client's traffic to the target storage container through the second-level load balancing.
  • the client's access request is parsed by the storage container, and finally the data can be stored or read in the distributed storage cluster.
  • multi-level load balancing method can reasonably allocate user data traffic between the load balancer and the storage container, and avoid data exceptions caused by the overload of a load balancer or a storage container. , which improves the overall stability of the distributed shared file system.
  • FIG. 1 is a schematic diagram of a system architecture of a distributed shared file system in an embodiment of the present application
  • FIG. 2 is a schematic diagram of a network configuration of a distributed shared file system in an embodiment of the present application
  • FIG. 3 is a schematic diagram of a data processing method of a distributed shared file system in an embodiment of the present application.
  • an embodiment of the present application provides a distributed shared file system, which may include a DNS server, a load balancer, a storage container (eg, a container implemented based on docker technology), and a distributed storage cluster (eg, ceph distributed type file system), in which the load balancer and the storage container can be deployed in the same service device or cluster, or in different service devices or clusters respectively, and one load balancer and at least one storage container can form an independent
  • the service units are connected to each other based on network communication.
  • the storage container is connected to the distributed storage cluster based on network communication.
  • the distributed shared file system provided by this application provides an application service that users can access and operate. Mount it to the distributed shared file system, thereby initiating various data access requests to the distributed shared file system, so as to realize the storage of data to the distributed storage cluster, and the read and write operations on the stored data.
  • the distributed shared file system can create a mount point for the file storage service, and feed back the mount point information to the client, where one mount point corresponds to at least one mount point.
  • the mount point information includes the uniquely identified domain name set by the distributed shared file system for the file storage service; at the same time, after the system completes the creation of the service unit, the load balancer in the service unit can be The IP address, the domain name of the mount point and its mapping relationship are configured in the DNS server.
  • the client performs the mount operation, it must first request the DNS server to resolve the mounted domain name.
  • the DNS server can feed back the IP address of the corresponding load balancer to the client based on the mapping relationship between the mounted domain name and the IP addresses of each load balancer terminal, so that the client can establish a communication connection with the corresponding load balancer based on the IP address, and realize the mounting of the file storage service.
  • the DNS server in this embodiment can implement a first-level load balancing strategy when performing domain name resolution.
  • the DNS server can monitor the current load status of each load balancer.
  • This load state can be characterized by operating parameters on the load balancer.
  • the load state can be represented by the number of connections of the load balancer. The more connections, the greater the load.
  • other parameters can also be used to characterize the load state according to needs or the difficulty of parameter acquisition.
  • the DNS server can select the load from the multiple load balancers according to the obtained load status. A smaller target load balancer, and the IP address of the target load balancer is fed back to the client, so that the client is mounted on the target load balancer.
  • the load balancer determination unit in the DNS server may identify the load balancer with the smallest number of connections among the load balancers, and assign The load balancer with the smallest number of connections is used as the assigned target load balancer.
  • multiple load balancers with the number of connections less than or equal to a certain fixed value can also be filtered out, and then one of these load balancers can be randomly selected as the target load balancer.
  • load balancers and storage containers can form service units.
  • the number of service units created for the same file storage service may be one or more.
  • a service unit can include a load balancer and one or more storage containers. After the file storage service is mounted to the load balancer, the client's access request will be sent to the load balancer first, and the load balancer will server to a storage container in the same service unit.
  • the load balancer can execute the second-level load balancing strategy, and select one of the storage containers to process the client's access request.
  • the second-level load balancing strategy may include selecting a storage container in combination with the weight coefficient and the data traffic actually processed by each storage container.
  • a corresponding weight coefficient may be preset for each storage container according to its service performance.
  • the weight factor can represent the capability of processing data traffic, and the higher the weight factor, the higher the capability of processing data traffic.
  • the service performance can be determined based on the maximum bandwidth carrying capacity, CPU, and other factors that affect the traffic processing capability.
  • the weight coefficient ratio is obtained according to the weight coefficient of each storage container.
  • the service unit includes three storage containers, and the weight coefficient ratio of these three storage containers is 5:4:2.
  • the load balancer When the load balancer selects storage containers based on the second-level load balancing strategy, it can obtain the weight coefficient ratio of each storage container in the service unit and the data traffic currently being processed by each storage container, and determine the target storage container.
  • the principle of determination is: After the access request is scheduled to the target storage container, the flow ratio of the existing data flow of each storage container after the change can be made closer to the above-mentioned coefficient ratio.
  • the third storage container can be used as the target storage container.
  • the traffic ratio of the existing data traffic can be changed to 10:7:3, which is closer to the weight coefficient ratio of 5 than the original 10:7:2 : 4: 2, in this way, data traffic can be reasonably allocated to each storage container, avoiding a state where some storage containers are fully loaded while others are idle.
  • each service unit may be managed by a daemon process in the system.
  • the daemon can monitor the status parameters of each storage container in the service unit.
  • the state parameters of the storage container may include information such as the working state, bandwidth, and number of connections of the storage container.
  • the daemon process is based on the obtained status parameters.
  • the working status of the storage container can be represented by status codes such as active and error.
  • active can represent normal work
  • error can represent abnormal work.
  • the status code can also represent unresponsive, reset and other states.
  • the daemon can remove the abnormal storage containers from the corresponding service unit, and add a corresponding number of storage containers to the corresponding service unit, so that the number of storage containers available in the service unit is increased. can be stabilized at a fixed value.
  • the storage container runs the Network Attached Storage (NAS for short) service, encapsulates the distributed storage cluster and exposes it to the user to provide the network storage service.
  • NAS Network Attached Storage
  • the received client can be accessed
  • the request is parsed, and based on the parsed content, the data in the distributed storage cluster is read, written, added, and deleted.
  • load balancing that is, distributing traffic, increasing throughput, and enhancing data processing capabilities
  • a service unit when the file storage service of the client needs to improve service performance, a service unit can be added under the corresponding mount point, and the IP address of the load balancer in the newly added service unit can be reported to the DNS server middle.
  • the DNS server implements the first-level load balancing strategy, it can have the opportunity to mount the client to the newly added load balancer.
  • this linear expansion method can achieve an approximate linear improvement of system service performance.
  • you need to reduce the service unit of the mount point you can first delete the IP address of the load balancer in the service unit to be reduced from the DNS server, so that no new access requests will be mounted to the load.
  • the service unit can be deleted to release the processing resources. In this way, high scalability of the system can be achieved.
  • the distributed shared file system further includes a management server for managing and coordinating the DNS server and each service device, and the management server is respectively connected with the DNS server and the server for deploying load balancers and storage containers.
  • the service device is connected based on network communication, and according to the user's request to create a file storage service received by the application service, a corresponding instruction is generated and sent to the service device to instruct the service device to create a load balancer and a storage container correspondingly, and at the same time create a new
  • the IP address configured by the load balancer; the management server can configure the mount domain name and other mount information for the file storage service, and feed it back to the user client, and at the same time send the mount domain name, the IP address of the load balancer and its corresponding relationship.
  • a corresponding control instruction is sent to the service device to instruct the service device to increase or decrease the service unit corresponding to the file storage service, and at the same time, add or decrease the service unit to the service unit.
  • the IP address of the load balancer is reported to the DNS server to update the records of the mounted domain name and IP address on the DNS server.
  • the distributed shared file system can be set on the network of the load balancer and the storage container in the service unit.
  • the load balancer and the corresponding client can be in the same subnet, so that the client can directly access the load balancer after being mounted on the load balancer. Also, the same load balancer can only be bound to one subnet. Through subnetting, the load balancers mounted on the clients of different users can be in different subnets, so as to achieve isolation between users.
  • Each storage container and distributed storage cluster can be located in the same network, such as the same 10G storage, so that fast data interaction can be performed between the storage container and the distributed storage cluster.
  • each storage container can also be in the same management network, which can be managed by a daemon.
  • the DNS server can be in a separate network environment, and can be connected to the subnet where the user's client is located through a route, thereby providing domain name resolution services for each user's client.
  • the load balancer can also maintain high availability by means of active and standby.
  • Each primary load balancer can have its own backup load balancer. When the primary load balancer cannot provide services, the backup load balancer can be upgraded to the primary load balancer and continue to provide services to ensure high system availability and stability. .
  • the distributed shared file system when creating a corresponding service unit for the file storage service of the client, can refer to the above-mentioned network division method to set the network of the load balancer and the storage container, which specifically includes: The load balancer is divided into the same subnet as the client, and the storage container is divided into the same network as the distributed storage cluster.
  • an embodiment of the present application further provides a data processing method for a distributed shared file system, and the method includes the following steps.
  • the DNS server When receiving a connection request initiated by a client, the DNS server allocates a target load balancer to the client according to the current load status of each load balancer, so as to mount the client to the target load balancer on the device.
  • the target load balancer receives the access request from the client, and identifies the current weight coefficient of each storage container, so as to schedule the access request to the target storage container according to the weight coefficient.
  • the target storage container parses the access request scheduled by the target load balancer, so as to process the data pointed to by the access request in the distributed storage cluster.
  • the technical solution provided by this application can distribute the user's data traffic to multiple storage containers for processing by means of multi-level load balancing. While utilizing idle resources, it reduces the overload of a certain node. possibility, thereby improving the overall stability of the system.
  • the DNS server can mount the client to the corresponding target load balancer through the first-level load balancing by judging the current load state of the load balancer.
  • the load balancer analyzes the weight coefficient of the back-end storage container, so as to schedule the client's traffic to the target storage container through the second-level load balancing.
  • the client's access request is parsed by the storage container, and finally the data can be stored or read in the distributed storage cluster.
  • multi-level load balancing method can reasonably allocate user data traffic between the load balancer and the storage container, and avoid data exceptions caused by the overload of a load balancer or a storage container. , which improves the overall stability of the distributed shared file system.
  • each embodiment can be implemented by means of software plus a necessary general hardware platform, and certainly can also be implemented by hardware.
  • the above-mentioned technical solutions can be embodied in the form of software products in essence or the parts that make contributions to the prior art, and the computer software products can be stored in a computer-readable storage medium, such as a read-only memory (Read -Only Memory, referred to as ROM)/Random Access Memory (Random Access Memory, referred to as RAM), magnetic disks, optical disks, etc., including several instructions to make a computer device (can be a personal computer, server, or network device, etc.) The methods described in various embodiments or portions of embodiments are performed.

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Software Systems (AREA)
  • Physics & Mathematics (AREA)
  • General Engineering & Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Data Mining & Analysis (AREA)
  • Databases & Information Systems (AREA)
  • Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
  • Computer And Data Communications (AREA)

Abstract

一种分布式共享文件系统及其数据处理方法，所述分布式共享文件系统包括DNS服务器、负载均衡器、存储容器以及分布式存储集群，其中：所述DNS服务器用于在接收到客户端发起的域名解析请求时，根据各个所述负载均衡器当前的负载状态，为所述客户端分配目标负载均衡器，以将所述客户端挂载至所述目标负载均衡器上(S1)；所述负载均衡器用于接收客户端的访问请求，并识别各个所述存储容器当前的权重系数，以根据所述权重系数将所述访问请求调度至目标存储容器处(S3)；所述存储容器用于解析所述负载均衡器调度的访问请求，以在所述分布式存储集群中对所述访问请求指向的数据进行处理(S5)。

Description

一种分布式共享文件系统及其数据处理方法
交叉引用
本申请要求于2020年11月20日递交的名称为“一种分布式共享文件系统及其数据处理方法”、申请号为202011310747.9的中国专利申请的优先权，其通过引用被全部并入本申请。
技术领域
本申请涉及互联网技术领域，特别涉及一种分布式共享文件系统及其数据处理方法。
背景技术
随着云计算、云存储等领域的不断发展以及数据的爆炸性增长，用户对于分布式存储的性能要求越来越高。然而现有的分布式共享文件系统，应对较大的数据流量时稳定性不够好。因此目前需要一种更加稳定的分布式共享文件系统。
发明内容
本申请的目的在于提供一种分布式共享文件系统及其数据处理方法，能够提高分布式共享文件系统的稳定性。
为实现上述目的，本申请实施例提供一种分布式共享文件系统，所述分布式共享文件系统包括域名系统(Domain Name System，简称DNS)服务器、负载均衡器、存储容器以及分布式存储集群，其中：所述DNS服务器，用于在接收到客户端发起的域名解析请求时，根据各个所述负载均衡器当前的负载状态，为所述客户端分配目标负载均衡器，以将所述客户端挂载至所述目标负载均衡器上；所述负载均衡器，用于接收客户端的访问请求，并识别各个所述存储容器当前的权重系数，以根据所述权重系数将所述访问请求调度至目标存储容器处；所述存储容器，用于解析所述负载均衡器调度的访问请求，以在所述 分布式存储集群中对所述访问请求指向的数据进行处理。
为实现上述目的，本申请实施例还提供一种分布式共享文件系统的数据处理方法，所述方法包括：DNS服务器在接收到客户端发起的域名解析请求时，根据各个负载均衡器当前的负载状态，为所述客户端分配目标负载均衡器，以将所述客户端挂载至所述目标负载均衡器上；所述目标负载均衡器接收所述客户端的访问请求，并识别各个存储容器当前的权重系数，以根据所述权重系数将所述访问请求调度至目标存储容器处；所述目标存储容器解析所述目标负载均衡器调度的访问请求，以在分布式存储集群中对所述访问请求指向的数据进行处理。
由上可见，本申请提供的技术方案，可以通过多级负载均衡的方式，将用户的数据流量分发到多个存储容器进行处理，在利用了闲置资源的同时，降低了某个节点负载过高的可能性，从而提高了系统整体的稳定性。具体地，DNS服务器可以通过对负载均衡器当前的负载状态进行判断，从而通过第一级的负载均衡，将客户端挂载至对应的目标负载均衡器上。负载均衡器通过对后端存储容器的权重系数进行分析，从而通过第二级的负载均衡，将客户端的流量调度至目标存储容器处。通过存储容器对客户端的访问请求进行解析，最终能够在分布式存储集群中对数据进行存储或者读取的处理。可见，引入多级负载均衡的方式，能够将用户的数据流量在负载均衡器和存储容器之间进行合理的分配，避免某个负载均衡器或者某个存储容器的负载过高而引发的数据异常，提高了分布式共享文件系统整体的稳定性。
附图说明
为了更清楚地说明本申请实施例中的技术方案，下面将对实施例描述中所需要使用的附图作简单地介绍，显而易见地，下面描述中的附图仅仅是本申请的一些实施例，对于本领域普通技术人员来讲，在不付出创造性劳动的前提下，还可以根据这些附图获得其他的附图。
图1是本申请实施例中分布式共享文件系统的系统架构示意图；
图2是本申请实施例中分布式共享文件系统的网络配置示意图；
图3是本申请实施例中分布式共享文件系统的数据处理方法示意图。
具体实施方式
为使本申请的目的、技术方案和优点更加清楚，下面将结合附图对本申请实施例作详细地描述。
请参阅图1，本申请实施例提供了一种分布式共享文件系统，可以包括DNS服务器、负载均衡器、存储容器(如，基于docker技术实现的容器)以及分布式存储集群(如，ceph分布式文件系统)，其中，负载均衡器和存储容器可部署在同一服务设备或集群中，也可分别部署在不同的服务设备或集群中，一个负载均衡器和至少一个存储容器可构成一个独立的服务单元，相互之间基于网络通信连接。存储容器与分布式存储集群基于网络通信连接。
本申请提供的分布式共享文件系统提供用户可访问操作的应用服务，用户可以通过客户端浏览器或本地安装的应用程序，访问该应用服务，基于应用服务创建文件存储服务，并将文件存储服务器挂载到该分布式共享文件系统，从而向分布式共享文件系统发起各项数据访问请求，从而实现将数据存储至分布式存储集群，以及对存储数据的读写操作。
具体地，当用户通过客户端创建了文件存储服务时，分布式共享文件系统可为该文件存储服务创建一个挂载点，并反馈挂载点信息至客户端，其中，一个挂载点对应至少一个服务单元，挂载点信息中包含分布式共享文件系统为该文件存储服务设定的唯一识别的域名；同时，当系统完成了服务单元的创建后，可将服务单元中的负载均衡器的IP地址、该挂载点域名及其映射关系配置在DNS服务器中。当客户端执行挂载操作时，需先向DNS服务器请求对挂载域名的解析。
当客户端发起针对接收到的挂载域名发起DNS解析请求时，DNS服务器可以基于挂载域名与各个负载均衡器的IP地址之间的映射关系，将对应的负载均衡器的IP地址反馈给客户端，从而使得客户端能够基于该IP地址与对应的负载均衡器建立通信连接，实现文件存储服务的挂载。
不同于普通的DNS服务器，本实施例中的DNS服务器在进行域名解析时，可以执行第一级的负载均衡策略。具体地，DNS服务器可以监控各个负载均衡器当前的负载状态。该负载状态可以通过负载均衡器上的运行参数来表征。例如，该负载状态可以通过负载均衡器的连接数表征，连接数越多，负载越大。当然，在实际应用中，还可以根据需要或者参数获取难易程度，通过其它的参 数来表征负载状态。当一个挂载域名对应有多个负载均衡器的IP地址时，DNS服务器在获取到各个负载均衡器当前的负载状态之后，便可以根据获取到的负载状态，从多个负载均衡器中选择负载较小的目标负载均衡器，并将该目标负载均衡器的IP地址反馈给客户端，从而将客户端挂载至目标负载均衡器上。
在一个实施例中，当负载均衡器当前的负载状态通过负载均衡器的连接数表征时，DNS服务器中的负载均衡器确定单元可以识别各个负载均衡器中连接数最小的负载均衡器，并将所述连接数最小的负载均衡器作为分配的所述目标负载均衡器。当然，在实际应用中，还可以筛选出连接数小于或者等于某个固定数值的多个负载均衡器，然后从这些负载均衡器中随机选择一个作为目标负载均衡器。
在分布式共享文件系统中，负载均衡器和存储容器可以构成服务单元。在分布式共享文件系统中，针对同一个文件存储服务创建的服务单元的数量可以是一个或者多个。在一个服务单元中，可以包括一个负载均衡器与一个或者多个存储容器，当文件存储服务挂载到负载均衡器后，客户端的访问请求将会先发送至该负载均衡器，并由负载均衡器转发至同一服务单元中的存储容器。负载均衡器在可以执行第二级的负载均衡策略，选择其中的一个存储容器来处理客户端的访问请求。其中，第二级的负载均衡策略可包含结合权重系数以及各存储容器当前实际处理的数据流量来选择存储容器。
具体地，可以根据各存储容器的服务性能，预先为其设置对应的权重系数。该权重系数可以表征处理数据流量的能力，权重系数越高，处理数据流量的能力也通常越高。其中服务性能可基于最大带宽承载量、CPU及其他影响流量处理能力的因素确定出。进而根据各存储容器的权重系数得到权重系数比，例如，服务单元中包含3个存储容器，这三个存储容器的权重系数比例为5：4：2。
当负载均衡器基于第二级的负载均衡策略选择存储容器时，可获取服务单元中各存储容器的权重系数比例和各存储容器当前正在处理的数据流量，确定目标存储容器，确定的原则就是，在将访问请求调度至目标存储容器处之后，能够使得各个存储容器变化后的现有数据流量的流量比例更接近于上述的系数比例。例如，上述的例子中，可以将第三个存储容器作为目标存储容器。将新的访问请求调度至第三个存储容器之后，变化后的现有数据流量的流量比例则 可以变为10：7：3，从而比原先的10：7：2更加接近于权重系数比例5：4：2，如此一来，可保证数据流量合理的分配至各存储容器，避免部分存储容器满负荷，而其他却空闲的状态。
在一个实施例中，各个服务单元可以由系统中的一个守护进程进行管理。该守护进程可以监控服务单元中各个存储容器的状态参数。在实际应用中，存储容器的状态参数可以包括存储容器的工作状态、带宽、连接数等信息。守护进程根据获取到的状态参数。
具体地，存储容器的工作状态可以通过active、error等状态码来表示。其中，active可以表征工作正常，error可以表征工作异常。当然，根据实际需要，状态码还可以表征未响应、复位中等状态。针对工作异常的存储容器，守护进程可以将出现异常的存储容器从对应的服务单元中剔除，并在对应的服务单元中新增对应数量的存储容器，从而使得服务单元中可用的存储容器的数量能够稳定在一个固定值。
在本实施例中，存储容器运行网络附属存储(Network Attached Storage，简称NAS)服务，对分布式存储集群进行封装并暴露给用户提供网络存储服务，通过NAS服务，可以对接收到的客户端的访问请求进行解析，并基于解析后的内容对分布式存储集群中的数据进行读、写、增、删等处理。分布式存储集群作为负载均衡的后端，以集群服务的方式提供高可用(即减少停工时间，保持其服务的高度可用性)与负载均衡(即分摊流量，增加吞吐，加强数据处理能力)的服务能力。
在一个实施例中，当客户端的文件存储服务需要提升服务性能时，可在相应的挂载点下新增服务单元，同时可以将新增的服务单元中负载均衡器的IP地址上报至DNS服务器中。这样，当DNS服务器执行第一级的负载均衡策略时，可以有机会将客户端挂载至新增的负载均衡器上，通过这种方式，可以快速实现分布式共享文件系统的拓展，满足扩大服务规模的需求，同时这种线性拓展的方式，可实现系统服务性能的近似线性提升。同样的，若需要对挂载点的服务单元进行缩减，可先将待缩减的服务单元中负载均衡器的IP地址从DNS服务器中删除，使得不会有新的访问请求被挂载到该负载均衡器，当存储容器上的流量处理完成后，即可将该服务单元删除，释放处理资源。如此，可实现系统的高拓展性。
在本申请的实施例中，分布式共享文件系统中还包含管理服务器，用于管理和协调DNS服务器和各服务设备，所述管理服务器分别与DNS服务器以及用于部署负载均衡器和存储容器的服务设备基于网络通信连接，并根据应用服务接收到的用户创建文件存储服务的请求，生成对应的指令发送至所述服务设备，以指示服务设备对应创建负载均衡器和存储容器，同时为新创建的负载均衡器配置的IP地址；管理服务器可为文件存储服务配置的挂载域名等挂载信息，并反馈给用户客户端，同时将挂载域名、负载均衡器的IP地址及其对应关系发送至DNS服务器。当检测到文件存储服务需要扩容或缩容时，发送对应的控制指令至所述服务设备，以指示服务设备增加或减少该文件存储服务对应的服务单元，同时将新增或减少的服务单元中负载均衡器的IP地址上报至DNS服务器，以更新DNS服务器上挂载域名与IP地址的记录。
请参阅图2，在针对客户端的文件存储服务挂载点创建对应的服务单元时分布式共享文件系统，可对服务单元中负载均衡器与存储容器的网络进行设置。
其中，负载均衡器与对应的客户端可以处于同一个子网中，如此，客户端在挂载至负载均衡器上后，可以直接访问负载均衡器。此外，同一个负载均衡器仅可以绑定至一个子网中。通过子网划分的方式，不同用户的客户端挂载的负载均衡器可以处于不同的子网内，从而实现用户之间的隔离。
各个存储容器与分布式存储集群可以位于同一个网络中，如位于同一个万兆存储，这样，存储容器与分布式存储集群之间可以进行快速的数据交互。此外，各个存储容器还可以处于同一个管理网络中，该管理网络可以由守护进程负责管理。
在分布式共享文件系统中，DNS服务器可以处于单独的一个网络环境中，与用户的客户端所在的子网可以通过路由进行连接，从而为各个用户的客户端提供域名解析服务。
在实际应用中，负载均衡器还可以通过主备的方式保持高可用。每个主负载均衡器均可以具备自身的备负载均衡器，当主负载均衡器无法提供服务时，备负载均衡器可以升级为主负载均衡器并继续提供服务，以保证系统的高可用性和稳定性。
在本申请的实施例中，分布式共享文件系统在为客户端的文件存储服务创建对应的服务单元时，可参照上述网络划分方式为对负载均衡器和存储容器 的网络进行设置，具体包含，将负载均衡器划分到与客户端相同的子网中，将存储容器划分到与分布式存储集群所在网络中。
请参阅图3，本申请实施例还提供了一种分布式共享文件系统的数据处理方法，所述方法包括以下步骤。
S1：DNS服务器在接收到客户端发起的连接请求时，根据各个负载均衡器当前的负载状态，为所述客户端分配目标负载均衡器，以将所述客户端挂载至所述目标负载均衡器上。
S3：所述目标负载均衡器接收所述客户端的访问请求，并识别各个存储容器当前的权重系数，以根据所述权重系数将所述访问请求调度至目标存储容器处。
S5：所述目标存储容器解析所述目标负载均衡器调度的访问请求，以在分布式存储集群中对所述访问请求指向的数据进行处理。
由上可见，本申请提供的技术方案，可以通过多级负载均衡的方式，将用户的数据流量分发到多个存储容器进行处理，在利用了闲置资源的同时，降低了某个节点负载过高的可能性，从而提高了系统整体的稳定性。具体地，DNS服务器可以通过对负载均衡器当前的负载状态进行判断，从而通过第一级的负载均衡，将客户端挂载至对应的目标负载均衡器上。负载均衡器通过对后端存储容器的权重系数进行分析，从而通过第二级的负载均衡，将客户端的流量调度至目标存储容器处。通过存储容器对客户端的访问请求进行解析，最终能够在分布式存储集群中对数据进行存储或者读取的处理。可见，引入多级负载均衡的方式，能够将用户的数据流量在负载均衡器和存储容器之间进行合理的分配，避免某个负载均衡器或者某个存储容器的负载过高而引发的数据异常，提高了分布式共享文件系统整体的稳定性。
本说明书中的各个实施例均采用递进的方式描述，各个实施例之间相同相似的部分互相参见即可，每个实施例重点说明的都是与其他实施例的不同之处。尤其，针对方法的实施例来说，均可以参照前述系统的实施例的介绍对照解释。
通过以上的实施例的描述，本领域的技术人员可以清楚地了解到各实施例可借助软件加必需的通用硬件平台的方式来实现，当然也可以通过硬件。基于这样的理解，上述技术方案本质上或者说对现有技术做出贡献的部分可以以 软件产品的形式体现出来，该计算机软件产品可以存储在计算机可读存储介质中，如只读存储器(Read-Only Memory，简称ROM)/随机存取存储器(Random Access Memory，简称RAM)、磁碟、光盘等，包括若干指令用以使得一台计算机设备(可以是个人计算机，服务器，或者网络设备等)执行各个实施例或者实施例的某些部分所述的方法。
以上所述仅为本申请的部分实施例，并不用以限制本申请，凡在本申请的精神和原则之内，所作的任何修改、等同替换、改进等，均应包含在本申请的保护范围之内。

Claims (10)

  1. 一种分布式共享文件系统，包括域名系统DNS服务器、负载均衡器、存储容器以及分布式存储集群，其中：
    所述DNS服务器，用于在接收到客户端发起的域名解析请求时，根据各个所述负载均衡器当前的负载状态，为所述客户端分配目标负载均衡器，以将所述客户端挂载至所述目标负载均衡器上；
    所述负载均衡器，用于接收客户端的访问请求，并识别各个所述存储容器当前的权重系数，以根据所述权重系数将所述访问请求调度至目标存储容器处；
    所述存储容器，用于解析所述负载均衡器调度的访问请求，以在所述分布式存储集群中对所述访问请求指向的数据进行处理。
  2. 根据权利要求1所述的系统，其中，所述分布式共享文件系统还用于在所述客户端创建文件存储服务时，为所述文件存储服务部署至少一个服务单元，并为所述文件存储服务设定挂载域名；其中，所述服务单元中包括一个负载均衡器和至少一个存储容器。
  3. 根据权利要求2所述的系统，其中，所述分布式共享文件系统还用于向所述客户端反馈所述挂载域名，并将所述挂载域名与所述服务单元中负载均衡器的IP地址之间的映射关系配置于所述DNS服务器中。
  4. 根据权利要求1所述的系统，其中，所述负载均衡器当前的负载状态通过负载均衡器的连接数表征；所述DNS服务器包括：
    负载均衡器确定单元，用于识别各个所述负载均衡器中连接数最小的负载均衡器，并将所述连接数最小的负载均衡器作为分配的所述目标负载均衡器。
  5. 根据权利要求1所述的系统，其中，所述负载均衡器包括：
    调度单元，用于从可选择的存储容器中识别出权重系数最大的存储容器，并将所述访问请求调度至所述权重系数最大的存储容器处；或者计算可选择的各个存储容器的权重系数的系数比例，并计算所述各个存储容器的现有数据流量的流量比例，根据所述系数比例和所述流量比例确定所述目标存储容器，以 使得将所述访问请求调度至所述目标存储容器处之后，所述各个存储容器的现有数据流量的流量比例更接近于所述系数比例。
  6. 根据权利要求1所述的系统，其中，若在所述分布式共享文件系统中新增服务单元，将新增的服务单元中负载均衡器的IP地址上报至所述DNS服务器中。
  7. 根据权利要求1所述的系统，其中，若在所述分布式共享文件系统中缩减服务单元，将待缩减的服务单元中负载均衡器的IP地址从所述DNS服务器中删除，并在所述待缩减的服务单元中存储容器上的流量处理完成后，将所述待缩减的服务单元删除。
  8. 根据权利要求6或7所述的系统，其中，各个所述服务单元由守护进程管理，其中，若服务单元中的存储容器出现异常，所述守护进程将出现异常的存储容器从对应的服务单元中剔除，并在所述对应的服务单元中新增对应数量的存储容器。
  9. 根据权利要求1所述的系统，其中，同一个负载均衡器仅绑定至一个子网中，并且不同用户的客户端通过不同的子网进行隔离。
  10. 一种分布式共享文件系统的数据处理方法，包括：
    DNS服务器在接收到客户端发起的域名解析请求时，根据各个负载均衡器当前的负载状态，为所述客户端分配目标负载均衡器，以将所述客户端挂载至所述目标负载均衡器上；
    所述目标负载均衡器接收所述客户端的访问请求，并识别各个存储容器当前的权重系数，以根据所述权重系数将所述访问请求调度至目标存储容器处；
    所述目标存储容器解析所述目标负载均衡器调度的访问请求，以在分布式存储集群中对所述访问请求指向的数据进行处理。
PCT/CN2021/071098 2020-11-20 2021-01-11 一种分布式共享文件系统及其数据处理方法 WO2022105048A1 (zh)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
CN202011310747.9A CN112445774A (zh) 2020-11-20 2020-11-20 一种分布式共享文件系统及其数据处理方法
CN202011310747.9 2020-11-20

Publications (1)

Publication Number Publication Date
WO2022105048A1 true WO2022105048A1 (zh) 2022-05-27

Family

ID=74737098

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2021/071098 WO2022105048A1 (zh) 2020-11-20 2021-01-11 一种分布式共享文件系统及其数据处理方法

Country Status (2)

Country Link
CN (1) CN112445774A (zh)
WO (1) WO2022105048A1 (zh)

Cited By (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN115174390A (zh) * 2022-07-19 2022-10-11 杭州视洞科技有限公司 一种基于openelb、openresty和负载均衡器的业务负载方法
CN115665263A (zh) * 2022-10-19 2023-01-31 上海浦东发展银行股份有限公司 一种流量调拨方法、装置、服务器及存储介质

Families Citing this family (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN113946294A (zh) * 2021-10-29 2022-01-18 蜂巢科技(南通)有限公司 一种分布式储存系统及其数据处理方法
CN114089917A (zh) * 2021-11-19 2022-02-25 中国电信集团系统集成有限责任公司 分布式对象存储集群及其扩容方法、装置及电子设备
CN114844912B (zh) * 2022-04-22 2022-12-09 北京志凌海纳科技有限公司 数据链路分配方法、装置及分布式块存储系统

Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2000112908A (ja) * 1998-10-06 2000-04-21 Toshiba Corp 負荷分散ｄｎｓシステム
CN102436401A (zh) * 2011-12-16 2012-05-02 北京邮电大学 负载均衡系统及方法
CN104202386A (zh) * 2014-08-27 2014-12-10 四川长虹电器股份有限公司 一种高并发量分布式文件系统及其二次负载均衡方法
CN106230992A (zh) * 2016-09-28 2016-12-14 中国银联股份有限公司 一种负载均衡方法和负载均衡节点
CN108933829A (zh) * 2018-07-10 2018-12-04 浙江数链科技有限公司 一种负载均衡方法及装置

Patent Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2000112908A (ja) * 1998-10-06 2000-04-21 Toshiba Corp 負荷分散ｄｎｓシステム
CN102436401A (zh) * 2011-12-16 2012-05-02 北京邮电大学 负载均衡系统及方法
CN104202386A (zh) * 2014-08-27 2014-12-10 四川长虹电器股份有限公司 一种高并发量分布式文件系统及其二次负载均衡方法
CN106230992A (zh) * 2016-09-28 2016-12-14 中国银联股份有限公司 一种负载均衡方法和负载均衡节点
CN108933829A (zh) * 2018-07-10 2018-12-04 浙江数链科技有限公司 一种负载均衡方法及装置

Cited By (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN115174390A (zh) * 2022-07-19 2022-10-11 杭州视洞科技有限公司 一种基于openelb、openresty和负载均衡器的业务负载方法
CN115665263A (zh) * 2022-10-19 2023-01-31 上海浦东发展银行股份有限公司 一种流量调拨方法、装置、服务器及存储介质

Also Published As

Publication number Publication date
CN112445774A (zh) 2021-03-05

Similar Documents

Publication Publication Date Title
WO2022105048A1 (zh) 一种分布式共享文件系统及其数据处理方法
US10171567B2 (en) Load balancing computer device, system, and method
KR100984384B1 (ko) 클러스터 노드들을 권위적 도메인 네임 서버들로서사용하여 액티브 부하 조절을 하는 시스템, 네트워크 장치,방법, 및 컴퓨터 프로그램 생성물
WO2022002209A1 (zh) 数据传输方法和代理服务器、存储介质及电子装置
US6965930B1 (en) Methods, systems and computer program products for workload distribution based on end-to-end quality of service
US6963917B1 (en) Methods, systems and computer program products for policy based distribution of workload to subsets of potential servers
US7631098B2 (en) Method, system and program product for optimized concurrent data download within a grid computing environment
EP2403219B1 (en) Method for name to address resolution
EP2088719B1 (en) Method and device for distributing file data
US8499086B2 (en) Client load distribution
US8578053B2 (en) NAS load balancing system
US7685312B1 (en) Resource location by address space allocation
JP2003256310A (ja) サーバ負荷分散システム、サーバ負荷分散装置、コンテンツ管理装置、及びサーバ負荷分散プログラム
CN107105013B (zh) 文件的处理方法、服务器、终端和系统
JP2007133896A (ja) サーバ負荷分散システム、サーバ負荷分散装置、コンテンツ管理装置、及びサーバ負荷分散プログラム
US20220318071A1 (en) Load balancing method and related device
CN109688229A (zh) 一种负载均衡集群下会话保持系统
JP2013090072A (ja) サービス提供システム
US9760370B2 (en) Load balancing using predictable state partitioning
CN114844912B (zh) 数据链路分配方法、装置及分布式块存储系统
JP2002259354A (ja) ネットワークシステム及び負荷分散方法
US7711780B1 (en) Method for distributed end-to-end dynamic horizontal scalability
CN214202379U (zh) 一种分布式共享文件系统
JP2007524877A (ja) データ記憶システム
KR100450605B1 (ko) 웹-어플리케이션 서버 및 그의 동적 컨텐츠 제공 방법

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 21893210

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 21893210

Country of ref document: EP

Kind code of ref document: A1