WO2022002209A1 - 数据传输方法和代理服务器、存储介质及电子装置 - Google Patents

数据传输方法和代理服务器、存储介质及电子装置 Download PDF

Info

Publication number
WO2022002209A1
WO2022002209A1 PCT/CN2021/104060 CN2021104060W WO2022002209A1 WO 2022002209 A1 WO2022002209 A1 WO 2022002209A1 CN 2021104060 W CN2021104060 W CN 2021104060W WO 2022002209 A1 WO2022002209 A1 WO 2022002209A1
Authority
WO
WIPO (PCT)
Prior art keywords
target
server
request
file
attribute information
Prior art date
Application number
PCT/CN2021/104060
Other languages
English (en)
French (fr)
Inventor
全锐
文永兴
唐文彬
马安伟
Original Assignee
中兴通讯股份有限公司
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 中兴通讯股份有限公司 filed Critical 中兴通讯股份有限公司
Priority to EP21834267.3A priority Critical patent/EP4178134A4/en
Priority to JP2023500001A priority patent/JP2023532947A/ja
Publication of WO2022002209A1 publication Critical patent/WO2022002209A1/zh

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/10—File systems; File servers
    • G06F16/18—File system types
    • G06F16/182—Distributed file systems
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L63/00—Network architectures or network communication protocols for network security
    • H04L63/02—Network architectures or network communication protocols for network security for separating internal from external traffic, e.g. firewalls
    • H04L63/0281—Proxies
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L67/00—Network arrangements or protocols for supporting network services or applications
    • H04L67/01—Protocols
    • H04L67/06—Protocols specially adapted for file transfer, e.g. file transfer protocol [FTP]
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L67/00—Network arrangements or protocols for supporting network services or applications
    • H04L67/01—Protocols
    • H04L67/10—Protocols in which an application is distributed across nodes in the network
    • H04L67/1095—Replication or mirroring of data, e.g. scheduling or transport for data synchronisation between network nodes
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L67/00—Network arrangements or protocols for supporting network services or applications
    • H04L67/01—Protocols
    • H04L67/10—Protocols in which an application is distributed across nodes in the network
    • H04L67/1097—Protocols in which an application is distributed across nodes in the network for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS]
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L67/00—Network arrangements or protocols for supporting network services or applications
    • H04L67/2866—Architectures; Arrangements
    • H04L67/289—Intermediate processing functionally located close to the data consumer application, e.g. in same machine, in same home or in same sub-network
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L67/00—Network arrangements or protocols for supporting network services or applications
    • H04L67/50—Network services
    • H04L67/56—Provisioning of proxy services
    • H04L67/564—Enhancement of application control based on intercepted application data

Definitions

  • the embodiments of the present disclosure relate to the technical field of file transmission under 5G intelligent networks, and in particular, to a data transmission method, a proxy server, a storage medium, and an electronic device.
  • FTP distributed server according to the scale can expand multiple FTP server instances to support larger scale.
  • the FTP reverse proxy is used to solve the network connectivity problem to ensure that the FTP client can transfer files between the FTP server that is not directly connected.
  • the single-instance FTP service has only one FTP server instance, and the upper limit of the support capacity is limited by the single-instance processing capability, which cannot meet the large-scale multi-connection scenario, and the existing implementation of multi-instance FTP server cannot solve high concurrency. Efficiency issues in connection scenarios.
  • Embodiments of the present disclosure provide a data transmission method, a proxy server, a storage medium, and an electronic device, so as to at least solve the problem of low file transmission efficiency in the related art.
  • a data transmission method including: a proxy server receiving a target request sent by a client, wherein the target request is a viewing request or a transmission request, and the viewing request is used to request viewing data
  • the target attribute information of the target file in the server is used to request to transfer the target file to the data server or obtain the target file from the data server; if the target request is a viewing request, the The proxy server obtains the target attribute information of the target file from the saved copy of the attribute information, and returns it to the client, wherein the attribute information copy records the target file identifier of the target file and the The corresponding relationship of target attribute information; in the case that the target request is a transmission request, the proxy server sends the target request to the target server in the data server, wherein the data server is a double-stacked
  • the FTPServer multi-instance distributed cluster server the data server includes a plurality of first servers, and the proxy server records the range of processing FTP users reported by each of the first servers.
  • a proxy server comprising: a receiving unit configured to receive a target request sent by a client, wherein the target request is a viewing request or a transmission request, and the viewing request is used for requesting to view the target attribute information of the target file in the data server, and the transmission request is used to request to transfer the target file to the data server or obtain the target file from the data server; the first obtaining unit is set to be in the target file When the request is a viewing request, obtain the target attribute information of the target file from the saved copy of the attribute information, and return it to the client, wherein the copy of the attribute information records the attribute information of the target file.
  • the corresponding relationship between the target file identifier and the target attribute information; the first sending unit is configured to send the target request to the target server in the data server when the target request is a transmission request, wherein,
  • the data server is a double-stacked FTPServer multi-instance distributed cluster server, the data server includes a plurality of first servers, and the proxy server records the range of processing FTP users reported by each of the first servers.
  • a computer-readable storage medium where a computer program is stored in the computer-readable storage medium, wherein the computer program is configured to execute any one of the above methods when running steps in the examples.
  • an electronic device comprising a memory and a processor, wherein the memory stores a computer program, the processor is configured to run the computer program to execute any of the above Steps in Method Examples.
  • FIG. 1 is a block diagram of a hardware structure of a mobile terminal according to a data transmission method according to an embodiment of the present disclosure
  • FIG. 2 is a flowchart of a data transmission method according to an embodiment of the present disclosure
  • FIG. 3 is a schematic diagram of the architecture and flow of a data transmission method according to an embodiment of the present disclosure
  • FIG. 4 is a flow chart of the shunting and addressing of a data transmission method according to an embodiment of the present disclosure
  • FIG. 5 is a flowchart of a distributed coordination algorithm of a data transmission method according to an embodiment of the present disclosure
  • FIG. 6 is a schematic diagram of distributed FTP cluster affinity and HA according to a data transmission method according to an embodiment of the present disclosure
  • FIG. 7 is a structural block diagram of a proxy server according to an embodiment of the present disclosure.
  • File transfer is a general-purpose technology.
  • file transfer is ubiquitous.
  • southbound out-of-band network elements due to the massive access characteristics of 5G, the number of access network elements has greatly increased.
  • File transfer is a common technology in network element performance management and version management. Since security can be obtained from the physical network of access network elements Therefore, the simple and efficient file transfer protocol FTP server has become a reasonable choice for file transfer, and a large number of connected network elements cause a large number of FTP concurrent connection requests.
  • the 5G low-latency intelligent network further requires FTP to achieve high-efficiency file transfer while supporting massive connections.
  • Distributed refers to the cooperation of multiple systems to complete a specific task. Distributed is to solve the problem of centralized management. It is too slow to superimpose all tasks on one node for processing. So split a big problem into multiple small problems, solve them separately, and finally work together.
  • the main work of distributed is to decompose tasks and disassemble functions.
  • the main usage scenario of a cluster is to share the pressure of requests, that is, to deploy the same application on several servers to share client requests.
  • the gateway server can connect to the external network and the internal network at the same time, and the internal network FTP users connect to the FTP proxy and access the external real FTP server through it.
  • FIG. 1 is a block diagram of a hardware structure of a mobile terminal according to a data transmission method according to an embodiment of the present disclosure.
  • the mobile terminal may include one or more (only one is shown in FIG.
  • processor 102 may include, but is not limited to, a processing device such as a microprocessor MCU or a programmable logic device FPGA) and a memory 104 configured to store data, wherein the above-mentioned mobile terminal may further include a transmission device 106 and an input/output device 108 configured as a communication function.
  • a processing device such as a microprocessor MCU or a programmable logic device FPGA
  • a memory 104 configured to store data
  • the above-mentioned mobile terminal may further include a transmission device 106 and an input/output device 108 configured as a communication function.
  • FIG. 1 is only a schematic diagram, which does not limit the structure of the above-mentioned mobile terminal.
  • the mobile terminal may also include more or fewer components than those shown in FIG. 1 , or have a different configuration than that shown in FIG. 1 .
  • the memory 104 may be configured to store computer programs, for example, software programs and modules of application software, such as computer programs corresponding to the data transmission methods in the embodiments of the present disclosure.
  • the processor 102 executes the computer programs stored in the memory 104 by running the computer programs.
  • Various functional applications and data processing implement the above-mentioned data transmission method.
  • Memory 104 may include high-speed random access memory, and may also include non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory.
  • the memory 104 may further include memory located remotely from the processor 102, and these remote memories may be connected to the mobile terminal through a network. Examples of such networks include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.
  • Transmission means 106 are arranged to receive or transmit data via a network.
  • the specific example of the above-mentioned network may include a wireless network provided by a communication provider of the mobile terminal.
  • the transmission device 106 includes a network adapter (Network Interface Controller, NIC for short), which can be connected to other network devices through a base station so as to communicate with the Internet.
  • the transmission device 106 may be a radio frequency (Radio Frequency, RF for short) module, which is configured to communicate with the Internet in a wireless manner.
  • RF Radio Frequency
  • FIG. 2 is a flowchart of a data transmission method according to an embodiment of the present disclosure. As shown in FIG. 2 , the process includes the following steps:
  • Step S202 the proxy server receives the target request sent by the client, wherein the target request is a viewing request or a transmission request, the viewing request is used for requesting to view the target attribute information of the target file in the data server, and the transmission request is used for Request to transmit the target file to the data server or obtain the target file from the data server;
  • the target request is a viewing request or a transmission request
  • the viewing request is used for requesting to view the target attribute information of the target file in the data server
  • the transmission request is used for Request to transmit the target file to the data server or obtain the target file from the data server
  • Step S204 when the target request is a viewing request, the proxy server obtains the target attribute information of the target file from the saved copy of the attribute information, and returns it to the client, wherein the The corresponding relationship between the target file identifier of the target file and the target attribute information is recorded in the attribute information copy;
  • Step S206 in the case that the target request is a transmission request, the proxy server sends the target request to a target server in the data server, wherein the data server is a double-stacked FTPServer multi-instance distribution.
  • the data server includes a plurality of first servers, and the proxy server records the scope of processing FTP users reported by each of the first servers.
  • the method further includes: acquiring file identifiers of all files in the data server and attribute information of all files; storing the data The file identifiers of all files in the server and the attribute information of all files are stored in the copy of the attribute information.
  • the method further includes: acquiring a correspondence between a user identifier and the first server, wherein different user identifiers correspond to different user identifiers the first server; save the corresponding relationship in the memory.
  • obtaining the attribute information of the target file from the copy of attribute information saved in the proxy server and returning it to the client includes: obtaining the attribute information the target file identifier of the target file in the target request; obtain the target attribute information corresponding to the target file identifier in the attribute information copy; and return the target attribute information to the client.
  • the method further includes: acquiring the target user identifier of the client; searching a memory in the target server corresponding to the target user ID.
  • the searching for the target server corresponding to the target user identifier in the memory includes: in the case where multiple servers corresponding to the target user identifier are found, selecting the server with the lowest load balance from the multiple servers
  • the target server is used as the target server, or the target server is determined from the plurality of servers by using a round-robin algorithm.
  • the method further includes: in the communication between the client and the target server A data transmission channel is established between the client and the target server; the target file is transmitted between the client and the target server through the data transmission channel.
  • the method further includes: in the case that the file size of the attribute information copy is smaller than a predetermined threshold, saving the attribute information copy in the memory; When the file size of the attribute information copy is greater than or equal to a predetermined threshold, the attribute information copy is stored in the memory.
  • the method further includes: Multiple FTPServer instances on a server form a one-layer stack structure; use a plurality of the first servers to form a two-layer stack structure to obtain the data server.
  • the method further includes: mounting the multiple FTPServer instances on the same first server in the same on the high-speed storage of the disk array LUN.
  • the file since in the process of file transmission, the file can be scheduled through the proxy server, the viewing request can directly return the target attribute information through the proxy server, and the transmission request can be sent to different target servers for processing. Therefore, The problem of low file transmission efficiency can be solved, and the effect of improving file transmission efficiency can be achieved.
  • the proxy server in the embodiment of the present disclosure may be an ftp reverse proxy server.
  • the first server may be a server node, and an FTP SERVER instance (FTPServer microservice) runs on the server node.
  • FTP SERVER instance FTP SERVER instance
  • a plurality of first servers form a double-layer stack structure to form a data server.
  • the FTPProxy reverse proxy is a primitive layer implementation of the FTP protocol. For the client, it is a standard FTP server, but many actual file operation requests are passed to the back-end FTPServer microservice cluster for processing.
  • the FTPProxy reverse proxy in the embodiment of the present disclosure focuses on local traffic distribution and cooperation with the FTPServer double-layer stack addressing design.
  • the reverse proxy is to shield the differences caused by multiple FTPServer distributed instances in the backend to the client, and cooperate with the offload addressing.
  • the FTP client does not need to pay attention to how many FTPServer instances there are and which server nodes they are running on. What is the address of each instance, whether an instance is stopped, etc.
  • the FTP client does not directly request the FTPServer microservice, but connects to the FTPProxy, which acts as a proxy for the FTPServer to provide FTP protocol capabilities.
  • FTPProxy provides a monitoring console, which can view the status of FTPProxy itself, all current FTP connection information, and the information and status of all ServerNode and FTPServer instances in the backend.
  • FTP reverse proxy can divide FTP requests into two categories by analyzing FTP requests:
  • View request file view request, such as: LIST, NLST, PWD, STAT. This type of request is mainly to view file information at the file system level.
  • Transfer request file operation and file transfer class requests, in addition to other requests of view class requests, such as: RETR, STOR, DELE, CWD, RNTO, etc. We collectively refer to this as a file transfer class of requests.
  • the reverse proxy server FTPProxy can provide the offload capability.
  • FTPProxy After receiving the request from the client, FTPProxy determines the type of the request. If it is a file viewing request, it will be processed locally and will not be passed to the back-end FTPServer. In order to support the processing of file viewing requests in FTPProxy, it is necessary to maintain a copy of the back-end FTPServer file directory tree and attribute information in FTPProxy. Because the single server node where each FTPServer in the backend is located shares the local file system, the file tree information for the same server node does not need to be merged in FTPProxy. The local file systems of each server node are not shared and are independent.
  • FTPProxy maintains a directory tree for all server nodes where the FTPServer is located on a server node basis. With subsequent FTP actions, the file system content of FTPServer will change, and the local cache of FTPProxy needs to be updated synchronously. Since all FTP requests go through the reverse proxy of FTPProxy, these changes can be made in the directory tree maintained by FTPProxy at the same time. Consistency of the content of the backend FTPServer file system.
  • the directory tree information maintained by FTPProxy can be stored in memory. If the backend FTPServer holds too many files, this part of the information can be persisted to storage to relieve memory pressure.
  • the client and the FTPServer instance will establish a file transfer channel directly.
  • the file transfer does not go through the proxy, but is directly connected and directly transferred, which reduces the one-time forwarding of FTPProxy, prevents FTPProxy from becoming a bottleneck, and further improves efficiency.
  • FTPProxy is responsible for distributing FTP requests to multiple FTPServer microservice instances on the backend. Because according to the double-stacked design of FTPServer, each FTPServer instance can support a certain range of FTP users. Therefore, FTPProxy determines which FTPServer instance to address to the backend according to the FTP user currently connected and logged in. Of course, there may be multiple such instances. , at this time, the request can be further balanced according to the load situation of the FTPServer to maximize the efficiency, or the Round-Robin polling algorithm can be used, which is simple and efficient.
  • micro-service architecture is mostly used for operation and maintenance management and network management. scale, and dual-machine cluster system HA. Therefore, the FTPServer in this solution, as a file transfer service component, is implemented in the form of a microservice container and supports multi-instance operation. At the same time, in order to improve the efficiency of file transfer, the FTPServer instance of the same server node shares the LUN of local efficient storage logical unit number, and multiple server nodes are stacked to form the entire FTP distributed cluster system to meet the requirements of high-concurrency and high-efficiency file transfer.
  • FTP server clusters should be implemented.
  • the embodiment of the present disclosure is designed to run the FTPServer in a container mode, which meets cloud-native requirements, supports multi-instance expansion, supports elastic scaling, and can pop up multiple instances to run according to configuration, load, such as the number of connections, connection activity, and CPU and memory usage.
  • a server node can run multiple FTPServer instances, and multiple server nodes can be deployed.
  • Such multiple FTPServer instances form an FTP distributed cluster, and the container management system, such as K8s, manages its life cycle and implements elastic scheduling and HA.
  • the disk array LUN attached to the local storage is shared with the FTPServer instance of the server node.
  • the biggest problem to be solved in an FTP server cluster is the consistency of the file system. Multiple FTPServer instances need to see a completely consistent file view, so that files can be uploaded and downloaded, and file operations can be performed correctly. In order to ensure this consistency, whether it is a file replication method or a distributed file system method, the efficiency problem is the most prominent problem. The increased IO and conflicts within the cluster greatly reduce the efficiency of file transfer. The embodiments of the present disclosure solve this problem.
  • the FTPServer instance of the same server is designed to share the disk array LUN attached to the local storage, and the distributed coordination mechanism is used to solve the file access conflict, so that the FTPServer instance of the same server sees a unified file system view .
  • This method utilizes the efficient IO of disk LUNs to support efficient file transfer.
  • the resource allocation of the server node is used to support multiple FTPServer instances.
  • the resource requirement of a single FTPServer instance is 1C2G (1 core CPU, 2G memory).
  • a high-end server can basically reach a configuration of 100C300G, which means that the same server node can support the horizontal expansion of 100 FTPServer instances, reaching 100,000 FTP. Concurrent connection support capability.
  • the main working directory of a specific FTP user is determined, and there is no overlap between files in different main working directories, that is, FTP operations on files are always limited to the main working directory of this user, so we
  • the main working directory of an FTP user can be bound to a fixed server node, and other FTP users can be bound to other server nodes with different main working directories. In this way, different server nodes handle different main working directories, and there is no requirement for file sharing views between multiple server nodes.
  • the division here is based on the home working directory of the FTP user, not the FTP user, because different FTP users can point to the same working directory. The net effect is that a specific working directory exists only on a specific server node.
  • the corresponding relationship between the main working directory of each FTP user and the server node is divided in advance, and which working directory a server node contains, it can process the file transfer request of the corresponding FTP user, and finally form the relationship between the server node and the FTP user.
  • All requests of FTP users are processed by a fixed server node (multiple FTPServer instances), which can continue to use the disk array LUN attached to the local storage, and can also use multiple FTPServer instances running on this server node. Clustering capabilities are supported for this user.
  • the FTPServer is divided into many categories.
  • the FTP user directory handled by a certain type of FTPServer (multiple instances) is determined, and does not intersect with other types of FTPServer, and runs on a fixed server node.
  • a single user running on a single server node can reach the order of magnitude of 100,000 concurrent connections, and different services generally use different FTP users. Therefore, combined with the multi-server node expansion in this solution, the entire system can provide more High order of magnitude concurrency support.
  • the FTP Servers of different server nodes can handle different FTP user ranges, and a specific FTPServer instance can only be run on a specific server node, and can be scaled on this node.
  • Other types of FTPServers in different working directory ranges can configure other labels to control their instances to run on other server nodes corresponding to the labels.
  • the number of server nodes corresponding to a label must be greater than 1.
  • Affinity means that once the first instance of a certain type of FTPServer is scheduled to a server node according to the labeling rules , other instances of this type of FTPServer will be scheduled to the same server node, even if there are multiple server nodes with the same label. This ensures that all similar FTPServer instances run on the same server node, can share the disk array LUN stored locally, and at the same time ensure HA.
  • a server node is Down, all FTPServer instances of this server node are switched to the one that conforms to the labeling rules as a whole. For some other server node, the disk array LUN is also reattached to the new server node.
  • This distributed coordination mechanism is realized through distributed locks across containers: when writing a file, a corresponding lock is generated according to the file name, and released after writing, and other read and write actions check whether the lock exists. Then other read and write actions to this file are not allowed.
  • This distributed coordination is a distributed lock based on temporary files and file sorting, and the lock is not reentrant.
  • a temporary directory is generated for each lock, corresponding to the specific files to be coordinated with reading and writing.
  • Each FTPServer instance to apply for a lock generates a temporary file in this directory according to its own container ID, and locks it. It is determined that if there are multiple containers applying at the same time (that is, there are multiple temporary files), the container that is successfully locked is selected according to the file name sorting fixed algorithm. In the end, only one container can acquire the lock, so as to achieve distributed coordination.
  • the response file viewing class operation is faster, and a large number of FTP concurrent connections can be supported.
  • the IO in the FTP server cluster is improved, and the file transfer efficiency is improved.
  • the FTPProxy reverse proxy is used to address multiple FTPServer instances in the backend, and file transfer requests can skip the proxy and directly connect and transfer directly, which improves the file transfer efficiency.
  • a double-stacked FTPServer multi-instance distributed cluster (data server) is designed.
  • affinity policy and distributed coordination similar FTPServer instances can share local magnetic LUNs for efficient storage, further improving file upload and download efficiency.
  • FIG. 3 is a schematic diagram of the structure and flow of the disclosed method.
  • FTPProxy can be deployed at the gateway to act as a reverse proxy and cooperate with the FTPServer instance that addresses the backend.
  • the working directory tree information in the local LUN is passed to FTPProxy for caching, so that it can quickly respond to file viewing requests.
  • FTPProxy maintains a working directory tree of all LUNs, which is dynamically updated with subsequent FTP operations.
  • FTPProxy provides a monitoring console to collect various information and status of the FTP distributed cluster system. It can view the status of FTPProxy itself, all current FTP connection information, and the information and status of all ServerNode and FTPServer instances in the backend.
  • the client FTPClient first connects to the FTPProxy for FTP user authentication, and the FTPProxy records the FTP user information corresponding to each FTP connection.
  • the FTPServer instance corresponding to the user is addressed, and the FTPClient and the FTPServer instance directly establish a file transfer channel for subsequent file transfer.
  • Each instance that forms an FTPServer distributed cluster runs on the corresponding ServerNode server node according to the labeling rules.
  • the server node is mounted with the corresponding disk array LUN, and the range of its working directory determines the range of FTP users that can be served.
  • FTPServer instances on the same ServerNode form a one-layer stack, and multiple ServerNodes form a two-layer stack to form the entire FTPServer distributed cluster. Can serve high concurrent connections of multiple FTP users.
  • the FTPServer instance runs on the ServerNode as a microservice in the form of a container. All FTPServer containers on the same ServerNode mount the same disk array LUN high-speed storage. Through distributed coordination and shared access, efficient IO is used to improve file transfer efficiency. Thus, the design purpose of high concurrency and high efficiency of the present disclosure is achieved.
  • Figure 4 is the flow chart of FTPProxy shunting and addressing
  • FTPProxy parses the request according to the FTP RFC protocol specification
  • step S403 if it is judged that it is a file transfer request, then go to step S405; otherwise, S404, directly respond through the local cache information;
  • FIG. 5 is a flowchart of a distributed coordination algorithm.
  • the FTPServer container of the same server node uses the same directory clusterlocks as the temporary directory for distribution locks.
  • All distributed locks provide a keyword (or file path), which is represented by hexadecimal code. Each lock creates a unique directory with this keyword under clusterlocks, and a directory is created according to the container ID that applies for the lock. file, so that a unique temporary file clusterlocks/ ⁇ lock keyword>/ ⁇ container ID> will be generated when each container applies for a lock
  • each lock has an expiration time of 60s (check whether it expires according to the last modification time of the file)
  • step 4 Check whether the lock directory (clusterlocks/ ⁇ lock keyword>) has files beginning with locked, if not, skip to step 4. If there is and the locked file has not expired, it means that the lock has been acquired, and the acquisition of the lock fails;
  • step 5 If there is only one file in the directory, if so, skip to step 5. Otherwise, sort multiple files, and then judge whether the first file name is the container ID. If it fails to acquire the lock, skip to step 6;
  • ServerNode-A There are three server nodes in the system: ServerNode-A, ServerNode-B, and ServerNode-C.
  • ServerNode-A is labeled: “FTP-A”, “FTP-C”, indicating that it can run the two types of FTPServer instances "FTP-A” and “FTP-C”;
  • ServerNode-B is labeled “FTP-A”
  • “FTP-B” is labeled “FTP-B”
  • “FTP-C” is labeled “FTP-B”
  • FTPServer instances There are three types of FTPServer instances: FTP-A, FTP-B, and FTP-C.
  • FTP-A can serve FTP users user-A1, user-A2, and user-A3; the disk array LUN where the corresponding working directory is located is LUN-A;
  • FTP-A is configured with 3 running instances: FTPServer-A- 1, FTPServer-A-2, FTPServer-A-3.
  • FTP-A can only run on the server node marked with "FTP-A", that is, ServerNode-A or ServerNode-B.
  • FIG. 6 is a schematic diagram of an optional distributed FTP cluster affinity and HA.
  • FTPServer instances such as FTP-A: FTPServer-A-1, FTPServer-A-2, and FTPServer-A-3 run on ServerNode-A marked with "FTP-A", and the corresponding magnetic array LUNs -A is also mounted on ServerNode-A.
  • FTPServer instances such as FTP-B: FTPServer-B-1, FTPServer-B-2 are running on ServerNode-B marked with "FTP-B", and the corresponding disk array LUN-B is also mounted on on ServerNode-B.
  • FTPServer instances such as FTP-C: FTPServer-C-1, FTPServer-C-2, FTPServer-C-3 are running on ServerNode-C marked with "FTP-C", and the corresponding magnetic array LUNs -C is also mounted on ServerNode-C
  • the cluster management system K8s finds that all FTP-A-type FTPServer instances are running on ServerNode-A: FTPServer-A-1, FTPServer-A-2, FTPServer -A-3. According to the labeling principle, it finds the server node ServerNode-B that can run the FTP-A instance (because it has the label "FTP-B"), and then mounts the corresponding magnetic array LUN-A to ServerNode-B first. Then, switch all FTP-A instances as a whole: FTPServer-A-1, FTPServer-A-2, FTPServer-A-3 to ServerNode-B to run. At this point, ServerNode-B runs two FTPServer instances, FTP-B and FTP-A, and mounts LUN-B and LUN-A at the same time.
  • the switched operating mode satisfies the label and affinity policy, and the corresponding local disk array LUN can be used.
  • HA can be achieved when an exception occurs.
  • the File Transfer Protocol is a set of standard protocols for file transfer on the network
  • the Logical Unit Number is an independent storage unit that can be directly identified by the server.
  • a data transmission apparatus is also provided, and the apparatus is used to implement the above-mentioned embodiments and preferred implementation manners, and what has been described will not be repeated.
  • the term "module” may be a combination of software and/or hardware that implements a predetermined function.
  • the apparatus described in the following embodiments is preferably implemented in software, implementations in hardware, or a combination of software and hardware, are also possible and contemplated.
  • FIG. 7 is a structural block diagram of a proxy server according to an embodiment of the present disclosure. As shown in FIG. 7 , the proxy server includes:
  • the receiving unit 702 is configured to receive a target request sent by the client, wherein the target request is a viewing request or a transmission request, and the viewing request is used to request to view the target attribute information of the target file in the data server, and the transmission request uses In requesting to transmit the target file to the data server or obtain the target file from the data server;
  • the first obtaining unit 704 is configured to obtain the target attribute information of the target file from the saved copy of the attribute information when the target request is a viewing request, and return it to the client, wherein the The corresponding relationship between the target file identifier of the target file and the target attribute information is recorded in the attribute information copy;
  • the first sending unit 706 is configured to send the target request to a target server in the data server when the target request is a transmission request, wherein the data server is a double-stacked FTPServer multi-instance
  • the data server includes a plurality of first servers
  • the proxy server records the scope of processing FTP users reported by each of the first servers.
  • the file can be scheduled through the proxy server, the viewing request can directly return the target attribute information through the proxy server, and the transmission request can be sent to different target servers for processing. Therefore, you can Solve the problem of low file transfer efficiency and achieve the effect of improving file transfer efficiency.
  • Embodiments of the present disclosure also provide a computer-readable storage medium, where a computer program is stored in the computer-readable storage medium, wherein the computer program is configured to execute the steps in any one of the above method embodiments when running.
  • the above-mentioned computer-readable storage medium may include, but is not limited to, a USB flash drive, a read-only memory (Read-Only Memory, referred to as ROM for short), and a random access memory (Random Access Memory, referred to as RAM for short) , mobile hard disk, magnetic disk or CD-ROM and other media that can store computer programs.
  • ROM Read-Only Memory
  • RAM Random Access Memory
  • An embodiment of the present disclosure also provides an electronic device, including a memory and a processor, where a computer program is stored in the memory, and the processor is configured to run the computer program to execute the steps in any one of the above method embodiments.
  • the above-mentioned electronic device may further include a transmission device and an input-output device, wherein the transmission device is connected to the above-mentioned processor, and the input-output device is connected to the above-mentioned processor.
  • modules or steps of the present disclosure can be implemented by a general-purpose computing device, and they can be centralized on a single computing device or distributed in a network composed of multiple computing devices
  • they can be implemented in program code executable by a computing device, so that they can be stored in a storage device and executed by the computing device, and in some cases, can be performed in a different order than shown here.
  • the described steps, or they are respectively made into individual integrated circuit modules, or a plurality of modules or steps in them are made into a single integrated circuit module to realize.
  • the present disclosure is not limited to any particular combination of hardware and software.

Abstract

本公开实施例提供了一种数据传输方法和代理服务器、存储介质及电子装置，上述方法包括：代理服务器接收客户端发送的目标请求，在目标请求为查看请求的情况下，代理服务器从保存的属性信息副本中获取目标文件的目标属性信息，并返回给客户端；在目标请求为传输请求的情况下，代理服务器将目标请求发送给数据服务器中的目标服务器，其中，数据服务器为双层堆叠的FTPServer多实例分布式集群服务器，数据服务器包括多个第一服务器，代理服务器中记录有每一个第一服务器上报的处理FTP用户的范围。

Description

数据传输方法和代理服务器、存储介质及电子装置
相关申请的交叉引用
本公开基于2020年7月1日提交的发明名称为“数据传输方法和代理服务器、存储介质及电子装置”的中国专利申请CN202010621938.0，并且要求该专利申请的优先权，通过引用将其所公开的内容全部并入本公开。
技术领域
本公开实施例涉及5G智能网络下文件传输技术领域，具体而言，涉及一种数据传输方法和代理服务器、存储介质及电子装置。
背景技术
现有支持FTP文件传输主要有如下几类方法：
高效单实例FTP服务器，单FTP服务器，利用本地文件系统，NIO等方式，高效支持FTP文件传输。
FTP分布式服务器，根据规模可以扩展多个FTP服务器实例，以支持更大规模。根据多FTP服务器实例间文件共享方式主要有如下3类：
(1)文件不共享；
(2)通过文件复制方式进行文件共享；
(3)通过NFS、CEPH等分布式文件系统方式进行文件共享；
一般通过FTP反向代理解决网络连通性问题，确保FTP客户端可以和不直通的FTP服务器之间传输文件。
然而，对于上述方法，单实例FTP服务，由于只有一个FTP服务器实例，支撑能力上限受限于单实例处理能力，无法满足大规模多连接场景，多实例FTP服务器现有实现无法很好解决高并发连接场景下效率问题。
也就是说，现有技术中数据传输的过程中，存在文件传输效率低的问 题。
发明内容
本公开实施例提供了一种数据传输方法和代理服务器、存储介质及电子装置，以至少解决相关技术中文件传输效率低的问题。
根据本公开的一个实施例，提供了一种数据传输方法，包括：代理服务器接收客户端发送的目标请求，其中，所述目标请求为查看请求或者传输请求，所述查看请求用于请求查看数据服务器中目标文件的目标属性信息，所述传输请求用于请求向所述数据服务器传输目标文件或者从所述数据服务器获取所述目标文件；在所述目标请求为查看请求的情况下，所述代理服务器从保存的属性信息副本中获取所述目标文件的所述目标属性信息，并返回给所述客户端，其中，所述属性信息副本中记录有所述目标文件的目标文件标识与所述目标属性信息的对应关系；在所述目标请求为传输请求的情况下，所述代理服务器将所述目标请求发送给所述数据服务器中的目标服务器，其中，所述数据服务器为双层堆叠的FTPServer多实例分布式集群服务器，所述数据服务器包括多个第一服务器，所述代理服务器中记录有每一个所述第一服务器上报的处理FTP用户的范围。
根据本公开的另一个实施例，提供了一种代理服务器，包括：接收单元，设置为接收客户端发送的目标请求，其中，所述目标请求为查看请求或者传输请求，所述查看请求用于请求查看数据服务器中目标文件的目标属性信息，所述传输请求用于请求向所述数据服务器传输目标文件或者从所述数据服务器获取所述目标文件；第一获取单元，设置为在所述目标请求为查看请求的情况下，从保存的属性信息副本中获取所述目标文件的所述目标属性信息，并返回给所述客户端，其中，所述属性信息副本中记录有所述目标文件的目标文件标识与所述目标属性信息的对应关系；第一发送单元，设置为在所述目标请求为传输请求的情况下，将所述目标请求发送给所述数据服务器中的目标服务器，其中，所述数据服务器为双层堆叠的FTPServer多实例分布式集群服务器，所述数据服务器包括多个第一 服务器，所述代理服务器中记录有每一个所述第一服务器上报的处理FTP用户的范围。
根据本公开的又一个实施例，还提供了一种计算机可读存储介质，所述计算机可读存储介质中存储有计算机程序，其中，所述计算机程序被设置为运行时执行上述任一项方法实施例中的步骤。
根据本公开的又一个实施例，还提供了一种电子装置，包括存储器和处理器，所述存储器中存储有计算机程序，所述处理器被设置为运行所述计算机程序以执行上述任一项方法实施例中的步骤。
附图说明
图1是根据本公开实施例的数据传输方法的移动终端的硬件结构框图；
图2是根据本公开实施例的数据传输方法的流程图；
图3是根据本公开实施例的数据传输方法的架构和流程示意图；
图4是根据本公开实施例的数据传输方法的分流与寻址流程图；
图5是根据本公开实施例的数据传输方法的分布式协调算法流程图；
图6是根据本公开实施例的数据传输方法的布式FTP集群亲和与HA示意图；
图7是根据本公开实施例的代理服务器的结构框图。
具体实施方式
下文中将参考附图并结合实施例来详细说明本公开的实施例。
需要说明的是，本公开的说明书和权利要求书及上述附图中的术语“第一”、“第二”等是用于区别类似的对象，而不必用于描述特定的顺序或先后次序。
文件传输
文件传输是通用技术，在5G智能网络场景下，文件传输无处不在。比如南向带外网元，由于5G海量接入特征，接入网元数大大增加，文件 传输是网元性能管理，版本管理中的常用技术，由于安全性可以从接入网元物理网络上得到保障，所以简单高效的文件传输协议FTP服务器成为了文件传输的合理选择，海量接入的网元引发大量FTP并发连接请求。同时5G低时延智能网络进一步要求FTP在支持海量连接的同时做到文件传输高效率。
分布式集群
分布式是指多个系统协同合作完成一个特定的任务。分布式是解决中心化管理的问题，把所有的任务叠加到一个节点处理，太慢了。所以把一个大的问题拆分为多个小的问题，并分别解决，最终协同合作。分布式的主要工作是分解任务，将职能拆解。集群主要的使用场景是为了分担请求的压力，也就是在几个服务器上部署相同的应用程序，来分担客户端请求。
FTP反向代理
因网络环境限制，需要从内网(不能直接连通外网)向外部的ftp上传、下载文件，只能在网关服务器上架设ftp反向代理，网关服务器可同时连通外网与内网，内网的FTP用户连接FTP代理，通过其访问外部真实的FTP服务器。
本公开实施例中所提供的方法实施例可以在移动终端、计算机终端或者类似的运算装置中执行。以运行在移动终端上为例，图1是本公开实施例的一种数据传输方法的移动终端的硬件结构框图。如图1所示，移动终端可以包括一个或多个(图1中仅示出一个)处理器102(处理器102可以包括但不限于微处理器MCU或可编程逻辑器件FPGA等的处理装置)和设置为存储数据的存储器104，其中，上述移动终端还可以包括设置为通信功能的传输设备106以及输入输出设备108。本领域普通技术人员可以理解，图1所示的结构仅为示意，其并不对上述移动终端的结构造成限定。例如，移动终端还可包括比图1中所示更多或者更少的组件，或者具有与图1所示不同的配置。
存储器104可设置为存储计算机程序，例如，应用软件的软件程序以 及模块，如本公开实施例中的数据传输方法对应的计算机程序，处理器102通过运行存储在存储器104内的计算机程序，从而执行各种功能应用以及数据处理，即实现上述的数据传输方法。存储器104可包括高速随机存储器，还可包括非易失性存储器，如一个或者多个磁性存储装置、闪存、或者其他非易失性固态存储器。在一些实例中，存储器104可进一步包括相对于处理器102远程设置的存储器，这些远程存储器可以通过网络连接至移动终端。上述网络的实例包括但不限于互联网、企业内部网、局域网、移动通信网及其组合。
传输装置106设置为经由一个网络接收或者发送数据。上述的网络具体实例可包括移动终端的通信供应商提供的无线网络。在一个实例中，传输装置106包括一个网络适配器(Network Interface Controller，简称为NIC)，其可通过基站与其他网络设备相连从而可与互联网进行通讯。在一个实例中，传输装置106可以为射频(Radio Frequency，简称为RF)模块，其设置为通过无线方式与互联网进行通讯。
在本实施例中提供了一种数据传输方法，图2是根据本公开实施例的数据传输方法的流程图，如图2所示，该流程包括如下步骤：
步骤S202，代理服务器接收客户端发送的目标请求，其中，所述目标请求为查看请求或者传输请求，所述查看请求用于请求查看数据服务器中目标文件的目标属性信息，所述传输请求用于请求向所述数据服务器传输目标文件或者从所述数据服务器获取所述目标文件；
步骤S204，在所述目标请求为查看请求的情况下，所述代理服务器从保存的属性信息副本中获取所述目标文件的所述目标属性信息，并返回给所述客户端，其中，所述属性信息副本中记录有所述目标文件的目标文件标识与所述目标属性信息的对应关系；
步骤S206，在所述目标请求为传输请求的情况下，所述代理服务器将所述目标请求发送给所述数据服务器中的目标服务器，其中，所述数据服务器为双层堆叠的FTPServer多实例分布式集群服务器，所述数据服务 器包括多个第一服务器，所述代理服务器中记录有每一个所述第一服务器上报的处理FTP用户的范围。
其中，在所述代理服务器接收所述客户端发送的所述目标请求之前，所述方法还包括：获取所述数据服务器中所有的文件的文件标识与所有的文件的属性信息；将所述数据服务器中所有的文件的文件标识与所有的文件的属性信息保存到所述属性信息副本中。
其中，在所述代理服务器接收所述客户端发送的所述目标请求之前，所述方法还包括：获取用户标识与所述第一服务器的对应关系，其中，不同的所述用户标识对应不同的所述第一服务器；将所述对应关系保存到内存中。
其中，所述在所述目标请求为查看请求的情况下，从所述代理服务器中保存的属性信息副本中获取所述目标文件的所述属性信息并返回给所述客户端包括：获取所述目标请求中的所述目标文件的所述目标文件标识；在所述属性信息副本中获取与所述目标文件标识对应的所述目标属性信息；将所述目标属性信息返回给所述客户端。
其中，在所述目标请求为传输请求的情况下，将所述目标请求发送给所述数据服务器中的所述目标服务器之前，所述方法还包括：获取所述客户端的目标用户标识；查找内存中与所述目标用户标识对应的所述目标服务器。
其中，所述查找内存中与所述目标用户标识对应的所述目标服务器包括：在查找到与所述目标用户标识对应的多个服务器的情况下，从所述多个服务器中选择负载均衡最低的服务器作为所述目标服务器，或者使用轮询算法从所述多个服务器中确定出所述目标服务器。
其中，在所述目标请求为传输请求的情况下，将所述目标请求发送给所述数据服务器中的所述目标服务器之后，所述方法还包括：在所述客户端与所述目标服务器之间建立数据传输通道；通过所述数据传输通道在所述客户端与所述目标服务器之间传输所述目标文件。
其中，在所述代理服务器接收客户端发送的目标请求之前，所述方法还包括：在所述属性信息副本的文件大小小于预定阈值的情况下，将所述属性信息副本保存在内存中；在所述属性信息副本的文件大小大于或等于预定阈值的情况下，将所述属性信息副本保存到存储器中。
其中，在所述目标请求为传输请求的情况下，所述代理服务器将所述目标请求发送给所述数据服务器中的所述目标服务器之前，所述方法还包括：使用位于同一个所述第一服务器上的多个FTPServer实例形成一层堆叠结构；使用多个所述第一服务器形成双层堆叠结构，得到所述数据服务器。
其中，在使用多个所述第一服务器形成双层堆叠结构，得到所述数据服务器之后，所述方法还包括：将同一个所述第一服务器上的多个所述FTPServer实例挂载在相同的磁阵LUN高速存储上。
通过本公开实施例，由于在文件的传输过程中，可以通过代理服务器进行文件的调度，查看请求可以直接通过代理服务器返回目标属性信息，而传输请求可以发送给不同的目标服务器进行处理，因此，可以解决文件传输效率低问题，达到提高文件传输效率效果。
下面结合具体示例对本公开进行说明。
本公开实施例中的代理服务器可以为ftp反向代理服务器。第一服务器可以为服务器节点，服务器节点上运行有FTP SERVER实例(FTPServer微服务)。多个第一服务器形成双层堆叠结构，组成数据服务器。
FTP反向代理FTPProxy分流寻址
FTPProxy反向代理是一个FTP协议的元语层实现，对于客户端而言其就是一个标准的FTP服务器，只是很多实际的文件操作请求由它传递到后端FTPServer微服务集群处理。本公开实施例的FTPProxy反向代理重点在本地分流和配合FTPServer双层堆叠寻址设计。
首先介绍反向代理，反向代理是为了屏蔽后端多个FTPServer分布式实例对客户端造成的差异，并配合分流寻址，FTP客户端不用关注有多少 FTPServer实例，各自运行在什么服务器节点，各实例地址是什么，某个实例是否停止了等细节。FTP客户端不直接请求FTPServer微服务，而是连接FTPProxy，由它来代理FTPServer提供FTP协议能力。FTPProxy提供了一个监控台，可以查看FTPProxy本身状态，当前所有FTP连接信息，后端所有ServerNode和FTPServer实例的信息与状态。
ftp反向代理通过分析FTP请求，可以将FTP请求分为2大类：
查看请求：文件查看类请求，如：LIST,NLST,PWD,STAT。这类请求主要是查看文件系统层面文件信息。
传输请求：文件操作和文件传输类请求，除了查看类请求的其他请求，如：RETR,STOR,DELE,CWD,RNTO等。我们统称为文件传输类请求。
为了缓解FTPServer的压力，并提升文件查看类请求的响应效率，可以在反向代理服务器FTPProxy提供分流能力。
FTPProxy在收到客户端的请求后，判断请求类型，如果是文件查看类请求，直接在本地处理，不再传递到后端的FTPServer。而要支持文件查看类请求在FTPProxy处理，需要在FTPProxy维护一份后端FTPServer文件目录树和属性信息副本。因为后端各FTPServer所在的单服务器节点是共享本地文件系统，所以对同一服务器节点而言这份文件树信息不用在FTPProxy合并。各服务器节点的本地文件系统不共享，各自独立，所以FTPProxy要针对所有FTPServer所在的服务器节点以服务器节点为单位维护一颗目录树。随着后续FTP动作，FTPServer的文件系统内容会发生变化，FTPProxy的本地缓存需要同步更新，由于所有FTP请求都经过FTPProxy反向代理，所以这些变化可以同时在FTPProxy维护的目录树中进行，保持和后端FTPServer文件系统内容的一致性。
为了提升效率，FTPProxy维护的目录树信息可以存放在内存中，如果后端FTPServer持有的文件太多，可以将这部分信息持久化到存储中，以缓解内存压力。
对于文件传输类请求，到达代理后，后续由客户端和FTPServer实例 直接建立文件传输通道，文件传输不通过代理，直连直传，减少FTPProxy的一次转发，避免FTPProxy成为瓶颈，进一步提升效率。
FTPProxy负责分发FTP请求到后端的多个FTPServer微服务实例。因为按照FTPServer双层堆叠设计，各FTPServer实例能支持的FTP用户是一定范围的，所以FTPProxy要根据当前连接登录的FTP用户来确定寻址到后端哪个FTPServer实例，当然这种实例可能存在多个，这时可以进一步根据FTPServer的负载情况均衡分配请求，以达到效率最大化，也可以采用Round-Robin轮询算法，简单高效。
在5G智能网络云网融合的趋势下，运维管理，网络管理等多采用微服务架构，多实例扩展成为了云原生对各服务组件的基本要求，以更好的支持分布式，高并发大规模，以及双机集群系统HA。所以本方案FTPServer作为文件传输服务组件，采用微服务容器方式实现，支持多实例运行。同时为了提升文件传输效率，同服务器节点FTPServer实例共享本地高效存储逻辑单元号LUN，多服务器节点再堆叠组成整个FTP分布式集群系统，达到高并发高效率文件传输要求。
单实例FTPServer能够支持的FTP并发连接数是有限的，其能力并不能随着资源的垂直扩展无上限提升，因为单实例内部的线程调度，锁等待等导致处理能力存在上限，一般1000左右并发连接就到顶了，再分配扩展资源也无法提升。所以为了支持更多并发连接，要实现FTP服务器集群。
本公开实施例设计FTPServer以容器方式运行，满足云原生要求，支持多实例扩展，支持弹缩，可根据配置，负载，比如连接数，连接活跃程度，CPU和内存的使用情况弹出多个实例运行。一个服务器节点可以运行多个FTPServer实例，可部署多个服务器节点。这样多个FTPServer实例组成了一个FTP分布式集群，由容器管理系统，比如K8s来管理其生命周期，实现弹缩调度和HA。
其中，同服务器节点FTPServer实例共享挂接在本地存储的磁阵LUN
FTP服务器集群最大要解决的问题是文件系统的一致性，多个 FTPServer实例需要看到完全一致的文件视图，这样文件上传下载，文件操作才能确保业务正确。为了保证这种一致性，无论是文件复制方式，还是分布式文件系统方式，效率问题都是最突出的问题，集群内部增加的IO和冲突大大降低了文件传输效率。本公开实施例对这一问题进行了解决。
为了解决文件系统一致性，设计同服务器的FTPServer实例共享挂接在本地存储的磁阵LUN，用分布式协调机制来解决文件访问冲突，这样同服务器的FTPServer实例看到的是统一的文件系统视图。此方法利用磁阵LUN的高效IO来支持文件高效传输。当然这需要对应FTPServer实例运行在同样服务器节点上，用服务器节点的资源高配来支持FTPServer多实例。比如单FTPServer实例的能力上限时资源要求为1C2G(1核CPU，2G内存)，目前高配服务器基本可以达到100C300G的配置，也就同服务器节点可以支持100个FTPServer实例的水平扩展，达到10万FTP并发连接支持能力。
运行在单服务器节点上的FTPServer多实例组成了一个FTP服务器集群，但是受限于服务器节点配置，此集群的处理能力也存在上限，一定程度上限制了分布式FTP服务器集群支持规模。所以需要打破单服务器节点的限制，利用多个服务器节点。而本地存储的磁阵LUN又无法跨多个服务器节点高效访问，否则就必须构建类NFS分布式文件系统，效率问题会出现。
结合FTP的业务特征，某个具体的FTP用户主工作目录是确定的，不同主工作目录下的文件不存在交叉，也就是FTP对文件的操作始终限制在此用户的主工作目录下，所以我们可以将某FTP用户的主工作目录绑定到某个固定的服务器节点，另外的FTP用户不同主工作目录绑定到其他服务器节点。这样，不同服务器节点处理不同的主工作目录，也就不存在多服务器节点间文件共享视图的要求了。这里划分的依据是FTP用户的主工作目录，而不是FTP用户，因为不同FTP用户可以指向同样的工作目录。最终的效果是某个具体的工作目录只存在于特定的服务器节点上。
提前划分好各FTP用户主工作目录和服务器节点的对应关系，某个服务器节点包含哪些工作目录，其就能处理对应的FTP用户的文件传输请求，最终形成服务器节点和FTP用户的关系，某个FTP用户的所有请求都由一个固定的服务器节点(的多FTPServer实例)来处理，可以继续利用挂接在本地存储的磁阵LUN，同时也能利用在此服务器节点上运行的多个FTPServer实例来为此用户支持集群能力。这样，就将FTPServer分成了很多类，某类FTPServer(多个实例)处理的FTP用户目录确定，并与其他类FTPServer不交叉，运行在固定的服务器节点上。
按照前面的资源估算，运行在单服务器节点的单用户可以达到10万并发连接这个数量级能力，而不同的业务一般使用不同的FTP用户，所以结合本方案中多服务器节点扩展，整个系统可以提供更高数量级的并发支持能力。
本公开实施例中不同服务器节点的FTPServer能处理的FTP用户范围是不同的，某个具体的FTPServer实例只能在特定的服务器节点运行，并在此节点弹缩。这需要利用容器管理系统K8s的标签机制，其可以通过标签来调度FTPServer实例运行的服务器节点。为每个服务器节点打上标签，同时配置某类FTPServer的标签，从而控制其实例运行在对应标签的服务器节点上。不同工作目录范围的其他类FTPServer可以配置其他标签，控制其实例运行在对应标签的其他服务器节点上。
按照本公开实施例中所提及的方案，某类FTPServer的实例可运行的标签对应的服务器节点只能是一个，才能利用本地存储的磁阵LUN，但是一旦该服务器节点失效，就无法保证HA了，所以某标签对应的服务器节点数必须大于1。但又需要确保所有同类FTPServer实例只能调度到同样的服务器节点，这要利用容器管理系统K8s的亲和策略，亲和是指一旦某类FTPServer第一个实例按照标签规则调度到某个服务器节点，后续该类FTPServer的其他实例都会调度到同样的服务器节点，即使存在同样标签的多个服务器节点。这样就确保所有同类FTPServer实例运行在同一服务器节点，可以共享本地存储的磁阵LUN，又同时保证了HA，当某服务 器节点Down机时，此服务器节点的所有FTPServer实例整体切换到符合标签规则的其他某个服务器节点，磁阵LUN也重新挂接到新服务器节点。
由于同一个服务器节点的所有FTPServer实例会对同一个磁阵LUN进行读写，所以需要一个分布式协调机制来协调这些FTPServer实例的IO操作不相互冲突。主要是对同一文件的读写不相互冲突，同时只能有一个FTPServer实例对同一文件进行写操作，写的时候不能同时读，其他无限制。同一FTPServer实例内部多线程间通过内存锁控制，无须分布式协调。
通过跨容器的分布式锁来实现这种分布式协调机制：写文件的时候，依据文件名生成对应的锁，写完后释放，其他读写的动作都查看是否存在这把锁，如果存在，则其他对此文件的读写动作不允许。
本分布式协调是一种基于临时文件和文件排序实现的分布式锁，该锁不可重入。申请锁时，为每一个锁生成一个临时目录，对应要协调读写的具体文件，各个要申请锁的FTPServer实例在此目录下根据自身容器ID生成临时文件，并进行加锁，加锁前后都进行判定，如果存在多个容器同时申请(也就存在多个临时文件)，根据文件名排序固定算法选择加锁成功的容器。最终达到只有一个容器获取这把锁的目的，从而实现分布式协调。
采用本公开实施例的方法，与现有技术相比，响应文件查看类操作更快，可以支持海量的FTP并发连接，同时在海量FTP连接情况下，由于没有文件复制或者网络文件系统，有效降低了FTP服务器集群内部的IO，提升了文件传输效率。
通过FTPProxy反向代理分流寻址后端多FTPServer实例，文件传输类请求跳过代理直连直传，提升文件传输效率。结合FTP业务特征，设计了双层堆叠的FTPServer多实例分布式集群(数据服务器)，利用亲和策略和分布式协调使得同类FTPServer实例可共享本地磁阵LUN高效存储，进一步提升文件上传下载效率。
如图3所示，图3为本公开方法架构和流程示意图。FTPProxy可以 部署在网关处，充当反向代理，并配合寻址后端的FTPServer实例。各ServerNode上第一个生成的FTPServer实例启动时将本地LUN中工作目录树信息传递给FTPProxy缓存下来，以便其快速响应文件查看类请求。FTPProxy维护有所有LUN的工作目录树，并随着后续的FTP操作动态更新。同时FTPProxy提供了一个监控台，收集FTP分布式集群系统各种信息和状态。其可以查看FTPProxy本身状态，当前所有FTP连接信息，后端所有ServerNode和FTPServer实例的信息与状态。
客户端FTPClient首先连接到FTPProxy，进行FTP用户认证，FTPProxy记录下各FTP连接对应的FTP用户信息。后续文件传输类请求发起时，寻址到对应此用户的FTPServer实例，由FTPClient和此FTPServer实例直接建立文件传输通道，进行后续的文件传输。
组成FTPServer分布式集群的各实例按照标签规则运行在对应的ServerNode服务器节点上，此服务器节点挂载对应的磁阵LUN，其工作目录范围决定了可服务的FTP用户范围。同一ServerNode上的FTPServer实例形成一层堆叠，多个ServerNode形成双层堆叠，从而组成整个FTPServer分布式集群。可以服务于多个FTP用户的高并发连接。
FTPServer实例以容器形态的微服务运行于ServerNode上，同一ServerNode上的所有FTPServer容器挂载同样的磁阵LUN高速存储，通过分布式协调共享访问，利用高效IO提升文件传输效率。从而达到本公开高并发高效率的设计目的。
如图4所示，图4为FTPProxy分流与寻址流程图
S401，FTP请求到达FTPProxy；
S402，FTPProxy根据FTP RFC协议规范解析出请求；
S403，判断如果是文件传输类请求，则到步骤S405；否则S404，通过本地缓存信息直接响应；
S405，根据当前FTP用户信息找出对应的FTPServer实例；如果FTPServer实例存在多个，可以根据负载均衡算法或者Round-Robin轮询 算法寻址到合适的FTPServer实例；
S406，FTPClient和FTPServer实例间建立直接文件传输通道；
S407，进行文件传输(上传或者下载)。
如图5所示，图5为分布式协调算法流程图。
同服务器节点的多FTPServer容器实例要实现对同一文件的读写不相互冲突，同时只能有一个FTPServer实例对同一文件进行写操作，写的时候不能同时读，这通过跨容器的分布式锁来实现，具体分布式锁算法如下：
前置条件：
同服务器节点的FTPServer容器用同一个目录clusterlocks作为分布锁临时目录。
所有分布锁提供一个关键字(或文件路径)，将关键字用16进制编码表示，每个锁在clusterlocks下以此关键字都建一个唯一的目录，目录下根据申请锁的容器ID建一个文件，这样对于每个容器申请一个锁时都会生成一个唯一的临时文件clusterlocks/<锁关键字>/<容器ID>
为防止容器申请到锁后异常退出，每个锁有一个过期时间60s(根据文件最后修改时间确认是否过期)
申请分布锁流程：
1、获取到文件锁对应目录；
2、检查锁目录(clusterlocks/<锁关键字>)是否存在locked开头的文件，如果无则跳到步骤4。如果有且locked文件未过期，说明锁已经被获取，则获取锁失败；
3、如果locked文件过期，则删除locked文件；
4、目录下是否只有一个文件，如果是则跳到步骤5。否则对多个文件进行排序，再判断第一个文件名是否为本容器ID，如果否则获取锁失败，是则跳到步骤6；
5、这一个文件的文件名是否为本容器ID，如果否则获取锁失败，是 则继续；
6、在文件名加上locked前缀；
7、sleep10ms，再次确认是否是唯一locked文件，如果不是则跳到步骤8，如果是则再次确认锁文件名是否是locked+本容器ID名，如果是则获取锁成功，否则获取锁失败；
8、如果存在多个locked文件，则再进行排序，第一个文件名对应的容器获取到锁，其余容器获取锁失败并清理掉自己生成的文件。
9、流程结束。
上述过程中，程序获取到分布式锁后，如果使用时间过长超过60s，需要在使用过程中需要更新锁文件的最后修改时间。
如以下表1所示，实例化了一个分布式FTPServer集群
表1
Figure PCTCN2021104060-appb-000001
系统存在3个服务器节点：ServerNode-A，ServerNode-B，ServerNode-C。
其中ServerNode-A打上标签：“FTP-A”，“FTP-C”，表示其可以运行“FTP-A”，“FTP-C”这2类FTPServer实例；ServerNode-B打上标签“FTP-A”，“FTP-B”，ServerNode-C打上标签“FTP-B”，“FTP-C”。
FTPServer实例有3类：FTP-A，FTP-B，FTP-C。其中FTP-A可服务 于FTP用户user-A1，user-A2，user-A3；其对应的工作目录所在的磁阵LUN为LUN-A；FTP-A配置有3个运行实例：FTPServer-A-1，FTPServer-A-2，FTPServer-A-3。FTP-A只能运行在打了“FTP-A”标签的服务器节点，即ServerNode-A或者ServerNode-B上。
图6为一种可选的分布式FTP集群亲和与HA示意图。
由于亲和原则：
所有的FTP-A这类FTPServer实例：FTPServer-A-1，FTPServer-A-2，FTPServer-A-3都运行在打了“FTP-A”标签的ServerNode-A上，同时对应的磁阵LUN-A也挂载到ServerNode-A上。
所有的FTP-B这类FTPServer实例：FTPServer-B-1，FTPServer-B-2都运行在打了“FTP-B”标签的ServerNode-B上，同时对应的磁阵LUN-B也挂载到ServerNode-B上。
所有的FTP-C这类FTPServer实例：FTPServer-C-1，FTPServer-C-2，FTPServer-C-3都运行在打了“FTP-C”标签的ServerNode-C上，同时对应的磁阵LUN-C也挂载到ServerNode-C上
这时，假设系统发生了异常，ServerNode-A宕机了，集群管理系统K8s发现原来在ServerNode-A运行了所有FTP-A类的FTPServer实例：FTPServer-A-1，FTPServer-A-2，FTPServer-A-3。根据标签原则，其找到能运行FTP-A类实例的服务器节点ServerNode-B(因为其打了标签“FTP-B”)，然后先将对应的磁阵LUN-A挂载到ServerNode-B上，然后，整体切换所有的FTP-A类实例：FTPServer-A-1，FTPServer-A-2，FTPServer-A-3到ServerNode-B运行起来。此时ServerNode-B就同时运行了FTP-B和FTP-A两类FTPServer实例，同时挂载了LUN-B，LUN-A。
这样，切换后的运行形态满足标签和亲和性策略，能使用对应的本地磁阵LUN，同时又在出异常时做到了HA。
如图6所示，当ServerNode-A节点down机后，FTPServer-A-1，FTPServer-A-2，FTPServer-A-3实例整体迁移到ServerNode-B节点运行， 集群迁移时将Lun-A挂载到ServerNode-B节点。
其中，File Transfer Protocol(FTP)是在网络上进行文件传输的一套标准协议，逻辑单元号(Logical Unit Number，LUN)是可以被服务器直接识别的独立存储单元。
通过以上的实施方式的描述，本领域的技术人员可以清楚地了解到根据上述实施例的方法可借助软件加必需的通用硬件平台的方式来实现，当然也可以通过硬件，但很多情况下前者是更佳的实施方式。基于这样的理解，本公开的技术方案本质上或者说对现有技术做出贡献的部分可以以软件产品的形式体现出来，该计算机软件产品存储在一个存储介质(如ROM/RAM、磁碟、光盘)中，包括若干指令用以使得一台终端设备(可以是手机，计算机，服务器，或者网络设备等)执行本公开各个实施例所述的方法。
在本实施例中还提供了一种数据传输装置，该装置用于实现上述实施例及优选实施方式，已经进行过说明的不再赘述。如以下所使用的，术语“模块”可以实现预定功能的软件和/或硬件的组合。尽管以下实施例所描述的装置较佳地以软件来实现，但是硬件，或者软件和硬件的组合的实现也是可能并被构想的。
图7是根据本公开实施例的代理服务器的结构框图，如图7所示，该代理服务器包括：
接收单元702，设置为接收客户端发送的目标请求，其中，所述目标请求为查看请求或者传输请求，所述查看请求用于请求查看数据服务器中目标文件的目标属性信息，所述传输请求用于请求向所述数据服务器传输目标文件或者从所述数据服务器获取所述目标文件；
第一获取单元704，设置为在所述目标请求为查看请求的情况下，从保存的属性信息副本中获取所述目标文件的所述目标属性信息，并返回给所述客户端，其中，所述属性信息副本中记录有所述目标文件的目标文件 标识与所述目标属性信息的对应关系；
第一发送单元706，设置为在所述目标请求为传输请求的情况下，将所述目标请求发送给所述数据服务器中的目标服务器，其中，所述数据服务器为双层堆叠的FTPServer多实例分布式集群服务器，所述数据服务器包括多个第一服务器，所述代理服务器中记录有每一个所述第一服务器上报的处理FTP用户的范围。
通过上述代理服务器，由于在文件的传输过程中，可以通过代理服务器进行文件的调度，查看请求可以直接通过代理服务器返回目标属性信息，而传输请求可以发送给不同的目标服务器进行处理，因此，可以解决文件传输效率低问题，达到提高文件传输效率效果。
本公开的实施例还提供了一种计算机可读存储介质，该计算机可读存储介质中存储有计算机程序，其中，该计算机程序被设置为运行时执行上述任一项方法实施例中的步骤。
在一个示例性实施例中，上述计算机可读存储介质可以包括但不限于：U盘、只读存储器(Read-Only Memory，简称为ROM)、随机存取存储器(Random Access Memory，简称为RAM)、移动硬盘、磁碟或者光盘等各种可以存储计算机程序的介质。
本公开的实施例还提供了一种电子装置，包括存储器和处理器，该存储器中存储有计算机程序，该处理器被设置为运行计算机程序以执行上述任一项方法实施例中的步骤。
在一个示例性实施例中，上述电子装置还可以包括传输设备以及输入输出设备，其中，该传输设备和上述处理器连接，该输入输出设备和上述处理器连接。
本实施例中的具体示例可以参考上述实施例及示例性实施方式中所描述的示例，本实施例在此不再赘述。
显然，本领域的技术人员应该明白，上述的本公开的各模块或各步骤可以用通用的计算装置来实现，它们可以集中在单个的计算装置上，或者分布在多个计算装置所组成的网络上，它们可以用计算装置可执行的程序代码来实现，从而，可以将它们存储在存储装置中由计算装置来执行，并且在某些情况下，可以以不同于此处的顺序执行所示出或描述的步骤，或者将它们分别制作成各个集成电路模块，或者将它们中的多个模块或步骤制作成单个集成电路模块来实现。这样，本公开不限制于任何特定的硬件和软件结合。
以上所述仅为本公开的优选实施例而已，并不用于限制本公开，对于本领域的技术人员来说，本公开可以有各种更改和变化。凡在本公开的原则之内，所作的任何修改、等同替换、改进等，均应包含在本公开的保护范围之内。

Claims (13)

  1. 一种数据传输方法，包括：
    代理服务器接收客户端发送的目标请求，其中，所述目标请求为查看请求或者传输请求，所述查看请求用于请求查看数据服务器中目标文件的目标属性信息，所述传输请求用于请求向所述数据服务器传输目标文件或者从所述数据服务器获取所述目标文件；
    在所述目标请求为查看请求的情况下，所述代理服务器从保存的属性信息副本中获取所述目标文件的所述目标属性信息，并返回给所述客户端，其中，所述属性信息副本中记录有所述目标文件的目标文件标识与所述目标属性信息的对应关系；
    在所述目标请求为传输请求的情况下，所述代理服务器将所述目标请求发送给所述数据服务器中的目标服务器，其中，所述数据服务器为双层堆叠的FTPServer多实例分布式集群服务器，所述数据服务器包括多个第一服务器，所述代理服务器中记录有每一个所述第一服务器上报的处理FTP用户的范围。
  2. 根据权利要求1所述的方法，其中，在所述代理服务器接收所述客户端发送的所述目标请求之前，所述方法还包括：
    获取所述数据服务器中所有的文件的文件标识与所有的文件的属性信息；
    将所述数据服务器中所有的文件的文件标识与所有的文件的属性信息保存到所述属性信息副本中。
  3. 根据权利要求1所述的方法，其中，在所述代理服务器接收所述客户端发送的所述目标请求之前，所述方法还包括：
    获取用户标识与所述第一服务器的对应关系，其中，不同的所述用户标识对应不同的所述第一服务器；
    将所述对应关系保存到内存中。
  4. 根据权利要求1所述的方法，其中，所述在所述目标请求为查看请求的情况下，从所述代理服务器中保存的属性信息副本中获取所述目标文件的所述属性信息并返回给所述客户端包括：
    获取所述目标请求中的所述目标文件的所述目标文件标识；
    在所述属性信息副本中获取与所述目标文件标识对应的所述目标属性信息；
    将所述目标属性信息返回给所述客户端。
  5. 根据权利要求1所述的方法，其中，在所述目标请求为传输请求的情况下，将所述目标请求发送给所述数据服务器中的所述目标服务器之前，所述方法还包括：
    获取所述客户端的目标用户标识；
    查找内存中与所述目标用户标识对应的所述目标服务器；
    将所述传输请求发送给所述目标服务器。
  6. 根据权利要求5所述的方法，其中，所述查找内存中与所述目标用户标识对应的所述目标服务器包括：
    在查找到与所述目标用户标识对应的多个服务器的情况下，从所述多个服务器中选择负载均衡最低的服务器作为所述目标服务器，或者使用轮询算法从所述多个服务器中确定出所述目标服务器。
  7. 根据权利要求5所述的方法，其中，在所述目标请求为传输请求的情况下，将所述目标请求发送给所述数据服务器中的所述目标服务器之后，所述方法还包括：
    在所述客户端与所述目标服务器之间建立数据传输通道；
    通过所述数据传输通道在所述客户端与所述目标服务器之间传输所述目标文件。
  8. 根据权利要求1所述的方法，其中，在所述代理服务器接收客户端发送的目标请求之前，所述方法还包括：
    在所述属性信息副本的文件大小小于预定阈值的情况下，将所述属性信息副本保存在内存中；
    在所述属性信息副本的文件大小大于或等于预定阈值的情况下，将所述属性信息副本保存到存储器中。
  9. 根据权利要求1所述的方法，其中，在所述目标请求为传输请求的情况下，所述代理服务器将所述目标请求发送给所述数据服务器中的所述目标服务器之前，所述方法还包括：
    使用位于同一个所述第一服务器上的多个FTPServer实例形成一层堆叠结构；
    使用多个所述第一服务器形成双层堆叠结构，得到所述数据服务器。
  10. 根据权利要求9所述的方法，其中，在使用多个所述第一服务器形成双层堆叠结构，得到所述数据服务器之后，所述方法还包括：
    将同一个所述第一服务器上的多个所述FTPServer实例挂载在相同的磁阵LUN高速存储上。
  11. 一种代理服务器，包括：
    接收单元，设置为接收客户端发送的目标请求，其中，所述目标请求为查看请求或者传输请求，所述查看请求用于请求查看数据服务 器中目标文件的目标属性信息，所述传输请求用于请求向所述数据服务器传输目标文件或者从所述数据服务器获取所述目标文件；
    第一获取单元，设置为在所述目标请求为查看请求的情况下，从保存的属性信息副本中获取所述目标文件的所述目标属性信息，并返回给所述客户端，其中，所述属性信息副本中记录有所述目标文件的目标文件标识与所述目标属性信息的对应关系；
    第一发送单元，设置为在所述目标请求为传输请求的情况下，将所述目标请求发送给所述数据服务器中的目标服务器，其中，所述数据服务器为双层堆叠的FTPServer多实例分布式集群服务器，所述数据服务器包括多个第一服务器，所述代理服务器中记录有每一个所述第一服务器上报的处理FTP用户的范围。
  12. 一种计算机可读存储介质，所述计算机可读存储介质中存储有计算机程序，其中，所述计算机程序被设置为运行时执行所述权利要求1至10任一项中所述的方。
  13. 一种电子装置，包括存储器和处理器，所述存储器中存储有计算机程序，所述处理器被设置为运行所述计算机程序以执行所述权利要求1至10任一项中所述的方法。
PCT/CN2021/104060 2020-07-01 2021-07-01 数据传输方法和代理服务器、存储介质及电子装置 WO2022002209A1 (zh)

Priority Applications (2)

Application Number Priority Date Filing Date Title
EP21834267.3A EP4178134A4 (en) 2020-07-01 2021-07-01 DATA TRANSMISSION METHOD, PROXY SERVER, STORAGE MEDIUM AND ELECTRONIC DEVICE
JP2023500001A JP2023532947A (ja) 2020-07-01 2021-07-01 データ転送方法、プロキシサーバ、記憶媒体及び電子デバイス

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
CN202010621938.0 2020-07-01
CN202010621938.0A CN113965560A (zh) 2020-07-01 2020-07-01 数据传输方法和代理服务器、存储介质及电子装置

Publications (1)

Publication Number Publication Date
WO2022002209A1 true WO2022002209A1 (zh) 2022-01-06

Family

ID=79315114

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2021/104060 WO2022002209A1 (zh) 2020-07-01 2021-07-01 数据传输方法和代理服务器、存储介质及电子装置

Country Status (4)

Country Link
EP (1) EP4178134A4 (zh)
JP (1) JP2023532947A (zh)
CN (1) CN113965560A (zh)
WO (1) WO2022002209A1 (zh)

Cited By (9)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN114584545A (zh) * 2022-03-03 2022-06-03 京东科技信息技术有限公司 数据管理方法、装置、系统、存储介质及电子设备
CN114598716A (zh) * 2022-04-02 2022-06-07 西湖大学 一种分布式文件存储系统、方法及电子设备
CN114615315A (zh) * 2022-03-22 2022-06-10 深圳壹账通创配科技有限公司 线上会话的通讯方法、装置、设备及存储介质
CN115002514A (zh) * 2022-05-27 2022-09-02 浙江大学 基于云原生控制器的spark视频转码系统及视频转码方法
CN115051982A (zh) * 2022-06-14 2022-09-13 北京天融信网络安全技术有限公司 基于ftp协议的信息处理方法、系统及存储介质
CN115174691A (zh) * 2022-06-22 2022-10-11 平安普惠企业管理有限公司 基于页面请求的大数据加载方法、装置、设备及介质
CN115242882A (zh) * 2022-09-20 2022-10-25 之江实验室 一种基于传输层路由访问k8s容器环境的方法及装置
CN115348310A (zh) * 2022-08-17 2022-11-15 中国电信股份有限公司 反向代理方法、装置、系统、电子设备及存储介质
CN117009310A (zh) * 2023-09-27 2023-11-07 苏州元脑智能科技有限公司 文件同步方法、装置、分布式全局内容库系统及电子设备

Families Citing this family (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN115250269B (zh) * 2022-07-26 2024-02-02 中银金融科技有限公司 一种文件分配方法及装置、存储介质及电子设备
CN115348258A (zh) * 2022-08-17 2022-11-15 中国建设银行股份有限公司贵州省分行 一种文件传输方法、装置、系统及电子设备

Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20110066703A1 (en) * 2009-05-20 2011-03-17 Creative Ad Technology Proprietary Limited Methods and systems for delivering media to client device
CN105100263A (zh) * 2015-08-20 2015-11-25 百度在线网络技术（北京）有限公司 一种反向代理方法及装置
CN105868333A (zh) * 2016-03-28 2016-08-17 金蝶软件（中国）有限公司 文件处理方法及装置
CN104065731B (zh) * 2014-06-30 2018-04-13 北京华电天益信息科技有限公司 一种ftp文件传输系统及传输方法
CN110120917A (zh) * 2019-06-28 2019-08-13 北京百度网讯科技有限公司 基于内容的路由方法及装置

Family Cites Families (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN101977236B (zh) * 2010-11-05 2015-05-20 北京世纪互联宽带数据中心有限公司 大文件多点分发系统

Patent Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20110066703A1 (en) * 2009-05-20 2011-03-17 Creative Ad Technology Proprietary Limited Methods and systems for delivering media to client device
CN104065731B (zh) * 2014-06-30 2018-04-13 北京华电天益信息科技有限公司 一种ftp文件传输系统及传输方法
CN105100263A (zh) * 2015-08-20 2015-11-25 百度在线网络技术（北京）有限公司 一种反向代理方法及装置
CN105868333A (zh) * 2016-03-28 2016-08-17 金蝶软件（中国）有限公司 文件处理方法及装置
CN110120917A (zh) * 2019-06-28 2019-08-13 北京百度网讯科技有限公司 基于内容的路由方法及装置

Non-Patent Citations (1)

* Cited by examiner, † Cited by third party
Title
See also references of EP4178134A4 *

Cited By (13)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN114584545A (zh) * 2022-03-03 2022-06-03 京东科技信息技术有限公司 数据管理方法、装置、系统、存储介质及电子设备
CN114615315A (zh) * 2022-03-22 2022-06-10 深圳壹账通创配科技有限公司 线上会话的通讯方法、装置、设备及存储介质
CN114598716A (zh) * 2022-04-02 2022-06-07 西湖大学 一种分布式文件存储系统、方法及电子设备
CN115002514A (zh) * 2022-05-27 2022-09-02 浙江大学 基于云原生控制器的spark视频转码系统及视频转码方法
CN115051982A (zh) * 2022-06-14 2022-09-13 北京天融信网络安全技术有限公司 基于ftp协议的信息处理方法、系统及存储介质
CN115051982B (zh) * 2022-06-14 2023-12-01 北京天融信网络安全技术有限公司 基于ftp协议的信息处理方法、系统及存储介质
CN115174691B (zh) * 2022-06-22 2023-09-05 山西数字政府建设运营有限公司 基于页面请求的大数据加载方法、装置、设备及介质
CN115174691A (zh) * 2022-06-22 2022-10-11 平安普惠企业管理有限公司 基于页面请求的大数据加载方法、装置、设备及介质
CN115348310A (zh) * 2022-08-17 2022-11-15 中国电信股份有限公司 反向代理方法、装置、系统、电子设备及存储介质
CN115242882B (zh) * 2022-09-20 2023-01-10 之江实验室 一种基于传输层路由访问k8s容器环境的方法及装置
CN115242882A (zh) * 2022-09-20 2022-10-25 之江实验室 一种基于传输层路由访问k8s容器环境的方法及装置
CN117009310A (zh) * 2023-09-27 2023-11-07 苏州元脑智能科技有限公司 文件同步方法、装置、分布式全局内容库系统及电子设备
CN117009310B (zh) * 2023-09-27 2024-01-23 苏州元脑智能科技有限公司 文件同步方法、装置、分布式全局内容库系统及电子设备

Also Published As

Publication number Publication date
EP4178134A4 (en) 2024-02-14
JP2023532947A (ja) 2023-08-01
CN113965560A (zh) 2022-01-21
EP4178134A1 (en) 2023-05-10

Similar Documents

Publication Publication Date Title
WO2022002209A1 (zh) 数据传输方法和代理服务器、存储介质及电子装置
US10171567B2 (en) Load balancing computer device, system, and method
US11775569B2 (en) Object-backed block-based distributed storage
US11314539B2 (en) Cloud storage based data processing method and system
CN107547661B (zh) 一种容器负载均衡实现方法
US20150213134A1 (en) Data query method and system and storage medium
US10146848B2 (en) Systems and methods for autonomous, scalable, and distributed database management
CN103905537A (zh) 分布式环境下管理工业实时数据存储的系统
CN105357296A (zh) 一种Docker云平台下弹性缓存系统
CN103237046A (zh) 支持混合云存储应用的分布式文件系统及实现方法
US20240039995A1 (en) Data access system and method, device, and network adapter
CN103207841A (zh) 基于键值对缓存的数据读写方法及装置
WO2018121334A1 (zh) 一种提供网页应用服务的方法、装置、电子设备及系统
CN107623703B (zh) 全局事务标识gtid的同步方法、装置及系统
WO2016095149A1 (zh) 一种数据压缩存储方法、装置，及分布式文件系统
WO2022105048A1 (zh) 一种分布式共享文件系统及其数据处理方法
CN112202918A (zh) 长连接通信的负载调度方法、装置、设备及存储介质
JP2016051446A (ja) 計算機システム、計算機、負荷分散方法及びそのプログラム
WO2021120633A1 (zh) 一种负载均衡方法及相关设备
WO2020252724A1 (zh) 日志处理方法、设备及计算机可读存储介质
KR101023622B1 (ko) 적응적 고성능 프락시 캐시 서버 및 캐싱방법
CN110049081A (zh) 用于搭建及使用高可用性Docker私库的方法和系统
US9432476B1 (en) Proxy data storage system monitoring aggregator for a geographically-distributed environment
CN110912966A (zh) 一种基于集装和隔离机制容器化的高速公路门架系统
US10148503B1 (en) Mechanism for dynamic delivery of network configuration states to protocol heads

Legal Events

Date Code Title Description
ENP Entry into the national phase

Ref document number: 2023500001

Country of ref document: JP

Kind code of ref document: A

NENP Non-entry into the national phase

Ref country code: DE

ENP Entry into the national phase

Ref document number: 2021834267

Country of ref document: EP

Effective date: 20230201