CN112653757A - File management system, method and equipment - Google Patents
File management system, method and equipment Download PDFInfo
- Publication number
- CN112653757A CN112653757A CN202011515429.6A CN202011515429A CN112653757A CN 112653757 A CN112653757 A CN 112653757A CN 202011515429 A CN202011515429 A CN 202011515429A CN 112653757 A CN112653757 A CN 112653757A
- Authority
- CN
- China
- Prior art keywords
- file
- metadata
- uploaded
- sftp
- downloaded
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
- 238000000034 method Methods 0.000 title claims abstract description 23
- 238000007726 management method Methods 0.000 claims description 94
- 238000012546 transfer Methods 0.000 claims description 14
- 238000012545 processing Methods 0.000 claims description 13
- 238000012795 verification Methods 0.000 claims description 13
- 238000004891 communication Methods 0.000 claims description 12
- 230000008569 process Effects 0.000 claims description 11
- 230000005540 biological transmission Effects 0.000 claims description 5
- 230000007246 mechanism Effects 0.000 claims description 5
- 238000005516 engineering process Methods 0.000 abstract description 7
- 230000004927 fusion Effects 0.000 abstract description 5
- 238000010586 diagram Methods 0.000 description 4
- 238000011144 upstream manufacturing Methods 0.000 description 3
- 230000002159 abnormal effect Effects 0.000 description 2
- 238000004458 analytical method Methods 0.000 description 2
- 230000006870 function Effects 0.000 description 2
- 230000004048 modification Effects 0.000 description 2
- 238000012986 modification Methods 0.000 description 2
- 230000011218 segmentation Effects 0.000 description 2
- 230000002776 aggregation Effects 0.000 description 1
- 238000004220 aggregation Methods 0.000 description 1
- 238000006243 chemical reaction Methods 0.000 description 1
- 238000013500 data storage Methods 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 230000018109 developmental process Effects 0.000 description 1
- 230000002349 favourable effect Effects 0.000 description 1
- 230000003993 interaction Effects 0.000 description 1
- 238000012544 monitoring process Methods 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 230000009466 transformation Effects 0.000 description 1
- 238000000844 transformation Methods 0.000 description 1
Images
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/01—Protocols
- H04L67/06—Protocols specially adapted for file transfer, e.g. file transfer protocol [FTP]
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/18—File system types
- G06F16/182—Distributed file systems
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/27—Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L63/00—Network architectures or network communication protocols for network security
- H04L63/08—Network architectures or network communication protocols for network security for authentication of entities
- H04L63/0876—Network architectures or network communication protocols for network security for authentication of entities based on the identity of the terminal or configuration, e.g. MAC address, hardware or software configuration or device fingerprint
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/01—Protocols
- H04L67/10—Protocols in which an application is distributed across nodes in the network
- H04L67/1097—Protocols in which an application is distributed across nodes in the network for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS]
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Signal Processing (AREA)
- Databases & Information Systems (AREA)
- Computer Networks & Wireless Communication (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Physics & Mathematics (AREA)
- Data Mining & Analysis (AREA)
- Computing Systems (AREA)
- Power Engineering (AREA)
- Computer Hardware Design (AREA)
- Computer Security & Cryptography (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention relates to the technical field of financial science and technology, and discloses a file management system, a method and equipment, which comprise the following steps: the FDSS is used for recording and storing the metadata of the file; the FPS is used for storing file resources of the file and/or determining target resources based on metadata of the file; the SFTP file system is used for storing metadata of a file to be uploaded to the FDSS and storing file resources to the FPS when a file uploading request is received; and/or determining metadata of the file to be downloaded from the FDSS when the file downloading request is received, and downloading the file to be downloaded from the FPS based on the metadata of the file to be downloaded. The metadata and the file resources are distributed and stored in the FDSS and the FPS, and the file management operation is executed through the SFTP, so that the fusion management of the metadata and the file resources is realized, and the high availability of the file service and the safety of the file storage are improved.
Description
Technical Field
The invention relates to the technical field of financial technology (Fintech), in particular to a file management system, method and equipment.
Background
With the development of computer technology, more and more technologies are applied in the financial field, and the traditional financial industry is gradually changing to financial technology (Fintech), but higher requirements are also put forward on the technologies due to the requirements of the financial industry on safety and real-time performance.
The current SFTP (Secure File Transfer Protocol) service scheme mainly includes a stand-alone scheme and a cluster scheme, where both schemes store files in a local File system, and for the stand-alone scheme, if a server fails, a File may be lost, and a plurality of SFTP servers need to be deployed in the cluster scheme, but it cannot be guaranteed that a back-end remote server cannot fail, so that the availability of the cluster scheme is not high enough.
Disclosure of Invention
The invention provides a file management system, a file management method and file management equipment, and aims to improve the usability of file service and the safety of file storage.
To achieve the above object, the present invention provides a file management system, including:
the file directory storage system FDSS is used for recording and storing the metadata of the file;
the file processing system FPS is used for storing file resources of the file and/or determining target file resources based on metadata of the file;
the file system comprises a Secure File Transfer Protocol (SFTP) file system, a file system and a file system, wherein the SFTP file system is used for storing metadata of a file to be uploaded to the FDSS and storing file resources to the FPS when receiving a file uploading request; and/or the device is used for determining the metadata of the file to be downloaded from the FDSS when a file downloading request is received, and downloading the file to be downloaded from the FPS based on the metadata of the file to be downloaded.
Optionally, the SFTP file system includes:
the device comprises a first receiving unit, a second receiving unit and a third receiving unit, wherein the first receiving unit is used for receiving a file uploading request, acquiring a corresponding file to be uploaded, and storing the file to be uploaded to the FPS so that the FPS can acquire metadata to be uploaded of the file to be uploaded;
the second receiving unit is used for receiving the metadata to be uploaded returned by the FPS;
the sending unit is used for sending the metadata to be uploaded to the FDSS through a message bus so that the FDSS can store the metadata to be uploaded and send prompt information of successful storage after successful storage;
and the feedback unit is used for deleting the file to be uploaded stored in the local storage unit after receiving the prompt message of successful storage, and feeding back a message of successful uploading to the corresponding client.
Optionally, the SFTP file system includes:
the third receiving module is used for receiving a file downloading request and determining metadata to be downloaded of a file to be downloaded based on the file downloading request;
the query module is used for querying file index information of a corresponding file to be downloaded from the FDSS based on the metadata to be downloaded;
the application module is used for applying for downloading the file to be downloaded from the FPS based on the file index information and storing the file to be downloaded to a local storage unit;
and the downloading module is used for acquiring the file to be downloaded from the local storage unit, transmitting the file to be downloaded to a corresponding client, and deleting the file to be downloaded stored in the local storage unit after the transmission is finished.
Optionally, the file management system further includes:
and the authentication service system ASS is used for verifying the legality of the login request sent by the client and accessing the corresponding client after the verification is successful.
Optionally, the authentication service system includes:
the login module is used for receiving the user name, the SFTP public key and the session information of the client, which are acquired by the SFTP file system based on a login request sent by the client;
and the verification module is used for constructing a public key authenticator by the user name, the SFTP public key and the session information, carrying out validity verification based on the public key authenticator and accessing the client through the SFTP file system after the verification is successful.
Optionally, the SFTP file system is connected to the FPS through a gateway; and the SFTP file system and the FDSS realize communication through a message bus.
Optionally, the FDSS is further used for creating folders and editing files.
Optionally, the FPS file system includes:
the distributed coordination service Zookeeper unit is used for determining a corresponding execution process based on a lock grabbing mechanism when receiving metadata to be downloaded or a file to be uploaded sent by SFTP;
the Haodu distributed file system HDFS unit is used for storing the file to be uploaded to an EPS and recording the actual storage position of the file to be uploaded;
and the distributed database Hbase unit is used for storing copy information of the file to be uploaded so as to ensure that the file to be uploaded is not lost.
In addition, to achieve the above object, the present invention further provides a file management method, including:
after a file management system is built, receiving a file management request sent by a client, wherein the file management request comprises a file uploading request and a file downloading request, and the file management system comprises a file directory storage system, a file processing system and a secure file transfer protocol file system;
and managing the files to be managed in the file management request according to a preset flow.
In addition, in order to achieve the above object, the present invention further provides a file management device, which includes a processor, a memory, and a file management program stored in the memory, wherein when the file management program is executed by the processor, the file management method is implemented.
Compared with the prior art, the invention provides a file management system, a method and equipment, comprising the following steps: the file directory storage system FDSS is used for recording and storing the metadata of the file; the file processing system FPS is used for storing file resources of the file and/or determining target file resources based on metadata of the file; the file system comprises a Secure File Transfer Protocol (SFTP) file system, a file system and a file system, wherein the SFTP file system is used for storing metadata of a file to be uploaded to the FDSS and storing file resources to the FPS when receiving a file uploading request; and/or the device is used for determining the metadata of the file to be downloaded from the FDSS when a file downloading request is received, and downloading the file to be downloaded from the FPS based on the metadata of the file to be downloaded. The metadata and the file resources are distributed and stored in the FDSS and the FPS, and the file management operation is executed through the SFTP, so that the fusion management of the metadata and the file resources is realized, and the high availability of the file service and the safety of the file storage are improved.
Drawings
Fig. 1 is a schematic diagram of a hardware configuration of a file management apparatus according to embodiments of the present invention;
FIG. 2 is a schematic diagram showing the components of a file management system according to a first embodiment of the present invention;
FIG. 3 is a schematic view of an operation scenario of a file management system according to a first embodiment of the present invention;
fig. 4 is a flowchart illustrating a file management method according to a second embodiment of the present invention.
The implementation, functional features and advantages of the objects of the present invention will be further explained with reference to the accompanying drawings.
Detailed Description
It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention.
Referring to fig. 1, fig. 1 is a hardware configuration diagram of a file management apparatus according to embodiments of the present invention. In this embodiment of the present invention, the file management apparatus may include a processor 1001 (e.g., a Central Processing Unit, CPU), a communication bus 1002, an input port 1003, an output port 1004, and a memory 1005. The communication bus 1002 is used for realizing connection communication among the components; the input port 1003 is used for data input; the output port 1004 is used for data output, the memory 1005 may be a high-speed RAM memory, or a non-volatile memory (non-volatile memory), such as a magnetic disk memory, and the memory 1005 may optionally be a storage device independent of the processor 1001. Those skilled in the art will appreciate that the hardware configuration depicted in FIG. 1 is not intended to be limiting of the present invention, and may include more or less components than those shown, or some components in combination, or a different arrangement of components.
The embodiment of the invention provides a file management system.
Referring to fig. 2, fig. 2 is a schematic diagram of a file management system according to a first embodiment of the present invention, and as shown in fig. 2, the file management system includes:
the file directory storage system FDSS is used for recording and storing the metadata of the file;
the File Directory Storage System (FDSS) has as its main responsibility to record and store metadata of a file, including file attributes (fs _ file _ attr) including id of the file, type of file (e.g., folder, normal file, soft link, other), authority of the file, file size, file index information, creation time, and modification time. It can be understood that the file attributes of different files are different, so that each file can be marked according to the file attributes, and the corresponding file is determined based on the file attributes.
In addition, the metadata further includes a file tree (fs _ file _ tree). Generally, a folder may have multiple child files, and the child folder includes grandchild files, which in turn may be like a tree if the parent file, child files, and grandchild files are represented by a hierarchy. That is, the file tree records the hierarchical relationship between the files, including the file id (identification), the parent file id, the name of the file (name), the absolute path of the file (absolute _ path), the creation time and the modification time of the file.
In the embodiment, the file tree is created by the FDSS, that is, the FDSS is also used for organizing the file tree according to the hierarchical relationship between the child file and the parent file. The file tree records file information and hierarchical relationship of files on each node. In the process of recording the hierarchical relationship, the FDSS firstly judges whether a parent file and a file to be uploaded exist, then inserts file information of the newly received file to be uploaded into a file attribute fs _ file _ attr, and records the hierarchical relationship between the files in a constructed file tree fs _ file _ tree according to a returned unique file id of the file to be uploaded. After the FDSS records the file tree, the hierarchical relationship between the newly received file to be uploaded and other files in the folder can be determined according to the file id on the file tree, and the hierarchical relationship is favorable for file analysis and can quickly determine the associated file.
Further, the FDSS is also used for creating folders and editing files. Editing the file includes creating the file, deleting the file, inquiring the file, updating the file, moving the file, etc. When creating a file, the FDSS receives file information such as a file name, file index information, a file path, file attribute information, and a file size transmitted from the SFTP.
And when the FDSS receives an instruction for creating the folder, creating an empty new folder, naming the folder according to the folder name specified by the user or a default folder name, and marking the file of the new folder. And after the FDSS receives the instruction of editing the file, acquiring specific editing operation and the target file to be edited, editing the target file to be edited based on the editing operation, and after the editing is finished, sending notification information by the SFTP for the SFTP to notify the FDSS to synchronously update the metadata of the edited target file to be edited.
The deployment architecture of the FDSS is distributed, file information is mapped into records in a database, high availability of the database is guaranteed through a main-standby synchronization mechanism between the databases, and the high availability of the FDSS is guaranteed through the distributed architecture, so that the high availability of the whole storage service is guaranteed. In this embodiment, the primary and standby synchronization is implemented by synchronously storing the files stored in the primary database to the secondary database.
Further, the FDSS and the SFTP file system realize communication through a message bus. A Message bus (MQ) is a cross-process communication mechanism for passing messages between upstream and downstream. MQ is a common message communication service of 'logical decoupling + physical decoupling' between an upstream and a downstream, and the message sending upstream only needs to rely on the MQ and does not depend on other services logically and physically. In this embodiment, the communication between the FDSS and the SFTP file system is realized through the message bus.
The file processing system FPS is used for storing file resources of the file and determining target resources based on metadata of the file;
specifically, the FPS (file process system) is constructed based on a large data platform, and a basic technical platform with the aid of the FPS can realize functions such as file storage, transmission, downloading, retrieval, authority control, content management and the like.
In this embodiment, the FPS file system includes:
the distributed coordination service Zookeeper unit is used for determining a corresponding execution process based on a lock grabbing mechanism when receiving metadata to be downloaded or a file to be uploaded sent by SFTP;
the Haodu distributed file system HDFS unit is used for storing the file to be uploaded to an EPS and recording the actual storage position of the file to be uploaded;
and the distributed database Hbase unit is used for storing copy information of the file to be uploaded so as to ensure that the file to be uploaded is not lost.
HBase is a distributed, column-oriented open database distributed database. HBase has the following characteristics: strong read-write consistency, but not "final consistency" data storage, making it very suitable for high-speed computing aggregation; automatic segmentation, namely, the automatic segmentation is dispersed in a cluster through a Region, and when the number of rows is increased, the Region can also automatically segment and redistribute; automatic failover; Hadoop/HDFS is integrated, and the HDFS can be used after being unpacked, so that the connection is simple and convenient.
Hadoop Distributed File System (HDFS) refers to a Distributed File System (Distributed File System) designed to fit on general purpose hardware (comfort hardware). HDFS has many similarities to existing distributed file systems. But at the same time, its distinction from other distributed file systems is also clear. HDFS is a highly fault tolerant system suitable for deployment on inexpensive machines. HDFS provides high throughput data access and is well suited for application on large-scale data sets.
The FPS file system is realized based on HDFS + Zookeeper + Hbase. The HBASE stores index information of all files, including file names corresponding to file _ id, compressed file sizes, HDFS paths for file storage, file storage time and the like. The HDFS is a location where a file is actually saved, and after the uploaded file is placed in a temporary file (tmp) directory, a merging process merges the tmp file into a large file of the HDFS directory. The number of file copies stored in the HDFS directory is greater than 1, and may be 3, for example, when the number of copies is 3, it is theoretically ensured that data is not lost when the HDFS hangs up two machines. The Zookeeper is responsible for the coordination of locks, generally, the operation of executing one file editing can be executed only by one process, and each process is coordinated in a way of locking.
The FPS stores files through the HDFS, and stores at least 2 copies, so that file resources are not lost, reliable storage of the file resources is guaranteed, and high availability of the FPS of the file processing system is improved. In addition, the embodiment adopts HBASE and HDFS to support mass file storage, and the bearing capacity of the file is also improved.
Therefore, the metadata is stored in the FDSS, the file resources are stored in the FPS, the load of the SFTP server is reduced, the dependence on the SFTP server is reduced, and meanwhile, the performance requirement on the SFTP server is also reduced, so that the SFTP server becomes a front-end processor only with protocol conversion and simple file interaction.
The SFTP file system is used for storing the metadata of the file to be uploaded to the FDSS and storing the file resource to the FPS when receiving the file uploading request; and/or the device is used for determining the metadata of the file to be downloaded from the FDSS when a file downloading request is received, and downloading the file to be downloaded from the FPS based on the metadata of the file to be downloaded.
I.e. the SFTP is used to perform file management operations on the file.
The SFTP file system is based on an SFTP server. SFTP (Secure File Transfer Protocol) is a network Transfer Protocol that provides File access and management based on a reliable data stream (data stream), and provides a more Secure File Transfer between a client and a server than the FTP Protocol.
The SFTP file system performs management operations on files based on the SFTP server, and multiple SFTP servers may be deployed. The SFTP file system is a Rmb file system rmbfilesys that communicates on a reliable message bus (Rmb). RmbFileSystems integrates distributed file storage and database technologies, file resources and metadata are stored in a distributed mode, file management operations are mapped into database operations of FDSS, files of SFTP are separated from linux disk storage media, distributed storage of the files is achieved, and accordingly high availability of overall services of the SFTP is guaranteed.
Further, the file management system further includes:
an authentication service system ass (authentication service system) for verifying the validity of the login request sent by the client and accessing the corresponding client after the verification is successful.
Specifically, the authentication service system includes:
the login module is used for receiving a user name (username), an SFTP public key and session information of the client, which are acquired by the SFTP file system based on a login request sent by the client; before sending a file management request, a client needs to send a login request to an SFTP server to log in the SFTP server, and after receiving the login request, the SFTP server obtains a user name (username), an SFTP public key and session information of the client based on the login request.
And the verification module is used for constructing a Public key Authenticator (Public key Authenticator) by the user name, the SFTP Public key and the session information, carrying out validity verification based on the Public key Authenticator, and accessing the client through the SFTP file system after the validity verification is successful. In this embodiment, if the SFTP public key in the public key authenticator is consistent with the stored preset public key of the user name, it is determined that the public key authenticator is valid; otherwise, if the SFTP public key in the public key authenticator is inconsistent with the stored preset public key of the user name, the public key authenticator is judged to be illegal. If the file is illegal, the corresponding client is not accessed, so that the safety of the file management system is ensured.
In the embodiment, the original authentication function of the SFTP server is transformed into the authentication service under the distributed micro-service architecture, and the SFTP public key of each user is stored in the database at the back end, so that the safety of the public key information is ensured, meanwhile, the micro-service architecture enhances the high availability of the authentication service, and the login and the use of the user cannot be influenced by single-point failure.
The file management operation comprises uploading, downloading, deleting, changing and the like. In actual operation, the SFTP file system is matched with the FDSS, the FPS and the ASS to realize management operation. Wherein the SFTP file system is connected to the FPS via a gateway, such as a Tencent Get Way (TGW); the SFTP file system and the FDSS realize communication through a message bus; the SFTP file system and the ASS realize communication through a message bus, so that the SFTP file system realizes communication with the FPS and the ASS through an HTTP protocol.
Referring to fig. 3, fig. 3 is a schematic view of an operation scenario of the file management system according to the first embodiment of the present invention.
Specifically, the SFTP file system includes:
the first receiving unit is used for receiving a file uploading request, acquiring a corresponding file to be uploaded, and storing the file to be uploaded to the FPS so that the FPS can acquire metadata to be uploaded of the file to be uploaded.
Specifically, after the file to be uploaded is written into a local storage unit, the file to be uploaded is stored to the FPS, so that the FPS can obtain file index information;
after a client sends a file uploading request, a preposed NGINX (proxy server) forwards the file uploading request to an SFTP server, so that the file uploading request is analyzed by the SFTP server receiving the file uploading request to determine a file to be uploaded. And after the File to be uploaded is written into a local storage unit, storing the File to be uploaded to the FPS so that the FPS can acquire File index information in metadata, wherein the File index information comprises (File Id and File hash value).
The second receiving unit is used for receiving the metadata to be uploaded returned by the FPS;
the sending unit is used for sending the metadata to be uploaded to the FDSS through a message bus so that the FDSS can store the metadata to be uploaded and send prompt information of successful storage after successful storage;
specifically, the file index information in the metadata to be uploaded and the file directory and the file name of the file to be uploaded are sent to the FDSS through a message bus, so that the FDSS stores the file index information, the file directory and the file name, and sends out prompt information of successful storage after successful storage;
and the feedback unit is used for deleting the file to be uploaded stored in the local storage unit after receiving the prompt message of successful storage, and feeding back a message of successful file uploading to the corresponding client.
In addition, the information of the parent file and the child file of the file can be acquired through the FPS, and if the corresponding parent file and/or the corresponding child file are/is stored in the FPS, the corresponding file tree is perfected according to the file information of the parent file and/or the child file.
Further, to implement the downloading operation, the SFTP file system includes:
the third receiving module is used for receiving a file downloading request, and determining metadata to be downloaded of a file to be downloaded based on the file downloading request, wherein the metadata to be downloaded comprises a file name and a file directory; after a client sends a file downloading request, the preposed NGINX forwards the file downloading request to an SFTP server, so that the file downloading request is analyzed by the SFTP server receiving the file downloading request to determine a file to be downloaded.
The query module is used for querying File index information of a corresponding File to be downloaded from the File directory storage system based on the File name and the File directory of the File to be downloaded, wherein the File index information comprises a File Id and a File hash value;
the application module is used for applying for downloading the file to be downloaded from the FPS based on the file index information and storing the file to be downloaded to a local storage unit;
and the downloading module is used for acquiring the file to be downloaded from the local storage unit, transmitting the file to be downloaded to a corresponding client, and deleting the file to be downloaded stored in the local storage unit after the transmission is finished.
Further, in this embodiment, the file in the local file system is operated through a FileChannel file pipeline, and the FileChannel provides operations such as file opening, uploading, and downloading. In this embodiment, an RmbFileChannel is created and used in an RmbFileSystem File system, and when a File is opened, an RmbFileChannel File pipeline may determine whether an opened path exists, that is, whether corresponding File information exists is queried from an FDSS according to a path of the File and a name of the File, and if the corresponding File information exists, index information (File Id and File hash) of the File is obtained, and the corresponding File is downloaded to a tem directory in time based on an FPS, and the RmbFileChannel points to the obtained File to be downloaded, so that accurate and fast reading of the File to be downloaded is achieved.
Further, when the SFTP file system creates a file, the RmbFileChannel writes the file to be created into the local file system first, and then the FPS uploads the file in time and saves the metadata into the FDSS.
Further, the file management system also comprises a log management module. The log management module is used for generating and storing logs for monitoring and management based on the logs, so that abnormal analysis can be performed based on the logs after abnormal conditions occur, and normal operation of the file management system is guaranteed.
The present embodiment provides a file management system according to the foregoing scheme, including: the file directory storage system FDSS is used for recording and storing the metadata of the file; the file processing system FPS is used for storing file resources of the file and/or determining target file resources based on metadata of the file; the file system comprises a Secure File Transfer Protocol (SFTP) file system, a file system and a file system, wherein the SFTP file system is used for storing metadata of a file to be uploaded to the FDSS and storing file resources to the FPS when receiving a file uploading request; and/or the device is used for determining the metadata of the file to be downloaded from the FDSS when a file downloading request is received, and downloading the file to be downloaded from the FPS based on the metadata of the file to be downloaded. The metadata and the file resources are distributed and stored in the FDSS and the FPS, and the file management operation is executed through the SFTP, so that the fusion management of the metadata and the file resources is realized, and the high availability of the file service and the safety of the file storage are improved.
Based on the file management system described in fig. 1, a second embodiment of the present invention provides a file management method, referring to fig. 4, fig. 4 is a flowchart illustrating the file management method according to the second embodiment of the present invention, and as shown in fig. 4, the file management method includes:
step S101: after a file management system is built, receiving a file management request sent by a client, wherein the file management request comprises a file uploading request and a file downloading request, and the file management system comprises a file directory storage system, a file processing system and a secure file transfer protocol file system;
step S102: and managing the files to be managed in the file management request according to a preset flow.
In this embodiment, a file management system is pre-established, where the file management system includes a directory storage system FDSS, a file processing system FPS, a secure file transfer protocol SFTP file system, and an authentication service system.
After the file management system is built, the authentication service system verifies the received file management request, and after the verification is passed, the file management request is accessed, and the file to be managed in the file management request is managed according to a preset flow. The file management request comprises a file uploading request, a file downloading request, a file deleting request and a file changing request.
If the file management request is a file uploading request, receiving the file uploading request, acquiring a corresponding file to be uploaded, writing the file to be uploaded into a local storage unit, and storing the file to be uploaded to the FPS so that the FPS can acquire file index information;
after a client sends a file uploading request, a preposed NGINX (proxy server) forwards the file uploading request to an SFTP server, so that the file uploading request is analyzed by the SFTP server receiving the file uploading request to determine a file to be uploaded. And after the File to be uploaded is written into a local storage unit, storing the File to be uploaded to the FPS so that the FPS can acquire File index information, wherein the File index information comprises (File Id and File hash value).
Receiving the file index information returned by the FPS;
sending the file index information, the file directory of the file to be uploaded and the file name to the FDSS through a message bus, so that the FDSS can store the file index information, the file directory and the file name, and sending prompt information of successful storage after successful storage;
and after receiving the prompt information of successful storage, deleting the file to be uploaded stored in the local storage unit, and feeding back a message of successful file uploading to the corresponding client.
Further, the information of the parent file and the child file of the file can be obtained through the FPS, and if the corresponding parent file and/or the corresponding child file are/is stored in the FPS, the corresponding file tree is perfected according to the file information of the parent file and/or the child file.
Further, if the file management request is a file downloading request, determining a file name and a file directory of a file to be downloaded based on the file downloading request; after a client sends a file downloading request, the preposed NGINX forwards the file downloading request to an SFTP server, so that the file downloading request is analyzed by the SFTP server receiving the file downloading request to determine a file to be downloaded.
Based on the File name and the File directory of the File to be downloaded, inquiring File index information of the corresponding File to be downloaded from the File directory storage system, wherein the File index information comprises a File Id and a File hash value;
applying for downloading the file to be downloaded from the FPS based on the file index information, and storing the file to be downloaded to a local storage unit;
and acquiring the file to be downloaded from the local storage unit, transmitting the file to be downloaded to a corresponding client, and deleting the file to be downloaded stored in the local storage unit after the transmission is finished.
According to the scheme, after the file management system is built, the file management request sent by the client is received, wherein the file management request comprises a file uploading request and a file downloading request, and the file management system comprises a file directory storage system, a file processing system and a secure file transfer protocol file system; and managing the files to be managed in the file management request according to a preset flow. Therefore, file management is carried out based on the file management system, the fusion management of metadata and file resources can be realized, and the high availability of file service and the safety of file storage are improved.
Compared with the prior art, the invention discloses a file management system, a method and equipment, comprising the following steps: the file directory storage system FDSS is used for recording and storing metadata of files, wherein the metadata comprises file attributes and file trees; the file processing system FPS is used for storing file resources of the file and determining target resources based on metadata of the file; and the Secure File Transfer Protocol (SFTP) file system is used for executing file management operation on the file. The metadata and the file resources are distributed and stored in the FDSS and the FPS, and the file management operation is executed through the SFTP, so that the fusion management of the metadata and the file resources is realized, and the high availability of the file service and the safety of the file storage are improved.
It should be noted that, in this document, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or system that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or system. Without further limitation, an element defined by the phrase "comprising an … …" does not exclude the presence of other like elements in a process, method, article, or system that comprises the element.
The above-mentioned serial numbers of the embodiments of the present invention are merely for description and do not represent the merits of the embodiments.
Through the above description of the embodiments, those skilled in the art will clearly understand that the method of the above embodiments can be implemented by software plus a necessary general hardware platform, and certainly can also be implemented by hardware, but in many cases, the former is a better implementation manner. Based on such understanding, the technical solution of the present invention may be embodied in the form of a software product, which is stored in a computer-readable storage medium (such as ROM/RAM, magnetic disk, and optical disk) as described above, and includes several instructions for enabling a terminal device to execute the method according to the embodiments of the present invention.
The above description is only for the preferred embodiment of the present invention and is not intended to limit the scope of the present invention, and all equivalent structures or flow transformations made by the present specification and drawings, or applied directly or indirectly to other related arts, are included in the scope of the present invention.
Claims (10)
1. A file management system, characterized in that the file management system comprises:
the file directory storage system FDSS is used for recording and storing the metadata of the file;
the file processing system FPS is used for storing file resources of the file and/or determining target file resources based on metadata of the file;
the file system comprises a Secure File Transfer Protocol (SFTP) file system, a file system and a file system, wherein the SFTP file system is used for storing metadata of a file to be uploaded to the FDSS and storing file resources to the FPS when receiving a file uploading request; and/or the device is used for determining the metadata of the file to be downloaded from the FDSS when a file downloading request is received, and downloading the file to be downloaded from the FPS based on the metadata of the file to be downloaded.
2. The file management system of claim 1, wherein the SFTP file system comprises:
the device comprises a first receiving unit, a second receiving unit and a third receiving unit, wherein the first receiving unit is used for receiving a file uploading request, acquiring a corresponding file to be uploaded, and storing the file to be uploaded to the FPS so that the FPS can acquire metadata to be uploaded of the file to be uploaded;
the second receiving unit is used for receiving the metadata to be uploaded returned by the FPS;
the sending unit is used for sending the metadata to be uploaded to the FDSS through a message bus so that the FDSS can store the metadata to be uploaded and send prompt information of successful storage after successful storage;
and the feedback unit is used for deleting the file to be uploaded stored in the local storage unit after receiving the prompt message of successful storage, and feeding back a message of successful uploading to the corresponding client.
3. The file management system of claim 1, wherein the SFTP file system comprises:
the third receiving module is used for receiving a file downloading request and determining metadata to be downloaded of a file to be downloaded based on the file downloading request;
the query module is used for querying file index information of a corresponding file to be downloaded from the FDSS based on the metadata to be downloaded;
the application module is used for applying for downloading the file to be downloaded from the FPS based on the file index information and storing the file to be downloaded to a local storage unit;
and the downloading module is used for acquiring the file to be downloaded from the local storage unit, transmitting the file to be downloaded to a corresponding client, and deleting the file to be downloaded stored in the local storage unit after the transmission is finished.
4. The file management system according to claim 1, further comprising:
and the authentication service system ASS is used for verifying the legality of the login request sent by the client and accessing the corresponding client after the verification is successful.
5. The document management system according to claim 4, wherein the authentication service system comprises:
the login module is used for receiving the user name, the SFTP public key and the session information of the client, which are acquired by the SFTP file system based on a login request sent by the client;
and the verification module is used for constructing a public key authenticator by the user name, the SFTP public key and the session information, carrying out validity verification based on the public key authenticator and accessing the client through the SFTP file system after the verification is successful.
6. The file management system of claim 1, wherein the SFTP file system is connected to the FPS through a gateway; and the SFTP file system and the FDSS realize communication through a message bus.
7. The file management system of claim 1, wherein the FDSS is further configured to create folders and edit files.
8. The file management system of claim 1, wherein the FPS file system comprises:
the distributed coordination service Zookeeper unit is used for determining a corresponding execution process based on a lock grabbing mechanism when receiving metadata to be downloaded or a file to be uploaded sent by SFTP;
the Haodu distributed file system HDFS unit is used for storing the file to be uploaded to an EPS and recording the actual storage position of the file to be uploaded;
and the distributed database Hbase unit is used for storing copy information of the file to be uploaded so as to ensure that the file to be uploaded is not lost.
9. A file management method, characterized in that the file management method comprises:
after a file management system is built, receiving a file management request sent by a client, wherein the file management request comprises a file uploading request and a file downloading request, and the file management system comprises a file directory storage system, a file processing system and a secure file transfer protocol file system;
and managing the files to be managed in the file management request according to a preset flow.
10. A file management apparatus comprising a processor, a memory and a file management program stored in said memory, said file management program, when executed by said processor, implementing the steps of the file management method according to claim 9.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202011515429.6A CN112653757B (en) | 2020-12-21 | 2020-12-21 | File management system, method and equipment |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202011515429.6A CN112653757B (en) | 2020-12-21 | 2020-12-21 | File management system, method and equipment |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN112653757A true CN112653757A (en) | 2021-04-13 |
| CN112653757B CN112653757B (en) | 2024-08-20 |
Family
ID=75358585
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202011515429.6A Active CN112653757B (en) | 2020-12-21 | 2020-12-21 | File management system, method and equipment |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN112653757B (en) |
Cited By (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113364887A (en) * | 2021-06-30 | 2021-09-07 | 北京天空卫士网络安全技术有限公司 | File downloading method based on FTP, proxy server and system |
| CN114173170A (en) * | 2021-12-06 | 2022-03-11 | Oppo广东移动通信有限公司 | Multi-device video sharing method, device, system, device and storage medium |
| CN114238241A (en) * | 2022-02-26 | 2022-03-25 | 杭州字节方舟科技有限公司 | Metadata processing method and computer system for financial data |
Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| KR20070038665A (en) * | 2005-10-06 | 2007-04-11 | 주식회사 솔루션박스 | Distributed file system and operating method thereof |
| CN104050248A (en) * | 2014-06-05 | 2014-09-17 | 北京捷成世纪科技股份有限公司 | File storage system and storage method |
| CN104731516A (en) * | 2013-12-18 | 2015-06-24 | 腾讯科技（深圳）有限公司 | Method and device for accessing files and distributed storage system |
| CN110177154A (en) * | 2019-06-17 | 2019-08-27 | 深圳前海微众银行股份有限公司 | A kind of file interaction processing method, apparatus and system |
| CN110209731A (en) * | 2019-04-25 | 2019-09-06 | 深圳壹账通智能科技有限公司 | Method of data synchronization, device and storage medium, electronic device |
| CN110213352A (en) * | 2019-05-17 | 2019-09-06 | 北京航空航天大学 | The unified Decentralized Autonomous storage resource polymerization of name space |
-
2020
- 2020-12-21 CN CN202011515429.6A patent/CN112653757B/en active Active
Patent Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| KR20070038665A (en) * | 2005-10-06 | 2007-04-11 | 주식회사 솔루션박스 | Distributed file system and operating method thereof |
| CN104731516A (en) * | 2013-12-18 | 2015-06-24 | 腾讯科技（深圳）有限公司 | Method and device for accessing files and distributed storage system |
| CN104050248A (en) * | 2014-06-05 | 2014-09-17 | 北京捷成世纪科技股份有限公司 | File storage system and storage method |
| CN110209731A (en) * | 2019-04-25 | 2019-09-06 | 深圳壹账通智能科技有限公司 | Method of data synchronization, device and storage medium, electronic device |
| CN110213352A (en) * | 2019-05-17 | 2019-09-06 | 北京航空航天大学 | The unified Decentralized Autonomous storage resource polymerization of name space |
| CN110177154A (en) * | 2019-06-17 | 2019-08-27 | 深圳前海微众银行股份有限公司 | A kind of file interaction processing method, apparatus and system |
Cited By (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113364887A (en) * | 2021-06-30 | 2021-09-07 | 北京天空卫士网络安全技术有限公司 | File downloading method based on FTP, proxy server and system |
| CN113364887B (en) * | 2021-06-30 | 2022-11-15 | 北京天空卫士网络安全技术有限公司 | File downloading method based on FTP, proxy server and system |
| CN114173170A (en) * | 2021-12-06 | 2022-03-11 | Oppo广东移动通信有限公司 | Multi-device video sharing method, device, system, device and storage medium |
| CN114238241A (en) * | 2022-02-26 | 2022-03-25 | 杭州字节方舟科技有限公司 | Metadata processing method and computer system for financial data |
| CN114238241B (en) * | 2022-02-26 | 2022-05-27 | 杭州字节方舟科技有限公司 | Metadata processing method and computer system for financial data |
Also Published As
| Publication number | Publication date |
|---|---|
| CN112653757B (en) | 2024-08-20 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US12061623B2 (en) | Selective synchronization of content items in a content management system | |
| CN112653757B (en) | File management system, method and equipment | |
| US7487551B2 (en) | Access to content addressable data over a network | |
| AU757667B2 (en) | Access to content addressable data over a network | |
| CN115731069A (en) | Method and system for detecting file quarticity of electronic files applied to information system of nuclear power plant | |
| CN111563083A (en) | Report data query method, device and system | |
| CN117376364A (en) | Data processing method and related equipment | |
| CN113810468A (en) | Method, system, device and storage medium for distributing request by gateway under K8s architecture |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |