CN116821073A

CN116821073A - Universal file storage method and system based on distributed system

Info

Publication number: CN116821073A
Application number: CN202310792311.5A
Authority: CN
Inventors: 余西; 李耀; 彭磊
Original assignee: Wuhan Zhongbang Bank Co Ltd
Current assignee: Wuhan Zhongbang Bank Co Ltd
Priority date: 2023-06-29
Filing date: 2023-06-29
Publication date: 2023-09-29

Abstract

The application relates to a general file storage method and a system based on a distributed system, wherein a client uploads storage data to an application server; the application server configures corresponding transfer parameters to the storage data according to the type of the object storage server; creating a connection corresponding to the object store according to different object store server types; judging the access authority URL corresponding to the authority type configuration according to the actual demand, linking the access authority URL to the stored data, and storing the access authority URL in a database; and judging whether the URL link needs access rights according to the actual requirements, and returning the corresponding URL link to the access terminal according to the judging result. Therefore, the object of the application is stable and reliable in storage, and the storage space is not limited, so that the problem of capacity expansion is not required, and the elastic expansion of the storage requirement can be realized; in addition, the OSS does not need the purchase, deployment and operation of the traditional hardware, and operation and maintenance work is saved. The method is applicable to three object storage modes, and greatly saves the workload of redevelopment due to the adjustment of object storage types.

Description

Universal file storage method and system based on distributed system

Technical Field

The present application relates to the field of computers, and in particular, to a method and a system for storing a general file based on a distributed system.

Background

At present, the market system stores pictures such as commodity pictures and the like on a file server, and as the business volume of the market system increases, the demand of commodity pictures, commodity detail pictures, commodity introduction videos and the like uploaded by a provider platform is gradually increased, and the traditional file server cannot meet the prior art; the traditional mode on the file server has poor expansibility, namely the file server cannot be elastically expanded when the space of the file server is insufficient, and manual deployment and maintenance are required; meanwhile, due to the increase of the access quantity of clients, when a large number of clients view commodity lists and commodity detail pages of the mall, static resources such as file server pictures and the like are read concurrently; under the background, OSS object storage can be used, the OSS object storage is stable and reliable, single-node faults are eliminated based on high-availability architecture design, and the continuity of service is ensured; meanwhile, the storage space has no upper limit, the problem of capacity expansion is not required, and the elastic expansion of the storage requirement can be realized; in addition, the OSS does not need the purchase, deployment and operation of the traditional hardware, and operation and maintenance work is saved.

The storage mode of the original file server adopted in the traditional mall project comprises files such as commodity pictures, commodity details, video introduction and the like of a provider platform and various excel documents.

With the continuous development of business and the promotion of various marketing activities, the access amount of clients in the mall is gradually increased, and the number of purchased goods in the mall is gradually increased. In order to rapidly meet the development of high-speed business, the system is updated at high speed, the defects of traditional file storage are gradually shown, the storage of a server is required to be continuously expanded, the manual maintenance cost is high, and the uploading efficiency of batch pictures is low.

Accordingly, attempts may be made to store files using object storage to address this problem.

Disclosure of Invention

The technical problems of the application are mainly solved by the following technical proposal:

a general file storage method based on a distributed system is characterized by comprising the steps that a client uploads storage data to an application server;

the application server configures corresponding transfer parameters to the storage data according to the type of the object storage server;

creating a connection corresponding to the object store according to different object store server types;

judging the access authority URL corresponding to the authority type configuration according to the actual demand, linking the access authority URL to the stored data, and storing the access authority URL in a database;

and judging whether the URL link needs access rights according to the actual requirements, and returning the corresponding URL link to the access terminal according to the judging result.

As a preferred alternative to this,

and the client selects the picture/video file to upload, and transmits the file information to the application server through the interface.

As a preferred alternative to this,

the application server assembles a request parameter, wherein the request parameter comprises a File stream, and the parameter Type field comprises tpye=S3, tpye=OSS and tpye=minio;

judging the type of the used object storage server, and storing the server according to different objects used by the user

As a preferred alternative to this,

judging the permission type according to actual requirements, selecting private/public read-write permission of the socket, and assembling corresponding parameters; the corresponding parameters include an uploaded File stream File, a bucket name bucketName, a File name fileName, a File type mediaType and a URL link expiration time

As a preferred alternative to this,

when selecting the private/public read-write rights of the socket, the method specifically comprises the following steps:

judging that if the socket is a private authority, the URL link expiration time field needs to be transmitted, and when the link is expired, the URL link cannot be accessed; if public read-write or public read, the URL link is permanently valid.

And assembling all parameters, calling a corresponding API and uploading the file stream to an object storage.

As a preferred alternative to this,

during access, whether the URL link needs access permission or not is judged according to actual requirements, and the URL link is returned to the client, wherein the method specifically comprises the following steps of

If access is required, encrypting the returned URL link in the application server to generate a new URL link and a corresponding key.

Storing the encrypted generated new URL link and the key into the redis, and setting the expiration time.

If the access authority is not required to be set, the URL is directly returned to the client for use.

As a preferred alternative to this,

setting an expiration time, wherein after expiration, the link cannot be accessed;

the default new link is valid all the time without setting the expiration time, and the new link of the deposit point in redis is destroyed or expired when the link is leaked or is no longer externally accessed.

As a preferred alternative to this,

and the access terminal displays the picture or the file according to the returned URL. If the file has the access right, the application server side checks the access right, and if the key is not right, the application server side prompts that the file cannot be accessed.

A universal file storage system based on a distributed system includes

Client side: the method comprises the steps of uploading stored data to an application server;

the application server: configuring corresponding transfer parameters to the storage data according to the type of the object storage server;

the creation module: creating a connection corresponding to the object store according to different object store server types;

and a storage module: judging the access authority URL corresponding to the authority type configuration according to the actual demand, linking the access authority URL to the stored data, and storing the access authority URL in a database;

and the access module is used for judging whether the URL link needs access permission according to the actual demand, and returning the corresponding URL link to the access terminal according to the judgment result.

An access terminal: for accessing stored data.

Therefore, the application has the following advantages:

1. the object storage is stable and reliable, the single node fault is eliminated based on the high-availability architecture design, and the continuity of the service is ensured; meanwhile, the storage space has no upper limit, the problem of capacity expansion is not required, and the elastic expansion of the storage requirement can be realized; in addition, the OSS does not need the purchase, deployment and operation of the traditional hardware, and operation and maintenance work is saved.

2. The method is applicable to three object storage modes, and greatly saves the workload of redevelopment due to the adjustment of object storage types.

Drawings

Fig. 1 is a flow chart of the method of the present application.

Detailed Description

The technical scheme of the application is further specifically described below through examples and with reference to the accompanying drawings.

Example 1:

the application relates to a universal file storage method based on a distributed system, which specifically comprises the following steps:

step 1, a client selects a picture/video file to upload, and file information is transmitted to an application server through an interface

Step 2, the application server receives the data request, wherein the request contains the information of the uploading file

Step 3, the application server assembles the request parameters, and transmits the request parameters into the parameter Type field stored in the file

Step 4, judging the type of the used object storage server, and transmitting parameters according to different object storage servers used by the user

Step 5, creating connection corresponding to object storage according to different object storage server types

And 6, judging the authority type according to the actual requirement, selecting the private/public read-write authority of the socket, assembling corresponding parameters such as Access Key information, and creating a corresponding OSSCLIENT connection example.

Step 6.1, judging that if the socket is a private authority, the URL link expiration time field needs to be transmitted, and when the link is expired, the URL link expiration time field cannot be accessed; if public read-write or public read, the URL link is permanently valid.

And 6.2, assembling all parameters, calling a corresponding API and uploading the file stream to the object storage.

And 7, storing the URL link of the file returned in the step 6.2 into a database.

And 8, judging whether the URL link needs access rights according to actual requirements, and returning the URL link to the client.

And 8.1, if the authority access is required, encrypting the URL link returned in the step 6.2 in an application server to generate a new URL link and a corresponding key.

Step 8.2, storing the new URL link and the key returned in step 8.1 into redis, and setting expiration time. If so, the link cannot be accessed after expiration; if not, the default new link is always active. When the link is leaked or no external access is performed, the new link of the storage point in the redis can be destroyed or expired directly, and the file and the address stored by the object are not affected.

And 8.2, if the access authority does not need to be set, the URL can be directly returned to the access terminal for use.

Step 9: and the access terminal displays the picture or the file according to the returned URL. If the file has the access right, the application server side checks the access right, and if the key is not right, the application server side prompts that the file cannot be accessed.

Example 2

The embodiment provides a general file storage system based on a distributed system, which comprises a client: the method comprises the steps of uploading stored data to an application server;

An access terminal: for accessing stored data.

Example 3

This embodiment relates to a specific case (using OSS), using the advantages of OSS:

1. stable, service availability up to 99.9%, automatic system scale expansion, no influence on external service, triple backup of data, and reliability up to 99.9%;

2. safety: multilevel security protection and Dos attack prevention; a multi-user isolation mechanism;

large scale, high performance: the system has a large amount of storage space, increases the space elasticity, and can support high-concurrency and high-flow read-write access in the same time.

3. High concurrency, massive data, high availability, high reliability and no need of self operation and maintenance are supported.

4. The application platform is wide, the RESTful API can be used for storing and accessing any type of data at any application, any time and any place, the capacity and the processing capacity are elastically expanded, and a plurality of storage types can be used for selecting and comprehensively optimizing the storage cost.

5. The minio, the S3 and the OSS are supported, and can be switched at will according to specific conditions without the need of re-docking development.

The specific steps are described in detail below.

S1, selecting commodity detail picture uploading by a system front end (Vue)

S2, transferring the picture file to the application server through interface call

S3, establishing connection through S3, controlling access permission of access item pictures, uploading expiration time and setting 7-day validity period of currently used private socket

S4, judging the file type of the uploaded picture, setting the file type as 'image/jpg', and generating a file name and a key value stored in the picture

S5, after calling the API interface to upload the picture information, returning to the generated URL

S6, if encryption access is needed, processing the URL returned in S5 on the application server, generating a new URL link by using the specific character string splice, generating an access key and expiration time corresponding to the new URL link in the redis, and returning the newly generated URL link and key to the access terminal for viewing

S7, when the access terminal checks the commodity, the access terminal synchronously transmits the new URL link and the key to the application terminal, and the application terminal verifies that the key is correct and can only be accessed

S8, if the link is not checked by the access terminal any more, the corresponding new link in the operable redis is invalid or deleted directly.

The OSS object storage server is high in availability, can be elastically expanded, and is large in file storage capacity and high in storage speed. The method can be freely switched according to different file stores used by users without redevelopment. And re-encrypting the returned URL at the server side, wherein access authority control can be added, and meanwhile, the expiration time can be set for the URL which is not expired for an ultra-long time, and when the URL link is leaked or is not externally accessed any more, the sharing link is operated to be directly invalid, so that the actual file storage in the object storage is not influenced.

It will be appreciated by those skilled in the art that embodiments of the present application may be provided as a method, system, or computer program product. Accordingly, the present application may take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Furthermore, the present application may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, and the like) having computer-usable program code embodied therein. The scheme in the embodiment of the application can be realized by adopting various computer languages, such as object-oriented programming language Java, an transliteration script language JavaScript and the like.

The present application is described with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems) and computer program products according to embodiments of the application. It will be understood that each flow and/or block of the flowchart illustrations and/or block diagrams, and combinations of flows and/or blocks in the flowchart illustrations and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general purpose computer, special purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.

These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instruction means which implement the function specified in the flowchart flow or flows and/or block diagram block or blocks.

These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.

While preferred embodiments of the present application have been described, additional variations and modifications in those embodiments may occur to those skilled in the art once they learn of the basic inventive concepts. It is therefore intended that the following claims be interpreted as including the preferred embodiments and all such alterations and modifications as fall within the scope of the application.

It will be apparent to those skilled in the art that various modifications and variations can be made to the present application without departing from the spirit or scope of the application. Thus, it is intended that the present application also include such modifications and alterations insofar as they come within the scope of the appended claims or the equivalents thereof.

Claims

1. A general file storage method based on a distributed system is characterized by comprising the steps that a client uploads storage data to an application server;

2. The method for storing a universal file based on a distributed system according to claim 1, wherein,

3. The method for storing a universal file based on a distributed system according to claim 1, wherein,

and judging the type of the used object storage server, and storing the servers according to different objects used by the user.

4. The method for storing a universal file based on a distributed system according to claim 1, wherein,

judging the permission type according to actual requirements, selecting private/public read-write permission of the socket, and assembling corresponding parameters; the corresponding parameters comprise an uploading File stream File, a bucket name bucketName, a File name fileName, a File type media type and a URL link expiration time.

5. The method for storing a universal file based on a distributed system according to claim 1, wherein when selecting a private/public read/write right of a socket, the method specifically comprises:

judging that if the socket is a private authority, the URL link expiration time field needs to be transmitted, and when the link is expired, the URL link cannot be accessed; if public read-write or public read, the URL link is permanently valid;

6. The method for storing universal files based on distributed system as claimed in claim 1, wherein, during access, determining whether the URL link requires access authority according to actual requirement, and returning the URL link to the client, specifically comprising

If access is required, encrypting the returned URL link in the application server to generate a new URL link and a corresponding key;

storing the encrypted generated new URL link and the secret key into the redis, and setting the expiration time;

7. The method for storing a universal file based on a distributed system as recited in claim 6, wherein,

8. The method for storing a universal file based on a distributed system as recited in claim 6, wherein,

and the access terminal displays the picture or the file according to the returned URL, and if the file has the access right, the application server terminal checks the access right, and if the key is not right, the application server terminal prompts that the file cannot be accessed.

9. A universal file storage system based on a distributed system, comprising

the access module is used for judging whether the URL link needs access permission according to actual requirements and returning the corresponding URL link to the access terminal according to a judgment result;

an access terminal: for accessing stored data.