RU2656739C1 - Data storage method and system - Google Patents

Abstract

FIELD: data processing.

SUBSTANCE: claimed invention relates to the field of data processing. Data storage system comprises at least one object persistence layer (OPL) consisting of at least one server and at least one storage system (SS), OPL contains a unique identifier (fs_id); user devices generating requests for managing data stored or transmitted to the OPL, each request containing one or more files, each of which contains a unique identifier (file_id) or an access identifier for the file stored in the OPL; at least one metadata management server for registering in the network of the OPL servers and processing requests for managing data from user devices; one or more routers for load balancing on the OPL.

EFFECT: technical result is to provide a unified management of a plurality of information flows containing user requests for storage or sending data to target storage devices determined by network access policy, and speeding up the process of requesting/exchanging information.

7 cl, 6 dwg

Description

FIELD OF TECHNOLOGY

The claimed technical solution relates to the field of data processing, in particular to structures for storing and accessing information with many layers of target information storage.

BACKGROUND

Currently, the solutions on the market allow you to choose the type of data storage system (SHD) at the software installation stage, for example, solutions such as DropBox, Google Drive, etc. In the process of working with these solutions, it is not possible or difficult to change the data storage layer on the fly. A certain data migration procedure is required, i.e. dynamic redistribution of data across storage layers, which is not provided for in the mentioned solutions.

As a rule, existing storage solutions are SAN (Storage area network) (block) [1] and NAS (Network Attached Storage) (file, network) systems [2]. The main drawback is the lack of functionality of the EFSS level (Enterprise File Sync and Share), which allows you to dynamically manage multiple information streams based on storage access control policies and optimize data storage and loading routes.

SUMMARY OF THE INVENTION

A technical problem solved by the claimed solution is the expansion of the existing functionality of data storage facilities by providing unified access to storage systems integrated into a single network, which will make it possible to optimally use hardware with reduced computing load when processing many user requests to receive or send files to SHD.

The technical result consists in providing unified control of a plurality of information streams containing user requests for storing or sending data to target information storage devices defined by a network access policy and accelerating the request / exchange of information.

The claimed solution is implemented by a data storage system that contains at least one target data storage layer (DSC), consisting of at least one server and at least one storage system, and the DSP contains a unique identifier (fs_id);

user devices generating control requests for data stored or transmitted to the TSX, each request containing one or more files, each of which contains a unique identifier (file_id), or an access identifier for a file stored in the TSX;

at least one metadata management server, designed to register on a network of TSH servers and process requests for data management from user devices;

one or more routers for load balancing on a central host network;

moreover

each DSP server contains at least a unique identifier (srv_id) and an address in the data network, and is designed to process requests for managing data generated by user devices based on a set of data type management policies;

SHD is one or more data storage devices connected to one or more of the mentioned server DSS, where each of the devices contains a unique identifier (stor_id) and is designed to store and manage the data type corresponding to the policy of the corresponding storage server;

when processing the aforementioned user requests, the metadata management server, based on the identifier contained in the user request, determines a device for storing information in the corresponding DSS corresponding to the said request and performs routing for user access to the said DSS.

In one of the private options for the implementation of the system, a centralized storage facility is selected from the group: cloud storage with online access on the Internet, local storage on a local network or an intranet.

In another private embodiment of the system, storage devices are selected from the group: solid-state storage devices (SSD), hard disk (HDD), flash drives, RAID array, optical drives (CD / DVD / MD / BlueRay), tape drives (LTFS).

In another particular embodiment of the system, the file management policy is based at least on access rights to the TSH.

In another particular embodiment of the system, the DSP server is connected to the router via HTTP / HTTPS.

In another particular embodiment of the system, the set of policies includes processing user request information, which includes analyzing the user access level and / or binding the user to a particular DSP and / or the network address of the incoming request and / or the user request file identifier.

The claimed solution is also implemented due to the method of forming a data storage structure, comprising the steps in which:

at least one target data storage layer (DSC) is formed by creating an information exchange route between at least one server and at least one data storage system (SHD), wherein the SHD is one or more data storage devices;

when creating the mentioned information exchange route, registration is carried out in the network of the DSS servers and one or more storage systems using a metadata management server that contains a database of identifiers, and during the registration process, each DSS server is assigned a unique identifier (srv_id) and an IP address in the network, each The storage associated with the aforementioned DSC server is assigned a unique identifier (stor_id);

moreover, a management policy is formed for each DSP server, on the basis of which for each storage connected to the mentioned DSS server, the storage and access support of a certain data type is formed;

moreover, the metadata management server when processing requests received from user devices, where each request contains one or more types of files, determines the storage system corresponding to this type of file in the corresponding DSP.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 illustrates the general scheme of the claimed system.

FIG. 2 illustrates the structure of a meta server database.

FIG. 3 illustrates a flowchart of a method for forming a DSC.

FIG. 4 illustrates a flow chart of the steps of sending a file to a DSP.

FIG. 5 illustrates a block diagram of receiving a file from a DSP.

FIG. 6 illustrates a general diagram of a device performing the functions of a system element.

DETAILED DESCRIPTION OF THE INVENTION

In FIG. 1 shows a General view of the claimed system (100) data storage. System (100) contains many clients (110) that generate requests for receiving / transmitting various data. Clients (110) interact using various computing devices, such as mobile phones, smartphones, tablets, thin clients, personal computers, laptops, game consoles, etc.

Requests from clients (110) are transmitted to a router (120), which also acts as a load balancer. Requests can be transmitted using protocols such as HTTP / HTTPS. For example, a Nginx server (https://nginx.org/en/) can be used as a router and a load balancer.

Router (120) exchanges data for processing user requests with servers TsSH (130-1) - (130-n). Each server (130) contains a specific set of parameters for its operation and is associated with one or more corresponding storage systems (160), connected through a corresponding plug-in (a software module that implements interaction with one or another type of storage systems, for example, Amazon S3 protocol plug-in, NFS plug-in, CIFS plugin, etc.) (150). Servers (130-1) - (130-n) contain information about the corresponding addresses in the data network, for example, the Internet, Intranet, in particular IP addresses, records of users who are allowed to access to the specified access policy (rights) the corresponding DSS, and information about the storage systems connected to the server (160).

The servers (130-1) - (130-n) exchange data with the router (120) using the HTTP / HTTPS data transfer protocols.

System (100) also contains a meta-request processing server or a metaserver (140). This server (140) is primarily intended for managing metadata, registering a network of servers (130) of the central logistic center and processing requests for data management from user devices (110). Server (140) contains the metadata database (200) shown in FIG. 2).

The metaserver database (200) contains the organization of records for the exchange of information flows of requests. Base (200) consists of a storage module for storage (201), a module for recording information about servers (202), a module for recording information about DSS (203), and a module for recording information about files (204). Each module contains a corresponding identifier for identification in the information exchange chain and information on the corresponding identifiers of the associated modules for organizing the process of exchanging / receiving files at the request of users (110). By the corresponding identifiers of the associated modules, the process of recognizing the corresponding access policies, such as the files corresponding to the policy and routing access to the TSH, is organized.

Storage module (201) contains a unique identifier (stor_id), which identifies one or more storage systems (160), which is a device for storing information. Also, the module (201) contains information about the types of storage systems (160) and the parameters of their settings.

The server module (202) contains a unique server identifier (130) in the system (100) (srv_id), with which information is processed by the corresponding server, the server name and its network address.

Modules (201) and (202) are connected with the DSP module (203), which contains information of the identifier of the information storage layer (fs_id) and the identifiers of the storage system (stor_id) and server (srv_id) corresponding to this layer, i.e. information for access to the storage system (160) and the corresponding server (130), forming the DSS.

Module (204) contains information about file identifiers (file_id) corresponding to the IDC identifier (fs_id) for one or more file types.

According to FIG. 3, the process (300) of the formation of the central heating plant is carried out as follows. For each information processing server (130-1) - (130-n), it is linked in the network (301) by registering a metaserver (140) in the database (200). During the registration, a unique identifier srv_id is generated for each server (130). One of the parameters of the server is its IPv4 address on the Internet or on the local network. This address will be used in the routing process as the target.

Next, at step (302), one or more storage systems (160) are linked to the corresponding server defined in step (301), for which purpose the mentioned storage system is registered in the metaserver database (200) (140). A unique identifier (stor_id) is created for each storage system. Storage is determined by type (file system, data transfer rate, data protection, storage capabilities, etc.), settings (which include parameters such as mount points, IP addresses, login, access password, etc.).

Next, access policies are created for the central hostel (303), which are stored on the meta-information servers, usually forming a route for the host hostel. Access policies (a set of policies) may include, but are not limited to: user access level, user binding to a specific DSP, network address of an incoming client request (110), user request file identifier. For each client device (110) or a group of devices connected by a single network, for example, a corporate one creates its own access policy, which may also depend on the types of files sent to the TSX or requested from the TSX. The access level of a particular device, which can be associated with the corresponding identifier and also depends on the type of storage and the type of files.

The access policy can also be tied by the biometric identifier of the user device, which is generated and stored on the meta-information servers (140) to access the corresponding DSS.

SHD (160) can be selected from a wide range of storage media, for example, solid-state storage devices (SSD), hard disk (HDD), flash drives, RAID array, optical drives (CD / DVD / MD / BlueRay ), tape drives (LTFS), etc.

Depending on the type of storage, it can be physically connected in read / write mode to only one server (block-device with a non-clustered file system) or to several servers (network storage, Amazon S3-compatible, etc.). Depending on the number of connections in the read / write mode, an additional registration of the DSP on the metaserver (140) is carried out, an additional identifier is obtained and, as a result, an additional storage route.

Access to the DSS can be via the Internet or a local area network. A DSS can be a cloud storage with access organization using standard tools, for example, a WEB browser, or using a software application installed on client devices (110).

After the access policies are set, a route of information flows (304) is formed, which will redirect user requests to the corresponding TSH request. The corresponding bundle from the data processing server and one or more storage systems (160) forms the DSP. The unique identifier TsSH fs_id is generated. It is this identifier that becomes the key of the storage route when determining the DSP during file transfer from / to the client (110). Due to the combination of records in the database (200) of the metaserver (140) and the established access policies, when clients (110) access the central logistic center, the corresponding link of the data processing server (130) and the connected storage system (160) determines the route for receiving / sending information to the aforementioned TSH.

As a router and load balancer, for example, a Nginx server can be used. Consider an example of setting storage paths.

Using the File_storage table, determine the available DSPs: fs_id, srv_id. For each DSS, from the Servers table and srv_id field, we determine the corresponding IPv4 server addresses. We configure Nginx:

In this example, fs_id_1, fs_id_2, fs_id_3, fs_id_4 are the unique (UUIDs) of the DSP, fs_id values for the corresponding DSP. And the network addresses 192.168.10.0/24 are the addresses of the corresponding servers.

In FIG. Figure 4 shows an example of the operation of the method of accessing the DSP to send files to it (400). A client device (110) generates a request to send one or more files (401). Before direct data transfer, based on a user request (110), a request is generated to create an object of type “download” to any of the servers (130) in the cluster, transmitting Meta-information about the file (402) to the server, which includes at least the name of the target file and its size. Based on the received request, the server (130), which received the request to create an object of type “download”, performs meta-information processing (403), in particular, to check the storage management policy and user access rights of the files being sent (404). Validation is performed by requesting an access policy (404) on the metaserver (140).

The server (130) obtains the final identifier of the corresponding DSF fs_id, based on the executed request. If for the corresponding user (110) the TSH is not defined in advance - the request goes to the table of companies, etc. until the CCH is determined. At least at one of the levels: company, group, user, file type — the TSH must be defined, which will be determined by the access control policy, otherwise access to the TSH will be denied (408).

In case of successful granting of the right to create a “download” (405), the server (130) determines the TSH (406). After receiving the final fs_id TSH, an entry is created in the Files table - registration of an object of the "download" type. When registering, a unique identifier for the “download” object is generated, and the fs_id CSX previously defined is recorded in the fs_id field. The result of creating the “download” object is an entry in the Files table, in which the file name f_name is already written.

As a response, the client receives the identifier of the object of type "download" from the Files table and the value fs_id is the UUID identifier of the TSH.

After determining the DSP and the corresponding storage (160) connected to the DSP, user request files are transferred to the corresponding storage (407). The PUT request to save the file comes from the client (110), having in the header part of the HTTP request the parameter “X-Server-Select” set to fs_id of the TSX.

The load router (120), based on the value of the “X-Server-Select” parameter, routes the request to the corresponding server (130), to which the storage system (160) of the TSX is connected. The server (130) that received the request, received fs_id in the “X-Server-Select” parameter, determines the exact storage location using the request to the metaserver (140), using fs_id to determine stor_id, and then save the file to the target storage (160) . At the end of file saving, the record of the “download” type is converted to the “file” type, thus completing the process of sending and saving the file.

In FIG. 5 shows a method of requesting data from a DSP (500), in particular, a request from clients (110) to download files. The user’s requests (501) to obtain a list of files, folders, etc., which can be described as Meta-information requests (502), are processed by any of the cluster servers (130) by querying the metaservers (140) and then responding to the user (110 ) At step (503), the server (130) determines access control policies for the DSP from the corresponding client device (110).

If the meta server (140) determines the appropriate DSP for accessing data, then the server (130) and the corresponding storage (160) are determined at the client’s request, in particular the processing of the corresponding DSS identifier fs_id.

Consider requesting a list of files in a folder. In response to the request, any of the servers (130) returns to the user an array of objects of the "file" type. For each record of the file type, the same ID is indicated.

An example of the JSON part of the server response by the list of files:

Next, access to the GLC is determined by processing the corresponding fs_id identifier. The client (110) sends a GET request, in which it indicates the file identifier and in the request header indicates the “X-Server-Select” parameter set to fs_id of the TSX.

The load router (120), based on the value of the “X-Server-Select” parameter, routes the request to the corresponding server (130), to which the storage system (160) of the central host is connected, corresponding to the client’s request (110) and a specific access control policy (step 506) . From the example of Nginx settings - in the “X-Server-Select” parameter the value “032019d0-cf70-438e-9e6e-7a94a6ef5f70” was received. If there is an entry in the Nginx configuration file of the form:

To Nginx will send this request to the server with the IP address 192.168.10.1 in accordance with the settings. As part of a complete solution, we will develop our own HTTP request router.

The server (130) that received the request, using fs_id in the “X-Server-Select” parameter, determines the exact storage location of the requested objects using the metaserver (140), using fs_id to determine stor_id by executing a SELECT query on the File_storage table in the database (200 ), and then reading and transferring the file from the target storage (160) to the client (507). The file is read by the corresponding server of the cluster (130), since the target storage system (160) is connected to it (step 508). Reading from a specific storage system (160) is carried out in accordance with the settings of the GSC obtained from the Storages table using the identifier stor_id, which, in turn, is obtained from the File_storages table using the fs_id key received in the request.

Mentioned files sent from clients (110) to be stored in the TSX and files requested from the TSX can be any type of data, in particular text, multimedia files, presentations, 3D models, CAD files, PDF, tables, etc. P. The access policy for the corresponding storage system (160) can be determined depending on the type of file, as well as the processing speed of the relevant information stored on it, the rules for storing information over time or other policies.

In FIG. 6 shows a general view of the device (600), which is used to implement the claimed technical solution, in particular, client devices (110), servers (130), metaservers (140).

The general scheme of a computing device that implements one or another program logic necessary for implementing the processes of information exchange and routing information flows consists of one or more processors (601), RAM (602), ROM (603), I / O facilities (604) I / O devices (605) and network access tools (606).

The processor (601) performs the main computing work when implementing the steps of the method in the above system to achieve the necessary technical effect.

Random access memory (RAM) (602) is designed to store instructions executed by one or more processors (601). A read only memory (ROM) is used to store data (603) and can be, for example, a hard disk drive (HDD), solid state drive (SSD), flash memory (NAND-flash, EEPROM, Secure Digital, etc.) , optical disc (CD, DVD, Blue Ray), a mini disc or a combination thereof.

Input / output (I / O) interfaces (604) are standard ports and devices for pairing devices and transmitting data, selected based on the required configuration of the device (600), in particular: USB (2.0, 3.0, USB-C, micro, mini), Ethernet, PCI, AGP, COM, LPT, PS / 2, SATA, Fire Wire, Lightning, etc.

I / O devices (605) are also selected from a well-known range of different devices, for example, a keyboard, touchpad, touch display, monitor, projector, mouse, joystick, trackball, light pen, stylus, sound output devices (speakers, headphones, built-in speakers buzzer) etc.

Network access tools (606) are selected from devices designed to implement the communication process between different devices via wired and / or wireless communication, in particular, such devices can be: GSM modem, Wi-Fi transceiver, Bluetooth or BLE module, GPS module, Glonass module, NFC, Ethernet module, etc.

For servers, I / O devices and Network Access Means can include specialized storage connectivity controllers, for example, FC NVA, iSCSI NVA, Ethernet, InfiniBand, etc.

The components of device (600) are interfaced via a common data bus.

The presented description of examples of implementation of the claimed technical solution reveals its preferred embodiment and should not be construed as limiting other, private methods for its implementation, which are determined by the scope of the requested legal protection and are obvious to specialists in this field of technology.

Information sources

1. M. Rouse. Storage area network (SAN) //

http://searchstorage.techtarget.com/definition/storage-area-network-SAN

2. J Frew. 7 Reasons to Use a NAS for Data Storage & Backups // http://www.makeuseof.com/tag/7-reasons-to-use-a-nas-for-data-storage-backups/

Claims (19)

1. A data storage system comprising: - at least one target data storage layer (DSC), consisting of at least one server and at least one storage system (DSS), and the DSC contains a unique identifier (fs_id); - user devices generating control requests for data stored or transmitted to the TSX, each request containing one or more files, each of which contains a unique identifier (file_id) or access identifier for the file stored in the TSX; - at least one metadata management server, designed to register on a network of TSH servers and process requests for data management from user devices; - one or more routers to balance the load on the central hostel; moreover - each DSP server contains at least a unique identifier (srv_id) and an address in the data transmission network and is designed to process requests for managing data generated by user devices based on a set of data type management policies; - SHD is one or more data storage devices connected to one or more of the mentioned server ЦСХ, where each of the devices contains a unique identifier (stor_id) and is designed to store and manage the data type corresponding to the policy of the corresponding server storage; - the metadata management server, when processing the aforementioned user requests based on the identifier contained in the user request, determines a device for storing information in the corresponding DSS corresponding to the said request and performs routing for user access to the said DSS. 2. The system according to claim 1, characterized in that the central host institution is selected from the group: cloud storage with online access to the Internet, local storage on a local network or intranet. 3. The system according to claim 1, characterized in that the storage devices of data storage systems are selected from the group: solid-state storage devices (SSD), hard disk (HDD), flash drives, RAID array, optical drives (CD / DVD / MD / BlueRay), tape drives (LTFS). 4. The system according to claim 1, characterized in that the file management policy is based at least on access rights to the TSH. 5. The system according to claim 1, characterized in that the DSP server is connected to the router via HTTP / HTTPS. 6. The system according to claim 1, characterized in that the set of policies includes the processing of user request information, which includes analysis of the user's access level, and / or user binding to a specific DSP, and / or network address of the incoming request, and / or identifier user request file. 7. A method of forming a data storage structure, comprising the steps of: - form at least one target data storage layer (DSC) by creating an information exchange route between at least one server and at least one data storage system (SHD), wherein the SHD is one or more data storage devices; - when creating the mentioned information exchange route, registration is carried out in the network of the DSS servers and one or more storage systems using a metadata management server that contains a database of identifiers, and during the registration process, each DSS server is assigned a unique identifier (srv_id) and an IP address in the network, each SHD associated with the aforementioned DSC server is assigned a unique identifier (stor_id); - moreover, a management policy is formed for each server of the DSP, on the basis of which for each storage connected to the mentioned server of the DSS, the support for storage and access of a certain type of data is formed; - moreover, the metadata management server when processing requests received from user devices, where each request contains one or more types of files, determines the storage system corresponding to this type of file in the corresponding DSP.

Images