CN110769025A - Method for accelerating data index of multi-tenant-oriented cloud storage system - Google Patents
Method for accelerating data index of multi-tenant-oriented cloud storage system Download PDFInfo
- Publication number
- CN110769025A CN110769025A CN201910840769.7A CN201910840769A CN110769025A CN 110769025 A CN110769025 A CN 110769025A CN 201910840769 A CN201910840769 A CN 201910840769A CN 110769025 A CN110769025 A CN 110769025A
- Authority
- CN
- China
- Prior art keywords
- index
- user
- information
- substep
- global
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Images
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/01—Protocols
- H04L67/10—Protocols in which an application is distributed across nodes in the network
- H04L67/1097—Protocols in which an application is distributed across nodes in the network for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS]
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/13—File access structures, e.g. distributed indices
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/16—File or folder operations, e.g. details of user interfaces specifically adapted to file systems
- G06F16/162—Delete operations
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/18—File system types
- G06F16/182—Distributed file systems
Abstract
The invention discloses a method for accelerating data indexing of a multi-tenant-oriented cloud storage system, which comprises the following steps: step 1: an index initialization stage, which initializes a global index and a local index according to user information; step 2: a new user adding system stage, which is used for establishing the information of the new user in the global index; and step 3: a user data indexing stage for indexing user data; and 4, step 4: and an index destroying stage, which is used for destroying the user metadata information. The invention realizes the classification of the metadata according to the users and the dynamic mounting of the user index information according to the online condition of the users, and has the advantages of quick data access under multiple tenants and low load of a metadata server.
Description
Technical Field
The invention relates to the field of data indexing, in particular to a method for accelerating data indexing of a multi-tenant-oriented cloud storage system.
Background
The advent of information technology has expanded the size and scope of information systems that generate large amounts of data each day, opening up the field of large data processing and analysis. The virtualization-based cloud computing environment can meet not only the increasing data storage requirements, but also the heavy and large-scale computing requirements of large data applications. The problem of cloud resource access efficiency is a very important problem, and directly influences the experience of a user on the service quality provided by a cloud provider. However, due to the increase of the amount of cloud computing data, the conventional data indexing method is not suitable for indexing data by multiple tenants in a big data context. Therefore, a new indexing method is needed to improve the access efficiency of the multi-tenant to the data.
At present, the prior related technical scheme is that a uniform global index is established for all data in a storage system according to the same rule; however, the scheme uniformly organizes the multi-tenant data index, and when the system runs, the whole global index information needs to be loaded, so that the load of the metadata server is increased. When a user indexes data, the organization mode can retrieve the global index, and the index time of the user is increased. It is not suitable for multi-tenant indexing of data in a big data context.
In the second prior art scheme, the Guangdong technical institute, Wen and Wen nations, and the like, which are invented in the 'A cloud storage system' of the second prior art scheme, apply for patent and obtain approval from the national intellectual property office in 2017-12-30 days, and are disclosed in 2019-03-01 days, with the publication numbers: 109413130A. The invention only solves the performance problem of cloud storage from the aspect of metadata, does not provide a scheme aiming at the index efficiency of multi-tenant users, and cannot ensure the service quality of the multi-tenant users in the cloud environment.
The technical problem to be solved by the invention is as follows:
1. aiming at the problems of increased load of a meta-server and low data indexing efficiency caused by unified organization of user data indexing information in a related scheme;
2. the problem that the user data indexes are uniformly organized by not distinguishing users in a storage system of a related scheme II is solved.
Disclosure of Invention
Aiming at the defects of the prior art, the invention provides a method for accelerating data indexing of a multi-tenant-oriented cloud storage system, which comprises the following steps:
step 1: an index initialization stage, which initializes a global index and a local index according to user information;
step 2: a new user adding system stage, which is used for establishing the information of the new user in the global index;
and step 3: a user data indexing stage for indexing user data;
and 4, step 4: and an index destroying stage, which is used for destroying the user metadata information.
The invention is further improved in that: the step 1 comprises the following substeps:
substep 1-1: initializing a global index root node;
substeps 1-2: acquiring corresponding information from a local metadata file according to the current online user information, and mounting the information into a global index;
substeps 1-3: the online user client stores the position of the user index information in the global index;
the invention is further improved in that: the step 2 comprises the following substeps:
substep 2-1: creating an index node for a new user;
substep 2-2: mounting the index node of the new user into the global index;
the invention is further improved in that: the step 3 comprises the following substeps:
substep 3-1: the user finds the position of the index information of the user in the global index according to the local index;
substep 3-2: accessing the global index to obtain the real position of the required data;
the invention is further improved in that: the step 4 comprises the following substeps:
substep 4-1: when the user is offline, updating the index information of the user into a metadata file, and unloading the index information corresponding to the user from the global index;
substep 4-2: when the user logs out, the index information corresponding to the user is unloaded from the global index, and the index information corresponding to the local metadata file is deleted.
The invention realizes the classification of the metadata according to the users and the dynamic mounting of the user index information according to the online condition of the users, and has the advantages of quick data access under multiple tenants and low load of a metadata server.
The data indexes are classified and organized according to users, and dynamic mounting is carried out according to the online condition of the users, so that the problems of the efficiency of the user data indexes and overlarge load of a metadata server are effectively solved, and the characteristics that the users can quickly and effectively access data can be realized on the basis of ensuring the quality of multi-tenant service.
The invention discloses a cloud storage system which comprises a metadata server, a data storage server and a client. The metadata server, the data storage server and the client are respectively connected in a one-to-one correspondence manner through a TCP/IP network protocol; the metadata server is constructed by adopting a centralized architecture and is used for storing and managing metadata information of all files in the cloud storage system; the data storage server is constructed by adopting a cluster type architecture and is used for storing file data blocks and providing data reading and writing service for the client; the client is used for a user to input an operation instruction so as to realize the writing, reading and deleting of the file. The technical scheme of the invention separates the control flow and the data flow of the file, supports the load balancing and data redundancy strategies of the storage resources and improves the system performance.
Drawings
FIG. 1 is a flow chart of the present invention;
FIG. 2 is a diagram of a cloud storage system architecture using the method of the present invention;
FIG. 3 is a cloud storage system read flow using the method of the present invention;
FIG. 4 is a cloud storage system write flow using the method of the present invention;
fig. 5 is a schematic diagram of a node for mounting index information corresponding to a user in the global index in step 2.
Detailed Description
The objects, technical solutions and advantages of the present invention will be described in further detail with reference to the accompanying drawings.
As shown in fig. 1 to 4, this embodiment provides a method for accelerating data indexing of a multi-tenant-oriented cloud storage system, which includes the following steps:
step 1: an index initialization stage, which initializes a global index and a local index according to user information; substep 1-1: initializing a global index root node;
substeps 1-2: acquiring corresponding information from a local metadata file according to the current online user information, and mounting the information into a global index;
substeps 1-3: the online user client stores the position of the user index information in the global index;
step 2: a new user adding system stage, which is used for establishing the information of the new user in the global index; substep 2-1: creating an index node for a new user;
substep 2-2: mounting the index node of the new user into the global index;
and step 3: a user data indexing stage for indexing user data;
substep 3-1: the user finds the position of the index information of the user in the global index according to the local index;
substep 3-2: accessing the global index to obtain the real position of the required data;
and 4, step 4: and an index destroying stage, which is used for destroying the user metadata information.
Substep 4-1: when the user is offline, updating the index information of the user into a metadata file, and unloading the index information corresponding to the user from the global index;
substep 4-2: when the user logs out, the index information corresponding to the user is unloaded from the global index, and the index information corresponding to the local metadata file is deleted.
Example 1
As shown in fig. 1, a method for accelerating data indexing of a multi-tenant-oriented cloud storage system includes the following steps:
step 1: initializing a global index root node;
substep 1-1: the metadata management server applies for a memory and creates an index information node;
substeps 1-2: initializing the node as a root node root of a global index;
step 2: user index information initialization
Substep 2-1: the method comprises the steps that a metadata management server obtains on-line user id information;
substep 2-2: creating a corresponding user information node according to the online user id information, and mounting the corresponding user information node into the global index;
substeps 2-3: acquiring index information of a corresponding user from a local metadata file according to the on-line user id information;
substeps 2-4: the index information corresponding to the user is mounted to the corresponding user information node in the global index, as shown in fig. 5.
And step 3: local index initialization
Substep 3-1: acquiring an information node address of a user in a global index;
substep 3-2: returning the information node address of the corresponding user to the corresponding client;
substep 3-3: after receiving the user information node position, the client stores the user information node position locally as local index information;
and 4, step 4: index access
Substep 4-1: a user accesses the local index to obtain the initial position of the user data index in the global index;
substep 4-2: accessing the global index according to the local index information, and finally obtaining the position information of the data;
and 5: index destruction
Substep 5-1: the user quits the system and sends a quit instruction 0 or 1 to the system;
substep 5-2: after receiving the quit instruction, the metadata management server judges the type of the instruction;
substep 5-3: the instruction of 0 represents that only the system is exited, and the metadata management server updates the user index information to a local metadata file and then unloads the user index information from the global index;
substeps 5-4: the instruction 1 indicates that the user logs off, the metadata management server updates the index information of the user to a local source data file, marks that the user logs off, unloads the index information of the user, and deletes the user information in the metadata file after a certain time threshold value is exceeded.
According to the method, the data index information can be classified according to users and the user index information can be unloaded dynamically, so that the users can index the data quickly, the load of a metadata server is reduced, and the system has quick data access and low load of a storage system under multiple tenants.
The above is only a preferred embodiment of the present invention, it should be noted that the above embodiment does not limit the present invention, and various changes and modifications made by workers within the scope of the technical idea of the present invention fall within the protection scope of the present invention.
Claims (5)
1. A method for accelerating data indexing of a multi-tenant-oriented cloud storage system is characterized by comprising the following steps: comprises the following steps:
step 1: an index initialization stage, which initializes a global index and a local index according to user information;
step 2: a new user adding system stage, which is used for establishing the information of the new user in the global index;
and step 3: a user data indexing stage for indexing user data;
and 4, step 4: and an index destroying stage, which is used for destroying the user metadata information.
2. The method for accelerating data indexing of the multi-tenant-oriented cloud storage system according to claim 1, wherein: the step 1 comprises the following substeps:
substep 1-1: initializing a global index root node;
substeps 1-2: acquiring corresponding information from a local metadata file according to the current online user information, and mounting the information into a global index;
substeps 1-3: and the online user client stores the position of the user index information in the global index.
3. The method for accelerating data indexing of the multi-tenant-oriented cloud storage system according to claim 1, wherein: the step 2 comprises the following substeps:
substep 2-1: creating an index node for a new user;
substep 2-2: and mounting the index node of the new user into the global index.
4. The method for accelerating data indexing of the multi-tenant-oriented cloud storage system according to claim 1, wherein: the step 3 comprises the following substeps:
substep 3-1: the user finds the position of the index information of the user in the global index according to the local index;
substep 3-2: and accessing the global index to acquire the real position of the required data.
5. The method for accelerating data indexing of the multi-tenant-oriented cloud storage system according to claim 1, wherein: the step 4 comprises the following substeps:
substep 4-1: when the user is offline, updating the index information of the user into a metadata file, and unloading the index information corresponding to the user from the global index;
substep 4-2: when the user logs out, the index information corresponding to the user is unloaded from the global index, and the index information corresponding to the local metadata file is deleted.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910840769.7A CN110769025B (en) | 2019-09-06 | 2019-09-06 | Method for accelerating data index of multi-tenant-oriented cloud storage system |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910840769.7A CN110769025B (en) | 2019-09-06 | 2019-09-06 | Method for accelerating data index of multi-tenant-oriented cloud storage system |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN110769025A true CN110769025A (en) | 2020-02-07 |
| CN110769025B CN110769025B (en) | 2022-04-22 |
Family
ID=69330285
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201910840769.7A Active CN110769025B (en) | 2019-09-06 | 2019-09-06 | Method for accelerating data index of multi-tenant-oriented cloud storage system |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN110769025B (en) |
Citations (9)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20070276801A1 (en) * | 2004-03-31 | 2007-11-29 | Lawrence Stephen R | Systems and methods for constructing and using a user profile |
| US20080235106A1 (en) * | 1994-05-31 | 2008-09-25 | Twintech E.U., Limited Liability Company | Software and Method That Enables Selection of One of A Plurality of Online Service Providers |
| CN101777057A (en) * | 2004-04-02 | 2010-07-14 | 易享信息技术（上海）有限公司 | Methods and systems for storing customer fields for multiple tenants in multi-tenant database system |
| US7788274B1 (en) * | 2004-06-30 | 2010-08-31 | Google Inc. | Systems and methods for category-based search |
| CN103218175A (en) * | 2013-04-01 | 2013-07-24 | 无锡成电科大科技发展有限公司 | Multi-tenant cloud storage platform access control system |
| CN103455512A (en) * | 2012-05-31 | 2013-12-18 | 上海博腾信息科技有限公司 | Multi-tenant data management model for SAAS (software as a service) platform |
| CN105808240A (en) * | 2016-03-01 | 2016-07-27 | 北京量邦信息科技股份有限公司 | Method for realizing user isolation under online programming environment |
| CN106960020A (en) * | 2017-03-09 | 2017-07-18 | 星环信息科技（上海）有限公司 | A kind of method and apparatus for creating concordance list |
| CN109413130A (en) * | 2017-12-30 | 2019-03-01 | 广东技术师范学院 | A kind of cloud storage system |
-
2019
- 2019-09-06 CN CN201910840769.7A patent/CN110769025B/en active Active
Patent Citations (9)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20080235106A1 (en) * | 1994-05-31 | 2008-09-25 | Twintech E.U., Limited Liability Company | Software and Method That Enables Selection of One of A Plurality of Online Service Providers |
| US20070276801A1 (en) * | 2004-03-31 | 2007-11-29 | Lawrence Stephen R | Systems and methods for constructing and using a user profile |
| CN101777057A (en) * | 2004-04-02 | 2010-07-14 | 易享信息技术（上海）有限公司 | Methods and systems for storing customer fields for multiple tenants in multi-tenant database system |
| US7788274B1 (en) * | 2004-06-30 | 2010-08-31 | Google Inc. | Systems and methods for category-based search |
| CN103455512A (en) * | 2012-05-31 | 2013-12-18 | 上海博腾信息科技有限公司 | Multi-tenant data management model for SAAS (software as a service) platform |
| CN103218175A (en) * | 2013-04-01 | 2013-07-24 | 无锡成电科大科技发展有限公司 | Multi-tenant cloud storage platform access control system |
| CN105808240A (en) * | 2016-03-01 | 2016-07-27 | 北京量邦信息科技股份有限公司 | Method for realizing user isolation under online programming environment |
| CN106960020A (en) * | 2017-03-09 | 2017-07-18 | 星环信息科技（上海）有限公司 | A kind of method and apparatus for creating concordance list |
| CN109413130A (en) * | 2017-12-30 | 2019-03-01 | 广东技术师范学院 | A kind of cloud storage system |
Non-Patent Citations (2)
| Title |
|---|
| HONGWEI LI: "Enabling Efficient Multi-Keyword Ranked Search Over Encrypted Mobile Cloud Data Through Blind Storage", 《IEEE》 * |
| 张涛等: "提高SQL Server数据库访问速度的方法", 《计算机时代》 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN110769025B (en) | 2022-04-22 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US11662936B2 (en) | Writing data using references to previously stored data | |
| CN103136243B (en) | File system duplicate removal method based on cloud storage and device | |
| US8572137B2 (en) | Data de-duplication in a distributed network | |
| US8224935B1 (en) | Systems and methods for efficiently synchronizing configuration data within distributed computing systems | |
| US11005717B2 (en) | Storage capacity evaluation method based on content delivery network application and device thereof | |
| US20180063055A1 (en) | Repartitioning a topic in a publish-subscribe message system | |
| US20100312749A1 (en) | Scalable lookup service for distributed database | |
| US11657025B2 (en) | Parallel processing of filtered transaction logs | |
| CN109981702B (en) | File storage method and system | |
| CN111083179B (en) | Internet of things cloud platform, and equipment interaction method and device based on same | |
| CN105677251A (en) | Storage system based on Redis cluster | |
| US10817203B1 (en) | Client-configurable data tiering service | |
| CN113032099B (en) | Cloud computing node, file management method and device | |
| CN109885577A (en) | Data processing method, device, terminal and storage medium | |
| CN106021566A (en) | Method, device and system for improving concurrent processing capacity of single database | |
| CN105187565A (en) | Method for utilizing network storage data | |
| CN105068875A (en) | Intelligence data processing method and apparatus | |
| US20110264903A1 (en) | Electronic device and method for loading configuration files using the same | |
| CN109446167A (en) | A kind of storage of daily record data, extracting method and device | |
| US11290397B2 (en) | Systems and methods for efficiently storing a distributed ledger of records | |
| CN110769025B (en) | Method for accelerating data index of multi-tenant-oriented cloud storage system | |
| CN116610670A (en) | State data storage method and device based on block chain | |
| CN115878042A (en) | Data storage system, data processing method and storage medium | |
| US10067678B1 (en) | Probabilistic eviction of partial aggregation results from constrained results storage | |
| EP3264254A1 (en) | System and method for a simulation of a block storage system on an object storage system |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |