CN107608867A - A kind of metadata performance test methods and system that mdtest is configured on cluster-based storage - Google Patents
A kind of metadata performance test methods and system that mdtest is configured on cluster-based storage Download PDFInfo
- Publication number
- CN107608867A CN107608867A CN201710775866.3A CN201710775866A CN107608867A CN 107608867 A CN107608867 A CN 107608867A CN 201710775866 A CN201710775866 A CN 201710775866A CN 107608867 A CN107608867 A CN 107608867A
- Authority
- CN
- China
- Prior art keywords
- cluster
- test
- based storage
- mdtest
- ssh
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Landscapes
- Storage Device Security (AREA)
Abstract
The invention provides a kind of metadata performance test methods and system that mdtest is configured on cluster-based storage, weighed by mdtest to carry out the test of the metadata performance of cluster-based storage, data guarantee is provided for the performance of product, enhance one's market competitiveness, and it is used to test by SSH connections cluster-based storage, not only configuration is simple, and SSH can make up network when proper use of in leak, ssh client is applied to kinds of platform, limiting factor is smaller, and the unfavorable impact to metadata is caused without frequent read-write of the consideration to storage system.The performance test flexible, convenient, practical, that metadata can be carried out well to different cluster-based storages of the present invention, reliable performance data parameter is provided for the metadata disposal ability of cluster-based storage product, improves cluster-based storage product competitiveness in the market.
Description
Technical field
The present invention relates to the technical field of test, and in particular to a kind of metadata that mdtest is configured on cluster-based storage
Can method of testing and system.
Background technology
With the development in epoch, the application of cluster storage system is more and more extensive, and cluster-based storage is by several " common stores
The cluster for being used to store of equipment " composition, the performance and capacity for forming each storage system of cluster-based storage can be by " collection
The mode of group " is able to be superimposed and extended.Once running into the bottleneck of storage system, two kinds of selections are just had：When：Using hardware more
Strengthen big single storage system；Second,：Formed " cluster of storage " using the storage system of several common performances." collection
Group " as an architecture being widely used, if use storage on form " cluster-based storage ", so that it may provide by than
Performance, capacity, reliability and the availability of the increased storage resource of example, breach a variety of limitations of stand-alone device.
The characteristics of cluster-based storage：Open architecture (high scalability), distributed operating system, naming space, Yi Guan
Rationality, load balancing, high-performance.The advantage of cluster-based storage is mainly reflected in the overall performance for improving parallel or subregion I/O, especially
It is workflow, reads intensive and mass file access, holistic cost is reduced by using more inexpensive server.
For cluster-based storage, the performance of metadata is to weigh a major criterion of properties of product, metadata
(Metadata), also known as broker data, relaying data, for describe data data (data about data), mainly describe
The information of data attribute (property), for supporting such as to indicate storage location, historical data, resource lookup, file record
Function.Metadata a kind of electronic type catalogue at last, in order to reach the purpose of scheduling, it is necessary to describing and collecting in data
Perhaps characteristic, and then reach the purpose for assisting data retrieval.Dublin Core (Dublin Core Metadata
Initiative, DCMI) be metadata a kind of application, be 2 months nineteen ninety-fives by International Library computer center (OCLC) and U.S.
State's national Super calculates and applies center (National Center for Supercomputing Applications, NCSA)
Combine the seminar of patronage, inviting 52 from librarian, computer expert, formulating specification jointly, create a set of description
The feature of network power subfile.
Mdtest is a benchmark test instrument for server metadata disposal ability, can be used for simulating to file
Or the open/stat/close operations of catalogue, it is then back to report.It supports MPI, is realized and coordinated without cryptographic acess by ssh
A large amount of clients are initiated to ask to server.For in general file system, substantial amounts of data file produces, and is to storage
The frequent read-write of system, it is high to the metadata performance requirement of cluster-based storage, seriously result even in metadata damage.
The content of the invention
Based on above mentioned problem, the present invention proposes a kind of metadata performance test side that mdtest is configured on cluster-based storage
Method and system.
The present invention provides following technical scheme：
On the one hand, the invention provides a kind of metadata performance test methods that mdtest is configured on cluster-based storage, bag
Include：
Step 101, MPI translation and compiling environments are installed in cluster test client；
Step 102, mdtest softwares are installed in the cluster test client；
Step 103, the ssh of machine between cluster test client and cluster-based storage is configured without cryptographic acess；
Step 104, the first configuration file of cluster-based storage node is changed, cluster test client is added in cluster-based storage node
IP and host name；The second configuration file is edited, writes the host name of clustered node machine and cluster test client；
Step 105, mdtest test script is performed in cluster-based storage node.
Wherein, the ssh of machine is specially without cryptographic acess between the configuration cluster test client and cluster-based storage：Institute
There is cluster test client to perform ssh without cryptographic acess order, check that the mandate of the cluster test client is literary by cat
Part, and the copy content in the authority to needing to be arranged on cluster test clients of the ssh without password login.
Wherein, test prevents from reporting an error without code entry after step 104 configuration finishes.
Wherein, the test script of the execution mdtest is the test script that configuration mdtest is completed in cluster-based storage node
And perform the test script or directly perform mdtest test script order.
In addition, present invention also offers a kind of metadata Performance Test System that mdtest is configured on cluster-based storage, institute
The system of stating includes：
Environment configurations module, for installing MPI translation and compiling environments in cluster test client, in the cluster test client
Mdtest softwares are installed；
Configuration module is accessed, for configuring the ssh of machine between cluster test client and cluster-based storage without cryptographic acess；
Profile module, for changing the first configuration file of cluster-based storage node, add cluster in cluster-based storage node and survey
Try the IP and host name of client；Edit the second configuration file, write clustered node machine and cluster test client
Host name；Execution module, for performing mdtest test script in cluster-based storage node.
Wherein, the ssh of machine is specially without cryptographic acess between the configuration cluster test client and cluster-based storage：Institute
There is cluster test client to perform ssh without cryptographic acess order, check that the mandate of the cluster test client is literary by cat
Part, and the copy content in the authority to needing to be arranged on cluster test clients of the ssh without password login.
Wherein, profile module test after configuration finishes without code entry, prevents from reporting an error.
Wherein, the test script of the execution mdtest is the test script that configuration mdtest is completed in cluster-based storage node
And perform the test script or directly perform mdtest test script order.
The invention provides a kind of metadata performance test methods and system that mdtest is configured on cluster-based storage, pass through
Mdtest weighs to carry out the test of the metadata performance of cluster-based storage, provides data guarantee for the performance of product, strengthens market
Competitiveness, and be used to test by SSH connections cluster-based storage, not only configuration is simple, and SSH can be made up when proper use of
Leak in network, ssh client are applied to kinds of platform, and limiting factor is smaller, without considering the frequent reading to storage system
Write the unfavorable impact caused to metadata.The present invention it is flexible, convenient, practical, can be good to different cluster-based storages
The performance test of metadata is carried out, reliable performance data parameter is provided for the metadata disposal ability of cluster-based storage product, carries
High cluster-based storage product competitiveness in the market.
Brief description of the drawings
Fig. 1 is flow chart of the method for the present invention.
Fig. 2 is the system architecture diagram of the present invention.
Embodiment
Technical scheme in order to illustrate the embodiments of the present invention more clearly, it will use below required in embodiment
Accompanying drawing is briefly described, it should be apparent that, drawings in the following description are only some embodiments of the present invention, for ability
For the those of ordinary skill of domain, on the premise of not paying creative work, it can also be obtained according to these accompanying drawings other attached
Figure.
In the prior art, in general file system, substantial amounts of data file produces, to the frequent of storage system
Read-write, test the metadata performance of cluster-based storage.
On the one hand, embodiments of the present invention provide a kind of metadata performance that mdtest is configured on cluster-based storage and surveyed
Method for testing, accompanying drawing 1 are flow chart of the method for the present invention, including：
Step 101, MPI translation and compiling environments are installed in cluster test client；
Mpi instruments, MPICH2 are managed using mpd services to process, and mpi programs are run using mpiexec, its
In-n represent that performing this order needs to call several nodes to go to complete ,-f includes the file of host name ,-hosts host name, leads
With CSV between machine name.Cluster client terminal install mpi, first mpi installation kits are decompressed, enter decompress catalogue with
This performs ./configure, 3 instructions of make, make install are installed.
Step 102, mdtest softwares are installed in the cluster test client；
Mdstest compressed packages are decompressed, mdstest decompression catalogues is entered and performs mpicc mdstest.1 and mpicc
Make instructions are performed after mdstest.c again, ls can view executable mdstest afterwards.
Step 103, the ssh of machine between cluster test client and cluster-based storage is configured without cryptographic acess；
Configurations of the ssh without cryptographic acess is completed by ssh-keygen-t rsa, is needing to install ssh without password login
Machine on perform respectively：Cd/root/.ssh/ is it finds that have after ssh-keygen-t rsa (always by enter key)：
Id_rsa id_rsa.pub known_hosts files, wherein, id_rsa is that key is put on a local, and id_rsa.pub is
Public key is, it is necessary to on the computer of access, so next, an authorized_keys file is generated, by id_
Rsa.pub contents are put into authorized_keys, it is necessary to instruct cp id_rsa.pub authorized_keys, pass through
Cat checks the authorized_keys files of every machine, and the copy content in authorized_keys files
To after needing to be arranged on machines of the ssh without password login and (there are several to copy several) can ssh without password login
Step 104, the first configuration file of cluster-based storage node is changed, cluster test client is added in cluster-based storage node
The IP and host name at end；The second configuration file is edited, writes the host name of clustered node machine and cluster test client；
Change cluster-based storage node /etc/hosts configuration files, etc/hosts files the inside includes all test visitors
IP and the host name at family end in cluster-based storage node according to the form of IP, host name, it is necessary to add client used in test
IP and host name, hosts files are edited with vi hosts, hosts files include the host name of all test clients, write-in
The host name of client used in clustered node machine and test, after configuration, ssh hostname once, prevent from reporting an error.
Step 105, mdtest test script is performed in cluster-based storage node.
Mdtest test the instructions such as (- f hosts./mdtest-n of mpiexec-n 2 are performed in cluster-based storage node
20-d/mnt/datapool/) with-f hosts to realize multiple client while call, altogether 2 tasks two section
Point operation, mdtest actual-the d/mnt/datapool/ (- n of execution order ./mdtest-n 20:Each task needs
Create/stat/remove file/directory number in each tree, need the create/ in each tree herein for each task
Stat/remove file/directory number is 20) path:/ mnt/datapool, 2 tasks, 40 files/directories.
Wherein, the test script of the execution mdtest is the test script that configuration mdtest is completed in cluster-based storage node
And perform the test script or directly perform mdtest test script order.
The invention provides a kind of metadata performance test methods that mdtest is configured on cluster-based storage, pass through mdtest
Weighed to carry out the test of the metadata performance of cluster-based storage, provide data guarantee for the performance of product, enhance one's market competitiveness,
And it is used to test by SSH connections cluster-based storage, not only configuration is simple, and SSH can be made up when proper use of in network
Leak, ssh client is applied to kinds of platform, and limiting factor is smaller, without considering that the frequent read-write to storage system causes
Unfavorable impact to metadata.The present invention is flexible, convenient, practical, can carry out member well to different cluster-based storages
The performance test of data, reliable performance data parameter is provided for the metadata disposal ability of cluster-based storage product, improves cluster
Store product competitiveness in the market.
On the other hand, embodiments of the present invention provide a kind of metadata performance that mdtest is configured on cluster-based storage
Test system, accompanying drawing 2 are system architecture diagram of the invention, and the system includes：
On the one hand, embodiments of the present invention provide a kind of metadata performance that mdtest is configured on cluster-based storage and surveyed
Method for testing, accompanying drawing 1 are flow chart of the method for the present invention, including：
Environment configurations module 201, for installing MPI translation and compiling environments in cluster test client, visitor is tested in the cluster
Family end is installed by mdtest softwares；
Mpi instruments, MPICH2 are managed using mpd services to process, and mpi programs are run using mpiexec, its
In-n represent that performing this order needs to call several nodes to go to complete ,-f includes the file of host name ,-hosts host name, leads
With CSV between machine name.Cluster client terminal install mpi, first mpi installation kits are decompressed, enter decompress catalogue with
This performs ./configure, 3 instructions of make, make install are installed.
Mdstest compressed packages are decompressed, mdstest decompression catalogues is entered and performs mpicc mdstest.1 and mpicc
Make instructions are performed after mdstest.c again, ls can view executable mdstest afterwards.
Configuration module 202 is accessed, the ssh for configuring machine between cluster test client and cluster-based storage visits without password
Ask；
Configurations of the ssh without cryptographic acess is completed by ssh-keygen-t rsa, stepped on needing installation ssh without password
Performed respectively on the machine of record：Cd/root/.ssh/ after ssh-keygen-t rsa (always by enter key) it finds that
Have：Id_rsa id_rsa.pub known_hosts files, wherein, id_rsa is that key is put on a local, id_rsa.pub
It is public key, it is necessary to on the computer of access, so next, an authorized_keys file is generated, by id_
Rsa.pub contents are put into authorized_keys, it is necessary to instruct cp id_rsa.pub authorized_keys, pass through
Cat checks the authorized_keys files of every machine, and the copy content in authorized_keys files
To after needing to be arranged on machines of the ssh without password login and (there are several to copy several) can ssh without password login
Profile module 203, for changing the first configuration file of cluster-based storage node, added in cluster-based storage node
The IP and host name of cluster test client；The second configuration file is edited, writes clustered node machine and cluster test visitor
The host name at family end；
Change cluster-based storage node /etc/hosts configuration files, etc/hosts files the inside includes all test visitors
IP and the host name at family end in cluster-based storage node according to the form of IP, host name, it is necessary to add client used in test
IP and host name, hosts files are edited with vi hosts, hosts files include the host name of all test clients, write-in
The host name of client used in clustered node machine and test, after configuration, ssh hostname once, prevent from reporting an error.
Execution module 204, for performing mdtest test script in cluster-based storage node.
Mdtest test the instructions such as (- f hosts./mdtest-n of mpiexec-n 2 are performed in cluster-based storage node
20-d/mnt/datapool/) with-f hosts to realize multiple client while call, altogether 2 tasks two section
Point operation, mdtest actual-the d/mnt/datapool/ (- n of execution order ./mdtest-n 20:Each task needs
Create/stat/remove file/directory number in each tree, need the create/ in each tree herein for each task
Stat/remove file/directory number is 20) path:/ mnt/datapool, 2 tasks, 40 files/directories.
Wherein, the test script of the execution mdtest is the test script that configuration mdtest is completed in cluster-based storage node
And perform the test script or directly perform mdtest test script order.
The invention provides a kind of metadata Performance Test System that mdtest is configured on cluster-based storage, pass through mdtest
Weighed to carry out the test of the metadata performance of cluster-based storage, provide data guarantee for the performance of product, enhance one's market competitiveness,
And it is used to test by SSH connections cluster-based storage, not only configuration is simple, and SSH can be made up when proper use of in network
Leak, ssh client is applied to kinds of platform, and limiting factor is smaller, without considering that the frequent read-write to storage system causes
Unfavorable impact to metadata.The present invention is flexible, convenient, practical, can carry out member well to different cluster-based storages
The performance test of data, reliable performance data parameter is provided for the metadata disposal ability of cluster-based storage product, improves cluster
Store product competitiveness in the market.
The foregoing description of the disclosed embodiments, those skilled in the art are enable to realize or using the present invention.To this
A variety of modifications of a little embodiments will be apparent for a person skilled in the art, and generic principles defined herein can
Without departing from the spirit or scope of the present invention, to realize in other embodiments.Therefore, the present invention will not be limited
The embodiments shown herein is formed on, but meets the most wide model consistent with principles disclosed herein and features of novelty
Enclose.
Claims (8)
- A kind of 1. metadata performance test methods that mdtest is configured on cluster-based storage, it is characterised in that：Step 101, MPI translation and compiling environments are installed in cluster test client；Step 102, mdtest softwares are installed in the cluster test client；Step 103, the ssh of machine between cluster test client and cluster-based storage is configured without cryptographic acess；Step 104, the first configuration file of cluster-based storage node is changed, cluster test client is added in cluster-based storage node IP and host name；The second configuration file is edited, writes the host name of clustered node machine and cluster test client；Step 105, mdtest test script is performed in cluster-based storage node.
- 2. according to the method for claim 1, it is characterised in that：Machine between the configuration cluster test client and cluster-based storage The ssh of device is specially without cryptographic acess：Ssh is performed without cryptographic acess order in all cluster test clients, is looked into by cat See the authority of the cluster test client, and the copy content in the authority to needing to be arranged to ssh On cluster test client without password login.
- 3. according to the method for claim 1, it is characterised in that：Test is stepped on without password after step 104 configuration finishes Land, prevent from reporting an error.
- 4. according to the method for claim 1, it is characterised in that：The test script of the execution mdtest is in cluster-based storage Node completes configuration mdtest test script and performs the test script or directly perform mdtest test script order.
- A kind of 5. metadata Performance Test System that mdtest is configured on cluster-based storage, it is characterised in that：The system includes：Environment configurations module, for installing MPI translation and compiling environments in cluster test client, installed in the cluster test client Mdtest softwares；Configuration module is accessed, for configuring the ssh of machine between cluster test client and cluster-based storage without cryptographic acess；Profile module, for changing the first configuration file of cluster-based storage node, add cluster in cluster-based storage node and survey Try the IP and host name of client；Edit the second configuration file, write clustered node machine and cluster test client Host name；Execution module, for performing mdtest test script in cluster-based storage node.
- 6. system according to claim 5, it is characterised in that：Machine between the configuration cluster test client and cluster-based storage The ssh of device is specially without cryptographic acess：Ssh is performed without cryptographic acess order in all cluster test clients, is looked into by cat See the authority of the cluster test client, and the copy content in the authority to needing to be arranged to ssh On cluster test client without password login.
- 7. system according to claim 5, it is characterised in that：Profile module test nothing after configuration finishes Code entry, prevent from reporting an error.
- 8. system according to claim 5, it is characterised in that：The test script of the execution mdtest is in cluster-based storage Node completes configuration mdtest test script and performs the test script or directly perform mdtest test script order.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201710775866.3A CN107608867A (en) | 2017-08-31 | 2017-08-31 | A kind of metadata performance test methods and system that mdtest is configured on cluster-based storage |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201710775866.3A CN107608867A (en) | 2017-08-31 | 2017-08-31 | A kind of metadata performance test methods and system that mdtest is configured on cluster-based storage |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN107608867A true CN107608867A (en) | 2018-01-19 |
Family
ID=61056893
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201710775866.3A Pending CN107608867A (en) | 2017-08-31 | 2017-08-31 | A kind of metadata performance test methods and system that mdtest is configured on cluster-based storage |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN107608867A (en) |
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20100235807A1 (en) * | 2009-03-16 | 2010-09-16 | Hitachi Data Systems Corporation | Method and system for feature automation |
| CN102541994A (en) * | 2011-11-18 | 2012-07-04 | 浪潮电子信息产业股份有限公司 | Method for deploying and sharing NFS (network file system) automatically and quickly |
| CN103475734A (en) * | 2013-09-25 | 2013-12-25 | 浪潮电子信息产业股份有限公司 | Linux cluster user backup migration method |
| CN104317726A (en) * | 2014-11-19 | 2015-01-28 | 浪潮电子信息产业股份有限公司 | Testing method for storage IO performance |
-
2017
- 2017-08-31 CN CN201710775866.3A patent/CN107608867A/en active Pending
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20100235807A1 (en) * | 2009-03-16 | 2010-09-16 | Hitachi Data Systems Corporation | Method and system for feature automation |
| CN102541994A (en) * | 2011-11-18 | 2012-07-04 | 浪潮电子信息产业股份有限公司 | Method for deploying and sharing NFS (network file system) automatically and quickly |
| CN103475734A (en) * | 2013-09-25 | 2013-12-25 | 浪潮电子信息产业股份有限公司 | Linux cluster user backup migration method |
| CN104317726A (en) * | 2014-11-19 | 2015-01-28 | 浪潮电子信息产业股份有限公司 | Testing method for storage IO performance |
Non-Patent Citations (1)
| Title |
|---|
| 闫鹏飞等: "基于广域存储系统的副本技术研究", 《网络新媒体技术》 * |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN108475360B (en) | Distributed computing dependency management system | |
| US10642725B2 (en) | Automated test generation for multi-interface enterprise virtualization management environment | |
| JP6280155B2 (en) | Techniques for network replication | |
| US9420034B2 (en) | Providing services to multiple tenants of an application | |
| US8290949B2 (en) | Resource name reconciliation in a configuration database | |
| US8402119B2 (en) | Real-load tuning of database applications | |
| US10397051B1 (en) | Configuration and testing of network-based service platform resources using a service platform specific language | |
| US20130086434A1 (en) | Configuration fault localization in shared resource environments | |
| US11374832B2 (en) | Integration and customization of third-party services with remote computing infrastructure | |
| US20190034464A1 (en) | Methods and systems that collect data from computing facilities and export a specified portion of the collected data for remote processing and analysis | |
| EP2095235A2 (en) | Application usage metering management method and system | |
| US20210124690A1 (en) | Memory-efficient virtual document object model for structured data | |
| US20180276096A1 (en) | On demand monitoring mechanism to identify root cause of operation problems | |
| US9830177B1 (en) | Computer implemented method and system, and computer program product, for determining compliance with a configuration parameter before deployment of a virtual appliance into a network appliance | |
| Joukov et al. | Application-storage discovery | |
| US11599404B2 (en) | Correlation-based multi-source problem diagnosis | |
| US8195876B2 (en) | Adaptation of contentious storage virtualization configurations | |
| US20220300491A1 (en) | Materialized views assistant | |
| US11023156B2 (en) | Dynamic API allocation based on data-tagging | |
| CN107608867A (en) | A kind of metadata performance test methods and system that mdtest is configured on cluster-based storage | |
| US11115494B1 (en) | Profile clustering for homogenous instance analysis | |
| Singh et al. | Hierarchical multi-log cloud-based search engine | |
| CN115485677A (en) | Secure data replication in a distributed data storage environment | |
| CN106301983A (en) | A kind of website based on fictitious host computer detection method and device | |
| JP2021531562A (en) | Two-step calculation memoing for large-scale entity resolution |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20180119 |