CN106844184A - A kind of distributed system metadata performance test methods, script and instrument - Google Patents
A kind of distributed system metadata performance test methods, script and instrument Download PDFInfo
- Publication number
- CN106844184A CN106844184A CN201710089997.6A CN201710089997A CN106844184A CN 106844184 A CN106844184 A CN 106844184A CN 201710089997 A CN201710089997 A CN 201710089997A CN 106844184 A CN106844184 A CN 106844184A
- Authority
- CN
- China
- Prior art keywords
- thread
- client
- distributed system
- test
- instruction
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/30—Monitoring
- G06F11/34—Recording or statistical evaluation of computer activity, e.g. of down time, of input/output operation ; Recording or statistical evaluation of user activity, e.g. usability assessment
- G06F11/3409—Recording or statistical evaluation of computer activity, e.g. of down time, of input/output operation ; Recording or statistical evaluation of user activity, e.g. usability assessment for performance assessment
- G06F11/3428—Benchmarking
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/30—Monitoring
- G06F11/3003—Monitoring arrangements specially adapted to the computing system or computing system component being monitored
- G06F11/3006—Monitoring arrangements specially adapted to the computing system or computing system component being monitored where the computing system is distributed, e.g. networked systems, clusters, multiprocessor systems
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- General Engineering & Computer Science (AREA)
- Physics & Mathematics (AREA)
- Quality & Reliability (AREA)
- General Physics & Mathematics (AREA)
- Computing Systems (AREA)
- Computer Hardware Design (AREA)
- Mathematical Physics (AREA)
- Debugging And Monitoring (AREA)
Abstract
The invention discloses a kind of distributed system metadata performance test methods, including：Thread enabled instruction is sent to client to be tested, obtains testing thread；Directory creating is sent to the test thread to instruct, obtain testing catalogue；Document creation is sent to the test catalogue to instruct；Statistics completes the time of the document creation instruction；With the quantity of documents for creating divided by the time, corresponding performance data is obtained.It can be seen that, the present invention creates the creation time of predetermined number file by counting according to many catalogues, multithreading, multi-client thought, comes the metadata performance of computation and measurement distributed system offer, the highest metadata performance of final exportable whole file system.The invention also discloses a kind of distributed system metadata performance test script and instrument, with technique effect same as mentioned above, will not be repeated here.
Description
Technical field
The present invention relates to distributed system metadata technical field of performance test, more particularly to a kind of distributed system unit number
According to performance test methods, script and instrument.
Background technology
It is well known that distributed file system has Metadata Service, data, services and the big part of client three.Distribution
Formula file system can carry out metadata IO first to file (including catalogue) operation, then carry out data IO.This operating process
Metadata performance is just shifted onto bottleneck point, particularly mass small documents IO scenes, due to storage architecture influence, metadata performance potential
Bottleneck point must be turned into.Metadata of distributed type file system performance can lift whole file system by metadata cluster concurrency performance
The metadata performance of system.The metadata performance of one distributed file system is how high, is surveyed there is presently no unified instrument
Amount metadata performance, the metadata UPS upper performance score that file system is provided in addition be how many also no instrument measure.
In sum, the metadata performance for how measuring distributed system offer is that those skilled in the art need solution at present
Technical problem certainly.
The content of the invention
In view of this, it is an object of the invention to provide a kind of distributed system metadata performance test methods, script and
Instrument, can accurately measure the metadata performance of distributed system offer.Its concrete scheme is as follows：
A kind of distributed system metadata performance test methods, including：
Thread enabled instruction is sent to client to be tested, obtains testing thread；
Directory creating is sent to the test thread to instruct, obtain testing catalogue；
Document creation is sent to the test catalogue to instruct；
Statistics completes the time of the document creation instruction；
With the quantity of documents for creating divided by the time, corresponding performance data is obtained.
Preferably, the client to be tested includes single client, 2 clients, 4 clients, 8 clients or 16
Individual client.
Preferably, test thread enabled instruction includes that single thread enabled instruction, dual-thread enabled instruction, 4 threads are opened
Dynamic instruction, 8 thread enabled instructions or 16 thread enabled instructions.
Preferably, the document creation instruction is to call open interfaces to create the text of predetermined number under corresponding test catalogue
Part.
Preferably, the file is empty file.
The invention also discloses a kind of distributed system metadata performance test script, including：
Thread enabled instruction, for starting test thread in client to be tested；
Directory creating is instructed, for creating test catalogue on the test thread；
Document creation is instructed, for the establishment file under the test catalogue；
Time statistics instruction, the time of the document creation instruction is completed for counting；
Performance data computations, divided by the time, corresponding performance data is obtained for the quantity of documents for creating.
The invention also discloses a kind of distributed system metadata performance testing tool, including above-mentioned distributed system unit
Data performance test script.
The invention discloses a kind of distributed system metadata performance test methods, including：Sent to client to be tested
Thread enabled instruction, obtains testing thread；Directory creating is sent to the test thread to instruct, obtain testing catalogue；To described
Test catalogue sends document creation instruction；Statistics completes the time of the document creation instruction；With create quantity of documents divided by
The time, obtain corresponding performance data.It can be seen that, it is of the invention according to many catalogues, multithreading, multi-client thought, by system
Meter creates the creation time of predetermined number file, carrys out the metadata performance of computation and measurement distributed system offer, final exportable
The highest metadata performance of whole file system.
The invention also discloses a kind of distributed system metadata performance test script, with skill same as mentioned above
Art effect, will not be repeated here.
The invention also discloses a kind of distributed system metadata performance testing tool, with above-mentioned script identical skill
Art effect, will not be repeated here.
Brief description of the drawings
In order to illustrate more clearly about the embodiment of the present invention or technical scheme of the prior art, below will be to embodiment or existing
The accompanying drawing to be used needed for having technology description is briefly described, it should be apparent that, drawings in the following description are only this
Inventive embodiment, for those of ordinary skill in the art, on the premise of not paying creative work, can also basis
The accompanying drawing of offer obtains other accompanying drawings.
Fig. 1 is a kind of schematic flow sheet of distributed system metadata performance test methods disclosed in the embodiment of the present invention；
Fig. 2 is a kind of structural representation of distributed system metadata performance test script disclosed in the embodiment of the present invention.
Specific embodiment
Below in conjunction with the accompanying drawing in the embodiment of the present invention, the technical scheme in the embodiment of the present invention is carried out clear, complete
Site preparation is described, it is clear that described embodiment is only a part of embodiment of the invention, rather than whole embodiments.It is based on
Embodiment in the present invention, it is every other that those of ordinary skill in the art are obtained under the premise of creative work is not made
Embodiment, belongs to the scope of protection of the invention.
It is shown in Figure 1 the embodiment of the invention discloses a kind of distributed system metadata performance test methods, including step
Rapid S1 to S5, wherein：
Step S1：Thread enabled instruction is sent to client to be tested, obtains testing thread.
The Metadata Service that distributed system is provided specifically includes single client and multi-client, in actual applications, many
Client includes 2 clients, 4 clients, 8 clients or 16 clients, certainly according to specific needs, multi-client
May also comprise 32 clients or 64 clients.
When the corresponding performance of Metadata Service for needing to provide the distributed system of multi-client is tested, it is necessary to
Client is called by the concurrent instruments of mpi, the client called is client to be tested.
Step S2：Directory creating is sent to the test thread to instruct, obtain testing catalogue.
The Metadata Service that distributed system is provided particularly may be divided into：Single client single thread data, services, single client
Multi-thread data service, multi-client single thread data, services and the service of multi-client multi-thread data.
It should be noted that in actual applications, multi-client include 2 clients, 4 clients, 8 clients or
16 clients, certainly according to specific needs, multi-client may also comprise 32 clients or 64 clients.
It should be further stated that, in actual applications, multithreading includes 2 threads, 4 threads, 8 threads or 16 threads,
Certainly according to specific needs, multithreading may also comprise 32 threads or 64 threads.
Therefore, thread enabled instruction includes single thread enabled instruction, dual-thread enabled instruction, 4 thread enabled instructions, 8 lines
Journey enabled instruction or 16 thread enabled instructions.
Step S3：Document creation is sent to the test catalogue to instruct.
In a kind of specific embodiment, document creation instruction is to call open interfaces to create under corresponding test catalogue
The file of predetermined number.
Predetermined number herein can be configured according to actual needs, for example, being created above by open interfaces
In the embodiment of predetermined number file, the file of predetermined number can be 100000 empty files, naturally it is also possible to for other
The empty file of quantity.
It should be noted that performance test is carried out for the Metadata Service that same distributed system is provided, if the survey
Examination includes：In single client single thread, single client multithreading, multi-client single thread and multi-client multithreading at least 2
Individual, then the total amount of the quantity of documents for creating should be identical, for example, it is desired to single client single thread, single thread of client 2,4 visitors
Family end single thread and 4 client dual-threads are tested, then single client single thread creates 100000 empty files, single client
End multithreading creates 50000 empty files of each directory creating, 4 clients in 100000 empty files, single client dual-thread
25000 empty files of each directory creating in the single thread of end, create 12500 skies in each catalogue in 4 client dual-threads
File, wherein, 4 client dual-threads refer to 4 clients, 2 threads of each client terminal start-up.
It should be further stated that, the file of establishment can be empty file, naturally it is also possible to create default standard text
Part.
Step S4：Statistics completes the time of the document creation instruction.
All Time to completing document creation instruction is counted.In previous example, in 4 client dual-threads
12500 empty files are created in each catalogue, may be different due to creating speed in different threads, therefore timing statisticses should be
Complete the time that 12500 last empty files create task.
Step S5：With the quantity of documents for creating divided by the time, corresponding performance data is obtained.
For the performance data test of single client single thread, i.e., with the quantity of documents for creating divided by timing statisticses；For
The performance data test of single client multithreading, multi-client single thread and multi-client multithreading, except the file with establishment
Quantity each thread can also be counted divided by timing statisticses to each client, i.e., complete file with establishment in the thread
Quantity completes the time of file divided by being created in the thread.
It is shown in Figure 2 the invention also discloses a kind of distributed system metadata performance test script, including thread opens
Dynamic instruction 11, directory creating instruction 12, document creation instruction 13, time statistics instruction 14 and performance data computations 15, its
In：
Thread enabled instruction 11, for starting test thread in client to be tested.
The Metadata Service that distributed system is provided specifically includes single client and multi-client, in actual applications, many
Client includes 2 clients, 4 clients, 8 clients and 16 clients, certainly according to specific needs, multi-client
May also comprise 32 clients and 64 clients.
When the corresponding performance of Metadata Service for needing to provide the distributed system of multi-client is tested, it is necessary to
Client is called by the concurrent instruments of mpi, the client called is client to be tested.
Directory creating instruction 12, for creating test catalogue on the test thread.
The Metadata Service that distributed system is provided particularly may be divided into：Single client single thread data, services, single client
Multi-thread data service, multi-client single thread data, services and the service of multi-client multi-thread data.
It should be noted that in actual applications, multi-client include 2 clients, 4 clients, 8 clients and
16 clients, certainly according to specific needs, multi-client may also comprise 32 clients and 64 clients.
It should be further stated that, in actual applications, multithreading includes 2 threads, 4 threads, 8 threads and 16 threads,
Certainly according to specific needs, multithreading may also comprise 32 threads and 64 threads.
Therefore, thread enabled instruction includes single thread enabled instruction, dual-thread enabled instruction, 4 thread enabled instructions, 8 lines
Journey enabled instruction, 16 thread enabled instructions.
Document creation instruction 13, for the establishment file under the test catalogue.
In a kind of specific embodiment, document creation instruction is to call open interfaces to create under corresponding test catalogue
The file of predetermined number.
Predetermined number herein can be configured according to actual needs, for example, being created above by open interfaces
In the embodiment of predetermined number file, the file of predetermined number can be 100000 empty files, naturally it is also possible to for other
The empty file of quantity.
It should be noted that performance test is carried out for the Metadata Service that same distributed system is provided, if the survey
Examination includes：In single client single thread, single client multithreading, multi-client single thread and multi-client multithreading at least 2
Individual, then the total amount of the quantity of documents for creating should be identical, for example, it is desired to single client single thread, single thread of client 2,4 visitors
Family end single thread and 4 client dual-threads are tested, then single client single thread creates 100000 empty files, single client
End multithreading creates 50000 empty files of each directory creating, 4 clients in 100000 empty files, single client dual-thread
25000 empty files of each directory creating in the single thread of end, create 12500 skies in each catalogue in 4 client dual-threads
File, wherein, 4 client dual-threads refer to 4 clients, 2 threads of each client terminal start-up.
It should be further stated that, the file of establishment can be empty file, naturally it is also possible to create default standard text
Part.
Time statistics instruction 14, the time of the document creation instruction is completed for counting.
All Time to completing document creation instruction is counted.In previous example, in 4 client dual-threads
12500 empty files are created in each catalogue, may be different due to creating speed in different threads, therefore timing statisticses should be
Complete the time that 12500 last empty files create task.
Performance data computations 15, divided by the time, corresponding performance number is obtained for the quantity of documents for creating
According to.
For the performance data test of single client single thread, i.e., with the quantity of documents for creating divided by timing statisticses；For
The performance data test of single client multithreading, multi-client single thread and multi-client multithreading, except the file with establishment
Quantity each thread can also be counted divided by timing statisticses to each client, i.e., complete file with establishment in the thread
Quantity completes the time of file divided by being created in the thread.
The invention also discloses a kind of distributed system metadata performance testing tool, including above-mentioned distributed system unit
Data performance test script.
In a particular embodiment, stored using NVDIMM media as metadata daily record, can guarantee that metadata will not lose
Lose, the metadata request that client sends in the form of the daily record storage in NVDIMM, metadata treatment is fast, and client can be quickly
Receive meta-data message response.Meanwhile, use SSD disk storage metadata, it is ensured that the processing speed of log in NVDIMM.
Finally, in addition it is also necessary to explanation, herein, such as first and second or the like relational terms be used merely to by
One entity or operation make a distinction with another entity or operation, and not necessarily require or imply these entities or operation
Between there is any this actual relation or order.And, term " including ", "comprising" or its any other variant meaning
Covering including for nonexcludability, so that process, method, article or equipment including a series of key elements not only include that
A little key elements, but also other key elements including being not expressly set out, or also include for this process, method, article or
The intrinsic key element of equipment.In the absence of more restrictions, the key element limited by sentence "including a ...", does not arrange
Except also there is other identical element in the process including the key element, method, article or equipment.
A kind of distributed system metadata performance test methods provided by the present invention, script and instrument are carried out above
It is discussed in detail, specific case used herein is set forth to principle of the invention and implementation method, above example
Illustrate that being only intended to help understands the method for the present invention and its core concept；Simultaneously for those of ordinary skill in the art, according to
According to thought of the invention, will change in specific embodiments and applications, in sum, this specification content
Should not be construed as limiting the invention.
Claims (7)
1. a kind of distributed system metadata performance test methods, it is characterised in that including：
Thread enabled instruction is sent to client to be tested, obtains testing thread；
Directory creating is sent to the test thread to instruct, obtain testing catalogue；
Document creation is sent to the test catalogue to instruct；
Statistics completes the time of the document creation instruction；
With the quantity of documents for creating divided by the time, corresponding performance data is obtained.
2. distributed system metadata performance test methods according to claim 1, it is characterised in that the visitor to be tested
Family end includes single client, 2 clients, 4 clients, 8 clients or 16 clients.
3. distributed system metadata performance test methods according to claim 2, it is characterised in that the test thread
Enabled instruction includes single thread enabled instruction, dual-thread enabled instruction, 4 thread enabled instructions, 8 thread enabled instructions or 16 threads
Enabled instruction.
4. distributed system metadata performance test methods according to claim 3, it is characterised in that the document creation
Instruction is to call open interfaces to create the file of predetermined number under corresponding test catalogue.
5. distributed system metadata performance test methods according to claim 4, it is characterised in that the file is sky
File.
6. a kind of distributed system metadata performance test script, it is characterised in that including：
Thread enabled instruction, for starting test thread in client to be tested；
Directory creating is instructed, for creating test catalogue on the test thread；
Document creation is instructed, for the establishment file under the test catalogue；
Time statistics instruction, the time of the document creation instruction is completed for counting；
Performance data computations, divided by the time, corresponding performance data is obtained for the quantity of documents for creating.
7. a kind of distributed system metadata performance testing tool, it is characterised in that including the distribution system described in claim 6
System metadata performance test script.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201710089997.6A CN106844184A (en) | 2017-02-20 | 2017-02-20 | A kind of distributed system metadata performance test methods, script and instrument |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201710089997.6A CN106844184A (en) | 2017-02-20 | 2017-02-20 | A kind of distributed system metadata performance test methods, script and instrument |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN106844184A true CN106844184A (en) | 2017-06-13 |
Family
ID=59129073
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201710089997.6A Pending CN106844184A (en) | 2017-02-20 | 2017-02-20 | A kind of distributed system metadata performance test methods, script and instrument |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN106844184A (en) |
Cited By (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN107729362A (en) * | 2017-09-05 | 2018-02-23 | 郑州云海信息技术有限公司 | A kind of storing performance testing method concurrent based on MPI |
| US10776009B2 (en) | 2019-01-03 | 2020-09-15 | International Business Machines Corporation | Journaling on an appendable non-volatile memory module |
| CN112241365A (en) * | 2020-07-24 | 2021-01-19 | 国家计算机网络与信息安全管理中心 | Method and device for efficient metadata construction and RPC performance improvement for Namenode |
Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102750221A (en) * | 2012-05-29 | 2012-10-24 | 兰雨晴 | Performance test method for Linux file system |
| CN104850584A (en) * | 2015-04-17 | 2015-08-19 | 北京赛思信安技术有限公司 | Test tool and method applied to metadata test of mass data storage system |
-
2017
- 2017-02-20 CN CN201710089997.6A patent/CN106844184A/en active Pending
Patent Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102750221A (en) * | 2012-05-29 | 2012-10-24 | 兰雨晴 | Performance test method for Linux file system |
| CN104850584A (en) * | 2015-04-17 | 2015-08-19 | 北京赛思信安技术有限公司 | Test tool and method applied to metadata test of mass data storage system |
Non-Patent Citations (1)
| Title |
|---|
| 许春聪: "CarrierFS_基于虚拟内存的分布式文件系统", 《华中科技大学学报(自然科学版)》 * |
Cited By (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN107729362A (en) * | 2017-09-05 | 2018-02-23 | 郑州云海信息技术有限公司 | A kind of storing performance testing method concurrent based on MPI |
| CN107729362B (en) * | 2017-09-05 | 2021-08-20 | 郑州云海信息技术有限公司 | MPI concurrency-based storage performance testing method |
| US10776009B2 (en) | 2019-01-03 | 2020-09-15 | International Business Machines Corporation | Journaling on an appendable non-volatile memory module |
| CN112241365A (en) * | 2020-07-24 | 2021-01-19 | 国家计算机网络与信息安全管理中心 | Method and device for efficient metadata construction and RPC performance improvement for Namenode |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| Li et al. | A performance comparison of SQL and NoSQL databases | |
| Jost et al. | Partitioning diversity for conservation analyses | |
| US8677366B2 (en) | Systems and methods for processing hierarchical data in a map-reduce framework | |
| US9477974B2 (en) | Method and systems for flexible and scalable databases | |
| CN106970929B (en) | Data import method and device | |
| CN106844184A (en) | A kind of distributed system metadata performance test methods, script and instrument | |
| Wang et al. | Distributed storage and index of vector spatial data based on HBase | |
| CN103714004A (en) | JVM online memory leak analysis method and system | |
| CN104123280A (en) | File comparison method and device | |
| CN104778182A (en) | Data import method and system based on HBase (Hadoop Database) | |
| US11853284B2 (en) | In-place updates with concurrent reads in a decomposed state | |
| Barbuzzi et al. | Parallel bulk Insertion for large-scale analytics applications | |
| CN105574164A (en) | Excel document data analysis method and device | |
| US20110179008A1 (en) | Hosting multiple logical databases contained in physical database | |
| Kougkas et al. | Chronolog: a distributed shared tiered log store with time-based data ordering | |
| Zhu et al. | An efficient bulk loading approach of secondary index in distributed log-structured data stores | |
| CN102521397A (en) | Data transmission method | |
| CN116028446A (en) | Time sequence data file management method, device, equipment and storage medium thereof | |
| TWI626548B (en) | Data collection and storage system and method thereof | |
| CN111522832B (en) | Big data real-time updating method, system, equipment and storage medium | |
| US9652308B2 (en) | Sharing a partitioned data set across parallel applications | |
| CN104978250A (en) | Iozone-based HBA card performance test method | |
| CN103150340A (en) | Fast loading system and method for multimedia data | |
| Schwarz et al. | The alice analysis facility prototype at gsi | |
| Hu et al. | External memory stream sampling |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20170613 |
|
| RJ01 | Rejection of invention patent application after publication |