CN103617004A - Tool and method for performing read-write tests on distributed file system - Google Patents
Tool and method for performing read-write tests on distributed file system Download PDFInfo
- Publication number
- CN103617004A CN103617004A CN201310584942.4A CN201310584942A CN103617004A CN 103617004 A CN103617004 A CN 103617004A CN 201310584942 A CN201310584942 A CN 201310584942A CN 103617004 A CN103617004 A CN 103617004A
- Authority
- CN
- China
- Prior art keywords
- node
- control
- file
- test result
- test
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Abstract
The invention discloses a tool and a method for performing read-write tests on a distributed file system. The method includes: aiming at the architecture features of the distributed file system, transmitting a control file to each storage node, allowing each storage node to execute read-write tests, recording the test results during the tests, returning the results to a control node, and allowing the control node to perform uniform calculation and evaluation. The tool mainly comprises the control mode and storage nodes. The tool and the method have the advantages that read-write instructions under high concurrency scenes are simulated, each control file only has a file name and file size, the data size generated by each control file is small and does not influence the read-write tests, coverage rate of system testing to the storage nodes is increased by the testing architecture, and the read-write performance of each distributed file system product can be reliably and accurately tested.
Description
Technical field
The present invention relates to computing machine high-performance computing sector, be specifically related to a kind of benchmark test of testing tool read and write to(for) distributed file system.
Technical background
Development along with information society, traditional industries are more and more and present the situation of expansion type accelerated development in the input of information application aspect, increasing data have been produced thus, the memory requirement that traditional storage architecture gradually cannot satisfying magnanimity data.And along with the develop rapidly of internet, text picture and video etc. and the traditional different unstructured data of structural data also rapid expansion, the storage of these destructuring mass datas has also proposed challenge to traditional centralized stores framework.How the continuous increase along with each enterprise, IT being dropped in addition, allow existing data produce to be worth and just do not tie up this proposition of storage resources and produced requirements at the higher level for the computing power of data mining and data analysis.
To sum up produced the concept of large data, comprised the storage of mass data and Distributed Calculation etc. technical field, wherein the storage of mass data generally adopts the mode of distributed storage at present.The Realization of Product of the distributed file system in distributed storage has much at present.The object of the invention of this instrument is that the lateral performance for these products relatively provides a unified standard and operation, can have to the performance of each distributed file system product the evaluation of an objective justice thus.
Summary of the invention
The technical problem to be solved in the present invention is: the present invention seeks to Technical Architecture characteristic for distributed file system, simulate it in the face of being difficult to realize concurrent write test and concurrent control of reading test under high concurrent, big data quantity test scene, improve accuracy and the unitarity of evaluating system.
Under distributed file system environment, each storage node is write to test assignment and distribute the concurrency that is difficult to realize filename continuity and task distribution.The control that each storage node is write to file generated size in test assignment is difficult to realize specifies according to control command.Be difficult to record and write each correlation results data of write performance in test process.Be difficult to feed back the test result of writing of each storage node.Under distributed file system environment, each storage node reading test assignment distribution is difficult to realize each node and specifies the concurrency of reading filename and task distribution.Be difficult to realize the removing of buffer memory and the record of reading to read in test process each related data of performance in each storage node reading test assignment.Be difficult to feed back each storage node reading test result.Control node and be difficult to calculate according to unified module according to the test result receiving, thereby obtain the write performance under unified standard and read performance test conclusion.
The technical solution adopted in the present invention is:
Distributed file system is read and write to an instrument for benchmark test, this instrument mainly comprises controls node and two parts of storage node, wherein:
Control node and comprise control documents generation strategy, control documents distribution module, communication module, test result receiver module, test result analysis module, under the high concurrent scene of control documents generation strategy simulation, specify the size of spanned file, according to storage node number, produce corresponding control documents quantity and corresponding spanned file name; By control documents distribution module, by communication module, control documents is distributed to each storage node and carries out; The test result of each storage node that received communication module is returned, is calculated and result of calculation is charged to file or is illustrated in terminal according to unified module by test result analysis module, wherein:
Control documents distribution module is responsible for simulating writing and reading order of file under high concurrent state, and according to cluster situation, generates control documents and be distributed to each storage node and carry out;
Communication module is responsible for providing and is controlled communicating by letter and the detection to fault node of node and storage node, and the control command that sends control node also receives the test result that storage node returns;
The test result of each storage node that test result receiver module received communication module is returned;
Test result analysis module is to be responsible for the collection of test result and the scheduling that node is controlled in processing, acceptance, the readwrite tests result of the storage node that reception is sent by communication module, and calculate the performance data of writing test and reading test according to relevant metric algorithm, and by result writing in files or be illustrated in terminal;
Storage node comprises that file writing module, file read module, test result logging modle, test result return to module, according to the control documents receiving by communication module, carry out corresponding file generated or file read operation, and record in the process of implementation relevant test data and final testing result is returned to control node, wherein:
File writing module is the file that the content in the control documents sending according to control node generates corresponding document size and filename, records the correlated performance data of write operation in generative process;
Content in the control documents that file read module sends according to control node reads corresponding file, and in reading process, records read operation correlated performance data;
Test result logging modle is responsible for recording corresponding data in read-write operation process;
Test result return module be responsible for by write test and read to have tested after result return to control node.
A kind of distributed file system is read and write to reference test method, characteristic for distributed file system framework, control documents is sent to each storage node, by each storage node, independently carry out the test of write and read, and logging test results in the process of writing test and reading to test, and test result is turned back to control node, by controlling node, unifiedly calculate and assess.
It is as follows that described method comprises that distributed file system is write testing process:
Control node receives to be write after test command, is generated the control documents of respective numbers by control documents generation strategy according to clustered node quantity, and every part of control documents comprises the filename that this memory node generates, the size of spanned file; Then by the high concurrent environment of this generation strategy module simulation, control documents is distributed to each back end; Each back end receives after control documents, according to control documents content, carries out spanned file operation, and in write operation process, by the test result logging modle on back end, is recorded the respective performances data of write operation process; After completing file generated operation, by writing test result on node, turn back to control node; Control node after receiving the test result that each memory node returns, by test result analysis module, collect result data and carry out analytic statistics, produce unified test result data; Configuration according to system will show in test result data writing in files or at control nodal terminal;
It is as follows that described method comprises that distributed file system is read testing process:
Control node receives to be read after test command, is generated the control documents of respective numbers by control documents generation strategy according to clustered node quantity, and every part of control documents comprises the filename that this memory node is specified file reading; Then by the high concurrent environment of this generation strategy module simulation, control documents is distributed to each back end; Each back end receives after control documents, according to control documents content, carries out file reading operation, before file reading, in order to guarantee test accuracy, can first the buffer memory of reading of this node be emptied to operation; In reading file operation process, by the test result logging modle on back end, recorded the respective performances data of read operation process; After completing file reading operation, by reading test result on node, turn back to control node; Control node after receiving the test result that each memory node returns, by test result analysis module, collect result data and carry out analytic statistics, produce unified test result data; Configuration according to system will show in test result data writing in files or at control nodal terminal.
Beneficial effect of the present invention is:
By control documents, simulate the read write command realizing under high concurrent scene, each control documents only has filename, two contents of file size, the data volume that produces very little, substantially can not exert an influence to the readwrite tests of distributed file system, and by this test structure, effectively improved the coverage rate of system testing to storage node, for the readwrite performance of reliably, accurately evaluating and testing each distributed file system product of system provides a unified testing tool.
Accompanying drawing explanation
Fig. 1 is that distributed file system is write test structure schematic diagram;
Fig. 2 is that distributed file system is read test structure schematic diagram.
Embodiment
With reference to the accompanying drawings, in conjunction with the embodiments to the detailed description of the invention.
Embodiment 1:
Distributed file system is read and write to an instrument for benchmark test, mainly comprised and control node and two parts of storage node, wherein:
Control node and comprise control documents generation strategy, control documents distribution module, communication module, test result receiver module, test result analysis module, under the high concurrent scene of control documents generation strategy simulation, specify the size of spanned file, according to storage node number, produce corresponding control documents quantity and corresponding spanned file name; By control documents distribution module, by communication module, control documents is distributed to each storage node and carries out; The test result of each storage node that received communication module is returned, is calculated and result of calculation is charged to file or is illustrated in terminal according to unified module by test result analysis module, wherein:
Control documents distribution module is responsible for simulating writing and reading order of file under high concurrent state, and according to cluster situation, generates control documents and be distributed to each storage node and carry out;
Communication module is responsible for providing and is controlled communicating by letter and the detection to fault node of node and storage node, and the control command that sends control node also receives the test result that storage node returns;
The test result of each storage node that test result receiver module received communication module is returned;
Test result analysis module is to be responsible for the collection of test result and the scheduling that node is controlled in processing, acceptance, the readwrite tests result of the storage node that reception is sent by communication module, and calculate the performance data of writing test and reading test according to relevant metric algorithm, and by result writing in files or be illustrated in terminal;
Storage node comprises that file writing module, file read module, test result logging modle, test result return to module, according to the control documents receiving by communication module, carry out corresponding file generated or file read operation, and record in the process of implementation relevant test data and final testing result is returned to control node, wherein:
File writing module is the file that the content in the control documents sending according to control node generates corresponding document size and filename, records the correlated performance data of write operation in generative process;
Content in the control documents that file read module sends according to control node reads corresponding file, and in reading process, records read operation correlated performance data;
Test result logging modle is responsible for recording corresponding data in read-write operation process;
Test result return module be responsible for by write test and read to have tested after result return to control node.
Embodiment 2:
A kind of distributed file system is read and write to reference test method, characteristic for distributed file system framework, control documents is sent to each storage node, by each storage node, independently carry out the test of write and read, and logging test results in the process of writing test and reading to test, and test result is turned back to control node, by controlling node, unifiedly calculate and assess.
Embodiment 3:
As shown in Figure 1, on the basis of embodiment 2, described in the present embodiment in method distributed file system to write testing process as follows:
Control node receives to be write after test command, is generated the control documents of respective numbers by control documents generation strategy according to clustered node quantity, and every part of control documents comprises the filename that this memory node generates, the size of spanned file; Then by the high concurrent environment of this generation strategy module simulation, control documents is distributed to each back end; Each back end receives after control documents, according to control documents content, carries out spanned file operation, and in write operation process, by the test result logging modle on back end, is recorded the respective performances data of write operation process; After completing file generated operation, by writing test result on node, turn back to control node; Control node after receiving the test result that each memory node returns, by test result analysis module, collect result data and carry out analytic statistics, produce unified test result data; Configuration according to system will show in test result data writing in files or at control nodal terminal.
Embodiment 4:
As shown in Figure 2, on the basis of embodiment 2, described in the present embodiment in method distributed file system to read testing process as follows:
Control node receives to be read after test command, is generated the control documents of respective numbers by control documents generation strategy according to clustered node quantity, and every part of control documents comprises the filename that this memory node is specified file reading; Then by the high concurrent environment of this generation strategy module simulation, control documents is distributed to each back end; Each back end receives after control documents, according to control documents content, carries out file reading operation, before file reading, in order to guarantee test accuracy, can first the buffer memory of reading of this node be emptied to operation; In reading file operation process, by the test result logging modle on back end, recorded the respective performances data of read operation process; After completing file reading operation, by reading test result on node, turn back to control node; Control node after receiving the test result that each memory node returns, by test result analysis module, collect result data and carry out analytic statistics, produce unified test result data; Configuration according to system will show in test result data writing in files or at control nodal terminal.
Claims (4)
1. distributed file system is read and write to an instrument for benchmark test, be it is characterized in that, this instrument mainly comprises controls node and two parts of storage node, wherein:
Control node and comprise control documents generation strategy, control documents distribution module, communication module, test result receiver module, test result analysis module, under the high concurrent scene of control documents generation strategy simulation, specify the size of spanned file, according to storage node number, produce corresponding control documents quantity and corresponding spanned file name; By control documents distribution module, by communication module, control documents is distributed to each storage node and carries out; The test result of each storage node that received communication module is returned, is calculated and result of calculation is charged to file or is illustrated in terminal according to unified module by test result analysis module, wherein:
Control documents distribution module is responsible for simulating writing and reading order of file under high concurrent state, and according to cluster situation, generates control documents and be distributed to each storage node and carry out;
Communication module is responsible for providing and is controlled communicating by letter and the detection to fault node of node and storage node, and the control command that sends control node also receives the test result that storage node returns;
The test result of each storage node that test result receiver module received communication module is returned;
Test result analysis module is to be responsible for the collection of test result and the scheduling that node is controlled in processing, acceptance, the readwrite tests result of the storage node that reception is sent by communication module, calculating is write test and is read the performance data of test, and by result writing in files or be illustrated in terminal;
Storage node comprises that file writing module, file read module, test result logging modle, test result return to module, according to the control documents receiving by communication module, carry out corresponding file generated or file read operation, and record in the process of implementation relevant test data and final testing result is returned to control node, wherein:
File writing module is the file that the content in the control documents sending according to control node generates corresponding document size and filename, records the correlated performance data of write operation in generative process;
Content in the control documents that file read module sends according to control node reads corresponding file, and in reading process, records read operation correlated performance data;
Test result logging modle is responsible for recording corresponding data in read-write operation process;
Test result return module be responsible for by write test and read to have tested after result return to control node.
2. one kind distributed file system is read and write to reference test method, it is characterized in that: for the characteristic of distributed file system framework, control documents is sent to each storage node, by each storage node, independently carry out the test of write and read, and logging test results in the process of writing test and reading to test, and test result is turned back to control node, by controlling node, unifiedly calculate and assess.
3. according to claim 2ly a kind of distributed file system is read and write to reference test method, it is characterized in that in described method that distributed file system is write testing process as follows:
Control node receives to be write after test command, is generated the control documents of respective numbers by control documents generation strategy according to clustered node quantity, and every part of control documents comprises the filename that this memory node generates, the size of spanned file; Then by the high concurrent environment of this generation strategy module simulation, control documents is distributed to each back end; Each back end receives after control documents, according to control documents content, carries out spanned file operation, and in write operation process, by the test result logging modle on back end, is recorded the respective performances data of write operation process; After completing file generated operation, by writing test result on node, turn back to control node; Control node after receiving the test result that each memory node returns, by test result analysis module, collect result data and carry out analytic statistics, produce unified test result data; Configuration according to system will show in test result data writing in files or at control nodal terminal.
4. according to claim 2ly a kind of distributed file system is read and write to reference test method, it is characterized in that in described method that distributed file system is read testing process as follows:
Control node receives to be read after test command, is generated the control documents of respective numbers by control documents generation strategy according to clustered node quantity, and every part of control documents comprises the filename that this memory node is specified file reading; Then by the high concurrent environment of this generation strategy module simulation, control documents is distributed to each back end; Each back end receives after control documents, according to control documents content, carries out file reading operation, before file reading, in order to guarantee test accuracy, can first the buffer memory of reading of this node be emptied to operation; In reading file operation process, by the test result logging modle on back end, recorded the respective performances data of read operation process; After completing file reading operation, by reading test result on node, turn back to control node; Control node after receiving the test result that each memory node returns, by test result analysis module, collect result data and carry out analytic statistics, produce unified test result data; Configuration according to system will show in test result data writing in files or at control nodal terminal.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201310584942.4A CN103617004A (en) | 2013-11-20 | 2013-11-20 | Tool and method for performing read-write tests on distributed file system |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201310584942.4A CN103617004A (en) | 2013-11-20 | 2013-11-20 | Tool and method for performing read-write tests on distributed file system |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN103617004A true CN103617004A (en) | 2014-03-05 |
Family
ID=50167707
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201310584942.4A Pending CN103617004A (en) | 2013-11-20 | 2013-11-20 | Tool and method for performing read-write tests on distributed file system |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN103617004A (en) |
Cited By (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2016065556A1 (en) * | 2014-10-29 | 2016-05-06 | 北京麓柏科技有限公司 | Software-defined storage system and method, and centralized control device thereof |
| CN106055486A (en) * | 2016-08-19 | 2016-10-26 | 浪潮(北京)电子信息产业有限公司 | Automatic operation maintenance method and platform of distributed file system |
| CN106897200A (en) * | 2017-02-22 | 2017-06-27 | 郑州云海信息技术有限公司 | A kind of distributed file system performance method of testing based on redis |
| CN106933739A (en) * | 2017-03-10 | 2017-07-07 | 郑州云海信息技术有限公司 | A kind of read-write hybrid test instrument based on hbase |
| CN107229564A (en) * | 2016-03-25 | 2017-10-03 | 阿里巴巴集团控股有限公司 | A kind of pressure simulation method and device |
| CN107480039A (en) * | 2017-09-22 | 2017-12-15 | 郑州云海信息技术有限公司 | The small documents readwrite performance method of testing and device of a kind of distributed memory system |
| CN108665938A (en) * | 2018-04-28 | 2018-10-16 | 百富计算机技术（深圳）有限公司 | It writes test method, read test method, readwrite tests method and terminal device |
| CN108874611A (en) * | 2017-05-12 | 2018-11-23 | 北京金山云网络技术有限公司 | A kind of construction method and device of test data |
Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20090222508A1 (en) * | 2000-03-30 | 2009-09-03 | Hubbard Edward A | Network Site Testing |
| CN102420727A (en) * | 2012-01-05 | 2012-04-18 | 北京邮电大学 | Distributed protocol test system and method |
-
2013
- 2013-11-20 CN CN201310584942.4A patent/CN103617004A/en active Pending
Patent Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20090222508A1 (en) * | 2000-03-30 | 2009-09-03 | Hubbard Edward A | Network Site Testing |
| CN102420727A (en) * | 2012-01-05 | 2012-04-18 | 北京邮电大学 | Distributed protocol test system and method |
Non-Patent Citations (1)
| Title |
|---|
| 卢华: "基于分布式的自动化协议测试系统的研究与实现", 《中国优秀硕士学位论文全文数据库信息科技辑》 * |
Cited By (9)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2016065556A1 (en) * | 2014-10-29 | 2016-05-06 | 北京麓柏科技有限公司 | Software-defined storage system and method, and centralized control device thereof |
| CN107229564A (en) * | 2016-03-25 | 2017-10-03 | 阿里巴巴集团控股有限公司 | A kind of pressure simulation method and device |
| CN107229564B (en) * | 2016-03-25 | 2020-12-11 | 阿里巴巴集团控股有限公司 | Pressure simulation method and device |
| CN106055486A (en) * | 2016-08-19 | 2016-10-26 | 浪潮(北京)电子信息产业有限公司 | Automatic operation maintenance method and platform of distributed file system |
| CN106897200A (en) * | 2017-02-22 | 2017-06-27 | 郑州云海信息技术有限公司 | A kind of distributed file system performance method of testing based on redis |
| CN106933739A (en) * | 2017-03-10 | 2017-07-07 | 郑州云海信息技术有限公司 | A kind of read-write hybrid test instrument based on hbase |
| CN108874611A (en) * | 2017-05-12 | 2018-11-23 | 北京金山云网络技术有限公司 | A kind of construction method and device of test data |
| CN107480039A (en) * | 2017-09-22 | 2017-12-15 | 郑州云海信息技术有限公司 | The small documents readwrite performance method of testing and device of a kind of distributed memory system |
| CN108665938A (en) * | 2018-04-28 | 2018-10-16 | 百富计算机技术（深圳）有限公司 | It writes test method, read test method, readwrite tests method and terminal device |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN103617004A (en) | Tool and method for performing read-write tests on distributed file system | |
| CN107480039B (en) | Small file read-write performance test method and device for distributed storage system | |
| JP5978401B2 (en) | Method and system for monitoring the execution of user requests in a distributed system | |
| US20140359624A1 (en) | Determining a completion time of a job in a distributed network environment | |
| CN110221983B (en) | Test method, test device, computer readable storage medium and computer equipment | |
| CN106055464B (en) | Data buffer storage testing schooling pressure device and method | |
| US11200149B2 (en) | Waveform based reconstruction for emulation | |
| Cheah et al. | Milieu: Lightweight and configurable big data provenance for science | |
| CN105306299A (en) | Streaming media server performance test method and test system | |
| CN102750221A (en) | Performance test method for Linux file system | |
| WO2017114472A1 (en) | Method and apparatus for data mining from core traces | |
| CN107391378A (en) | The generation method and device of a kind of test script | |
| CN104636401B (en) | A kind of method and device of SCADA system data rewind | |
| CN108829802B (en) | Associated log playback method and device | |
| CN110297743B (en) | Load testing method and device and storage medium | |
| CN104317957A (en) | Open platform and system for processing reports and report processing method | |
| US11151013B2 (en) | Systems and methods for performance evaluation of input/output (I/O) intensive enterprise applications | |
| CN117077588B (en) | Hardware acceleration simulation debugging system | |
| CN105069139A (en) | File access method, file access device and server | |
| Jayakumar et al. | INTERNATIONAL JOURNAL OF COMPUTER ENGINEERING & TECHNOLOGY (IJCET) | |
| Luo et al. | ScalaiOExtrap: Elastic I/O tracing and extrapolation | |
| CN105279007A (en) | Multi-core processor simulation method and apparatus | |
| CN103902304A (en) | Method and device for evaluating Web application and system | |
| JP2007249949A (en) | Device for storing variable value to provide context for test result to be formatted | |
| US20140325468A1 (en) | Storage medium, and generation apparatus for generating transactions for performance evaluation |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| C10 | Entry into substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20140305 |
|
| RJ01 | Rejection of invention patent application after publication |