CN104794007A - Mass data parallel processing testing method based on electric large data platform - Google Patents
Mass data parallel processing testing method based on electric large data platform Download PDFInfo
- Publication number
- CN104794007A CN104794007A CN201510213675.9A CN201510213675A CN104794007A CN 104794007 A CN104794007 A CN 104794007A CN 201510213675 A CN201510213675 A CN 201510213675A CN 104794007 A CN104794007 A CN 104794007A
- Authority
- CN
- China
- Prior art keywords
- large data
- data platform
- electric power
- response time
- virtual user
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Landscapes
- Test And Diagnosis Of Digital Computers (AREA)
Abstract
The invention provides a mass data parallel processing testing method based on an electric large data platform. The method includes the steps that the operation method of single files is packaged into a Java program capable of running; virtual users are started, the Java program capable of running concurrently runs, and the response time of the single users, the network throughout of the electric large data platform, and the occupation condition of disk IO, CPU and memory resources are observed and recorded; the number of the virtual users running the Java program capable of running is gradually increased, and the average response time of the current single users, the network throughout of the electric large data platform, and the occupation condition of the disk IO, CPU and memory resources are observed and recorded; when the average response time of the single users is increased to 10 times of an initial value, increasing of the virtual users stops, the load condition is the maximum load number of the electric large data platform, and the pressure testing process is completed. Due to the method, an assigned number of testing loads concurrently executed by the electric large data platform are guaranteed, and a testing measurement is controllable and reliable.
Description
Technical field
The present invention relates to the method for testing of parallel data processing, be specifically related to a kind of method of testing based on the parallel processing of electric power large data platform mass data.
Background technology
The large data environment of electric power is progressively formed, and the data scale of construction is just at undergoes rapid expansion, and data type is diversified gradually, and data age improves constantly.The large data platform of electric power has carried out revolutionary reconstruct to whole power information Technical Architecture, changes into data-centered, calculating is used for data by the method for operation of infosystem, forms the flexibility association between service logic, computation structure, data model.
" large data " just technological incorporation type under the new situation develop and apply the concentrated reflection of intelligent theory, the Technical Architecture of the data characteristics such as have that the reply data scale of construction is huge, data type is various, value density is low and processing speed is fast, and with the application model that the content value-added service of high added value inside and outside industry is target, its core is exactly information resource development and utilization.Electric power large data refer to be collected by various acquisition of information channels such as sensor, smart machine, video monitoring equipment, audio communication device, mobile terminals, magnanimity, structuring, semi-structured, non-structured, and there is the business datum set of incidence relation each other.Along with the development of science and technology and the popularization of bulk information system and strengthened research, Consumer's Experience demand constantly increases, the data accumulated in this process and service also get more and more, and be tending towards magnanimity, traditional database cannot meet so huge call data storage.
Along with the scope of business of the large data platform of electric power covers Various types of data excavating resource and business innovation, integrated data resource consolidation, calculating middleware services, Intelligent Information Processing service, provide the intelligent decision analysis ability towards typical services application.Need the method for a set of correspondence or system to test the charge capacity of the large data platform of electric power, better to utilize the large data platform of electric power.
Summary of the invention
On the basis of existing technology, the concurrency test process that the present invention uses, it is a kind of Load pressure test process, by progressively increasing system load, the change of system performance under observation different loads, determine when meeting performance index, the ultimate load that system can bear, may produced problem under the performance bottleneck that discovery system may exist and high load condition.Load pressure test can determine system performance is in failure state under what loading condition, obtains with this maximum service rank that system can provide.Propose a kind of method of testing based on the parallel processing of electric power large data platform mass data, based on the large data platform of electric power, Load pressure test is carried out to the parallel processing of mass data.Comprise the following steps: be encapsulated as respectively can run java applet by the read operation method of Single document and write operation method; Start multiple Virtual User, concurrent running carries out the run java applet of read operation to Single document or Single document is carried out to the run java applet of write operation, the situation that takies of the response time of run java applet that observed and recorded average treatment unique user runs and the resource such as network throughput, disk I/O, CPU, internal memory of the large data platform cluster of electric power; Increase gradually to run and the run java applet of read operation is carried out to Single document or Single document is carried out to the number of Virtual User of run java applet of write operation, the response time of run java applet that under observed and recorded current virtual number of users, average treatment unique user runs and the situation that takies of the resource such as network throughput, disk I/O, CPU, internal memory of the large data platform cluster of electric power; When the response time of the run java applet that average treatment unique user runs is increased to 10 times of initial value, terminate the Load pressure test of the large data platform parallel processing of electric power.
The object of the invention is to adopt following technical proposals to realize:
Based on a method of testing for electric power large data platform mass data parallel processing, its improvements are, described method comprises:
(1) be encapsulated as can run java applet by the method for operating of Single document;
(2) start Virtual User, concurrent running can run java applet, the network throughput of the response time of observed and recorded unique user and the large data platform cluster of electric power, disk I/O, CPU and memory source take situation;
(3) increase gradually to run and can run the Virtual User number of java applet, the response time of the current average unique user of observed and recorded and the network throughput of the large data platform cluster of electric power, disk I/O, CPU and memory source take situation;
(4) when the response time of average unique user is increased to 10 times of initial value, stop increasing Virtual User, this loading condition is the ultimate load of the large data platform of electric power, completes pressure test sequence.
Preferably, in described step (1), the operation of Single document can realize reading different size, dissimilar Single document, write, increase, delete, revising and search operation.
Further,
Described read operation be from the distributed memory system of the large data platform of electric power by file download to Virtual User client;
Said write be operating as from Virtual User client by files passe to the distributed memory system of the large data platform of electric power;
Described increase operation, deletion action and retouching operation are process accordingly the file in the large data platform distributed memory system of electric power; With
Described search operation is search the file in the large data platform distributed memory system of electric power and result feedback is returned Virtual User client.
Preferably, in described step (2), Virtual User startup number needs takes into full account distributed factor, start the large data platform nodes of electric power with the client of the order of magnitude, ensure that the charge capacity of each client is less than its network throughput, and the load of each client of equilibrium allocation.
Preferably, when increasing Virtual User in described step (3), when the response time of the run java applet that average treatment unique user runs is less than 3 times of initial value, every 1/3 response time increases by 2 times that last Virtual User increases number; When being not less than 3 times of initial value, every 1/2 response time increases last Virtual User and increases number.
Preferably, the response time of the average unique user of described step (4) does not exceed 10 times of initial value, then return step (3).
Compared with the prior art, beneficial effect of the present invention is:
1. the advantage that the present invention makes full use of parallel processing carries out Load pressure test, and ensure the test load of electric power large data platform concurrence performance specified quantity, test result is controlled, reliable.
2. the present invention is while carrying out extreme pressure test, running performance index, saving resource and the time of the large data platform of electric power under acquisition different loads condition.
3. the Virtual User number flexibility and changeability that increases newly at set intervals of the present invention, the load pressure also flexibility and changeability that each Virtual User runs, the performance index of the large data platform of electric power under random loading condition can be tested, can adjust rapidly test load amount close to during system processing limit, effective anti-locking system is paralysed because of overload.
4. startup of the present invention and the large data platform nodes of electric power are with the client of the order of magnitude, fully expand available network bandwidth capacity, break the network throughput test bottleneck of distributed environment, are beneficial to the load limit measuring the large data platform of electric power.
5. the present invention can the load of each client terminal start-up of equilibrium allocation, highlights the superiority of parallel environment, can ensure that the charge capacity of each client is all less than its network throughput, and realizes test procedure and run and coordinate.
Accompanying drawing explanation
Fig. 1 is a kind of method of testing process flow diagram based on the parallel processing of electric power large data platform mass data provided by the invention.
Embodiment
Below in conjunction with accompanying drawing, the specific embodiment of the present invention is described in further detail.
A kind of method of testing based on the parallel processing of electric power large data platform mass data of the present invention, idiographic flow is as follows:
Step 1: be encapsulated as can run java applet by the method for operating of Single document;
Wherein, the operation of Single document can realize reading different size, dissimilar Single document, writes, increases, deletes, revises, the operation such as search.
Read operation be from the distributed memory system of the large data platform of electric power by file download to Virtual User client, write operation be from Virtual User client by files passe to the distributed memory system of the large data platform of electric power, increase operation, deletion action, retouching operation are process accordingly the file in the large data platform distributed memory system of electric power, and search operation is search the file in the large data platform distributed memory system of electric power and result feedback is returned Virtual User client.
Step 2: start multiple Virtual User, concurrent running can run java applet, the situation that takies of the resource such as network throughput, disk I/O, CPU, internal memory of the response time of the average unique user of observed and recorded and the large data platform cluster of electric power;
Wherein, Virtual User starts number needs will take into full account distributed factor, starts the large data platform nodes of electric power with the client of the order of magnitude, ensures that the charge capacity of each client is all less than its network throughput, and the load of each client of equilibrium allocation.
Step 3: increase the number run and can run the Virtual User of java applet gradually, the situation that takies of the response time of the current average unique user of observed and recorded and the resource such as network throughput, disk I/O, CPU, internal memory of the large data platform cluster of electric power;
Wherein, when increasing Virtual User, when the response time of the run java applet that average treatment unique user runs is less than 3 times of initial value, every 1/3 response time increases last 2 times of increasing Virtual User number; When being not less than 3 times of initial value, every 1/2 response time increases the last Virtual User number increased.
Step 4: when the response time of average unique user is increased to 10 times of initial value, stops increasing Virtual User, can think that this loading condition is the ultimate load of the large data platform of electric power, complete pressure test sequence.
Finally should be noted that: above embodiment is only in order to illustrate that technical scheme of the present invention is not intended to limit; those of ordinary skill in the field still can modify to the specific embodiment of the present invention with reference to above-described embodiment or equivalent replacement; these do not depart from any amendment of spirit and scope of the invention or equivalent replacement, are all applying within the claims of the present invention awaited the reply.
Claims (6)
1. based on a method of testing for electric power large data platform mass data parallel processing, it is characterized in that, described method comprises:
(1) be encapsulated as can run java applet by the method for operating of Single document;
(2) start Virtual User, concurrent running can run java applet, the network throughput of the response time of observed and recorded unique user and the large data platform cluster of electric power, disk I/O, CPU and memory source take situation;
(3) increase gradually to run and can run the Virtual User number of java applet, the response time of the current average unique user of observed and recorded and the network throughput of the large data platform cluster of electric power, disk I/O, CPU and memory source take situation;
(4) when the response time of average unique user is increased to 10 times of initial value, stop increasing Virtual User, this loading condition is the ultimate load of the large data platform of electric power, completes pressure test sequence.
2. a kind of method of testing based on the parallel processing of electric power large data platform mass data as claimed in claim 1, it is characterized in that, in described step (1), the operation of Single document can realize reading different size, dissimilar Single document, write, increase, delete, revising and search operation.
3. a kind of method of testing based on the parallel processing of electric power large data platform mass data as claimed in claim 2, is characterized in that,
Described read operation be from the distributed memory system of the large data platform of electric power by file download to Virtual User client;
Said write be operating as from Virtual User client by files passe to the distributed memory system of the large data platform of electric power;
Described increase operation, deletion action and retouching operation are process accordingly the file in the large data platform distributed memory system of electric power; With
Described search operation is search the file in the large data platform distributed memory system of electric power and result feedback is returned Virtual User client.
4. a kind of method of testing based on the parallel processing of electric power large data platform mass data as claimed in claim 1, it is characterized in that, in described step (2), Virtual User startup number needs takes into full account distributed factor, start the client of the large data platform nodes of electric power with the order of magnitude, ensure that the charge capacity of each client is less than its network throughput, and the load of each client of equilibrium allocation.
5. a kind of method of testing based on the parallel processing of electric power large data platform mass data as claimed in claim 1, it is characterized in that, when increasing Virtual User in described step (3), when the response time of the run java applet that average treatment unique user runs is less than 3 times of initial value, every 1/3 response time increases by 2 times that last Virtual User increases number; When being not less than 3 times of initial value, every 1/2 response time increases last Virtual User and increases number.
6. a kind of method of testing based on the parallel processing of electric power large data platform mass data as claimed in claim 1, it is characterized in that, the response time of the average unique user of described step (4) does not exceed 10 times of initial value, then return step (3).
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201510213675.9A CN104794007A (en) | 2015-04-29 | 2015-04-29 | Mass data parallel processing testing method based on electric large data platform |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201510213675.9A CN104794007A (en) | 2015-04-29 | 2015-04-29 | Mass data parallel processing testing method based on electric large data platform |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN104794007A true CN104794007A (en) | 2015-07-22 |
Family
ID=53558817
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201510213675.9A Pending CN104794007A (en) | 2015-04-29 | 2015-04-29 | Mass data parallel processing testing method based on electric large data platform |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN104794007A (en) |
Cited By (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN106445763A (en) * | 2016-09-09 | 2017-02-22 | 中国南方电网有限责任公司电网技术研究中心 | Power distribution and utilization big data platform test method and system |
| CN107229564A (en) * | 2016-03-25 | 2017-10-03 | 阿里巴巴集团控股有限公司 | A kind of pressure simulation method and device |
| CN107480015A (en) * | 2017-07-04 | 2017-12-15 | 网易（杭州）网络有限公司 | Load test approach, device, system, storage medium and pressure survey server |
Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102567203A (en) * | 2011-12-28 | 2012-07-11 | 创新科软件技术（深圳）有限公司 | Method and system for testing performance of distributed file system (DFS) |
| CN102609472A (en) * | 2012-01-18 | 2012-07-25 | 深圳市同洲视讯传媒有限公司 | Method and system for implementing performance test of distributed database system |
| CN104331477A (en) * | 2014-11-04 | 2015-02-04 | 哈尔滨工业大学 | Method for testing concurrency property of cloud platform based on federated research |
-
2015
- 2015-04-29 CN CN201510213675.9A patent/CN104794007A/en active Pending
Patent Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102567203A (en) * | 2011-12-28 | 2012-07-11 | 创新科软件技术（深圳）有限公司 | Method and system for testing performance of distributed file system (DFS) |
| CN102609472A (en) * | 2012-01-18 | 2012-07-25 | 深圳市同洲视讯传媒有限公司 | Method and system for implementing performance test of distributed database system |
| CN104331477A (en) * | 2014-11-04 | 2015-02-04 | 哈尔滨工业大学 | Method for testing concurrency property of cloud platform based on federated research |
Cited By (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN107229564A (en) * | 2016-03-25 | 2017-10-03 | 阿里巴巴集团控股有限公司 | A kind of pressure simulation method and device |
| CN107229564B (en) * | 2016-03-25 | 2020-12-11 | 阿里巴巴集团控股有限公司 | Pressure simulation method and device |
| CN106445763A (en) * | 2016-09-09 | 2017-02-22 | 中国南方电网有限责任公司电网技术研究中心 | Power distribution and utilization big data platform test method and system |
| CN107480015A (en) * | 2017-07-04 | 2017-12-15 | 网易（杭州）网络有限公司 | Load test approach, device, system, storage medium and pressure survey server |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN111324427B (en) | Task scheduling method and device based on DSP | |
| CN109684228B (en) | Performance test method, device, system and storage medium | |
| US20140215170A1 (en) | Block Compression in a Key/Value Store | |
| CN111813805A (en) | Data processing method and device | |
| CN107729570B (en) | Data migration method and device for server | |
| CN110069557B (en) | Data transmission method, device, equipment and storage medium | |
| CN111666131A (en) | Load balancing distribution method and device, computer equipment and storage medium | |
| CN114327857A (en) | Operation data processing method and device, computer equipment and storage medium | |
| CN104794007A (en) | Mass data parallel processing testing method based on electric large data platform | |
| CN110275681A (en) | A kind of date storage method and data-storage system | |
| CN103631873A (en) | Data compression method and memory system | |
| CN112685148A (en) | Asynchronous communication method and device of mass terminals, computer equipment and storage medium | |
| CN115098426B (en) | PCIE equipment management method, interface management module, PCIE system, equipment and medium | |
| CN111159265B (en) | ETL data migration method and system | |
| CN105049524A (en) | Hadhoop distributed file system (HDFS) based large-scale data set loading method | |
| CN105068875A (en) | Intelligence data processing method and apparatus | |
| CN109213745B (en) | Distributed file storage method, device, processor and storage medium | |
| CN112860412B (en) | Service data processing method and device, electronic equipment and storage medium | |
| CN115033551A (en) | Database migration method and device, electronic equipment and storage medium | |
| CN106302634A (en) | A kind of data migration method and system | |
| CN112380088A (en) | Test method and device and electronic equipment | |
| TWI706343B (en) | Sample playback data access method, device and computer equipment | |
| CN108259583B (en) | Data dynamic migration method and device | |
| CN113448739B (en) | Data processing method and device | |
| CN115576924A (en) | Data migration method |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| C10 | Entry into substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| CB02 | Change of applicant information |
Address after: 100192 Beijing city Haidian District Qinghe small Camp Road No. 15 Applicant after: China Electric Power Research Institute Applicant after: GLOBAL ENERGY INTERCONNECTION RESEARCH INSTITUTE Applicant after: State Grid Corporation of China Applicant after: State Grid Shandong Electric Power Company Address before: 100192 Beijing city Haidian District Qinghe small Camp Road No. 15 Applicant before: China Electric Power Research Institute Applicant before: State Grid Smart Grid Institute Applicant before: State Grid Corporation of China Applicant before: State Grid Shandong Electric Power Company |
|
| CB02 | Change of applicant information | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20150722 |
|
| RJ01 | Rejection of invention patent application after publication |