New! View global litigation for patent families

CN103297285A - Distributed cluster performance test system, method and device - Google Patents

Distributed cluster performance test system, method and device Download PDF

Info

Publication number
CN103297285A
CN103297285A CN 201210043849 CN201210043849A CN103297285A CN 103297285 A CN103297285 A CN 103297285A CN 201210043849 CN201210043849 CN 201210043849 CN 201210043849 A CN201210043849 A CN 201210043849A CN 103297285 A CN103297285 A CN 103297285A
Authority
CN
Grant status
Application
Patent type
Prior art keywords
pressure
module
cluster
request
tested
Prior art date
Application number
CN 201210043849
Other languages
Chinese (zh)
Other versions
CN103297285B (en )
Inventor
金敬亭
胡星
乔敏
Original Assignee
百度在线网络技术(北京)有限公司
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date

Links

Abstract

The invention provides a distributed cluster performance test system, method and device. The system comprises an upstream cluster, a tested module, a back end cluster, a network bridge, a pressure tool and a monitoring module. The network bridge is used for receiving a first pressure request sent by the upstream cluster, sending the first pressure request to the tested module, receiving a second pressure request generated by the tested module according to the first pressure request, forwarding the second pressure request to the back end cluster and storing the first pressure request and communication data packs returned by the back end cluster according to the second pressure request. The pressure tool is used for sending preset pressure to the tested module according to the first pressure request, and the tested module forwards the preset pressure to the network bridge, so that corresponding packs in the stored communication data packs are checked and are returned to tested module. The monitoring module and the tested module are placed on the same server. The monitoring module is used for monitoring performance data of the server and carrying out performance analysis according to the performance data. The network bridge is used for simulating a real cluster, performance test accuracy and test efficiency are improved, and cost is saved.

Description

分布式集群的性能测试系统、方法和装置 Test System distributed cluster, methods and apparatus

技术领域 FIELD

[0001] 本发明涉及网络搜索技术领域,尤其涉及一种分布式集群的性能测试系统、方法和装置。 [0001] The present invention relates to a network search technology, and more particularly relates to a distributed cluster performance test systems, methods and apparatus.

背景技术 Background technique

[0002] 对于类似网络搜索引擎的分布式集群系统,现有的性能测试在真实环境中进行,需要将大量机器资源搭建后端集群环境。 [0002] For similar Internet search engines distributed cluster system, the existing performance tests carried out in the real world, you need a lot of machine resources to build the back-end cluster environment. 这种方法存在的问题是:(1)浪费机器资源,人力维护成本高;(2)线下无法搭建和线上一样数据量规模的环境,数据量和线上相差很远; The problem with this method is that: (1) a waste of machine resources, high cost of maintenance manpower; (2) the amount of data lines can not be built and the size of the online environment, as data volume and line very far;

[3]线下搭建的后端集群的网络拓扑和线上不一致。 [3] The network topology inconsistencies and rear lines of the line structures clusters.

[0003] 而且,现有技术中需要根据不同的被测模块,准备不同的压力输入源、压力工具和监控工具,通用性差,需要较多的开发和维护成本。 [0003] Further, according to the prior art require different test modules, to prepare different pressure input source, pressure monitoring tools and tools, poor versatility, require more development and maintenance costs.

发明内容 SUMMARY

[0004] 本发明旨在至少解决上述技术问题之一。 [0004] The present invention aims to solve at least one of the technical problems described above.

[0005] 为此,本发明的第一个目的在于提出一种可以提高测试效率和测试性能准确性、通用性好、成本低的分布式集群的性能测试系统。 [0005] For this purpose, a first object of the present invention is to provide a system can improve the performance test performance test efficiency and test accuracy, versatility, low cost distributed cluster.

[0006] 本发明的第二个目的在于提出一种网桥。 [0006] A second object of the present invention is to provide a bridge.

[0007] 本发明的第三个目的在于提出一种分布式集群的性能测试方法。 [0007] A third object of the present invention is to propose a method for testing the performance of the distributed cluster.

[0008] 为了实现上述目的,根据本发明的第一方面实施例的分布式集群的性能测试系统包括:上游集群,所述上游集群用于发送第一压力请求;被测模块;后端集群,所述后端集群包括多个下游集群;网桥,所述网桥分别与所述上游集群、被测模块和后端集群相连,所述网桥用于接收所述第一压力请求并将所述第一压力请求发送至所述被测模块,接收所述被测模块根据所述第一压力请求生成的第二压力请求并将所述第二压力请求转发至所述多个下游集群,以及保存所述第一压力请求以及所述多个下游集群根据所述第二压力请求返回的通信数据包;压力工具,所述压力工具用于根据所述网桥保存的所述第一压力请求向所述被测模块发送预定压力,其中所述被测模块将所述预定压力转发至所述网桥,所述网桥根据所述预定压力在所述保存的通信数据包中 Performance Testing Distributed Cluster [0008] To achieve the above object, according to a first aspect of the embodiment of the present invention comprises: an upstream cluster, the cluster is configured to send a first pressure upstream of the request; module under test; the rear end of the cluster, the rear end of the cluster comprises a plurality of downstream clusters; bridge, said bridge respectively with the upstream cluster, the cluster is connected to the module under test and a rear end, said bridge for receiving the first request and the pressure said first pressure request to the module under test, said second pressure receiving module according to the first measured pressure generated request requests and forwards the request to the second pressure downstream of the plurality of clusters, and storing the first request and the pressure downstream of the plurality of cluster communication data packets in accordance with a request to return to the second pressure; pressure tool, the pressure means for the bridge request according to the stored pressure to the first the test module sends a predetermined pressure, wherein said test module forwarding the predetermined pressure to the bridge, the bridge in accordance with the predetermined pressure stored in the communication data packet 找所述预定压力相应的数据包并返回至所述被测模块;以及监控模块,所述监控模块与所述被测模块位于同一个服务器上,用于监控所述服务器的性能数据,并根据所述性能数据进行性能分析。 Get the predetermined pressure corresponding to the data packet and returns the module under test; and a monitoring module, the monitoring module and the module under test is located on the same server, for monitoring the performance data of the server, and in accordance with the performance data for performance analysis.

[0009] 根据本发明实施例的分布式集群的性能测试系统,网桥分别与上游集群、被测模块和多个下游集群(后端集群)相连,通过一个网桥可以真实模拟线上拓扑和数据量,无需搭建真实的集群环境即可对被测模块的性能进行测试,提高测试准确性和测试效率,节约成本。 [0009] The performance test distributed cluster system embodiment of the present invention, a bridge, a plurality of test modules, and a downstream cluster (Cluster rear end) is connected upstream of the cluster, respectively, through a bridge line can simulate real topology and the amount of data without the need to build a real cluster environment on the performance of the module under test to test, improve test accuracy and test efficiency and cost savings.

[0010] 为了实现上述目的,根据本发明的第二方面实施例的网桥包括:模式选择模块,所述模式选择模块用于接收用户输入的录制模式或测试模式;通信模块,所述通信模块用于在所述录制模式下接收上游集群发送的第一压力请求并将所述第一压力请求发送至被测模块,以及接收所述被测模块根据所述第一压力请求生成的第二压力请求并将所述第二压力请求转发至后端集群,以及接收所述后端集群根据所述第二压力请求返回的通信数据包;所述通信模块还用于在所述测试模式下接收所述被测模块转发的预定压力;录制模块,所述录制模块与所述通信模块相连,用于保存所述第一压力请求和所述返回的通信数据包;以及模拟模块,所述模拟模块分别与所述通信模块和所述录制模块相连,用于根据所述预定的压力在所述录制模块保存的通信数据包中查找 Bridge [0010] To achieve the above object, according to the second aspect of the embodiments of the present invention comprises: a mode selection module, the mode selector means for receiving a recording mode or a test mode input by a user; a communication module, the communication module means for receiving a first pressure upstream of the cluster transmitted in the first recording mode request and the request to the module under pressure, and the second pressure receiving module according to the first measured pressure generated request pressure request and forwarding the request to the second backend cluster, and the cluster receiving the rear end of the communication request packet to return data based on the second pressure; the communication module is further configured to receive the test mode said test module forwarding the predetermined pressure; recording module, said recording module is connected with the communication module, configured to store the first request and the pressure of the return communication data packets; and a simulation module, each of said simulation module and the communication module is connected to the recording module, for searching the stored recording data packet communication modules in accordance with the predetermined pressure 述预定的压力对应的返回数据包,其中所述通信模块还用于将所述预定压力对应的返回数据包发送至所述被测模块。 Said predetermined pressure corresponding to the return packet, wherein the communication module is further configured to return to the predetermined pressure corresponding to a data packet to the module under test.

[0011] 根据本发明实施例的网桥,通过与上游集群、被测模块和后端集群相连,可以真实地模拟线上拓扑和数量,无需搭建真实的集群环境即可对被测模块的性能进行测试,提高测试准确性和测试效率,节约成本。 [0011] The bridge embodiment of the present invention, the cluster is connected to the upstream, the rear end of the module under test and the cluster can realistically simulate topology and line number, without the need to build the real cluster environment performance of the module under test test, improve test accuracy and test efficiency and cost savings.

[0012] 为了实现上述目的,根据本发明的第三方面实施例的分布式集群的性能测试方法,包括以下步骤:(a)在录制模式下,网桥接收上游集群发送的第一压力请求,并将所述第一压力请求发送至被测模块,以及保存所述第一压力请求;(b)所述被测模块接收所述第一压力请求,并根据所述第一压力请求生成第二压力请求,以及将所述第二压力请求发送至所述网桥;(c)所述网桥接收所述第二压力请求并将所述第二压力请求发送至后端集群,其中所述后端集群包括多个下游集群;(d)所述多个下游集群根据所述第二压力请求返回相应的数据包至所述网桥;(e)所述网桥接收所述并保存所述返回的数据包;(f)在测试模式下,通过压力工具根据所述网桥保存的第一压力请求发送预定压力至所述被测模块;(g)所述被测模块将所述预定压力转发至所述网桥,并接收所述 [0012] To achieve the above object, according to a third aspect of the present invention, performance test method for distributed cluster of embodiment, comprising the steps of: (a) in the recording mode, receiving a first pressure upstream of the cluster bridge request sent, and the first pressure request to the module under test, and saving the first pressure request; (b) the test module receives the first pressure request and generating a second request from the first pressure (c) after receiving the bridge request and the second pressure of the second pressure request to the rear end of the cluster, wherein said; pressure request, and the second pressure request to the bridge a downstream end cluster includes a plurality of clusters; (d) downstream of the plurality of clusters request to return the corresponding data packet to the bridge according to the second pressure; (e) the said bridge receives and stores the return a data packet; (f) in the test mode, by the pressure means the pressure of the bridge request according to a first predetermined pressure stored in the send to the module under test; (G) the test module forwarding the predetermined pressure to the bridge, and receiving the 桥根据所述预定压力在所述保存的通信数据包中查找到的相应的数据包;(h)在测试过程中,监控所述被测模块所在的服务器的性能数据,并根据所述性能数据进行性能分析。 The bridge to find the predetermined pressure stored in said communication packets into a corresponding data packet; (H) during the test, the monitored performance data server where the module under test, and based on the performance data performance analysis.

[0013] 根据本发明实施例的分布式集群的性能测试方法,通过网桥抓取上游集群、被测模块以及后端集群之间的通信数据包,从而通过一个网桥可以真实模拟线上拓扑和数据量,无需搭建真实的集群环境即可对被测模块进行性能测试,提高测试准确性和测试效率,节约成本。 [0013] The performance test method for distributed clusters embodiment of the present invention, by gripping the upstream bridge cluster, communication data packets between the rear end of the module under test and the cluster, which can simulate the real topology through a bridge line and the amount of data without the need to build a real cluster environment to test modules for performance testing, improve test accuracy and test efficiency and cost savings.

[0014] 本发明附加的方面和优点将在下面的描述中部分给出,部分将从下面的描述中变得明显,或通过本发明的实践了解到。 [0014] This additional aspects and advantages of the invention will be set forth in part in the description which follows, from the following description in part be apparent from, or learned by practice of the present invention.

附图说明 BRIEF DESCRIPTION

[0015] 本发明上述的和/或附加的方面和优点从下面结合附图对实施例的描述中将变得明显和容易理解,其中, The above-described invention and / or additional aspects and advantages of the drawings from the following description of embodiments will become apparent and more readily appreciated [0015], wherein,

[0016] 图1是根据本发明一个实施例的分布式集群的性能测试系统的录制模式结构框图; [0016] FIG. 1 is a block diagram according to the recording mode of the distributed cluster system performance tests according to one embodiment of the present invention;

[0017] 图2是根据本发明一个实施例的分布式集群的性能测试系统的测试模式的结构框图; [0017] FIG. 2 is a block diagram according to the test pattern distributed cluster system performance tests according to one embodiment of the present invention;

[0018] 图3是根据本发明一个实施例的监控模块的结构框图; [0018] FIG. 3 is a block diagram showing a monitoring module according to an embodiment of the present invention;

[0019] 图4是根据本发明另一个实施例的监控模块的结构框图; [0019] FIG. 4 is a block diagram of the monitor module according to another embodiment of the present invention;

[0020] 图5是根据本发明一个实施例的网桥的结构框图;以及 [0020] FIG. 5 is a block diagram of the embodiment of a bridge according to the embodiment of the present invention; and

[0021] 图6是根据本发明一个实施例的分布式集群的性能测试方法的流程图。 [0021] FIG. 6 is a flowchart of the performance test method of the embodiment of a distributed cluster embodiment of the present invention. 具体实施方式 detailed description

[0022] 下面详细描述本发明的实施例,所述实施例的示例在附图中示出,其中自始至终相同或类似的标号表示相同或类似的元件或具有相同或类似功能的元件。 [0022] Example embodiments of the present invention is described in detail below, exemplary embodiments of the embodiment shown in the accompanying drawings, wherein same or similar reference numerals designate the same or similar elements or elements having the same or similar functions. 下面通过参考附图描述的实施例是示例性的,仅用于解释本发明,而不能理解为对本发明的限制。 By following with reference to the embodiments described are exemplary only for explaining the present invention and should not be construed as limiting the present invention. 相反,本发明的实施例包括落入所附加权利要求书的精神和内涵范围内的所有变化、修改和等同物。 In contrast, embodiments of the present invention includes all variations that fall within the appended claims the spirit and terms, modifications and equivalents thereof.

[0023] 在本发明的描述中,需要理解的是,术语“第一”、“第二”等仅用于描述目的,而不能理解为指示或暗示相对重要性。 [0023] In the description of the present invention, it is to be understood that the terms "first," "second," and the like for illustrative purposes only, and not intended to indicate or imply relative importance. 在本发明的描述中,需要说明的是,除非另有明确的规定和限定,术语“相连”、“连接”应做广义理解,例如,可以是固定连接,也可以是可拆卸连接,或一体地连接;可以是机械连接,也可以是电连接;可以是直接相连,也可以通过中间媒介间接相连。 In the description of the present invention, it should be noted that, unless otherwise expressly specified or limited, the term "coupled", "connected" are to be broadly understood, for example, may be a fixed connection, the connection may be detachable or integrally connected; may be a mechanical connector, it may be electrically connected; may be directly connected, can also be connected indirectly through an intermediary. 对于本领域的普通技术人员而言,可以具体情况理解上述术语在本发明中的具体含义。 Those of ordinary skill in the art, be appreciated that the specific circumstances of the specific meanings in the present invention. 此外,在本发明的描述中,除非另有说明,“多个”的含义是两个或两个以上。 Further, in the description of the present invention, unless otherwise specified, the meaning of "more" is two or more.

[0024] 流程图中或在此以其他方式描述的任何过程或方法描述可以被理解为,表示包括一个或更多个用于实现特定逻辑功能或过程的步骤的可执行指令的代码的模块、片段或部分,并且本发明的优选实施方式的范围包括另外的实现,其中可以不按所示出或讨论的顺序,包括根据所涉及的功能按基本同时的方式或按相反的顺序,来执行功能,这应被本发明的实施例所属技术领域的技术人员所理解。 [0024] In the flowchart in any process or method or otherwise described in this description may be understood as representing modules comprises one or more steps for implementing specific logical functions or processes executable instructions, fragment or portion, and the scope of the preferred embodiment of the present invention includes other implementations, which may be shown or discussed in order not press, comprising a substantially simultaneous manner or in reverse order, depending upon the functionality to perform the functions involved it should be understood that embodiments skilled in the art of the present invention.

[0025] 下面参考说明书附图描述根据本发明实施例的分布式集群的可靠性的测试系统。 [0025] The following description with reference to the accompanying drawings distributed cluster reliability test system according to an embodiment of the present invention from the description.

[0026] 一种分布式集群的性能测试系统,包括:上游集群,用于发送第一压力请求;被测模块;后端集群,包括多个下游集群;网桥,分别与上游集群、被测模块和后端集群相连,用于接收第一压力请求并将第一压力请求发送至被测模块,接收被测模块根据第一压力请求生成的第二压力请求并将第二压力请求转发至多个下游集群,以及保存第一压力请求以及多个下游集群根据第二压力请求返回的通信数据包;压力工具,用于根据网桥保存的第一压力请求向被测模块发送预定压力,其中被测模块将预定压力转发至网桥,网桥根据预定压力在保存的通信数据包中查找相应的数据包并返回至被测模块;以及监控模块,与被测模块位于同一个服务器上,用于监控该服务器的性能数据,并根据该性能数据进行性能分析。 [0026] A distributed cluster performance testing system, comprising: an upstream cluster, a request for transmitting a first pressure; module under test; the rear end of the cluster, the cluster comprising a plurality of downstream; bridge, respectively upstream cluster, measured and a rear end connected to the cluster module, receiving a request for a first pressure and a first pressure request to the module under test, the test module receives the request and a second pressure in the first pressure according to a second request of forwarding the request to the plurality of generated communication data packets back downstream of the cluster, and storing the first request and the pressure downstream of the plurality of clusters according to a second pressure request; pressure means, according to a first bridge request stored pressure a predetermined pressure to the module under test, wherein the test the predetermined pressure is forwarded to the module bridge, the bridge in accordance with a predetermined pressure to find the communication data stored in the corresponding data packet and returns the packet to the module under test; and a monitoring module, and the module under test on the same server, for monitoring performance data of the server, and performance analysis based on the performance data.

[0027] 图1为本发明一个实施例的分布式集群的性能测试系统的录制过程的结构框图。 [0027] FIG. 1 is a block diagram showing the recording process performance test system according to one embodiment of a distributed cluster of the present invention. 如图1所示,根据本发明实施例的分布式集群的性能测试系统在录制过程中包括上游集群110、被测模块120、网桥130和后端集群140,其中,后端集群140包括多个下游集群141,网桥130分别与上游集群110、被测模块120和多个下游集群141相连。 As shown, the system performance test distributed cluster embodiment of the present invention comprises an upstream cluster 110, the module under test 120, the bridge 130, and the rear end of the recording cluster 140, wherein a rear end 140 includes a plurality of the cluster 1 downstream clusters 141, 130, respectively upstream bridge cluster 110, 120 is connected to the module under test and a plurality of clusters 141 downstream.

[0028] 具体地,上游集群110用于发送第一压力请求。 [0028] Specifically, the upstream pressure of the cluster 110 is configured to send a first request. 网桥130接收该第一压力请求并将该第一压力请求(二进制包)发送至被测模块120。 Pressure bridge 130 receives the first request and the first pressure request (binary packet) is sent to the module under test 120. 被测模块120接收该第一压力请求,并根据该第一压力请求生成第二压力请求,例如第一压力请求为搜索“百度、专利、审查意见”,生成的第二压力请求为搜索“专利、审查意见”。 A first test module 120 receives the request pressure and a second pressure request generation request according to the first pressure, a first pressure e.g. search request "Baidu, patents, review comments," a second pressure generated request is for "Patent review comments. " 然后,被测模块120将第二压力请求发送至多个下游集群141。 Then, the second test module 120 requests to a plurality of downstream pressure 141 clusters. 多个下游集群141根据第二压力请求返回相应的通信数据包至网桥130。 A plurality of clusters 141 downstream pressure according to a second request returns a communication packet corresponding to the bridge 130. 在上述过程中,网桥130保存上游集群110发送的第一压力请求以及多个下游集群141返回的通信数据包,这个过程即为数据的录制过程。 In the above process, the bridge 130 to save a first pressure upstream of the cluster 110 sends the request and the plurality of downstream communication packet returned from the cluster 141, the process is the recording process data. [0029] 图2为本发明一个实施例的分布式集群的性能测试系统的测试过程的结构框图。 [0029] FIG. 2 is a block diagram of the testing process Testing System distributed cluster of one embodiment of the present invention. 如图2所示,在测试过程中该分布式集群的性能测试系统仅使用被测模块120和网桥130,此外还包括压力工具150和监控模块160。 2, during the test performance of the test system using distributed cluster only the module under test 120 and bridge 130, also includes a pressure monitoring module 160 and the tool 150. 利用录制过程保存下来的数据包进行数据回放,模拟被测模块整个后端集群。 Using the stored recording data packets down the data playback, the entire rear end of the module under test simulated clusters.

[0030] 具体地,首先,压力工具150根据录制过程保存下来的第一压力请求数据包向被测模块120发送预定压力。 [0030] Specifically, first, the pressure tool 150 according to a first request packet recording storage pressure down a predetermined pressure transmitted to the module under test 120. 被测模块120再将该预定压力转发至网桥130,网桥130根据该预定压力在录制过程中保存下来的通信数据包中查找到该预定压力对应的数据包并将其返回至被测模块120。 The test module 120 then forwards the predetermined pressure to the bridge 130, the bridge module under test 130 based on the predetermined pressure is preserved during the recording process communication packet to find the data packets corresponding to a predetermined pressure and returned to the 120. 监控模块160与被测模块120布置在同一个服务器170上,用于监控该服务器170的性能数据,并根据该性能数据进行性能分析。 The monitoring module 160 and the module under test 120 is disposed on the same server 170, to monitor the performance of the data server 170, and performance analysis based on the performance data.

[0031] 应理解,对于压力工具150来说,不同模块的通信协议不一样,进行保存下来的二进制数据包的结构不一样。 [0031] It should be understood, for the pressure means 150 is the communication protocol different modules are not the same, structural preserved binary data packet is not the same. 压力工具150要根据模块特点去包含很多请求包的二进制数据包里找出每一个请求包。 Pressure to the tool 150 to the request packet contains a lot of binary data package module according to the characteristics of each request packet to find. 由此,在本发明的一个实施例中,在录制数据包时,每次通信包都具有如下的记录格式: Thus, in one embodiment of the present invention, when recording packet, each communication packet has the following recording format:

[0032] [0032]

魔鬼数字Dump Id I Key 版本号保留字段&长度 n,誦AU自政 Dump Id I Key magic number version number & reserved field length n, self-governance AU recite

4字节8字节8字节2字节2字节4字节 Dump的数据包体 4 bytes 8 bytes 8 bytes 2 bytes 2 bytes 4 bytes of the packet body Dump

[0033] 其中,魔鬼数字用于标识一个协议包的头,避免二进制数据错乱;Dump ID用于标识所保存的通信数据包的编号;key标识为上文描述的保证回放真实性存在的key ;版本号用于区别不同模块;保留字段用于后续扩展使用,例如存放dump的时间戳,方便后续按照时间戳回放;包长度用于标识存取的数据包的实际长度;Dump的数据包体用于实际存放通信数据包,如压力请求数据包或返回的通信数据包。 [0033] wherein, the magic number for identifying a first protocol packet, to avoid confusion binary data; Dump ID number for identifying the stored communication packets; key identification key to ensure the presence of the authenticity of the playback as described above; the version number used to distinguish between different modules; field is reserved for future extensions, for example to store the dump time stamp, the time stamp according to facilitate subsequent playback; packet length for identifying the actual length of the access packets; packet body by dump storing actual communication data packet, such as a pressure or a request packet communication packets returned.

[0034] 由此,压力工具150根据上述固定的记录格式进行解析,可以很容易的解析每一条记录,进而读取每次请求包,然后通过压力控制模块发送给被测模块120。 [0034] Accordingly, the pressure fixing tool 150 according to the above-described recording format parsing, can easily parse each record, and thus each read request packet, and then transmits through a pressure control module 120 to the module under test.

[0035] 此外,在本发明的一个实施例中,网桥按照key标识&value对的形式建立词典来保存第一压力请求和返回的通信数据包,其中value为返回的通信数据包,key标识即为上述的通信包中包含的key标识。 [0035] Further, in one embodiment of the present invention, the bridges establish dictionary identification key in the form & value of the request to save a first pressure and return communication packet, which is the return value of the communication packet data, i.e., identification key identified as a key above communication included in the package. 由此,通过解析压力请求包或者对压力请求包处理后做签名而得到一个key,然后通过这个唯一的key在词典中查找捆绑的返回数据包,以此达到流量回放和线上运行流量一致。 Thus, by analyzing the request packet to the pressure or pressure process request packet do get a signature key, and then look for return packet bundle in a dictionary by this unique key, in order to achieve consistent flow rate and playback operation flow line.

[0036] 图3为本发明一个实施例的监控模块的结构框图。 [0036] FIG 3 a block diagram showing a monitoring module according to the present embodiment of the invention. 如图3所示,监控模块160包括总控单元161和多个分控单元162。 As shown in FIG. 3, the monitoring module 160 includes a master control unit 161 and a plurality of sub-control unit 162. 总控单元161用于配置监控项、控制监控的启动和停止以及根据监控到的性能数据进行性能分析。 The master control unit 161 configured for monitoring items, start and stop control and monitoring performance analysis according to the monitored performance data. 每个分控单元162均与总控单元161相连,用于根据总控单元161的控制周期性地收集被测模块120的性能数据,并对性能数据进行格式化。 Each sub-control unit 162 are connected to the master control unit 161, for collecting performance data of the module under test 120 based on the control of the master control unit 161 periodically, and performance data format. 在本发明的一个实施例中,总控单元161为一台总控机器,多个分控单元162为多台性能测试机器。 In one embodiment of the present invention, the master control unit 161 is a master control machine, a plurality of sub-control unit 162 is a plurality of machine performance testing.

[0037] 具体地,分控单元162包括如下的功能函数: [0037] Specifically, the sub-control unit 162 includes a function of a function:

[0038] (l)monitor_vmstat (fiIePath, period) [0038] (l) monitor_vmstat (fiIePath, period)

[0039] 功能:周期性地收集vmstat数据; [0039] Function: vmstat periodically collect data;

[0040] (2)monitor_module (fiIePath, period, module_name)[0041] 功能:周期性地收集某个模块的cpu、内存、文件句柄/socket句柄等占用情况; [0040] (2) monitor_module (fiIePath, period, module_name) [0041] Function: a module collected periodically cpu, memory, file handles / socket handle other occupation;

[0042] (3)monitor_iostat (filePath, period, module_name) [0042] (3) monitor_iostat (filePath, period, module_name)

[0043] 功能:周期性地收集iostat数据; [0043] Function: iostat periodically collect data;

[0044] (4)monitor_net (filePath, period, module_name) [0044] (4) monitor_net (filePath, period, module_name)

[0045] 功能:周期性地收集网卡数据,采用sar命令; [0045] Function: periodically collect data card using sar command;

[0046] (5) format_vmstat (filePathln, tablePath) [0046] (5) format_vmstat (filePathln, tablePath)

[0047] 功能:对vmstat数据进行格式化; [0047] Function: vmstat data format;

[0048] (6)format_module(filePathln, tablePath); [0048] (6) format_module (filePathln, tablePath);

[0049] 功能:对模块的性能数据进行格式化; [0049] Function: performance data formatting module;

[0050] (7) format_iostat (filePathln, tablePath) [0050] (7) format_iostat (filePathln, tablePath)

[0051] 功能:对模块的iostat数据进行格式化; [0051] Function: iostat data module format;

[0052] (8) format_net (filePathln, tablePath) [0052] (8) format_net (filePathln, tablePath)

[0053] 功能:对网络数据进行格式化。 [0053] Function: the network data format.

[0054] 总控单元161包括如下的功能函数: [0054] The master control unit 161 includes the following performance function:

[0055] (I)install_trust(machineList) [0055] (I) install_trust (machineList)

[0056] 功能:建立总控单元和各个分控单元之间的双向信任关系,方便工具进行ssh和SCP操作; [0056] Function: establish two-way trust relationship between the master control unit and the respective sub-control unit, and a convenient tool for ssh SCP operation;

[0057] (2) deploy (machineList, dirList) [0057] (2) deploy (machineList, dirList)

[0058] 功能:将监控角本部署到各个分控单元上; [0058] Function: This deployment monitor angle to each sub-control unit;

[0059] (3) start_monitor (machineList, dirList) [0059] (3) start_monitor (machineList, dirList)

[0060] 功能:批量启动各个分控单元; [0060] Function: Start Bulk respective sub-control unit;

[0061] (4) stop_monitor (machineList, dirList) [0061] (4) stop_monitor (machineList, dirList)

[0062] 功能:批量停止各个分控单元; [0062] Function: Stop the respective sub-volume control unit;

[0063] (5)get_machine_info(machineList) [0063] (5) get_machine_info (machineList)

[0064] 功能:获取各个分控单元上的机器物理参数; [0064] Function: Get the machine physical parameters on each sub-control unit;

[0065] (6)get_average(tablePathln, secsj tcibIePcithOut) [0065] (6) get_average (tablePathln, secsj tcibIePcithOut)

[0066] 功能:对格式化后的数据求其在不同时间段内的平均值; [0066] Function: find the average of the different time formatted data;

[0067] (7) get_min (tablePathln, secs, tablePathOut) [0067] (7) get_min (tablePathln, secs, tablePathOut)

[0068] 功能:对格式化后的数据求其在不同时间段内的最小值; [0068] Function: seeking the minimum value in different time periods of the formatted data;

[0069] (8) get_max (tablePathln, secs, tablePathOut) [0069] (8) get_max (tablePathln, secs, tablePathOut)

[0070] 功能:对格式化后的数据求其在不同时间段内的最大值; [0070] Function: seeking the maximum in different time periods of the formatted data;

[0071] (9) get_rate (tablePathln, secs, tablePathOut) [0071] (9) get_rate (tablePathln, secs, tablePathOut)

[0072] 功能:对格式化后的数据求某一列数据在不同区间的比例。 [0072] Function: seeking the ratio of a column of data in different sections of the formatted data.

[0073]图4为本发明另一个实施例的监控模块的结构框图。 [0073] FIG. 4 PCU configuration block diagram of another embodiment of the present invention. 如图4所示,在图3所示的实施例的基础上,该监控模块160还包括日志处理单元163。 4, on the basis of the embodiment shown in FIG. 3, the monitor module 160 the processing unit 163 further includes a log. 日志处理单元163与总控单元161相连,用于对获取到的性能数据进行解析和统计。 A log processing unit 163 is connected to the master control unit 161, configured to obtain performance data parsed and statistics. 具体地可包括如下的功能函数: Specifically, the function may include a function:

[0074] (I) get_average_press(logPath, secs) [0074] (I) get_average_press (logPath, secs)

[0075] 功能:获取不同时间段内压力的平均值; [0075] Function: Get the average pressure over different time periods;

[0076] (2) get_max_press (logPath, secs)[0077] 功能:获取不同时间段内压力的最大值; [0076] (2) get_max_press (logPath, secs) [0077] Function: Get the maximum pressure of the different time periods;

[0078] (3) get_min_press (logPath, secs) [0078] (3) get_min_press (logPath, secs)

[0079] 功能:获取不同时间段内压力的最小值; [0079] Function: Get different time periods of the minimum pressure;

[0080] (4)get_average_parm(logPath, secs, key) [0080] (4) get_average_parm (logPath, secs, key)

[0081] 功能:获取不同时间段内某个参数的平均值; [0081] Function: obtaining an average value of a parameter over different time periods;

[0082] (5) get_max_parm(logPath, secs, key) [0082] (5) get_max_parm (logPath, secs, key)

[0083] 功能:获取不同时间段内某个参数的最大值; [0083] Function: Get the maximum value of a parameter over different time periods;

[0084] (6)get_min_parm(logPath, secs, key) [0084] (6) get_min_parm (logPath, secs, key)

[0085] 功能:获取不同时间段内某个参数的最小值; [0085] Function: obtaining the minimum value of a parameter over different time periods;

[0086] (7) get_rate_parm(logPath, secs, key) [0086] (7) get_rate_parm (logPath, secs, key)

[0087] 功能:获取某个参数在不同区间的比例。 [0087] Function: Gets a parameter in the proportion of different intervals.

[0088] 应理解,上述总控单元、分控单元以及日志处理单元的功能函数仅为示意性的例子。 [0088] It should be understood, illustrative examples of the above-described functions of the total function of the control unit, a log processing unit, and a sub-control unit only. 本领域的技术人员可根据实际需要配置不同的监控项,也就是使用不同的功能函数,这些修改和变化均应包含在本发明的保护范围内。 Those skilled in the art may be configured according to the actual requirements monitored item, which is a function of different features, such modifications and variations be included within the scope of the present invention.

[0089] 根据本发明实施例的分布式集群的性能测试系统,网桥分别与上游集群、被测模块和多个下游集群(后端集群)相连,通过一个网桥可以真实模拟线上拓扑和数据量,无需搭建真实的集群环境即可对被测模块的性能进行测试,提高测试准确性和测试效率,节约成本。 [0089] The performance test distributed cluster system embodiment of the present invention, a bridge, a plurality of test modules, and a downstream cluster (Cluster rear end) is connected upstream of the cluster, respectively, through a bridge line can simulate real topology and the amount of data without the need to build a real cluster environment on the performance of the module under test to test, improve test accuracy and test efficiency and cost savings.

[0090] 下面参考说明书附图描述根据本发明实施例的网桥。 [0090] The following description with reference to the accompanying drawings bridge according to an embodiment of the present invention.

[0091] 图5为本发明一个实施例的网桥的结构框图。 [0091] FIG. 5 is a block diagram showing an embodiment of a bridge according to the present invention. 如图5所示,根据本发明实施例的网桥包括模式选择模块310、通信模块320、录制模块330和模拟模块340。 5, the selection module 310, communication module 320 according to an embodiment of the present invention, the bridge includes a mode, the recording module 330 and simulation module 340.

[0092] 具体地,模式选择模块310用于接收用户输入的录制模式或测试模式。 [0092] Specifically, the mode selection module 310 for recording mode or a test mode to receive user input. 在录制模式下,通信模块320接收上游集群发送的第一压力请求并将该第一压力请求发送至被测模块,以及接收被测模块根据第一压力请求生成的第二压力请求并将该第二压力请求转发至后端集群,以及接收后端集群根据第二压力请求返回的通信数据包;录制模块330与通信模块320相连,保存第一压力请求和返回的通信数据包。 In the recording mode, the communication module 320 receives a first pressure upstream of the cluster and the request sent to the first pressure request module under test, and a second pressure receiving request generating module under test according to a first request and the second pressure forwarding the request to the rear end of the second pressure cluster, and a rear end received data packet according to a second trunked communication request to return pressure; the recording module 330 and the communication module 320 is connected to the first storage request and the return pressure communication packet. 在测试模式下,通信模块320接收被测模块转发的预定压力。 In the test mode, the communication module 320 receives a predetermined pressure test module forwarding. 模拟模块340分别与通信模块320和录制模块330相连,根据预定压力在录制模块330中保存的通信数据包中查找到该预定压力对应的返回数据包。 Simulation module 340 are connected with the communication module 330 and recording module 320, in accordance with a predetermined pressure to find recording module 330 stored in the communication data packet to a predetermined pressure corresponding to the return packet. 然后,通信模块320将该数据包发送至被测模块。 Then, the communication module 320 transmits the data packet to the module under test.

[0093] 应理解,不同模块的通信协议不一样,进行保存下来的二进制数据包的结构不一样。 [0093] It should be understood, the communication protocol different modules are not the same, for the saved binary data packet structure is different. 在进行性能测试时,要根据模块特点去包含很多请求包的二进制数据包里找出每一个请求包。 During performance testing, to contain a lot of binary data package to the request packet to identify each packet in accordance with a request module characteristics. 由此,在本发明的一个实施例中,在录制数据包时,每次通信包都具有如下的记录格式: Thus, in one embodiment of the present invention, when recording packet, each communication packet has the following recording format:

[0094] [0094]

魔鬼数字Dump IdI Key 版本号保留字段&长度4字节8字节8字节2字节2字节4字节 Dump IdI Key magic number version number & reserved field length 8 bytes 8 bytes 4 bytes 2 bytes 2 bytes 4 bytes

[0095] 其中,魔鬼数字用于标识一个协议包的头,避免二进制数据错乱;Dump ID用于标识所保存的通信数据包的编号;key标识为上文描述的保证回放真实性存在的key ;版本号用于区别不同模块;保留字段用于后续扩展使用,例如存放dump的时间戳,方便后续按照时间戳回放;包长度用于标识存取的数据包的实际长度;Dump的数据包体用于实际存放通信数据包,如压力请求数据包或返回的通信数据包。 [0095] wherein, the magic number for identifying a first protocol packet, to avoid confusion binary data; Dump ID number for identifying the stored communication packets; key identification key to ensure the presence of the authenticity of the playback as described above; the version number used to distinguish between different modules; field is reserved for future extensions, for example to store the dump time stamp, the time stamp according to facilitate subsequent playback; packet length for identifying the actual length of the access packets; packet body by dump storing actual communication data packet, such as a pressure or a request packet communication packets returned. 由此,在进行性能测试时,根据上述固定的记录格式进行解析,可以很容易的解析每一条记录,进而读取每次请求包,然后发送预定压力给被测模块。 Accordingly, during the performance test, carried out according to the fixed recording format parsing, can easily parse each record, and thus each read request packet, and then sends the module under test to a predetermined pressure.

[0096] 此外,在本发明的一个实施例中,模拟模块240按照key标识&value对的形式建立词典来保存第一压力请求和返回的通信数据包,其中value为返回的通信数据包,key标识即为上述的通信包中包含的key标识。 [0096] Further, in one embodiment of the present invention, the simulation module 240 establish a dictionary in the form of key identifier & value pairs to store a first pressure request and return communication data packet, wherein the value for the return communication data packet, key identifier that is, a communication identifier of the key included in the package. 由此,可以先通过解析压力请求包或者对压力请求包处理后做签名而得到一个key,然后模拟模块240通过这个唯一的key在词典中查找捆绑的返回数据包,以此达到流量回放和线上运行流量一致。 Thereby, it is possible to request packets by parsing a pressure or pressure to obtain a key signature after a request to do packet processing and analog blocks 240 through this unique key look in the dictionary returns data packet bundle, in order to achieve flow line and playback agreement on the run flow.

[0097] 根据本发明实施例的网桥,通过与上游集群、被测模块和后端集群相连,可以真实地模拟线上拓扑和数量,无需搭建真实的集群环境即可对被测模块的性能进行测试,提高测试准确性和测试效率,节约成本。 [0097] The bridge embodiment of the present invention, the cluster is connected to the upstream, the rear end of the module under test and the cluster can realistically simulate topology and line number, without the need to build the real cluster environment performance of the module under test test, improve test accuracy and test efficiency and cost savings.

`[0098] 下面参考说明书附图描述根据本发明实施例的分布式集群的性能测试方法。 `[0098] The following description with reference to the accompanying drawings distributed cluster performance test method according to an embodiment of the present invention.

[0099] 图6为本发明一个实施例的分布式集群的性能测试方法的流程图。 [0099] Test Method 6 is a flowchart of the performance of a distributed cluster according to the present embodiment of the invention.

[0100] 如图6所示,根据本发明实施例的分布式集群的性能测试方法,包括下述步骤。 [0100] As shown in FIG 6, the performance test method for distributed clusters embodiment of the present invention, comprises the following steps.

[0101] 步骤S101,在录制模式下,网桥接收上游集群发送的第一压力请求,并将该第一压力请求发送至被测模块,以及保存该第一压力请求。 [0101] step S101, and in the recording mode, a first pressure upstream of the cluster bridge receives transmission request, and the first pressure request to the module under test, and saving the first pressure request.

[0102] 步骤S102,被测模块接收第一压力请求,并根据该第一压力请求生成第二压力请求,以及将第二压力请求发送至网桥。 [0102] Step S102, the test module receives a first pressure request and generating a second pressure request according to the request of the first pressure and the second pressure request to the bridge.

[0103] 例如,第一压力请求为搜索“百度、专利、审查意见”,生成的第二压力请求为搜索“专利、审查意见”。 [0103] For example, a first pressure request is a search for "Baidu, patents, review comments," a second pressure generated request is for "Patent review comments." ` `

[0104] 步骤S103,网桥接收第二压力请求并将第二压力请求发送至后端集群,其中后端集群包括多个下游集群。 [0104] Step S103, the bridge receives a second pressure request and a second request to the rear pressure cluster, wherein the cluster comprises a plurality of downstream rear end clusters.

[0105] 步骤S104,多个下游集群根据第二压力请求返回相应的数据包至网桥。 [0105] step S104, a request to return a plurality of downstream clusters corresponding data packet in accordance with a second pressure to the bridge.

[0106] 步骤S105,网桥接收并保存多个下游集群返回的数据包。 [0106] Step S105, the bridge receives the packet and stores a plurality of clusters returned downstream.

[0107] 上述步骤SlOl至步骤S105即为数据的录制过程。 [0107] The steps SlOl to step S105 is the data recording process. 通过录制过程保存的数据包,即可模拟被测模块整个后端集群,实现后续的被测模块的性能测试。 By recording saved data packet, the entire rear end of the module under test can be simulated cluster, subsequent testing for performance of the module under test.

[0108] 步骤S106,在测试模式下,通过压力工具根据网桥保存的第一压力请求发送预定压力至被测模块。 [0108] step S106, in the test mode, predetermined by pressure means pressure request to the module under test according to a first stored pressure bridge.

[0109] 应理解,对于压力工具来说,不同模块的通信协议不一样,进行保存下来的二进制数据包的结构不一样。 [0109] It should be understood, for the pressure means, the communication protocol different modules are not the same, structural preserved binary data packet is not the same. 压力工具要根据模块特点去包含很多请求包的二进制数据包里找出每一个请求包。 Pressure to the tool to the request packet contains a lot of binary data package module according to the characteristics of each request packet to find. 由此,在本发明的一个实施例中,在录制数据包时,每次通信包都具有如下的记录格式: Thus, in one embodiment of the present invention, when recording packet, each communication packet has the following recording format:

[0110] [0110]

Figure CN103297285AD00121

[0111] 其中,魔鬼数字用于标识一个协议包的头,避免二进制数据错乱;Dump ID用于标识所保存的通信数据包的编号;key标识为上文描述的保证回放真实性存在的key ;版本号用于区别不同模块;保留字段用于后续扩展使用,例如存放dump的时间戳,方便后续按照时间戳回放;包长度用于标识存取的数据包的实际长度;Dump的数据包体用于实际存放通信数据包,如压力请求数据包或返回的通信数据包。 [0111] wherein, the magic number for identifying a first protocol packet, to avoid confusion binary data; Dump ID number for identifying the stored communication packets; key identification key to ensure the presence of the authenticity of the playback as described above; the version number used to distinguish between different modules; field is reserved for future extensions, for example to store the dump time stamp, the time stamp according to facilitate subsequent playback; packet length for identifying the actual length of the access packets; packet body by dump storing actual communication data packet, such as a pressure or a request packet communication packets returned. 由此,压力工具根据上述固定的记录格式进行解析,可以很容易的解析每一条记录,进而读取每次请求包,然后通过压力发送控制模块发送预定压力给被测模块。 Accordingly, the pressure fixing tool according to the above recording format parsing, can easily parse each record, and thus each read request packet, and then sends the module under test to a predetermined pressure by the pressure transmitting control module.

[0112] 步骤S107,被测模块将预定压力转发至网桥。 [0112] step S107, the module under test to a predetermined pressure to the bridge forwards.

[0113] 步骤S108,网桥根据预定压力在所保存的通信数据包中查找到相应的返回数据包。 [0113] Step S108, to find a bridge according to a predetermined pressure stored in the communication data packet corresponding to the return packet.

[0114] 在本发明的一个实施例中,按照key标识&value对的形式建立词典来保存第一压力请求和返回的通信数据包,其中value为返回的通信数据包,key标识即为上述的通信包中包含的key标识。 [0114] In one embodiment of the present invention, the key identification in the form of establishing the dictionary & value of pressure to a first storage request and return communication packet, which is the return value of the communication packet data, the identification key is the above-mentioned communication identification key included in the package. 由此,可以先通过解析压力请求包或者对压力请求包处理后做签名而得到一个key,然后通过这个唯一的key在词典中查找捆绑的返回数据包,以此达到流量回放和线上运行流量一致。 Thereby, it is possible to request packets by parsing a pressure or pressure after a request to do a signature obtained by treating a packet key, and then look for return packet bundle in a dictionary by this unique key, in order to achieve playback operation flow rate and flow lines consistent.

[0115] 步骤S109,在测试过程中,监控被测模块所在服务器的性能数据,并根据该性能数据进行性能分析。 [0115] step S109, during the test, measured to monitor the performance data of the server module, and performance analysis based on the performance data.

[0116] 应理解,不同模块需要监控的性能指标不一样,在本发明的一个实施例中提供通用的监控工具,用户只需要配置监控项,监控工具自动监控相应的数据,性能测试结束后,自动地根据要求对这些监控数据进行分析。 [0116] It should be understood, the different modules required performance monitor not the same, to provide a common embodiment monitoring tools in one embodiment of the invention, the user only needs to configure the monitored item, monitoring tool automatically monitors the appropriate data, after the performance test, The requirements for such automatic monitoring data analysis.

[0117] 具体地,通用监控工具分为两个单元,总控单元和多个分控单元。 [0117] In particular, the generic monitoring tool is divided into two units, a plurality of sub-master control units and control units. 总控单元负责监控的部署、启动、停止以及数据的分析工作。 Master Control Unit is responsible for monitoring the deployment, start, stop, and analysis of data. 每个分控单元均与总控单元相连,根据总控单元的控制指令周期性地收集数据,并对收集到的数据进行格式化。 Each sub-control units are connected to the master control unit, the data collected periodically according to a control command of the master control unit, and the data collected to be formatted.

[0118] 在本发明的一个实施例中,分控单元收集的数据可包括:vmstat数据;CPU、内存和文件句柄的占用情况;iostat数据;被测模块的日志;以及网络数据等。 [0118] In one embodiment of the present invention, the data collection sub-control unit may include: vmstat transactions; occupancy CPU, memory, and file handle; the iostat transactions; log module under test; and a data network.

[0119] 总控单元的具体功能可包括:建立所述总控单元与各个所述分控单元之间的双向信任关系;将监控脚本部署到各个所述分控单元上;批量启动或停止所述多个分控单元;获取各个所述分控单元的机器物理参数;以及根据所格式化后的性能数据计算其在不同时间段内的平均值、最小值、最大值以及某一列数据在不同区间的比例等。 Specific Function [0119] The master control unit may include: establishing a bidirectional trust relationship between the master control unit and the respective sub-control unit; monitor script to deploy the respective sub-control unit; start or stop the batch said plurality of sub-control units; each of said sub-control means acquires the physical parameters of the machine; and calculating the performance data which is formatted different time periods the average, minimum, maximum, and a column of data in different proportion of the interval.

[0120] 在本发明的另一个实施例中,通用监控工具还可包括日志处理单元。 [0120] In another embodiment of the present invention, the universal monitoring tool may further comprise a log processing unit. 该日志处理单元与总控单元相连,用于对获取到的性能数据进行解析和统计。 The log processing unit is connected to the master control unit, configured to obtain performance data parsed and statistics. 例如,获取不同时间段内压力的平均值、最大值和最小值等。 For example, the pressure obtaining different time average, maximum and minimum value.

[0121] 根据本发明实施例的分布式集群的性能测试方法,通过网桥抓取上游集群、被测模块以及后端集群之间的通信数据包,从而通过一个网桥可以真实模拟线上拓扑和数据量,无需搭建真实的集群环境即可对被测模块进行性能测试,提高测试准确性和测试效率,节约成本。 [0121] The performance test method for distributed clusters embodiment of the present invention, by gripping the upstream bridge cluster, communication data packets between the rear end of the module under test and the cluster, which can simulate the real topology through a bridge line and the amount of data without the need to build a real cluster environment to test modules for performance testing, improve test accuracy and test efficiency and cost savings.

[0122] 综上,根据本发明实施例的分布式集群的性能测试的系统及方法和网桥,至少具有以下有益效果: [0122] In summary, the method and system according to the embodiment and the performance test bridge embodiment of the present invention is distributed cluster having at least the following advantages:

[0123] 1、拓扑结构和线上一致;[0124] 根据线上集群各个模块的连接关系拓扑结构,构建线下测试集群,保证被测模块所处的上下游环境和线上一致。 [0123] 1, and consistent line topology; [0124] The connection relationship of each module line topology of the cluster, the cluster constructed to test the line to ensure that the module under test is located on the same line and the downstream environment.

[0125] 2、数据量和线上相当; [0125] 2, a considerable amount of data and lines;

[0126] 模拟集群使用的数据包来源于被测模块运行态dump下来或者通过网桥在调研环境中dump下来,数据量和数据内容与线上基本一致。 [0126] Simulation data packet used in the cluster from the module under test operating mode or down dump dump down through the bridge in the research environment, the amount of data and consistent content online.

[0127] 3、数据流回放和线上一致; [0127] 3, consistent with the data stream and playback online;

[0128] 模拟集群使用的数据包在dump下来时对每个数据包都做了标记。 [0128] Data packets used in the simulated cluster down to dump each packet been tagged. 每个请求和返回结果用一个唯一的标识建立词典,回放时,网桥根据请求包获得唯一的标识,然后通过这个唯一的标识在词典中查找捆绑的返回数据包,以此达到流量回放和线上运行流量一致。 With each request and returns the result to establish a unique identification dictionary, during playback, the bridge according to obtain a unique identification request packet and return packet lookup bundled in the dictionary by the unique identifier, in order to achieve flow line and playback agreement on the run flow.

[0129] 4、提高性能测试准确性; [0129] 4, to improve the accuracy of the performance test;

[0130] 使用模拟线上拓扑和数据量的真实集群,提高性能测试准确性。 [0130] Analog line topology using real cluster and the data amount, to improve the accuracy of the performance test.

[0131] 5、提高测试效率; [0131] 5, improve test efficiency;

[0132] 无需搭建真实的集群环境,提高测试效率。 [0132] without the need to build a real cluster environment, improve test efficiency.

[0133] 6、节约成本; [0133] 6, cost savings;

[0134] 无需使用真实集群环境,使用一台机器模拟整个后端集群,大量节省机器资源,节约成本。 [0134] without using real clustered environment, the use of a machine to simulate the entire back-end cluster, save a lot of machine resources and cost savings.

[0135] 应当理解,本发明的各部分可以用硬件、软件、固件或它们的组合来实现。 [0135] It should be understood that various portions of the present invention may be implemented in hardware, software, firmware or a combination thereof to achieve. 在上述实施方式中,多个步骤或方法可以用存储在存储器中且由合适的指令执行系统执行的软件或固件来实现。 In the above-described embodiment, a plurality of steps or methods may be implemented in software or firmware and executed by a suitable system executing instructions stored in a memory with. 例如,如果用硬件来实现,和在另一实施方式中一样,可用本领域公知的下列技术中的任一项或他们的组合来实现:具有用于对数据信号实现逻辑功能的逻辑门电路的离散逻辑电路,具有合适的组合逻辑门电路的专用集成电路,可编程门阵列(PGA),现场可编程门阵列(FPGA)等。 For example, if implemented in hardware, as in another embodiment, the present technique may be any one of the following well-known in the art, or their combination thereof: a logic gate circuit for implementing logic functions upon data signals discrete logic circuits having appropriate combinational logic gate circuit ASIC, a programmable gate array (PGA), a field programmable gate array (FPGA) and the like.

[0136] 在本说明书的描述中,参考术语“一个实施例”、“一些实施例”、“示例”、“具体示例”、或“一些示例”等的描述意指结合该实施例或示例描述的具体特征、结构、材料或者特点包含于本发明的至少一个实施例或示例中。 [0136] In the description of the present specification, reference to the term "one embodiment," "some embodiments", "an example", "a specific example", or "some examples" means that a description of the exemplary embodiment or embodiments described a particular feature, structure, material, or characteristic is included in at least one embodiment of the present invention, embodiments or examples. 在本说明书中,对上述术语的示意性表述不一定指的是相同的实施例或示例。 In the present specification, a schematic representation of the above terms necessarily referring to the same embodiment or example. 而且,描述的具体特征、结构、材料或者特点可以在任何的一个或多个实施例或示例中以合适的方式结合。 Furthermore, the particular features, structures, materials, or characteristics described embodiments or examples may be at any one or more in a proper manner.

[0137] 尽管已经示出和描述了本发明的实施例,对于本领域的普通技术人员而言,可以理解在不脱离本发明的原理和精神的情况下可以对这些实施例进行多种变化、修改、替换和变型,本发明的范围由所附权利要求及其等同限定。 [0137] While there has been illustrated and described embodiments of the present invention, those of ordinary skill in the art, to be understood that various changes may be made to these embodiments without departing from the principles and spirit of the present invention, modifications, substitutions and modifications, the scope of the invention being indicated by the appended claims and their equivalents.

Claims (17)

1.一种分布式集群的性能测试系统,其特征在于,包括: 上游集群,所述上游集群用于发送第一压力请求; 被测模块; 后端集群,所述后端集群包括多个下游集群; 网桥,所述网桥分别与所述上游集群、被测模块和后端集群相连,所述网桥用于接收所述第一压力请求并将所述第一压力请求发送至所述被测模块,接收所述被测模块根据所述第一压力请求生成的第二压力请求并将所述第二压力请求转发至所述多个下游集群,以及保存所述第一压力请求以及所述多个下游集群根据所述第二压力请求返回的通信数据包; 压力工具,所述压力工具用于根据所述网桥保存的所述第一压力请求向所述被测模块发送预定压力,其中所述被测模块将所述预定压力转发至所述网桥,所述网桥根据所述预定压力在所述保存的返回的通信数据包中查找所述预定压力相应的 Performance Test 1. A distributed system clusters, wherein, comprising: an upstream cluster, the cluster is configured to send a first pressure upstream of the request; module under test; the rear end of the cluster, said cluster comprising a plurality of downstream rear end cluster; bridge, said bridge respectively with the upstream cluster, the cluster is connected to the module under test and a rear end, said bridge means for receiving said first pressure and said first pressure request to the request the module under test, said second pressure receiving module according to the first measured pressure generated request requests and forwards the request to the second pressure downstream of the plurality of clusters, and storing the first request and the pressure downstream of said plurality of cluster communication packet according to the request returned by the second pressure; pressure tool, said pressure transmitting means for requesting a predetermined pressure to the module under test according to the stored first pressure bridge, wherein said test module forwarding said predetermined pressure to said bridge, said bridge searching the predetermined pressure stored in the return packet communication data according to the respective predetermined pressure 据包并返回至所述被测模块;以及监控模块,所述监控模块与所述被测模块位于同一个服务器上,用于监控所述服务器的性能数据,并根据所述性能数据进行性能分析。 And returning said packet to the module under test; and a monitoring module, the monitoring module and the module under test on the same server, for monitoring the performance data of the server, and performance analysis based on the performance data .
2.如权利要求1所述的分布式集群的性能测试系统,其特征在于,所述通信数据包包括: 魔鬼数字,用于标识协议包头; Dump ID,用于标识所保存的通信数据包的编号; key标识,用于标识压力·请求及其对应的返回数据包; 版本号,用于区别不同的被测模块; 保留字段,用于扩展所述通信数据包; 包长度,用于标识所述通信数据包的实际长度;和Dump的数据包体,用于存放所述通信数据包。 2. The distributed cluster performance test system according to claim 1, wherein said data packet communication comprising: a magic number, identifies the protocol header; Dump ID, for identifying the communication packet is stored No; key identifier for identifying the request and returns the pressure-corresponding data packet; version number for distinguishing the different test modules; field is reserved for extending the communication data packet; packet length, for identifying the actual length of said communication packets; Dump and packet body, and for storing the communication data packet.
3.如权利要求2所述的分布式集群的性能测试系统,其特征在于,所述网桥按照key标识&value对的形式建立词典以保存所述第一压力请求和所述返回的通信数据包,其中value为所述返回的通信数据包,key标识为所述第一压力请求包中的key标识。 3. The performance test of the distributed cluster system as claimed in claim 2, characterized in that, the bridges establish dictionary identification key in the form & value of the pressure to preserve the first request and the return communication packet , wherein said return value is a data communication packet, the key is identified as a first pressure request key identification packet.
4.如权利要求1所述的分布式集群的性能测试系统,其特征在于,所述监控模块具体包括: 总控单元,所述总控单元用于配置监控项、控制监控的启动和停止以及根据所述性能数据进行性能分析;以及多个分控单元,每个分控单元均与所述总控单元相连,用于根据所述总控单元的控制周期性地收集所述被测模块所在的服务器的性能数据,并对所述性能数据进行格式化。 4. A distributed cluster of performance test system according to claim 1, wherein the monitoring module comprises: a master control unit, the master control unit is configured for monitoring items, and the control stop and start monitoring performance analysis based on the performance data; and a plurality of sub-control units, each sub-control units are connected to the master control unit, for collecting the periodically according to the master control unit controlling the module under test is located performance data servers, and the performance data format.
5.如权利要求1或4所述的分布式集群的性能测试系统,其特征在于,所述性能数据包括: vmstat 数据; CPU、内存和文件句柄/socket句柄等占用情况; iostat 数据; 被测模块运行的日志;以及网络数据。 5. A distributed cluster performance test system of claim 1 or claim 4, wherein the performance data comprises: vmstat transactions; the CPU, memory, file handles, and / socket handle other occupation; the iostat transactions; measured log module operation; and a data network.
6.如权利要求4所述的分布式集群的性能测试系统,其特征在于,所述总控单元具体包括: 建立所述总控单元与各个所述分控单元之间的双向信任关系; 将监控脚本部署到各个所述分控单元上; 批量启动或停止所述多个分控单元; 获取各个所述分控单元的机器物理参数;以及根据所格式化后的性能数据计算其在不同时间段内的平均值、最小值、最大值以及某一列数据在不同区间的比例。 6. A distributed cluster of performance test system according to claim 4, wherein the master control unit comprises: establishing a bidirectional trust relationship between the master control unit and the respective sub-control means; and deployed to monitor script on each of the sub-control unit; batch start or stop of the plurality of sub-control unit; machine acquires physical parameter of the respective sub-control unit; and a calculation based on the performance data which is formatted at different times within a segment average, minimum, maximum, and a column of data in a different ratio range.
7.如权利要求4所述的分布式集群的性能测试系统,其特征在于,所述监控模块还包括: 日志处理单元,所述日志处理单元与所述总控单元相连,用于对所述性能数据进行解析和统计。 7. A distributed cluster of performance test system according to claim 4, wherein the monitoring module further comprises: a log processing unit, a log processing unit and the master control unit is connected to the performance data is parsed and statistics.
8.一种网桥,其特征在于,包括: 模式选择模块,所述模式选择模块用于接收用户输入的录制模式或测试模式; 通信模块,所述通信模块用于在所述录制模式下接收上游集群发送的第一压力请求并将所述第一压力请求发送至被测`模块,以及接收所述被测模块根据所述第一压力请求生成的第二压力请求并将所述第二压力请求转发至后端集群,以及接收所述后端集群根据所述第二压力请求返回的通信数据包;所述通信模块还用于在所述测试模式下接收所述被测模块转发的预定压力; 录制模块,所述录制模块与所述通信模块相连,用于保存所述第一压力请求和所述返回的通信数据包;以及模拟模块,所述模拟模块分别与所述通信模块和所述录制模块相连,用于根据所述预定的压力在所述录制模块保存的通信数据包中查找所述预定的压力对应的返回数据包 A bridge, characterized by comprising: a mode selection module, the mode selector means for receiving a recording mode or a test mode input by a user; a communication module, a communication module for receiving the said recording mode a first pressure upstream of the cluster and the first transmission request requests the second pressure sent to the pressure `test module, and receiving the test request generation module according to the first pressure and said second pressure request forwarding the request to the rear end of the cluster, and the cluster receiving the rear end of the communication request packet to return data based on the second pressure; the communication module is further for receiving a predetermined pressure in said test mode said test module forwarding ; recording module, connected to the recording module and the communication module, configured to store the first request and the pressure of the return communication data packets; and a simulation module, the analog module and the communication module, respectively, and the recording module is connected, a return packet corresponding to a predetermined pressure according to said predetermined pressure to find the record stored in the packet communications module ,其中所述通信模块还用于将所述预定压力对应的返回数据包发送至所述被测模块。 Wherein the communication module is further configured to return to the predetermined pressure corresponding to a data packet to the module under test.
9.如权利要求8所述的网桥,其特征在于,所述通信数据包包括: 魔鬼数字,用于标识协议包头; Dump ID,用于标识所保存的通信数据包的编号; key标识,用于标识压力请求及其对应的返回数据包; 版本号,用于区别不同的被测模块; 保留字段,用于扩展所述通信数据包; 包长度,用于标识所述通信数据包的实际长度;和Dump的数据包体,用于存放所述通信数据包。 9. A bridge according to claim 8, wherein said data packet communication comprising: a magic number, identifies the protocol header; Dump ID, a number that identifies the stored communication packets; key identifier, and the return pressure is used to identify the corresponding request packet; version number for distinguishing the different test modules; field is reserved for extending the communication data packet; packet length, for identifying the communication packet actual length; Dump and packet body, and for storing the communication data packet.
10.如权利要求9所述的网桥,其特征在于,所述模拟模块还用于按照key标识&value对的形式建立词典以保存所述第一压力请求和所述返回的通信数据包,其中value为所述返回的通信数据包,key标识为所述第一压力请求包中的key标识。 10. A bridge according to claim 9, characterized in that said analog module is further configured to establish a first dictionary to store the request and the return pressure communication packet in accordance with the key identifier & value pairs, wherein the return value is a communication packet, key identifier is a first pressure request key identification packet.
11.一种分布式集群的性能测试方法,其特征在于,包括以下步骤: (a)在录制模式下,网桥接收上游集群发送的第一压力请求,并将所述第一压力请求发送至被测模块,以及保存所述第一压力请求;(b)所述被测模块接收所述第一压力请求,并根据所述第一压力请求生成第二压力请求,以及将所述第二压力请求发送至所述网桥; (c)所述网桥接收所述第二压力请求并将所述第二压力请求发送至后端集群,其中所述后端集群包括多个下游集群; (d)所述多个下游集群根据所述第二压力请求返回相应的数据包至所述网桥; (e)所述网桥接收所述并保存所述返回的数据包; (f)在测试模式下,通过压力工具根据所述网桥保存的第一压力请求发送预定压力至所述被测模块; (g)所述被测模块将所述预定压力转发至所述网桥,并接收所述网桥根据所述预定压力在所述保存 A performance test method for distributed clusters, characterized by, comprising the steps of: (a) in the recording mode, the bridge receiving a first pressure upstream of the cluster transmits a request, and the request to a first pressure the module under test, and saving the first pressure request; (b) the test module receives the request first pressure and a second pressure generation request according to the request of the first pressure and the second pressure request to the bridge; (c) the bridge request and the second pressure receiving said second pressure request to the rear end of the cluster, wherein the cluster comprises a plurality of downstream rear clusters; (D ) downstream of the plurality of clusters according to the second pressure request returns the corresponding data packet to the bridge; (e) the bridge receives the data packet and store the returned; (f) in a test mode under pressure by a first pressure means in accordance with a request to save the bridge sends a predetermined pressure to the module under test; (G) of the module under the predetermined pressure is forwarded to the bridge, and receiving the the bridge said predetermined pressure stored in the 的通信数据包中查找到的相应的返回数据包; (h)在测试过程中,监控所述被测模块所在的服务器的性能数据,并根据所述性能数据进行性能分析。 Communication packet to find the corresponding return packets; (H) during the test, the monitored performance data server where the module under test, and performance analysis based on the performance data.
12.如权利要求11所述的分布式集群的性能测试方法,其特征在于,所述通信数据包包括: 魔鬼数字,用于标识协议包头; Dump ID,用于标识所保存的通信数据包的编号; key标识,用于标识压力请求及其对应的返回数据包; 版本号,用于区别不同的被测模块; 保留字段,用于扩展所述通信数据包; 包长度,用于标识所述通信数据包的实际长度;和Dump的数据包体,用于存放所述通信数据包。 12. The performance test method of claim distributed cluster, wherein said data packet communication comprising: a magic number, identifies the protocol header; Dump ID, for identifying the communication packet is stored No; key identifier for identifying the request and its corresponding pressure return packets; version number for distinguishing the different test modules; field is reserved for extending the communication data packet; packet length, for identifying the the actual length of the communication packets; Dump and packet body, and for storing the communication data packet.
13.如权利要求12所述的分布式集群的性能测试方法,其特征在于,还包括步骤: 所述网桥按照key标识&value对的形式建立词典以保存所述第一压力请求和所述返回的通信数据包,其中value为所述返回的通信数据包,key标识为所述第一压力请求包中的key标识。 13. The method of testing the performance of the distributed cluster of claim 12, wherein further comprising the step of: establishing said bridge dictionary identification key in the form & value of the pressure to preserve the first request and the return communication packet, wherein said return value is a data communication packet, the request key identification key identification packet to the first pressure.
14.如权利要求11所述的分布式集群的性能测试方法,其特征在于,所述步骤(h)通过通用监控工具实现,其中所述通用监控工具包括: 总控单元,所述总控单元用于配置监控项、控制监控的启动和停止以及根据所述性能数据进行性能分析;以及多个分控单元,每个分控单元均与所述总控单元相连,用于根据所述总控单元的控制周期性地收集所述被测模块所在的服务器的性能数据,并对所述性能数据进行格式化。 14. The method of performance testing distributed cluster of claim 11, wherein said step (h) implemented by a general purpose monitoring tools, wherein the common monitoring tool comprising: a master control unit, the master control unit monitored item is used to configure, control, and start and stop monitoring performance analysis based on the performance data; and a plurality of sub-control units, each sub-control units are connected to the master control unit, according to the master control the control unit periodically collecting said performance data server where the module under test, and the performance data format.
15.如权利要求14所述的分布式集群的性能测试方法,其特征在于,所述性能数据包括: vmstat 数据; CPU、内存和文件句柄的占用情况; iostat 数据; 被测模块的运行日志;以及网络数据。 15. The method of testing the performance of the distributed cluster of claim 14, wherein the performance data comprises: vmstat transactions; occupancy CPU, memory, and file handle; the iostat transactions; running log of the module under test; and network data.
16.如权利要求14所述的分布式集群的性能测试方法,其特征在于,所述总控单元具体用于: 建立所述总控单元与各个所述分控单元之间的双向信任关系; 将监控脚本部署到各个所述分控单元上; 批量启动或停止所述多个分控单元; 获取各个所述分控单元的机器物理参数;以及根据所格式化后的性能数据计算其在不同时间段内的平均值、最小值、最大值以及某一列数据在不同区间的比例。 16. The method of testing the performance of the distributed cluster of claim 14, wherein the master control unit is configured to: establish two-way trust relationship between the master control unit and the respective sub-control unit; monitor script to deploy the respective points on the control unit; batch start or stop of the plurality of sub-control units; each of said sub-control means acquires the physical parameters of the machine; and calculating the performance data which is different formatted period average, minimum, maximum, and a column of data in a different ratio range.
17.如权利要求14所述的分布式集群的性能测试方法,其特征在于,所述通用监控工具还包括: 日志处理单元, 所述日志处理单元与所述总控单元相连,用于对所述性能数据进行解析和统计。 17. The method of testing the performance of the distributed cluster of claim 14, wherein the monitoring the universal tool further comprises: a log processing unit, a log processing unit is connected to the master control unit for the analyzing said performance data and statistics.
CN 201210043849 2012-02-23 2012-02-23 Test System distributed cluster, methods and apparatus CN103297285B (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN 201210043849 CN103297285B (en) 2012-02-23 2012-02-23 Test System distributed cluster, methods and apparatus

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN 201210043849 CN103297285B (en) 2012-02-23 2012-02-23 Test System distributed cluster, methods and apparatus

Publications (2)

Publication Number Publication Date
CN103297285A true true CN103297285A (en) 2013-09-11
CN103297285B CN103297285B (en) 2016-03-16

Family

ID=49097633

Family Applications (1)

Application Number Title Priority Date Filing Date
CN 201210043849 CN103297285B (en) 2012-02-23 2012-02-23 Test System distributed cluster, methods and apparatus

Country Status (1)

Country Link
CN (1) CN103297285B (en)

Cited By (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN104461856A (en) * 2013-09-22 2015-03-25 阿里巴巴集团控股有限公司 Performance test method, device and system based on cloud computing platform
CN105515883A (en) * 2014-09-22 2016-04-20 阿里巴巴集团控股有限公司 Test method and system
CN106055486A (en) * 2016-08-19 2016-10-26 浪潮(北京)电子信息产业有限公司 Automatic operation maintenance method and platform of distributed file system

Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20070118633A1 (en) * 2005-11-10 2007-05-24 Kumar C P V Cluster command testing
CN101127801A (en) * 2007-08-29 2008-02-20 中兴通讯股份有限公司 A method for testing relay performance of group call
US8037109B2 (en) * 2003-06-30 2011-10-11 Microsoft Corporation Generation of repeatable synthetic data
CN202033748U (en) * 2011-04-22 2011-11-09 阿里巴巴集团控股有限公司 Search engine performance test system

Patent Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US8037109B2 (en) * 2003-06-30 2011-10-11 Microsoft Corporation Generation of repeatable synthetic data
US20070118633A1 (en) * 2005-11-10 2007-05-24 Kumar C P V Cluster command testing
CN101127801A (en) * 2007-08-29 2008-02-20 中兴通讯股份有限公司 A method for testing relay performance of group call
CN202033748U (en) * 2011-04-22 2011-11-09 阿里巴巴集团控股有限公司 Search engine performance test system

Cited By (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN104461856A (en) * 2013-09-22 2015-03-25 阿里巴巴集团控股有限公司 Performance test method, device and system based on cloud computing platform
CN105515883A (en) * 2014-09-22 2016-04-20 阿里巴巴集团控股有限公司 Test method and system
CN106055486A (en) * 2016-08-19 2016-10-26 浪潮(北京)电子信息产业有限公司 Automatic operation maintenance method and platform of distributed file system

Also Published As

Publication number Publication date Type
CN103297285B (en) 2016-03-16 grant

Similar Documents

Publication Publication Date Title
US6724729B1 (en) System analyzer and method for synchronizing a distributed system
US6560723B1 (en) Automatic communication protocol test system with message/sequence edit function and test method using the same
CN102143022A (en) Cloud measurement device and method for IP network
CN101051954A (en) Detecting system and method for radio sensor network algorithm
CN101227340A (en) Embedded type network quality detecting device and method and evaluation system and evaluation method thereof
CN101846985A (en) Remote data acquisition system and method
CN101447991A (en) Test device used for testing intrusion detection system and test method thereof
CN101072131A (en) Network operation monitoring method and system
CN101789889A (en) Method and system for automatic testing communication equipment
CN101197736A (en) Wireless net member equipment monitoring system and monitoring method thereof
CN103152393A (en) Charging method and charging system for cloud computing
US20120079324A1 (en) Firmware tracing in a storage data communication system
CN1863102A (en) System and method for testing apparatus based on managing information base
CN101275987A (en) Equipment test system and method
CN102035673A (en) Fiber channel-based general automatic test system
CN1735047A (en) Fault self-diagnosis method and device
CN102571466A (en) Method, device and system for testing service
CN101141328A (en) Method and device for simulating Diameter server terminal
CN1592238A (en) Method and appatatus for processing wide band device fault positioning requesting
CN101145993A (en) A multi-point access interface test method and its test system
JP2002041331A (en) Server performance measuring device
CN101873351A (en) Pressure test method and system for automobile remote monitoring system
CN102104502A (en) Ethernet equipment performance test platform based on Linux system
CN102801587A (en) Large-scale network-oriented virtualized monitoring system and dynamic monitoring method thereof
CN101217593A (en) A performance test device and method of signaling transmission network

Legal Events

Date Code Title Description
C06 Publication
C10 Entry into substantive examination
C14 Grant of patent or utility model