WO2015135182A1

WO2015135182A1 - Method and apparatus for evaluating performance of real-time system

Info

Publication number: WO2015135182A1
Application number: PCT/CN2014/073377
Authority: WO
Inventors: Siyan ZHAO; Saradindu KAR; Yaotian ZHENG
Original assignee: Telefonaktiebolaget L M Ericsson (Publ)
Priority date: 2014-03-13
Filing date: 2014-03-13
Publication date: 2015-09-17

Abstract

The disclosure provides a method (100) for evaluating a performance of a real-time system. The method comprises: creating (S110) a first real-time system and a second real-time system; causing (S120) the first real-time system to execute a task, so as to obtain a plurality of first performance parameters; causing (S130) the second real-time system to execute said task, so as to obtain a plurality of second performance parameters; and calculating (S140) a first performance metric based on the plurality of first performance parameters and a second performance metric based on the plurality of second performance parameters.

Description

METHOD AND APPARATUS FOR EVALUATING PERFORMANCE OF

REAL-TIME SYSTEM

TECHNICAL FIELD

The disclosure generally relates to real-time system, and more particularly, to a method and apparatus for evaluating a performance of a real-time system.

BACKGROUND

With the development of distributed computation techniques, real-time systems are becoming increasingly popular. For example, they play an important role in the so-called "Big Data" analysis.

Typically, a real-time system can be created based at least on a real-time framework, a real-time data processing algorithm and a database. There are currently many kinds of real-time frameworks, such as S4, Storm and

Spark-streaming, many kinds of real-time data processing algorithms, such as Clustream, Clustree, Ozaboost, Incmine and Moment, and many kinds of databases, such as Structured Query Language (SQL) databases (e.g., Postresql, Mysql, etc.) and NoSQL databases (e.g., Hbase, Cassandra, etc.). The

performance of a real-time system depends on the real-time framework, real-time data processing algorithm and database used in the system.

Furthermore, the performance of a real-time system is also affected by its data throughput and available resources.

It is desired to find a best real-time system for executing a task. Thus, there is a need for a solution for evaluating a performance of a real-time system in an efficient manner. SUMMARY

It is an object of the disclosure to provide a method and apparatus for evaluating a performance of a real-time system as well as associated computer program and computer program storage product. According to an aspect of the disclosure, a method for evaluating a performance of a real-time system is provided. The method comprises: creating a first real-time system and a second real-time system; causing the first real-time system to execute a task, so as to obtain a plurality of first performance parameters;

causing the second real-time system to execute said task, so as to obtain a plurality of second performance parameters; and calculating a first performance metric based on the plurality of first performance parameters and a second performance metric based on the plurality of second performance parameters.

In an embodiment, the first real-time system is created by selecting a first real-time framework from a plurality of real-time frameworks, a first data processing algorithm from a plurality of data processing algorithms and a first database from a plurality of databases. The second real-time system is created by selecting a second real-time framework from the plurality of real-time frameworks, a second data processing algorithm from the plurality of data processing algorithms and a second database from the plurality of databases.

In an embodiment, said causing the first real-time system to execute a task comprises: generating a first data stream for the first real-time system at a first data throughput and causing the first real-time system to execute said task using a first distributed computation resource. Said causing the second real-time system to execute said task comprises: generating a second data stream for the second real-time system at a second data throughput and causing the second real-time system to execute said task using a second distributed computation resource. In an embodiment, the first distributed computation resource comprises a first number of distributed computation nodes in the first real-time system. The second distributed computation resource comprises a second number of distributed computation nodes in the second real-time system. In an embodiment, each of the plurality of first performance parameters and the plurality of second performance parameters is obtained by using trace system scripts.

In an embodiment, each of the plurality of first performance parameters and the plurality of second performance parameters comprises one or more of a Central Processing Unit (CPU) occupation, a memory occupation, a framework processing time length, an algorithm execution time length, a storage space and a storage time length required to accomplish the task.

In an embodiment, said calculating the first performance metric based on the plurality of first performance parameters comprises: assigning one of a first set of predetermined weights to each of the plurality of first performance parameters; and calculating, as the first performance metric, a weighted average of the plurality of first performance parameters. Said calculating the second

performance metric based on the plurality of second performance parameters comprises: assigning one of a second set of predetermined weights to each of the plurality of second performance parameters; and calculating, as the second performance metric, a weighted average of the plurality of second performance parameters. In an embodiment, the method further comprises: comparing the first

performance metric and the second performance metric; and selecting one of the first real time system and the second real-time system based on the comparison.

The disclosure also provides an apparatus for evaluating a performance of a real-time system as well as associated computer program and computer program storage product. The above embodiments of the method are also applicable for the apparatus, computer program and computer program storage product.

With the embodiments of the disclosure, the first and second real-time systems are created and caused to execute the same task. The performance parameters associated with each real-time system are obtained and then the performance metrics for the first and second real-time systems are calculated based on their respective performance parameters. An advantage with the embodiments is to make it possible to evaluate the performance of each real-time system efficiently. Further, the performance metrics of these systems can be compared to decide which of them has a better performance, which allows selecting a real-time system most suitable for executing the task.

BRIEF DESCRIPTION OF THE DRAWINGS The above and other objects, features and advantages will be more apparent from the following description of embodiments with reference to the figures, in which:

Fig. 1 is a flowchart illustrating a method for evaluating a

performance of a real-time system according to an

embodiment of the disclosure;

Fig. 2 is a schematic diagram of an apparatus for evaluating a

performance of a real-time system according to an

embodiment of the disclosure; and

Fig. 3 is a schematic diagram of an arrangement that can be used in an apparatus for evaluating a performance of a real-time system according to an embodiment of the disclosure.

DETAILED DESCRIPTION

The embodiments of the disclosure will be detailed below with reference to the drawings. It should be noted that the following embodiments are illustrative only, rather than limiting the scope of the disclosure.

Fig. 1 is a flowchart illustrating a method 100 for evaluating a performance of a real-time system according to an embodiment of the disclosure. The method 100 can be performed by e.g., a control server in a distributed computation

environment and comprises the steps as described below.

At step S110, a first real-time system and a second real-time system are created, denoted as RS1 and RS2, respectively. It is to be noted here that the disclosure is not limited to two systems and more than two systems can be created in this step.

As described above, there are a plurality of candidate real-time frameworks, such as S4, Storm and Spark, a plurality of candidate real-time data processing algorithms, such as Clustream and Clustree, and a plurality of candidate

databases, such as Structured Query Language (SQL) databases (e.g., Postresql, Mysql, etc.) and NoSQL databases (e.g., Hbase, Cassandra, etc.), for creating a real-time system. In the step S110, the real-time system RS1 can be created by selecting a first real-time framework (denoted as FR1 ) from a plurality of real-time frameworks, a first data processing algorithm (denoted as AL1 ) from a plurality of data processing algorithms and a first database (denoted as DB1 ) from a plurality of databases. Similarly, the real-time system RS2 can be created by selecting a second real-time framework (denoted as FR2) from the plurality of real-time frameworks, a second data processing algorithm (denoted as AL2) from the plurality of data processing algorithms and a second database (denoted as DB2) from the plurality of databases.

In an example, the system RS1 can be created based on the framework S4, the algorithm Clustream and the database Postgresql, i.e., FR1 =S4, AL1 =Clustream and DB1 = Postgresql. The system RS2 can be created based on the framework Storm, the algorithm Clustream and the database Hbase, i.e., FR2=Storm, AL2=Clustream and DB2=Hbase. Here, the control server may determine the configurations of the systems RS1 and RS2 (i.e., the framework, algorithm and database used for creating each of the systems) based on a configuration file received from a configuration server. For example, the configuration file may contain information on the framework, algorithm and database used for creating each of the systems RS1 and RS2.

At step S120, the first real-time system RS1 is caused to execute a task, so as to obtain a plurality of first performance parameters. Here, the task can be e.g., at least one of online data filtering, data aggregation, correlation of events, frequent pattern analysis, online clustering, online classification and the like.

In the step 120, a first data stream (denoted as TR1 ) is generated for the first real-time system RS1 at a first data throughput (denoted as TH1 ). Here, the control server may include a data generator for generating the data stream TR1 for inputting to the system RS1 .

The first real-time system RS1 is caused to execute the task using a first distributed computation resource. In particular, the first distributed computation resource comprises a first number (denoted as N1 ) of distributed computation nodes in the first real-time system RS1 . Here the distributed computation nodes can be physical computation nodes or virtual nodes each representing an executing instance of the task. Here, the data throughput TH1 and the number N1 can also be determined based on the configuration file as described above. For example, in addition to the information on the framework, algorithm and database used for creating each of the systems, the configuration file may contain information on the data throughput and the number of distributed computation nodes in each of the systems. Here, the plurality of first performance parameters can be obtained by using trace system scripts. The trace system scripts are software/programs to enable real-time monitoring of the system performance without interrupting the execution of the task. The plurality of first performance parameters may comprise one or more of a Central Processing Unit (CPU) occupation, a memory occupation, a framework processing time length, an algorithm execution time length, a storage space and a storage time length required for the system RS1 to accomplish the task.

Similarly, at step S130, the second real-time system RS2 is caused to execute the task, so as to obtain a plurality of second performance parameters.

In the step 130, a second data stream (denoted as TR2) is generated for the second real-time system RS2 at a second data throughput (denoted as TH2). Here, the control server may include a data generator for generating the data stream TR2 for inputting to the system RS2.

The second real-time system RS2 is caused to execute the task using a second distributed computation resource. In particular, the second distributed

computation resource comprises a second number (denoted as N2) of distributed computation nodes in the second real-time system RS2. Again, the distributed computation nodes here can be physical computation nodes or virtual nodes each representing an executing instance of the task. Here, the data throughput TH2 and the number N2 can also be determined based on the configuration file as described above.

The plurality of second performance parameters can also be obtained by using trace system scripts. The plurality of second performance parameters may comprise one or more of a CPU occupation, a memory occupation, a framework processing time length, an algorithm execution time length, a storage space and a storage time length required for the system RS2 to accomplish the task. At step S140, a first performance metric (denoted as PM1 ) is calculated based on the plurality of first performance parameters, and a second performance metric (denoted as PM2) is calculated based on the plurality of second performance parameters.

In particular, one of a first set of predetermined weights can be assigned to each of the plurality of first performance parameters and a weighted average of the plurality of first performance parameters can be calculated as the first

performance metric PM1 .

Similarly, one of a second set of predetermined weights can be assigned to each of the plurality of second performance parameters and a weighted average of the plurality of second performance parameters can be calculated as the second performance metric PM2. Here at least one weight in the second set of predetermined weights can be different from its corresponding weight in the first set of predetermined weights. These weights can be determined by using a Support Vector Machine (SVM) Classifier based on e.g., historical performance evaluation data. Then, the first performance metric PM1 can be compared with the second performance metric PM2 and one of the first real time system RS1 and the second real-time system RS2 can be selected based on the comparison.

Next, a specific example will be given for further explaining the above method 100.

In this example, the configurations of the real-time systems RS1 and RS2 are listed in Table 1 below. It is to be noted that, as discussed above, the principle of the disclosure can be generalized readily to more than two systems. Hence, a third real-time system RS3 is also shown in Table 1 for the purpose of illustration. Table 1 - Configurations of Real-time Systems

As shown in the above Table 1 , the system RS1 is configured based on the framework S4, the algorithm Clustream and the database Postgresql. The data stream TR1 is input into the system RS1 at the data throughput TH1 =1 Mbps. In the system RS1 , the data stream TR1 and the task (e.g., an online clustering task) are distributed to N1 =2 distributed computation nodes for execution. The system RS2 is created based on the framework Storm, the algorithm Clustream and the database Hbase. The data stream TR2 is input into the system RS2 at the data throughput TH2=1 Mbps. In the system RS2, the data stream TR2 and the task are distributed to N2=4 distributed computation nodes for execution. The system RS3 is created based on the framework S4, the algorithm Clustree and the database Postgresql. A data stream TR3 is input into the system RS3 at a data throughput TH3=2Mbps. In the system RS3, the data stream TR3 and the task are distributed to N3=2 distributed computation nodes for execution.

The respective performance parameters obtained for the systems RS1 , RS2 and RS3 are listed in Table 2 below.

Table 2 - Performance Parameters

As shown in the above Table 2, in order to accomplish the task, the system RS1 requires a CPU occupation of 50%, a memory occupation of 50%, a framework processing time length of 120s, an algorithm execution time length of 50s, a storage space of 2G bits and a storage time length of 30s; the system RS2 requires a CPU occupation of 40%, a memory occupation of 40%, a framework processing time length of 80s, an algorithm execution time length of 30s, a storage space of 1.5G bits and a storage time length of 20s; and the system RS3 requires a CPU occupation of 60%, a memory occupation of 60%, a framework processing time length of 100s, an algorithm execution time length of 35s, a storage space of 3.2G bits and a storage time length of 40s.

In order to calculate the performance metric associated with each of the systems, some of performance parameters are normalized first. For example, the framework processing time length can be normalized to 900 seconds for S4 and 600 seconds for Storm; the algorithm execution time length can be normalized to 900 seconds for Clustream and 600 seconds for Clustree; the storage space can be normalized to 20 G bits; and the storage time length can be normalized to 300 seconds. Then, the normalized performance parameters can be obtained, as listed in Table 3 below (the parameters of CPU occupation and memory occupation can be converted into decimals directly).

Table 3 - Normalized Performance Parameters

Then, one of a first set of predetermined weights is assigned to each of the normalized performance parameters listed in the row associated with the system RS1 in Table 3. For example, the first set of predetermined weights may contain the following weights to be assigned to the respective normalized performance parameters associated with the system RS1 :

{0.12, 0.08, 0.35, 0.25, 0.12, 0.08}.

A weighted average of the normalized performance parameters associated with the system RS1 is calculated as the performance metric PM1 : PM1 =(0.12^*0.5+0.08^*0.5+0.35^*0.134+0.25^*0.056+0.12^*0.1 +0.08^*0.1 )/ (0.12+0.08+0.35+0.25+0.12+0.08) = 0.1809

Similarly, one of a second set of predetermined weights is assigned to each of the normalized performance parameters listed in the row associated with the system RS2 in Table 3. For example, the second set of predetermined weights may contain the following weights to be assigned to the respective normalized performance parameters associated with the system RS2:

{0.1 , 0.1 , 0.29, 0.31 , 0.1 , 0.1 }.

A weighted average of the normalized performance parameters associated with the system RS2 is calculated as the performance metric PM2:

PM2=(0.1 ^*0.4+0.1 ^*0.4+0.29^*0.134+0.31 ^*0.034+0.1 ^*0.075+0.1 ^*0.067)/

(0.1 +0.1 +0.29+0.31 +0.1 +0.1 ) = 0.1436

Further, one of a third set of predetermined weights is assigned to each of the normalized performance parameters listed in the row associated with the system RS3 in Table 3. For example, the third set of predetermined weights may contain the following weights to be assigned to the respective normalized performance parameters associated with the system RS3:

{0.12, 0.08, 0.35, 0.25, 0.12, 0.08}.

A weighted average of the normalized performance parameters associated with the system RS3 is calculated as the performance metric PM3:

PM3=(0.12^*0.6+0.08^*0.6+0.35^*0.112+0.25^*0.058+0.12^*0.16+0.08^*0.134)/

(0.12, 0.08, 0.35, 0.25, 0.12, 0.08) = 0.20362

Here, a smaller value of the performance metric represents a better performance of the system. Therefore, the real-time system RS2 having the lowest

performance metric value can be determined as the most suitable system for executing the task.

Fig. 2 is a schematic diagram of an apparatus 200 for evaluating a performance of a real-time system according to an embodiment of the disclosure.

As shown in Fig. 2, the apparatus 200 includes a communication unit 210 for communicating with other entities such as a configuration server and a number of distributed computation nodes. The apparatus 200 further includes an

arrangement 220 for implementing the method described above with reference to Fig. 1 . The apparatus 200 may further comprise one or more memories 230 and one or more further functional units 240.

The arrangement 220 can be implemented as a pure hardware solution or as a combination of software and hardware, e.g., by one or more of: a processor or a micro processor and adequate software and memory for storing of the software, a Programmable Logic Device (PLD) or other electronic component(s) or processing circuitry configured to perform the actions described above, and illustrated, e.g., in Fig. 1 . The arrangement 220 may be implemented and/or described as follows.

Referring to Fig. 2, the apparatus 200 includes a creating unit 221 for creating a first real-time system and a second real-time system.

The apparatus 200 further includes a first causing unit 222 for causing the first real-time system to execute a task, so as to obtain a plurality of first performance parameters.

The apparatus 200 further includes a second causing unit 223 for causing the second real-time system to execute said task, so as to obtain a plurality of second performance parameters. The apparatus 200 further includes a calculating unit 224 for calculating a first performance metric based on the plurality of first performance parameters and a second performance metric based on the plurality of second performance parameters. It is to be noted here that the respective units 221 -224 can be implemented as software components or hardware circuits.

In an embodiment, the first real-time system is created by selecting a first real-time framework from a plurality of real-time frameworks, a first data processing algorithm from a plurality of data processing algorithms and a first database from a plurality of databases and the second real-time system is created by selecting a second real-time framework from the plurality of real-time frameworks, a second data processing algorithm from the plurality of data processing algorithms and a second database from the plurality of databases. In an embodiment, the first causing unit 222 is configured to: generate a first data stream for the first real-time system at a first data throughput and cause the first real-time system to execute said task using a first distributed computation resource. The second causing unit 223 is configured to: generate a second data stream for the second real-time system at a second data throughput and cause the second real-time system to execute said task using a second distributed computation resource.

In an embodiment, the first distributed computation resource comprises a first number of distributed computation nodes in the first real-time system, and wherein the second distributed computation resource comprises a second number of distributed computation nodes in the second real-time system.

In an embodiment, each of the plurality of first performance parameters and the plurality of second performance parameters is obtained by using trace system scripts.

In an embodiment, each of the plurality of first performance parameters and the plurality of second performance parameters comprises one or more of a Central Processing Unit (CPU) occupation, a memory occupation, a framework

processing time length, an algorithm execution time length, a storage space and a storage time length required to accomplish the task.

In an embodiment, the calculating unit 224 is configured to: assign one of a first set of predetermined weights to each of the plurality of first performance

parameters; calculate, as the first performance metric, a weighted average of the plurality of first performance parameters; assign one of a second set of

predetermined weights to each of the plurality of second performance parameters; and calculate, as the second performance metric, a weighted average of the plurality of second performance parameters. In an embodiment, the apparatus 200 further includes a comparing unit (not shown) for comparing the first performance metric and the second performance metric and a selecting unit (not shown) for selecting one of the first real time system and the second real-time system based on the comparison.

Fig. 3 shows an embodiment of an arrangement 300 which may be used in the apparatus 200. The arrangement 300 includes a processor 310, e.g., a Digital Signal Processor (DSP). The processor 310 may be a single unit or a plurality of units to perform different actions of procedures described herein. The

arrangement 300 may also comprise an input/output unit 330 for

receiving/sending signal from/to other entities.

Furthermore, the arrangement 300 includes at least one computer program storage product 320 in the form of a non-volatile or volatile memory, e.g., an Electrically Erasable Programmable Read-Only Memory (EEPROM), a flash memory and a hard drive. In other words the computer program storage product 320 comprises or is, irrespective of embodiment, a non-transitory computer readable medium. The computer program storage product 320 includes a computer program 340, which includes code/computer readable instructions, which when executed by the processor 310 in the arrangement 300 causes the arrangement 300 or the apparatus 200 in which it is included to perform the actions, e.g., of the procedure described earlier in conjunction with Fig. 1 .

The computer program product 340 may be configured as a computer program code structured in computer program modules 340A - 340D.

Hence, in an exemplifying embodiment where the arrangement 300 is used in the apparatus 200, the code in the computer program of the arrangement 300 includes a creating part 340A for creating a first real-time system and a second real-time system. The code in the computer program of the arrangement 300 further includes a first causing part 340B for causing the first real-time system to execute a task, so as to obtain a plurality of first performance parameters. The code in the computer program of the arrangement 300 further includes a second causing part 340C for causing the second real-time system to execute said task, so as to obtain a plurality of second performance parameters. The code in the computer program of the arrangement 300 further includes a calculating part 340D for calculating a first performance metric based on the plurality of first performance parameters and a second performance metric based on the plurality of second performance parameters.

The computer program modules could essentially perform the actions of the flow illustrated in Fig. 1 , to emulate the arrangement 220 in the apparatus 200. In other words, when the different computer program modules are executed in the processor 310, they may correspond, e.g., to the units 221 - 224 of Fig. 2.

Although the code means in the embodiments disclosed above in conjunction with Fig. 3 are implemented as computer program modules which when executed in the processing unit causes the device to perform the actions described above in conjunction with the figures mentioned above, at least one of the code means may in alternative embodiments be implemented at least partly as hardware circuits.

The processor may be a single CPU (Central processing unit), but could also comprise two or more processing units. For example, the processor may include general purpose microprocessors; instruction set processors and/or related chips sets and/or special purpose microprocessors such as Application Specific

Integrated Circuit (ASICs). The processor may also comprise board memory for caching purposes. The computer program may be carried by a computer program product connected to the processor. The computer program product may comprise a computer readable medium on which the computer program is stored. For example, the computer program product may be a flash memory, a

Random-access memory (RAM), a Read-Only Memory (ROM), or an EEPROM, and the computer program modules described above could in alternative embodiments be distributed on different computer program products in the form of memories.

The disclosure has been described above with reference to embodiments thereof. It should be understood that various modifications, alternations and additions can be made by those skilled in the art without departing from the spirits and scope of the disclosure. Therefore, the scope of the disclosure is not limited to the above particular embodiments but only defined by the claims as attached.

Claims

1. A method (100) for evaluating a performance of a real-time

system, comprising:

creating (S110) a first real-time system and a second real-time system;

causing (S120) the first real-time system to execute a task, so as to obtain a plurality of first performance parameters;

causing (S130) the second real-time system to execute said task, so as to obtain a plurality of second performance parameters; and calculating (S140) a first performance metric based on the plurality of first performance parameters and a second

performance metric based on the plurality of second performance parameters.

2. The method (100) of claim 1 , wherein

the first real-time system is created by selecting a first real-time

framework from a plurality of real-time frameworks, a first data processing algorithm from a plurality of data processing

algorithms and a first database from a plurality of databases, and the second real-time system is created by selecting a second real-time framework from the plurality of real-time frameworks, a second data processing algorithm from the plurality of data processing algorithms and a second database from the plurality of

databases.

3. The method (100) of claim 1 , wherein

said causing (S120) the first real-time system to execute a task comprises: generating a first data stream for the first real-time system at a first data throughput and causing the first real-time system to execute said task using a first distributed computation resource, and

said causing (S130) the second real-time system to execute said task

comprises: generating a second data stream for the second real-time system at a second data throughput and causing the second real-time system to execute said task using a second distributed computation resource.

The method (100) of claim 1 , wherein the first distributed computation resource comprises a first number of distributed computation nodes in the first real-time system, and wherein the second distributed computation resource comprises a second number of distributed computation nodes in the second real-time system.

5. The method (100) of claim 1 , wherein each of the plurality of first performance parameters and the plurality of second performance parameters is obtained by using trace system scripts.

6. The method (100) of claim 1 , wherein each of the plurality of first performance parameters and the plurality of second performance parameters comprises one or more of a Central Processing Unit (CPU) occupation, a memory occupation, a framework processing time length, an algorithm execution time length, a storage space and a storage time length required to accomplish the task.

7. The method (100) of claim 1 , wherein

said calculating (S140) the first performance metric based on the plurality of first performance parameters comprises:

assigning one of a first set of predetermined weights to each of the plurality of first performance parameters; and

calculating, as the first performance metric, a weighted average of the plurality of first performance parameters, and

said calculating (S140) the second performance metric based on the plurality of second performance parameters comprises:

assigning one of a second set of predetermined weights to each of the plurality of second performance parameters; and calculating, as the second performance metric, a weighted average of the plurality of second performance parameters.

8. The method (100) of claim 1 , further comprising: comparing the first performance metric and the second performance metric; and

selecting one of the first real time system and the second real-time system based on the comparison.

An apparatus (220) for evaluating a performance of a real-time system, comprising:

a unit (221 ) for creating a first real-time system and a second real-time system;

a unit (222) for causing the first real-time system to execute a task, so as to obtain a plurality of first performance parameters; a unit (223) for causing the second real-time system to execute said task, so as to obtain a plurality of second performance parameters; and

a unit (224) for calculating a first performance metric based on the plurality of first performance parameters and a second performance metric based on the plurality of second performance parameters.

10. The apparatus (220) of claim 9, wherein

the first real-time system is created by selecting a first real-time

framework from a plurality of real-time frameworks, a first data processing algorithm from a plurality of data processing algorithms and a first database from a plurality of databases, and the second real-time system is created by selecting a second real-time framework from the plurality of real-time frameworks, a second data processing algorithm from the plurality of data processing algorithms and a second database from the plurality of databases.

11 . The apparatus (220) of claim 9, wherein

said unit (222) for causing the first real-time system to execute a task is configured to: generate a first data stream for the first real-time system at a first data throughput and cause the first real-time system to execute said task using a first distributed computation resource, and said unit (223) for causing the second real-time system to execute said task is configured to: generate a second data stream for the second real-time system at a second data throughput and cause the second real-time system to execute said task using a second distributed computation resource.

12. The apparatus (220) of claim 9, wherein the first distributed

computation resource comprises a first number of distributed computation nodes in the first real-time system, and wherein the second distributed computation resource comprises a second number of distributed computation nodes in the second real-time system.

13. The apparatus (220) of claim 9, wherein each of the plurality of first performance parameters and the plurality of second performance parameters is obtained by using trace system scripts.

14. The apparatus (220) of claim 9, wherein each of the plurality of first performance parameters and the plurality of second performance parameters comprises one or more of a Central Processing Unit (CPU) occupation, a memory occupation, a framework processing time length, an algorithm execution time length, a storage space and a storage time length required to accomplish the task.

15. The apparatus (200) of claim 9, wherein said unit (224) for

calculating the first performance metric and the second performance metric is configured to:

assign one of a first set of predetermined weights to each of the plurality of first performance parameters;

calculate, as the first performance metric, a weighted average of the plurality of first performance parameters;

assign one of a second set of predetermined weights to each of the plurality of second performance parameters; and

calculate, as the second performance metric, a weighted average of the plurality of second performance parameters.

16. The apparatus (220) of claim 9, further comprising:

a unit for comparing the first performance metric and the second performance metric; and

a unit for selecting one of the first real time system and the second real-time system based on the comparison.

17. An apparatus (200) for evaluating a performance of a real-time system, comprising a processor (220) and a memory (230), said memory (230) comprising instructions executable by said processor (220) whereby said apparatus (200) is operative to: create a first real-time system and a second real-time system; cause the first real-time system to execute a task, so as to obtain a plurality of first performance parameters;

cause the second real-time system to execute said task, so as to obtain a plurality of second performance parameters; and calculate a first performance metric based on the plurality of first performance parameters and a second performance metric based on the plurality of second performance parameters.

18. The apparatus (200) of claim 17, wherein

the first real-time system is created by selecting a first real-time

framework from a plurality of real-time frameworks, a first data processing algorithm from a plurality of data processing algorithms and a first database from a plurality of databases, and the second real-time system is created by selecting a second real-time framework from the plurality of real-time frameworks, a second data processing algorithm from the plurality of data processing algorithms and a second database from the plurality of

databases.

19. The apparatus (200) of claim 17, wherein

said causing the first real-time system to execute a task comprises:

generating a first data stream for the first real-time system at a first data throughput and causing the first real-time system to execute said task using a first distributed computation resource, and

said causing the second real-time system to execute said task comprises: generating a second data stream for the second real-time system at a second data throughput and causing the second real-time system to execute said task using a second distributed

computation resource.

20. The apparatus (200) of claim 17, wherein the first distributed

21 . The apparatus (200) of claim 17, wherein each of the plurality of first performance parameters and the plurality of second performance parameters is obtained by using trace system scripts.

22. The apparatus (200) of claim 17, wherein each of the plurality of first performance parameters and the plurality of second performance parameters comprises one or more of a Central Processing Unit (CPU) occupation, a memory occupation, a framework processing time length, an algorithm execution time length, a storage space and a storage time length required to accomplish the task.

23. The apparatus (200) of claim 17, wherein

said calculating the first performance metric based on the plurality of first performance parameters comprises:

said calculating the second performance metric based on the plurality of second performance parameters comprises:

assigning one of a second set of predetermined weights to each of the plurality of second performance parameters; and

calculating, as the second performance metric, a weighted average of the plurality of second performance parameters.

The apparatus (200) of claim 17, wherein said memory further contains instructions executable by said processor whereby said apparatus is operative to:

compare the first performance metric and the second

performance metric; and

select one of the first real time system and the second real-time system based on the comparison.

A computer program (340) comprising computer readable instructions which, when run on an apparatus for evaluating a performance of a real-time system, cause the apparatus to:

create a first real-time system and a second real-time system; cause the first real-time system to execute a task, so as to obtain a plurality of first performance parameters;

A computer program storage product (320) comprising computer readable storage means storing the computer program (340) according to claim 25.