KR100929912B1 - System and Method for Performance Evaluation of Situational Awareness Server - Google Patents

Abstract

It provides a system and method that can comprehensively evaluate the performance of scalability, processing speed, and stability of a situation-aware server using scenarios. The performance evaluation system of the situation awareness server includes a plurality of evaluation modules, each evaluation module including a scenario generation unit for storing a scenario including a user's behavior and a change in situation, and an event according to the progress of the scenario stored in the scenario generation unit. An event generation unit for outputting an event, a service management unit which receives an event from the event generation unit, delivers the event to the situation awareness server, and virtually performs a service according to a command from the service management unit, and notifies the scenario generation unit of the result. Contains a service agent.

Description

Performance evaluation system and method for context-aware server

The present invention relates to a system and method for evaluating performance of a situation awareness server, and more particularly, to a system and method for comprehensively evaluating performance of a situation awareness server such as scalability, processing speed, and stability using a scenario. .

The present invention is derived from the research conducted as part of the IT growth engine technology development project of the Ministry of Information and Communication and the Ministry of Information and Communication Research and Development. [Task Management Number: 2006-S-026-02, Title: URC Server Framework for Active Services] Development of the URC Server Framework for Proactive Robotic Services].

In recent years, active researches on ubiquitous related technologies such as ubiquitous computing and ubiquitous network have been actively conducted in the information and communication field. Ubiquitous means that users can easily get the information they want anytime / anywhere. In the coming ubiquitous era, smart objects with computing and communication capabilities will be able to recognize and adapt to dynamic environment changes, that is, context-aware.

'Situation' is defined as 'information that defines the state of an entity existing in the real world', where an entity means an interaction between a person, a place, or a person and a service. In addition, 'situational awareness' refers to acquiring changes in the user's current location, behavior and tasks, temperature, time, and emotional state from the surrounding environment. The context awareness service refers to providing a service suitable for a user's situation based on the context information as described above. This context-aware service recognizes the user's situation by collecting and exchanging the situation information, and provides the user with the appropriate service through the process of interpretation and reasoning, and will emerge as the main service in the ubiquitous era in the future. It is clear.

In the ubiquitous era, the major objects and devices around everyday life will have the computing and communication ability and the ability to recognize the surroundings.In addition, intelligent communication between objects and devices will allow users to comfortably receive appropriate and useful services for the situation. Could be.

Two representative methods of constructing a context aware framework for providing such services are known. First, a method of providing a service desired by a user by collaborating with each other such as various sensors or actuators installed in a situation. Since the terminal is aware of its state and determines which other terminal to associate with in order to provide a service required by the user, the terminal should have excellent performance. Therefore, the price of the terminal rises and a lot of load is applied.

Second, the situation-aware server is installed, and the situation-aware server is responsible for a task requiring a lot of load, and a method of requiring low performance in the terminal. The context aware server collects and manages the sensed data and derives the service required by the user. The context aware server provides context information to the context-aware application and allows the context aware application to control the actuator. Research on such a situation-aware server is actively in progress, and an example of a conventional situation-aware server is as follows.

The Scarlet-Context-aware infrastructure of the Illinois Institute of Technology focuses on how to provide contextual information to context-aware applications. In particular, research has been conducted to transfer contextual information between heterogeneous platforms. And Gaia of Illinois University allows context-aware applications to obtain and infer a variety of contextual information by the context-aware service structure. In Gaia, a context provider collects situations from sensors or data sources and provides them to context applications. In particular, it has a context synthesizer to deduce context information collected by context providers as contextual information of higher concepts. It can be abstracted and provided to context-aware applications. In addition, SOCAM at the University of Singapore facilitates the development of context awareness services and systems. SOCAM has a context interpreter and provides logical posture services for situations.

Various researches are being conducted on the context aware server itself, including the above. However, the context aware server is responsible for the processing of a large number of terminals, and also requires a large number of operations at the same time in order to collect the context information collected from various sources to provide a user with a service suitable for the situation in real time. However, unlike active research on the situation awareness server itself, there is a lack of research on evaluating the performance of the situation awareness server.

The present invention is to solve the above problems, to provide a system and method that can comprehensively evaluate the performance, such as scalability, processing speed, stability of the context aware server using a virtual service agent based on the scenario The purpose.

In order to achieve the above object, a performance evaluation system of a situation recognition server according to the present invention includes a plurality of evaluation modules, and the evaluation module includes a scenario generation unit for storing a scenario including a user's behavior and a change of situation, and a scenario According to the progress of the scenario stored in the generation unit, the event generation unit for outputting the event, the service receiving unit receives the event from the event generation unit, and delivers the event to the situation-aware server, and virtually performs the service according to the instructions from the service management unit And a virtual service agent for notifying the scenario generator of the result.

Preferably, the virtual service agent implements a plurality of virtual devices, and notifies the scenario generation unit of the completion of the operation of the virtual device corresponding to the instruction from the service management unit. More preferably, the virtual service agent stores operation time according to the operation of each virtual device.

Also, preferably, the service manager verifies the validity of the event input from the event generator and transmits the event to the situation recognition server, and the plurality of evaluation modules include a scenario generator, an event generator, a service manager, and a virtual service agent. Share one or more.

In addition, in the method for evaluating the performance of the situation recognition server by the situation recognition server performance evaluation system having a plurality of evaluation modules according to the present invention, each evaluation module includes: (a) a scenario generation unit for changing a user's behavior and situation; Storing a scenario including a; (b) outputting an event according to the progress of the scenario; and (c) receiving, by the service manager, an event and delivering the event to a situation recognition server; And (d) the virtual service agent performing the service virtually according to the instruction from the service management unit and notifying the scenario generation unit of the result.

Preferably, in step (d), the scenario generation unit notifies the completion of the operation of the virtual device corresponding to the instruction from the service management unit, and step (d), the operation duration time stored in correspondence with the operation of each virtual device. Based on the determination of the completion of the operation of the virtual device, step (c) validates the input event and delivers it to the situation recognition server.

In addition, preferably, the scalability of the situation awareness server is evaluated by measuring the degree to which the performance of the situation awareness server decreases as the number of the plurality of evaluation modules increases, and the number of scenarios that the situation awareness server performs for a predetermined time is determined. The measurement evaluates the processing speed of the situation awareness server.

The situation that the situation awareness server needs to collect and process continuously changes, and in order to reflect the situation change on the situation awareness server immediately, the processing speed of the situation change of the situation awareness server must be fast. In addition, the context aware server communicates with a large number of devices, and is used in connection with a plurality of context aware applications, so it must be easy to expand and provide a stable service continuously. Therefore, scalability, processing speed, and stability of the context aware server are important measures of performance.

According to the present invention, it is possible to evaluate the scalability, processing speed, stability, etc. of the situation awareness server comprehensively and inexpensively and comprehensively, so that it can be used as an important tool in the development of the situation awareness server.

DETAILED DESCRIPTION Hereinafter, exemplary embodiments will be described with reference to the accompanying drawings to help understand the present invention. The following examples are provided to more easily understand the present invention, and the present invention is not limited by these examples.

1 is an overall configuration diagram including a performance evaluation system of a situation recognition server according to the present invention.

The context aware server 200 recognizes the situation and provides a service through data exchange with the context aware applications 221, 222, 223, and 224 using the context aware server manager 210. In addition, the situation recognition server performance evaluation system 100 according to the present invention is connected to the situation recognition server 200 to evaluate the performance of the situation recognition server 200.

First, the configuration of the situation recognition server 200 will be described. The context recognition server 200 is connected to various terminals, for example, a TV 231, an RFID sensor 232, a microphone 233, and a speaker 234. Each terminal is connected to the context aware applications 221, 222, 223, and 224 in the context aware server 200 to deliver a sensed value or receive instructions to provide a service. For example, in the case of the TV 231, when the TV is turned on, the TV 231 transmits the information that the TV is turned on to the context aware application 221, and the context aware application 221 transmits the information to the context aware server manager 210. do. On the contrary, when the instruction for turning off the TV is transmitted from the situation recognition server manager 210 to the situation recognition application 221, the situation recognition application 221 transmits the instruction to the TV 231 so that the TV 221 is turned off. The context recognition server 200 decomposes and fuses the context information to derive new context information, and provides context information to the context awareness applications 221 to 224. The context aware application receives an event that stores context information from the context aware server and provides a service suitable for the context.

The context aware applications 221 ˜ 224 are connected to the terminals 231 ˜ 234 and are subjects that use context information in association with the context aware server manager 210. It is a kind of program that receives an event from the context aware server management unit 210 and performs a given task according to the event, and issues a command to an actuator installed in the environment or receives context change information from a terminal and stores the situation aware server management unit 210. The information will be corrected. In FIG. 1, the context recognition applications 221 ˜ 224 are installed in the context awareness server 200, but may be installed outside the context awareness server 200 or may be installed inside each terminal.

In FIG. 1, four terminals have been described as an example, but the situation recognition server 200 is actually connected to various terminals, the number of connected terminals is increasing, and further, the situation information from the terminals installed at a plurality of places is obtained. It is necessary to collect and analyze the situation information to give instructions to the terminal to provide a service suitable for the situation information in real time, so a large number of operations are required. Therefore, in order to provide a stable service, it is essential to evaluate the performance of the situational awareness server 200. In the prior art, there is no special study on such performance evaluation.

The configuration of the situation recognition server performance evaluation system 100 according to the present invention will be described. The situation recognition server performance evaluation system 100 has n evaluation modules 100-1, 100-2, and 100-n. For example, the evaluation module 1 100-1 includes a scenario generator 110, an event generator 120, a service manager 130, and a virtual service agent 140. There is no particular limitation on the number of evaluation modules, and each evaluation module may operate independently of each other or in association with each other. Some components of the evaluation module may be shared between the evaluation modules. Although not shown, the control and calculation unit controls the n evaluation modules, aggregates and analyzes the performance evaluation results, and notifies the evaluator of the evaluation results.

The scenario generating unit 110 transmits a scenario or an event to the event generating unit 120 according to the stored scenario so that the event generating unit 120 generates an event, and also changes the situation from the virtual service agent 140. Receive the message for the following scenario. Scenarios here include changes in user behavior and situations. An example of creating a scenario stored in the scenario generating unit 110 will be described. Assume that a user performs the following actions sequentially.

1. The user enters the room.

2. Turn on the TV by saying “TV”.

3. Turn off the TV by saying "TV off."

4. The user leaves the room.

The TV 231, the RFID sensor 232, the microphone 233, and the speaker 234 shown in FIG. 1 are installed in the room, and the situation recognition application 224 connected to the speaker 234 is used to voice the situation change. If synthesis can tell, the user's scenario is determined as follows:

Scenario 1. A user enters a room.

Scenario 2. A voice saying "Hello OOO" is made through the speaker.

Scenario 3. Turn on the TV by saying "TV".

Scenario 4. A voice is heard through the speaker.

Scenario 5. Turn off the TV by saying "TV off."

Scenario 6. A voice is heard through the speaker.

Scenario 7. The user leaves the room.

This scenario represents a series of processes that the situation awareness server 200 actually performs in operation. The determined scenario is stored in the scenario generator 110, and the event generator 120 generates an event representing a virtual situation according to the determined scenario. Of course, the event may be prepared in advance according to the scenario in the scenario generator 110 and delivered to the event generator 120. Meanwhile, the scenario generator 110 may store only an event instead of a scenario, or the event generator 120 may be part of the scenario generator 110.

You create an event to change the situation, and when you notice the change, you will move on. Therefore, according to the example of the scenario described above, the operations performed by the scenario generator 110 in sequence are as follows.

Task 1. Raise the UserEntered (USER_ID) event.

Task 2. Wait for the completion of voice speech

Task 3. Raising the SpeechReceived ("Tivory") Event

Task 4. Wait for the TV to turn on

Task 5. Raising the SpeechReceived ("TV Off") Event

Task 6. Wait for the TV to turn off

Task 7. Raising the UserLeft (USER_ID) Event

Four event-related tasks (tasks 1, 3, 5, and 7) among the above tasks are events generated by the scenario generator 110 through the event generator 120. The event generator 120 generates a virtual situation event and transmits the event to the service manager 130 according to a scenario or an event received from the scenario generator. By the event, the situation change can be delivered to the situation recognition server 200 through the service manager 130, and the situation recognition server 200 transmits the situation change according to the delivered event to the service manager 130. Events include, for example, the user's situation (location, name, voice, body temperature, blood pressure, pulse rate, etc.), spatial situation, time situation, environmental situation, and the like.

The service manager 130 transmits the event transmitted from the event generator 120 to the situation recognition server 200, at which time the service manager 130 verifies the validity of the event. For example, if an RFID sensor is installed to recognize a user's location, if the user is near the RFID sensor, the RFID sensor continues to send a message that the user has been recognized. The service manager 130 filters redundant and unnecessary information by validating these events. The scenario generator 110 generates an event using the event generator, and the service manager 130 processes the event by validating the generated event and transmits the event to the situation recognition server 200.

And, the task (tasks 2, 4, 6) related to three waits among the above tasks is a procedure in which the scenario generator 110 waits for a signal from the virtual service agent 140 to indicate that the procedure is completed. For example, when the situation recognition server 200 is actually operating, when the user enters the room, the RFID sensor 232 detects this and the user in the situation recognition server manager 210 through the situation recognition application 222. Notify you that you have entered the room. Thereafter, the context awareness server manager 210 instructs the microphone 233 to speak a voice "hello" through the context awareness application 223. Then, the situation recognition server manager 210 waits for a signal from the situation recognition application 223 indicating that voice speech is completed. In the present invention, the actual operation process is implemented by using the virtual service agent 140.

The virtual service agent 140 implements a virtual device of the virtual TV 141, the virtual RFID sensor 142, the virtual microphone 143, and the virtual speaker 144 to receive a corresponding virtual signal from the service manager 130. The device is simulated to operate, and the scenario generator 140 is notified of the completion of the operation. In order to accurately evaluate the performance of the context aware server, a signal transceiving process and a time required with the virtual terminals should be considered. The present invention implements the virtual service agent 140 without directly accessing the terminals to the context aware server 200. You can perform a satisfactory performance evaluation simply and inexpensively. The virtual service agent 140 will be described with reference to FIG. 2.

2 is a conceptual diagram illustrating a virtual service agent. The left side of the arrow in Fig. 2 shows the configuration of the actual device. For example, the TV service agent 331 is connected to the TV and is responsible for transmitting and receiving signals with the TV. When two TVs of the TV1 231-1 and the TV2 231-2 are connected, the TV service agent is connected. The TV1 driver 2311 and the TV2 driver 2312 are provided between and. The actual device includes terminals 231-1 and 231-2, which are physical devices, and drivers 2311 and 2312 that operate the terminals. Since the situation awareness server uses a plurality of terminals manufactured by various manufacturers, a TV service agent connected through the terminals TV1 and TV2 and the drivers 2311 and 2312 in order to use the various terminals in a common manner ( 331).

However, since installing the actual terminals for the performance evaluation of the situation awareness server requires a lot of resources and space, in the present invention, the virtual service agent (see the right side of FIG. 140 may be implemented to perform performance evaluation as if the actual device was operating. The virtual service agent 140 may be connected to various types of devices capable of accessing a situation recognition server, such as a virtual TV service agent 141, a virtual RFID sensor service agent 142, a virtual microphone service agent 143, and a virtual speaker service agent 144. And virtual service agents for all of these. The virtual service agent 142 stores the operation time required according to the operation of each implemented virtual device, and after completion of the operation time for the completion of the operation of the virtual device corresponding to the command from the service management unit 130, the scenario. The generation unit 110 is notified. A procedure performed by the situation recognition server performance evaluation system described with reference to FIGS. 1 and 2 according to the scenario and the work order described above will be described with reference to FIG. 3.

3 is a flowchart for explaining an example of evaluating the performance of the situation awareness server. The arrows indicate the flow of signals, and the downward order indicates the process of subordinated. The scenario generator 110, the event generator 120, the virtual service agent 140, and the service manager 130 are included in an evaluation module installed in the situation recognition server performance evaluation system 100. In addition, the context aware server manager 210 and the context aware application 220 are installed in the context aware server 200.

First, the scenario generator 110 transmits the scenario 1 that “the user enters the room” to the event generator 120. The event generator 120 generates a UserEntered event (task 1) and delivers it to the service manager 130, and the service manager 130 transmits the validity of the event to the situation recognition server manager 210. The context aware server manager 210 interprets the UserEntered event, stores the changed location of the user in the context aware store, and delivers the UserEntered event to the corresponding context aware application 220. Here, the corresponding situation awareness application is the situation awareness application 224 connected to the speaker 234 in FIG.

The context-aware application 220 actually needs to request the speaker to speak a "hello" voice according to the UserEntered event, and to request the voice service from the virtual service agent 140 without operating the actual device for performance evaluation. do. That is, the context aware application 220 requests the voice of "hello" to the virtual service agent 140 through the context aware server manager 210 and the service manager 130. When the virtual service agent 140 notifies the scenario generation unit 110 that the voice has ended after a predetermined time required for the voice utterance according to the voice utterance request, scenario 2 and task 2 are completed.

Next, the scenario generator 110 transmits the scenario 3 to the event generator 120 that the user says 'TV'. The event generator 120 generates a SpeechReceived event (operation 3) and delivers it to the service manager 130, and the service manager 130 transmits the validity of the event to the situation recognition server manager 210. The context aware server manager 210 transmits the SpeechReceived event to the context aware application 220. The context aware application 220 requests "TVON" to the virtual service agent 140 through the context aware server manager 210 and the service manager 130. When the virtual service agent 140 notifies the scenario generator 110 that the TV is turned on after a predetermined time required for the TVON and the voice "Turn on the TV" according to the TVON request, scenario 4 and task 4 are completed. do. Similar sequential procedures are followed for the following scenarios 5-7 and 5-7.

The above example illustrates a procedure in which one evaluation module 100 performs performance evaluation according to one scenario. The performance evaluation system according to the present invention includes n evaluation modules as shown in FIG. 1 and performs performance evaluation through signal transmission and reception with a situation recognition server according to each scenario. In addition, performance evaluation may be performed by connecting the plurality of situation recognition server performance evaluation systems 100 to one situation recognition server 200.

The present invention evaluates the scalability, processing speed, and stability of the situation recognition server 200 by using n evaluation modules provided in the situation recognition server performance evaluation system 100. While increasing the number of evaluation modules, by measuring the degree to which the performance of the situation recognition server 200 decreases according to the weighted load, it is possible to determine the number of evaluation modules that can be stably performed by the situation recognition server scalability Evaluate. Then, the situation recognition server evaluates how many scenarios can be executed in a certain period of time to evaluate the processing speed (throughput). In addition, stability is assessed by measuring the performance of complex scenarios that are infinitely repeated or require long periods of time.

The situation recognition server performance evaluation system 100 according to the present invention is divided into evaluation module units, so that it is easy to increase or decrease the number of evaluation modules and to freely change or change scenarios. In addition, the same evaluation module can be used to compare and evaluate multiple situational awareness servers, and to derive objective data on the minimum performance that the situational awareness server should have.

The system and method for evaluating the performance of the situational awareness server according to the present invention can perform various types of performance evaluation on the situational awareness server based on a scenario without connecting actual devices. By evaluating the performance, we will make a significant contribution to the research of the context aware server.

1 is an overall configuration diagram including a performance evaluation system of a situation awareness server according to the present invention,

2 is a conceptual diagram illustrating a virtual service agent.

3 is a flowchart for explaining an example of evaluating the performance of the situation awareness server.

Claims (11)

As a performance evaluation system of the situation awareness server, Including a plurality of evaluation modules, The evaluation module, A scenario generator configured to store a scenario including a user's behavior and a change of situation; An event generator for outputting an event according to the progress of the scenario stored in the scenario generator; A service manager which receives an event from the event generator and delivers the event to a context aware server; And A virtual service agent that virtually performs a service according to an instruction from the service manager and notifies the scenario generator to the result; And the plurality of evaluation modules share one or more of the scenario generator, the event generator, the service manager, and the virtual service agent. The method according to claim 1, The virtual service agent implements a plurality of virtual devices, and notifies the scenario generation unit of the completion of the operation of the virtual device corresponding to the instruction from the service management unit. The method according to claim 2, The virtual service agent is a performance evaluation system of the situation awareness server, characterized in that for storing the operation time required according to the operation of each virtual device. The method according to claim 1, The service manager, the performance evaluation system of the situation recognition server, characterized in that to verify the validity of the event input from the event generation unit to the situation recognition server. delete As a method of evaluating the performance of the situation awareness server by the situation recognition server performance evaluation system having a plurality of evaluation modules, each evaluation module includes: (a) storing, by the scenario generation unit, a scenario including a change in behavior and a situation of a user; (b) outputting an event as the scenario progresses; (c) receiving, by the service manager, an event and delivering the event to a context aware server; And (d) virtual service agent performing a service virtually according to a command from the service management unit and notifying the scenario generation unit of the result; And evaluating the scalability of the situation awareness server by measuring a degree to which the performance of the situation awareness server decreases as the number of the plurality of evaluation modules increases. The method according to claim 6, In step (d), And notifying the scenario generation unit of the completion of the operation of the virtual device corresponding to the instruction from the service management unit. The method according to claim 7, In the step (d), it is determined that the completion of the operation of the virtual device based on the time required for the operation stored in response to the operation of each virtual device. The method according to claim 6, The step (c), the method of evaluating the performance of the situation recognition server, characterized in that the validity of the input event delivered to the situation recognition server. delete The method according to claim 6, And evaluating the processing speed of the situation awareness server by measuring the number of scenarios executed by the situation awareness server for a predetermined time.

Images