KR100730635B1 - System and method for service-oriented automatic remote control, remote server and remote control agent - Google Patents

Abstract

A system and a method for performing service-oriented automatic remote control, and a remote server and a remote control agent thereof are provided to enable a user to receive a service providing a remotely controllable device connected to Internet at anytime and anywhere by automating remote control in a service-oriented way based on a semantic network technology and an AI(Artificial Intelligence) planning technology. A remote control agent(200) requests the service to the remote device. The remote server(300) controls the remote device based on remote device/service information by a service request of the remote control agent, and includes a control knowledge-base(310) and a device agent(320) performing the requested service by controlling the remote device. The controlling knowledge repository includes a knowledge-base(317) storing the remote device/service information, a search service module(311) searching the information corresponding to the service request from the knowledge-base, and an inferring module(313) generating a planning domain by generating/using an inference model for performing the requested service based on device/domain knowledge details received from the search service module and generating a service plan by setting an object task.

Description

Service-oriented automatic remote control system, remote control method, remote server and remote control agent {SYSTEM AND METHOD FOR SERVICE-ORIENTED AUTOMATIC REMOTE CONTROL, REMOTE SERVER AND REMOTE CONTROL AGENT}

1 is a block diagram of a conventional web-based remote control system.

2 is a block diagram of a remote control system according to an embodiment of the present invention.

3 is a detailed configuration diagram of a remote control system according to an embodiment of the present invention.

4 is a flowchart of a remote control method according to an embodiment of the present invention.

The present invention relates to a remote control system, a remote control method, a remote server and a remote control agent, and more particularly, a remote control system, a remote control method, a remote server and a remote control agent to control a remote device to perform a target service. It is about.

Thanks to the development of computer networking technology, various researches and technical developments on internet-based remote control systems have been conducted. These technologies focus primarily on enabling users to remotely control devices connected to the network, such as mobile robots, digital home appliances and unmanned mobile vehicles, via the Internet. It also leverages the power of Internet technology to enable users to control their networked devices whenever they want, wherever they are in the world, enabling them to be used in a variety of fields such as industry, education, entertainment, entertainment and security. To provide.

Recently, as the World Wide Web technology is widely used, remote control systems using the same have been developed. All of these systems are based on the Hyper Text Transport Protocol (HTTP) to take advantage of Common Gateway Interface (CGI) or Java technology to control remote devices. Examples include Mercury, the first remote excavation system developed at the University of Southern California, Xaiver, an indoor mobile robot at Carnegie Mellon Univ., And Roger Williams University. PumaPaint, a web-based painting system developed by Univ.

In these systems, each control command provides behavior-based control of the remote. Examples of remotely controlled robotic arms on the web include "OPEN-GRIPPER", "GRASP", "LIFT-UP", "LIFT-DOWN", and "Movement". You can assume a control command like "(MOVE)".

More specifically, as shown in the configuration diagram of the conventional web-based remote control system of Figure 1, while the user monitors the remote robot arm 130 through the real-time feedback procedure 112 based web camera 140 The control command and control parameters in the proper order must be manually selected and transmitted to the web server 110 performing the robot arm device control procedure 111 through the user interface 101 on the web browser 100.

For example, in order to perform a remote "CLEAR-0BJECT" task using a robot arm, the user checks the target and location to be moved and the state of the robot arm while viewing the web camera and sequentially executes the following commands. Should be done as In general, to perform a task, "move arm to target", "lower", "open forceps", "pinch", "lift", "move arm to target position", "lower", "open forceps" "," Such as "lift" must be processed continuously, this process is complicated, and there is a problem that errors are likely to occur in the process.

Therefore, the technical problem to be achieved by the present invention is to enable the service-oriented automation of the remote control system processed by the user operation.

According to another aspect of the present invention, a remote control system includes a remote control system for performing a service by controlling a remote device, the remote control agent requesting a service for the remote device; And a remote server controlling the remote device based on the remote device information and the service information according to the service performance request of the remote control agent.

In addition, a remote control method according to an aspect of the present invention for achieving the above technical problem, a remote control method for performing a target service by controlling a remote device, a) remote device information and services to perform the target service Searching for information; b) creating a service plan based on the retrieved remote device information and the service information; And c) creating and executing a remote device control process via the service plan.

In addition, the remote control method according to an aspect of the present invention for achieving the above technical problem, is connected to a remote control agent, the remote control of the remote server to perform the target service by controlling the remote device in response to the request of the remote control agent A control method, comprising: a) searching for remote device information and service information to perform a target service; b) creating a service plan based on the retrieved remote device information and the service information; And c) controlling the remote device according to the remote device control process generated through the service plan to perform the target service.

In addition, the remote server according to the characteristics of the present invention for achieving the above-described technical problem, is connected to the remote control agent, the remote server as a remote server to control the remote device according to the request of the remote control agent to perform the target service, Control knowledge storage for storing device information and service information; And a device control agent controlling the remote device based on the stored remote device information and the service information to perform the requested service.

In addition, a remote control agent according to an aspect of the present invention for achieving the above technical problem is a remote control agent requesting a remote device to perform a target service through a remote server, the remote device information and service information from the remote server Search module for searching for; A process configuration module for generating a device control process through a service plan generated at the remote server based on the remote device information and the service information; And a process execution module that executes a device control process so that the remote server controls the remote device to perform a target service.

Hereinafter, the present invention will be described in detail with reference to the accompanying drawings.

In the following description, for convenience of description, a remote control system according to an embodiment of the present invention uses a semantic web language, OWL (Web Ontology Language) and OWL-S (OWL for Services) as a web-based remote control system. Although a case will be described as an example, it is obvious that the scope of the present invention is not necessarily limited to the web-based system and the OWL and OWL-S. The scope of the present invention includes various network environments applicable as obvious to those skilled in the art based on the content described in this specification, and the language most suitable for each environment.

As shown in FIG. 2, the remote control system according to the embodiment of the present invention includes a remote control agent 200 and a remote server 300.

For example, when the remote control system according to the embodiment of the present invention is a web-based remote control system, semantic web means meanings of functions and interfaces of a web-based remote device and a sensor to be controlled by the system. It can be described as a language of Web Ontology Language (OWL) and OWL-S (OWL for Services).

In this case, OWL is a web ontology language that is not only used to display information to a person but also a language designed to be used to implement an application that can directly process information. OWL is rich in vocabulary and formal semantics, which makes it excellent for creating machine-interpretable web content, and includes three sublanguages with different expressions: OWL Lite, OWL, and OWL FULL.

OWL-S also defines a service ontology that defines a web service, a service profile ontology that expresses what the service does, and how to describe the process of the service in detail. works may include a service model ontology and a service grounding ontology describing how to access it. As a standardized form for describing web services, the upper ontology of OWL-S is used, and each web service is created using the classes and attributes defined in the upper ontology of OWL-S.

In this case, the service profile briefly describes the web service structure, the functions performed by the service, and the service features. The service profile includes inputs, preconditions, outputs provided after the service, and Define the effect. This allows you to define what the web service does. The service model defines a process flow, a composition hierarchy, and a process definition, and specifically defines IOPE (input, output, precondition, and effect) of each process. In addition, service grounding defines a communication protocol, a port number, etc., and describes all the information necessary for the web service to actually perform, such as data structures, protocols, and physical addresses of services. It handles the actual operation of web services, and software agents can access and run web services through these defined protocols.

When the remote control agent 200 receives the service request from the user through the user interface, the remote control agent 200 infers based on the OWL and OWL-S specifications to configure and execute the required service-oriented automatic control process. At this time, the remote control agent communicates with the device control service and the sensor monitoring service on the remote server 300 (Web server according to this example) through a simple object access protocol (SOAP), which is a web messaging protocol, based on the generated control process. The service requested by the user can be executed.

Hereinafter, a more specific embodiment will be described with reference to FIG. 3.

The remote control system according to the embodiment of the present invention may include a remote control agent 200 and a remote server 300, and the remote server 300 may include a control knowledge store and a device control agent. At this time, the control knowledge store and the device control agent may be provided separately in a plurality of servers according to its function. Each component is interconnected based on a network such as the Internet.

The remote control agent 200 may include an interface module 201, a search module 205, a process configuration module 203, a process execution module 207, and a communication module 209.

The communication module 209 may communicate with a remote process, including a web service protocol stack having one or more of SOAP, eXtensible Markup Language (XML), and Hyper Text Markup Language (HTTP).

The interface module 201 provides an interface through which a user can input a desired service task.

The search module 205 retrieves knowledge necessary for accomplishing the service task input through the interface module 201 from the control knowledge store 310, which will be described later, on a semantic basis.

The process configuration module 203 constructs a device control process for accomplishing a user request service task based on the knowledge retrieved by the discovery module 205. The process execution module 207 executes the device control process generated by the process configuration module 203 in communication with the device control agent 320 at a remote location, which will be described later, via the communication module 209.

The control knowledge store 310 may include a search service module 311, an inference module 313, a communication module 315, and a knowledge base 317.

The knowledge base 317 includes a domain knowledge base 319 and a device knowledge base 318, such that the domain knowledge base 319 is, for example, an OWL-S Composite Process and a data flow within the process. The domain knowledge for service task configuration described as a flow, and the device knowledge base 318 may, for example, function of a remote device described as an OWL-S Atomic Process and Service Grounding; Stores knowledge of the control interface. In this case, the domain knowledge base 319 and the device knowledge base 318 may be configured as one base or may be separately configured and included in the knowledge base 317.

The search service module 311 provides a semantic based search service for knowledge stored in the knowledge base 317, which is meant for the remote control agent 200 through the communication module 315 included in the control knowledge store 310. Provides control knowledge search service based

The inference module 313 provides an OWL knowledge base query processing function for a semantic based search service.

The device control agent 320 includes a control service module 321 and a communication module 323, and the control service module 321 implements a control function for a remote device or sensor connected to the device control agent 320. Thus, the control service is provided to the remote control agent 200 through the communication module 323.

4 is an overall flowchart of a remote control method according to an embodiment of the present invention. As shown, the remote control process according to the embodiment of the present invention includes a control knowledge search step S100, a planning step S200, and a control process execution step S300.

First, in the control knowledge search step S100, the domain knowledge base 319 and the device knowledge base 318 may be described as OWL-S as described above to store the domain knowledge and the device knowledge as described above.

Next, when a specific service execution request is requested from the user through the interface module 201 included in the remote control agent 200 (S101), a semantic-based control knowledge search process is executed (S103). More specifically, the discovery module 205 of the remote control agent 200 may be configured from the domain knowledge base 319 and the device knowledge base 318, respectively, for the domain knowledge specification, the composite process and the device service specification, and the single process for configuring the control process. Search and data flow specification and service grounding specification (S105, S106, S107, S108). The search service module 311 of the control knowledge store 310 provides the semantic based search service for the knowledge stored in the knowledge base 317 to the remote control agent 200 through the communication module 315.

Examples of a remote home security check service request of a user described by OWL, a domain knowledge specification and a device service specification for a corresponding service are as follows. In this example, the home security inspection service is described as a process of moving a mobile robot equipped with a camera to a designated place in the house, taking a picture of the place, and transmitting the picture to an outside user.

<Example of Home Security Check Service Request>

<Embodiment of home security check service domain knowledge specification searched on a semantic basis according to the service request from domain knowledge base>

<Data flow from CaptrureImage to ImageToBeSent in the composite process "ReportHomeStatus" process in the domain knowledge specification above>

<Embodiment of Robot Drive Device Service Specification Searched from Device Knowledge Base for "MoveTo" Single Service on Domain Knowledge Specification>

<Service Grounding for "RobotMoveToService" Service in the Device Service Specification>

<Web service definition file containing information for executing RobotMoveToService on the service grounding>

When the control knowledge search step S100 as described above is finished, the planning step S200Z is executed. Planning largely includes parsing and inference model generation, planning domain creation / meaning expansion, and planning.

First, the language inference module of the control knowledge repository receives the domain knowledge specification and the device service specification searched in the above-described control knowledge search step and parses each of them to generate an inferable object model (S201 and S203). In this example, we take the domain knowledge specification and device service knowledge specification described in OWL as inputs, and generate an OWL object model that can be parsed and inferred. More specifically, parsing a file written in OWL and executing it using a Java library. Can create Java objects.

Next, the inference module creates a planning domain and extends its meaning. For example, in order to plan using the Hierarchical Task Network (HTN) method, all processes described in OWL must be converted into planning operators and methods. This can be called domain creation. In this case, a parsing inference model may be received as an input to generate a planning method, a planning operator, and an initial state and a target task for performing an actual known planning algorithm (S205, S207, S209, and S211).

In this case, HTN is a type of artificial intelligence planning and is based on a hierarchical task model. The basic idea of HTN-based service configuration is to break down complex tasks into subtasks, and then task decomposition into the simplest level of unit tasks that can be executed directly. For this purpose, the HTN performs planning in units of tasks having a hierarchical structure called an operator and a method. At this time, the operator is mapped to a single process (Atomic Process) of OWL-S as a minimum unit of directly executable tasks. Methods are tasks that can be subdivided into subtasks and are mapped to the OWL-S Composite Process. That is, like the OWL-S hierarchical structure, the HTN also has a hierarchical structure of tasks through methods.

Planning domains are created through the creation of planning methods and planning operators, which follow a domain extension algorithm. In addition, more flexible planning domain generation may be possible through semantic expansion based on subsumpton relation inference of the parsed domain knowledge specification and the device service specification object model (S213).

The following is an example of a planning domain generation and semantic extension algorithm for performing HTN planning.

<Example of Planning Domain Semantic Extension Algorithm for HTN Planning>

That is, a planning domain is created based on the inference of the parsing of the OWL object model of the parsed domain knowledge specification and the device service specification, where S is the device service specification of the OWL-S process model, and K is the OWL-S process model. The domain knowledge specification, extended domain knowledge where K 'is derived through an extension algorithm, where C is a composite process of K' and T is a single process in C, for every single process T in C, If there is a device service that matches T in S, then it provides that device service for T. If there is no device service that exactly matches T, it replaces it with a similar service to extend the planning domain semantics.

The following is an example of a planning domain generation algorithm for HTN planning.

<Example of Planning Domain Generation Algorithm for HTN Planning>

Hereinafter, more specific embodiments of the planning operator generation and planning method generation algorithm will be described.

<Example of HTN Planning Operator Generation Algorithm for Device Service Specification>

<Example of HTN Planning Method Generation Algorithm for Domain Knowledge Specification>

That is, if a service has only output, it is regarded as a service for sensing only. If a service has an effect, it is regarded as a service for performing a specific action, and a planning operator is created. Creates a planning method for the case where the service process type is parallel with a single service, and creates a planning domain.

Next, in the planning stage, the inference module may generate a service plan by performing an appropriate AI planning algorithm (for example, an HTN planning algorithm) by inputting the planning method and the planning operator, the initial value, and the target task generated previously. (S217).

The control process execution step S300 largely includes the detailed steps of generating the control process and executing the control process.

In the control processor generation step, the process configuration module of the remote control agent generates a web service process executable through the actual web protocol by inputting the service plan generated through the above-described planning step. At this time, not only the service plan but also the data flow specification and service grounding specification retrieved from the domain knowledge base and the device knowledge base in the control knowledge search step are needed to create an executable web service process. In addition, the data flow specification and the service grounding specification are respectively reflected in the OWL inference model generated through the OWL parsing and the inference model generation process, so that the data flow and the service grounding information are obtained through the OWL inference model in the actual control process generation process (S205). , S301, S305).

Finally, in the execution of the control process, the process execution module of the remote control agent communicates using the device control service and SOAP using the execution initial value generated through the information input from the user with the initial value input together with the service request. The control process is executed (S303, S307).

What has been described above is only an embodiment of the present invention, and the scope of the present invention is not limited thereto, and various modifications can be made to those skilled in the art. For example, as described above, the scope of the present invention is not limited to the web-based technology, and it is obvious that the present invention can be applied to a technology that assumes various network environments. The scope of the present invention includes all technical components recognized as the claims described below and equivalents thereof.

As described above, in the conventional web-based remote control technology, the user has to perform the unit behavior-based remote control of the device connected to the Internet through the HTTP and HTML-based user interfaces which are the web protocols, respectively. Specifically, in the existing web-based remote control system, in order to perform a desired task, the user had to manually select an appropriate device control command and perform it through a web browser while monitoring the remote device through a web camera or various sensors.

However, according to the present invention, the use of semantic network technology and artificial intelligence planning technology brings the effect of service-oriented automation of complex and error-prone manual remote control technology. Therefore, based on the present invention, a user can remotely control and receive a service anytime, anywhere through a device connected to the Internet such as a mobile robot, a digital home appliance, and an unmanned mobile vehicle.

Claims (39)

In the remote control system to control the remote device to perform the target service, A remote control agent requesting a service to the remote device; And A remote server controlling the remote device based on remote device information and service information according to a service execution request of the remote control agent; The remote server, A knowledge base for storing the remote device information and the service information as device knowledge and domain knowledge, respectively; Search service module for retrieving information from the knowledge base according to the request of the remote control agent, Inference that generates a planning domain from the inference model generated by generating an inference model for performing a target service by inputting the device knowledge specification and the domain knowledge specification received from the search service module, and sets a target task to generate a service plan Control Knowledge Repository comprising a module; Wow And a device control agent that controls the remote device based on the stored remote device information and the service information to perform the requested service. Remote control system. The method of claim 1, The remote device information, One or more of the functions of the remote device, the remote device control interface, and the single process information that the remote device executes directly. Remote control system. The method of claim 1, The service information, Domain knowledge for configuring service tasks, including one or more of a complex process consisting of a plurality of single processes—processes directly executed by a remote device—and data flow within the process. Remote control system. delete delete delete The method of claim 1, The reasoning module With the planning methods (tasks that can be divided into subtasks), planning operators (directly executable task minimum units), initial states, and target tasks generated through planning domain generation and semantic expansion from the inference model, To create a service plan through an AI planning algorithm. Remote control system. The method of claim 7, wherein The AI planning algorithm, Hierarchical Task Network (HTN) planning algorithm Remote control system. The method of claim 1, The knowledge base Device knowledge base for storing the device knowledge and A domain knowledge base that stores the domain knowledge Remote control system. The method according to any one of claims 1 to 3, The device control agent, A control service module for controlling a connected remote device according to a device control process generated for executing a target service; Remote control system. The method according to any one of claims 1 to 3, The remote control agent, A process configuration module for generating a device control process for the remote device through a service plan generated at the remote server to perform a desired service; A process execution module for executing a device control process so that the remote server controls the remote device to perform a target service; Remote control system. A remote control agent that requests a remote device to perform a target service through a remote server, A search module for searching for remote device information and service information from the remote server; A process configuration module for generating a device control process through a service plan generated at the remote server based on the remote device information and the service information; And A process execution module for executing a device control process so that the remote server controls the remote device to perform a target service; The service plan, The remote server stores the remote device information and the service information as device knowledge and domain knowledge, respectively, and generates an inference model for performing a target service by inputting the device knowledge specification and the domain knowledge specification, and planning from the generated inference model. A service plan is created by creating a domain and setting up target tasks. Remote control agent. delete The method of claim 12, Further comprising an interface module to which the target service is input from the user Remote control agent. In the remote server connected to the remote control agent, to control the remote device in accordance with the request of the remote control agent to perform the target service, A control knowledge repository for storing remote device information and service information; And A device control agent for controlling the remote device based on the stored remote device information and the service information to perform the requested service; The control knowledge storage, A knowledge base for storing the remote device information and the service information as device knowledge and domain knowledge, respectively; A search service module for retrieving information from the knowledge base in response to a request to the remote control agent; And An inference module for generating an inference model for performing a target service by inputting the device knowledge specification and the domain knowledge specification received from the search service module, generating a planning domain from the generated inference model, and setting a target task. Containing Remote server. The method of claim 15, The remote device information One or more of the functions of the remote device, the remote device control interface, and the single process information that the remote device executes directly. Remote server The method of claim 15, The service information is Domain knowledge for configuring service tasks, including one or more of a complex process consisting of a plurality of single processes—processes that are run directly by the remote device—and data flow within the process. Remote server. delete delete The method of claim 15, The inference module, Planning methods (tasks that can be divided into subtasks), planning operators (directly executable task minimum units), initial states, and target tasks generated through planning domain generation and semantic expansion from the inference model are inputted. To create a service plan through an intelligent planning algorithm. Remote server. The method of claim 20, The AI planning algorithm, Hierarchical Task Network (HTN) planning algorithm Remote server. The method of claim 15, The knowledge base A device knowledge base for storing the device knowledge; And A domain knowledge base that stores the domain knowledge Remote server. The method according to any one of claims 15 to 17, The device control agent, A control service module for controlling a connected remote device according to a device control process generated for executing a target service; Remote server. In the remote control method to control the remote device to perform the target service, a) storing the remote device information and the service information as the device knowledge and the domain knowledge, respectively, to search for the device knowledge and the domain knowledge corresponding to the remote device information and the service information for performing the target service; b-1) receiving the device knowledge and domain knowledge corresponding to the target service as a result of the discovery; b-2) generating an inference model using the received device knowledge and domain knowledge, generating a planning domain from the generated inference model, and setting a target task to generate a service plan; And c) creating and executing a remote device control process via the service plan; Remote control method. The method of claim 24, The remote device information, One or more of the functions of the remote device, the remote device control interface, and the single process information that the remote device executes directly. Remote control method. The method of claim 24, The service information, Domain knowledge for configuring service tasks, including one or more of a complex process consisting of a plurality of single processes—processes directly executed by a remote device—and data flow within the process. Remote control method. delete delete The method of claim 24, Step b-2), With planning methods (tasks that can be split into subtasks), planning operators (minimum units of directly executable tasks), initial states, and target tasks generated through planning domain generation and semantic extensions from the inference model, To create a service plan through artificial intelligence planning algorithm Remote control method. The method of claim 29, The AI planning algorithm Hierarchical Task Network (HTN) planning algorithm Remote control method. The method of claim 24, wherein C), c-1) generating a device control process for the remote device necessary to perform a target service based on the service plan; c-2) causing the remote device to be controlled and performing a target service by executing the device control process; Remote control method. In the remote control method of the remote server connected to the remote control agent, to control the remote device according to the request of the remote control agent to perform the target service, a) storing the remote device information and the service information as the device knowledge and the domain knowledge, respectively, to search for the device knowledge and the domain knowledge corresponding to the remote device information and the service information for performing the target service; b-1) receiving the device knowledge and domain knowledge corresponding to the target service as a result of the discovery; b-2) generating an inference model using the received device knowledge and domain knowledge, generating a planning domain from the generated inference model, and setting a target task to generate service planning; And c) controlling the remote device according to the remote device control process generated through the service plan to perform the target service; Remote control method. 33. The method of claim 32, The remote device information, One or more of the functions of the remote device, the remote device control interface, and a single process directly executed by the remote device. Remote control method. 33. The method of claim 32, The service information, Domain knowledge for configuring service tasks, including one or more of a complex process consisting of a plurality of single processes—processes that are run directly by the remote device—and data flow within the process. Remote control method. delete delete 33. The method of claim 32, Step b-2), With planning methods (tasks that can be split into subtasks), planning operators (minimum units of directly executable tasks), initial states, and target tasks generated through planning domain generation and semantic extensions from the inference model, To create a service plan through artificial intelligence planning algorithm Remote control method. The method of claim 37, The AI planning algorithm Hierarchical Task Network (HTN) planning algorithm Remote control method. The method of any one of claims 32 to 34, wherein C), Controlling a remote device connected through a network according to a remote device control process generated by the remote control agent to perform a target service; Remote control method.

Images