KR20120000437A - Apparatus and method for augmented reality interface agent - Google Patents

Abstract

PURPOSE: An apparatus and a method for augmented reality interface agent are provided to enable the formation of a human-friendly interface by grasping user intention through the external information of augmented reality environment and user information. CONSTITUTION: A client unit(100) adjusts and outputs reality images and virtual 3D(Dimensional) images based on user information and tracking information. A server unit(200) reasons the situation which is subject to user intention and transmits the situation to the client unit. Service software(300) provides various function modules for various works according to the situation. The client unit comprises an image acquisition unit, an extracting unit for extracting user's location and direction, an image generator, and a matching unit.

Description

Augmented Reality-based Interface Agent System Apparatus and Method {Apparatus and Method for Augmented Reality Interface Agent}

The present invention relates to an augmented reality-based interface agent, and in particular, an augmented reality-based interface that allows a user to appropriately perform a predetermined process through a 3D virtual character after grasping a user's intention using user information and tracking information. An agent system apparatus and method.

The interface between a computer and a user, which is widely used nowadays, is mainly based on a direct manipulation method centering on a graphical user interface.

The direct manipulation method is a method in which a user transmits a direct command inside a program using a toolbar such as a menu, an icon, a point device, and the like provided in a graphical user interface.

However, the direct manipulation method causes a problem that the toolbar for manipulating each function is also complicated as the functions provided by the program become huge.

In addition, the direct manipulation method is cumbersome to acquire knowledge about various tools provided by the program in order to achieve a desired goal.

Therefore, a process for helping a user understand or perform a task on behalf of the user was required. Accordingly, an interface agent was developed. The interface agent is an intelligent process that acts as an intermediary for interaction between a person and a program and successfully grasps and executes a user's purpose. In other words, it is a program that assists a user to perform a task or performs on behalf of a task by grasping the user's intention with prior knowledge of the task that the user wants to perform. I can pursue it.

However, since the process using the interface agent is dependent on the software in which the agent exists and determines the user's situation by using only information on the internal state of the system, there are a lot of limitations in recognizing and using the real-time situation of the user. There is this.

Therefore, an object of the present invention is to solve the above problems, augmented reality based to extend the interface agent to the augmented reality space to provide a more human-friendly interface by interacting with the user via the virtual character as a medium It is to provide an interface agent system apparatus and method.

According to a feature of the present invention for achieving the above object, a client unit for matching and outputting the real image and the virtual 3D image based on the user information and tracking information; A server unit inferring a situation intended by a user and delivering the same to the client unit; And service software that provides various functional modules to perform appropriate tasks according to the inferred situation at the request of the server unit.

The client unit may include an image acquisition unit that acquires an image of a surrounding environment, an extraction unit that extracts a current user's position and direction through the image information obtained by the image acquisition unit, and the server according to the information extracted by the extraction unit. And an image generator for generating a virtual 3D image based on the information received from the unit, and a matching unit for matching the information generated by the image generator and the actual image information to be displayed as one information.

The server unit may include an external state model unit configured to determine a location and a direction of the user using the image information provided by the image acquisition unit, an internal state model unit configured to determine the intention of the user by the service software, and the external state. An inference unit for inferring the user's intention through the internal state model unit and a position and direction of the user by a model unit, a work plan unit for generating a work plan for the current situation inferred by the inference unit, and in the work plan And performing a task through a function call of the service software, and transmitting a task to the image generator to generate the virtual 3D image.

The client unit is a portable device equipped with a camera so as to capture video information.

Another aspect of the invention, the client unit comprises the steps of extracting the location and direction of the user through image acquisition; Deriving, by the server unit, the current situation based on the extracted position and direction of the user and prior knowledge of the task; Executing, by the server unit, a work plan generated according to the inferred current situation by a function call of service software, and simultaneously transmitting a situation about the work being performed to the client unit; And generating, by the client unit, the virtual 3D image by using the information on the work being performed, and providing the user with the real image information.

According to the present invention, the user's intention is identified through external information and user information of the augmented reality environment, and a virtual character is provided as a medium in a user-oriented communication method to the user according to a process generated to satisfy the identified intention. In addition to providing a friendly interface, the user has the advantage of being able to properly plan and perform a given task.

1 is an overall configuration diagram of augmented reality based interface agent system apparatus according to an embodiment of the present invention
2 is a flowchart illustrating an operation of an apparatus for augmented reality based interface agent system according to an exemplary embodiment of the present invention.

Hereinafter, exemplary embodiments of an apparatus and method for augmented reality-based interface agent system according to the present invention will be described in detail with reference to the accompanying drawings.

First, prior to describing the present invention, the core studies among the studies related to the interface agent and augmented reality will be described.

First is the 'persona project'. The project, conducted by Microsoft Research, is a peedy assistant program that works with living things. The parrot peedy is an interface agent system that selects and plays music through conversations with users from the collection of audio CDs held by the PC. The system includes voice language processing that receives voice and interprets it as a high-dimensional event, a dialogue management part that accepts the interpreted event and determines how the character behaves, and responds to the input of animation motion and peedy. It is the video and audio output part that talks to the user with the output of voice and sound effects. Through this, peedy accepts and interprets human words with anthropomorphic traits and shows the user visually and aurally according to the situation.

Second is the Sony Computer Science Lab Project. This is a study on the interaction of facial expressions between computer and human.

Third, 'Shared Space: AR Interface'. It augments virtual objects on the real world through augmented reality interfaces to help people understand and easily interact between computers, people and their surroundings.

Next, the Augmented Reality Interface Agent (ARIA) system of the present invention will be described based on the above studies.

That is, the ARIA system refers to an enhanced interface agent that operates based on an augmented reality environment. Through this, the work can be performed through the interaction of human, interface, and software in the augmented reality environment. Detailed descriptions are as follows.

1 is a block diagram of an apparatus for augmented reality-based interface agent system according to a preferred embodiment of the present invention.

1, an ARIA system includes a client unit 100 and a server unit 200.

First, the client unit 100. The client unit 100 is a device that a user directly manipulates, for example, a mobile device such as a PDA and a mobile phone, and is provided with an augmented reality environment.

The configuration of the client unit 100 is as follows.

First, an image tracker 102 for acquiring an image of a surrounding environment is provided.

An extraction unit 104 is provided to extract the current user's position and direction through the image information acquired by the image acquisition unit 102.

An image generator 106 for generating a virtual 3D image based on the information received from the server unit 200 is provided. The information includes the location and direction information of the user extracted by the extraction unit 104, that is, tracking information and the user information.

The matching unit 108 is provided to match the information generated by the image generating unit 106 with the actual image information and display it on the display unit 110 for the user to see.

Transmitting and receiving unit 120 for transmitting and receiving a variety of information is provided. The transceiver 120 transmits and receives information to and from the server unit 200 using a wired or wireless communication method.

Next, the configuration of the server unit 200 will be described.

The server unit 200 is configured with an external state model unit 202 and an internal state model unit 204 to grasp the intention of the user.

The external state model unit 202 is a model for identifying the state of the user. This is determined by receiving the information extracted by the extraction unit 104 of the client unit 100, and finally based on the image information provided from the image acquisition unit 102 to obtain the image of the surrounding environment.

The internal state model unit 204 is to determine the state of the system. This includes knowing the state of the system as well as prior knowledge of the tasks to meet the user's intentions. To this end, the internal state model 204 is connected to the service software 300 including a plurality of service modules 310. The service software 300 will be described later.

And an inference unit 206 for inferring a current situation based on the information obtained through the external state model unit 202 and the internal state model unit 204.

An action planner 208 is used to generate a future work plan for the current state by using prior knowledge of a task to be performed on the current situation inferred by the inference unit 206.

An action perfomer 210 for performing a task according to the task plan and transmitting a situation of the performed task to the client unit 100 is provided. The task performing unit 210 performs a task through a function call of the service software 300.

The service software 300 refers to a practical application program that implements various functions provided by the ARIA to a user. In other words, the service module 310 enables the user to efficiently perform this process in accordance with various scenarios. Accordingly, the service software 300 provides a protocol for communicating with the ARIA, and various functions are registered in the ARIA for the process to be performed.

Using augmented reality-based interface agent system device of such a configuration

2 is a flowchart illustrating an operation of an apparatus for augmented reality-based interface agent system according to an exemplary embodiment of the present invention.

First, a user connects his mobile device to ARIA (s100). The mobile device stores a profile for the user, and is assigned an ID for itself in the ARIA when accessing the ARIA. Here, instead of directly operating the mobile device to the ARIA, the mobile device may be automatically connected to the ARIA and assigned an ID when entering a predetermined place.

Once the mobile device and the ARIA are connected, the mobile device is in a state capable of communicating with the ARIA.

The user acquires the surrounding image or the front image by using the image acquisition unit 102 of the mobile device he has, for example, a camera, for communication with the ARIA (S102).

The image acquired by the image acquisition unit 102 is transferred to the extraction unit 104.

The extraction unit 104 extracts the current user's position and direction through the image information acquired by the image acquisition unit 102 (S104). For reference, in order to extract the location and direction information of the user, the mobile device stores information for knowing coordinates, maps, direction information, etc. for each region.

The extraction unit 104 transmits the extracted location and direction information of the user to the external state model unit 202 of the server unit 200 through the transmission and reception unit 120 (s106).

The server unit 200 determines the external state for the user based on the information transmitted from the external state model unit 202 through the transceiver 120. In addition, the internal state model unit 204 of the server unit 200 infers the user information to determine the user's intention (s108).

All information grasped by the external state model unit 202 and the internal state model unit 204 is transmitted to the inference unit 206. In this manner, the inference unit 206 may infer a current situation regarding a place where the user is located and a task that the user wants to perform (S110).

Once the inference unit 206 infers the current situation, the inferred information is transferred to the job planning unit 208. Here, a future work plan for the current state is generated by using prior knowledge about a task to be performed on the current situation inferred by the inference unit 206 (S112).

The action plan in which the action planner 208 is generated is transmitted to the action execution unit 210.

The task execution unit 210 generates a process by calling the service module 310 corresponding to the task to be performed through a function call of the service software 300 (s114).

The process transfers to the client unit 100 (s116).

The image generating unit 106 of the client unit 100 generates a virtual 3D image based on the information received from the work performing unit 210 (S118).

The generated 3D image is transferred to the matching unit 108. The matching unit 108 matches the information generated by the image generating unit 106 with the actual image information acquired by the image obtaining unit 102 (s120), and displays the display unit 110 so that the user can view it. (S122).

The information displayed on the display 110 is information to be provided to the user with respect to a process to be performed by the user according to the user's intention. This is provided through 3D virtual characters.

A process to be performed by the user as described above will be described by way of example.

When a user enters a building with a mobile device (i.e., client unit) 100 owned by the user, the client unit 100 is automatically connected to the server unit 200 of the ARIA system and assigned an ID for the mobile device. . When the connection is complete, the client unit 100 provides a welcome message and a virtual character provided by the server unit 200 to provide a progress process according to a user's purpose.

According to the progress process, the server unit 200 infers the intention of the user based on the state, location information, and user information of the user received according to the operation of the client unit 100 as described above. In this case, the user may provide the purpose of visiting the building through the conversation with the server unit 200.

The server unit 200 provides the information through the 3D virtual character according to a predetermined scenario according to the purpose of the visit of the user.

In other words, if the user is an employee residing in the building, the server unit 200 records access / commute information of the user by using access, schedule, and postal management software, and the client unit 100 records information about a meeting schedule or mail of the day. Provide through.

On the other hand, if the external visitor is a meeting attendee, the client unit 100 provides information for guiding the visitor to the meeting room and calling the meeting attendees using a schedule and conference room management software.

In addition, in case of other users, it provides only information on convenience facilities in the building, and provides a function of inventing an alarm using security management software when the visitor is present in an unacceptable space.

As described above, the present invention infers the intention of the user and then provides a progress process according to the user's purpose through the virtual character, so that the processor according to the real-time situation of the user can be properly progressed.

The scope of the present invention is not limited to the embodiments described above, but is defined by the claims, and various changes and modifications can be made by those skilled in the art within the scope of the claims. It is self evident.

100: client unit 102: image acquisition unit
104: extraction unit 106: image generation unit
108: matching unit 200: server unit
202: external state model unit 204: internal state model unit
206: Reasoning Department 208: Work Planning Department
210: work performing unit 300: service software

Claims (5)

A client unit for matching and outputting a real image and a virtual 3D image based on user information and tracking information;
A server unit inferring a situation intended by a user and delivering the same to the client unit; And
Augmented reality-based interface agent system device comprising a service software that provides a variety of functional modules to perform a proper operation according to the inferred situation at the request of the server unit. The method of claim 1, wherein the client unit,
Image acquisition unit for obtaining an image of the surrounding environment,
Extraction unit for extracting the current position and direction of the user through the image information obtained by the image acquisition unit,
An image generator for generating a virtual 3D image based on the information received from the server unit according to the information extracted by the extractor;
Augmented reality-based interface agent system device characterized in that it comprises a matching unit for matching the information generated by the image generating unit and the real image information to be displayed as one information. The method of claim 1, wherein the server unit,
An external state model unit which grasps the position and direction of the user using the image information provided by the image acquisition unit;
An internal state model unit which grasps the intention of the user by the service software;
An inference unit for inferring the position and direction of the user by the external state model unit and the intention of the user through the internal state model unit;
A job planning unit for generating a job plan with respect to the current situation inferred by the reasoning unit;
A task execution unit configured to perform a task by calling a function of the service software according to the task plan, and to transmit the performed task to the image generator to generate a virtual 3D image. Interface agent system device. The method of claim 2,
The client unit is augmented reality-based interface agent system device, characterized in that the portable device equipped with a camera to capture the image information. Extracting, by the client unit, the position and the direction of the user through image acquisition;
Deriving, by the server unit, the current situation based on the extracted position and direction of the user and prior knowledge of the task;
Executing, by the server unit, a work plan generated according to the inferred current situation by a function call of service software, and simultaneously transmitting a situation about the work being performed to the client unit; And
And generating, by the client unit, the virtual 3D image by using the information on the work being performed, and providing the user by integrating the virtual 3D image with the real image information.

Images