KR100491432B1 - Interactive exhibition system using multi-modal user interface - Google Patents

Abstract

The present invention relates to an interactive digital exhibition system using a multi-modal user interface, comprising: a rod having a red ball and a blue band; And a camera mounted on an upper portion of the screen to capture an image of a user holding a rod; a gaze tracking module configured to calculate a direction of a gaze by analyzing a position and contrast of a user's eyes in an image provided by the camera; A 3D input module including a behavior recognition module for recognizing the horizontal and vertical positions of the rods in the image and tracking the movement of the rods by calculating the distance between the rod and the screen using the size of the red ball shown in the image; And a 3D GUI toolkit for receiving the direction of the gaze and the movement of the rod from the 3D input module and changing the display state of the exhibit according to the 3D input module. By doing so, there is an effect that a human-computer interaction can be made simple and a cost-effective exhibition system can be realized.

Description

Interactive exhibition system with multi-modal user interface {INTERACTIVE EXHIBITION SYSTEM USING MULTI-MODAL USER INTERFACE}

The present invention relates to an interactive exhibition system using a multi-modal user interface. More particularly, the present invention relates to a user's gaze and behavior using a PC camera and a rod. It's about an interactive exhibition system that lets you view your exhibits at.

Today, many museums use interactive exhibition systems. Therefore, it is necessary to build a system that reflects the behavior of visitors to help users to use the interactive exhibition system conveniently.

Since most interactive display systems rely on computer systems, the main concern is Human-Computer Interaction (HCI), especially the new Non-Conventional, easy to use and easy to use system. Multi-Modal) in the user interface.

This interface replaces the hassle of having to learn how to use the computer separately for users visiting the interactive display system, so that the system can be used more efficiently. In other words, the interactive exhibition system is very useful because it is designed to reflect the behavior of museum visitors. This is the main reason why research on interactive exhibition systems using a multi-modal user interface is conducted.

Currently, devices such as Head Mounted Display (HMD), Data Gloves, and Cave Automated Virtual Environment (CAVE) are used in virtual reality programs. These devices create realistic three-dimensional screens from a three-dimensional perspective, bringing deep appeal to users. However, it is not so commercially suitable for installation in museums. It also has the disadvantage of being complicated and expensive for the general public to use.

Many studies on eye tracking have been conducted. These studies focus on tracking the position of the eye.

Template matching method compares the initial template with the template image over time to find out the deformed part of the face [ ALYuile, DSCohen and PWHalinan, Feature extraction from face using deformable template, Proceedings of IEEE Computer Society Conference On Computer Vision and Pattern Recognition, pp . 104-109, 1989 ].

Positioning methods find out where the gaze is located [ JKKo, N.Kim and RSRamakrishna, Facial Feature Tracking for Eye-Head Controlled Human Computer Inteface, IEEE TENCON, 1999 ].

Eye-head controlled systems and eye tracking systems are based on off-site technology. Color-based methods also reduce the gaze range. This method examines the color of the face surface to identify areas of similar color distribution [ J. Yang and A. Waibel, A Real-time Face Tracker, Proceedings of the Third IEEE Workshop on Applications of Computer Vision, pp. 142-147, 1996, Rogerio Schmidt Feris, Teofilo Emidio de Campos, Roberto Marcondes Cesar Junior Detection and Tracking of Facial Feature in Video Sequences, Lecture Notes in Artificial Intelligence, vol. 1793, pp. 197-206, April 2000 ]. To get better results this way, it's a good idea to give the same person the same lighting. Most eye tracking methods have a process of deriving facial features. This method is an integral part of the system because it helps visitors to use the interactive display system more easily and conveniently. The eye tracking method can be applied not only to this, but also to virtual reality (VR), augmented reality (AR), and various other computing environments.

Behavior recognition system is based on rod tracking technology. There is a way to use a magnetic sensor to pinpoint the rod, but it is expensive. Therefore, there is a need for new technologies that are simple and inexpensive.

First, there is a method of attaching an infrared LED to the end of the rod and using the infrared camera to recognize the movement of the rod [ JYChung, NKKim, GJKim and CMPark, A Low Cost Real-Time Motion Tracking System for VR Application, International Conference on Virtual Systems and Multimedia 2001, 2001 ].

Another option would be to use a PC camera and a color-based rod. Using a simple structured rod as a three-dimensional pointing tool, not a complicated tool, it can be easily handled by anyone. Eye recognition using an inexpensive PC camera makes it possible to look around objects and easily find deep information for easy pointing. Helps you figure out

In order to solve the above problems, an object of the present invention is to provide an intuitive, simple and inexpensive interactive exhibition system that can be easily enjoyed by visitors such as museums using color-based rods and PC cameras that provide 3DOF (Degree Of Freedom). do.

In order to achieve the above object, the present invention provides an interactive display system for displaying an exhibit on a computer screen, comprising: a rod having a red ball and a blue band; And a camera mounted on an upper portion of the screen to capture an image of a user holding the rod, and analyzing the position and contrast of the eye of the user in the image provided by the camera to calculate the direction of the gaze. A tracking module, and a behavior recognition module for recognizing the horizontal and vertical positions of the rods in the image and tracking the movement of the rods by calculating the distance between the screen and the rods using the size of the red ball shown in the image. Input module; And a 3D GUI toolkit which receives the direction of the eye and the movement of the rod from the 3D input module and changes the display state of the exhibit accordingly. Provide a system.

As described above, the interactive exhibition system according to the present invention includes a PC camera and a rod, and tracks the eyes of the user and recognizes the behavior. In particular, the rod is for recognizing the user's behavior, and should be designed to express the user's behavior. The user's behavior is recognized by the interface. If the user wants to see the exhibits at various angles on the screen, the object on the screen can be rotated. The user simply grabs the rod and draws a circle to rotate the object he wants to see on the interface. The user can also use this rod as a tool for commands to enlarge or reduce an object. Pull the rod to enlarge the size of the object, and push the rod to shrink it. The rods are also designed to perform simple interactive tasks such as selecting an object or executing a process, making it easy to draw with a rod to select an object from the menu. PC camera is used to recognize the user's gaze and the movement of the rod.

The eye tracking technique and the behavior recognition technique may be used independently, or the user may command the system by combining the eye tracking and the rod. This method of conversation is called 3D conversation from the point of view of human and computer talking in three-dimensional space. The toolkit provides an application program interface (API) for programmers who want to create such interactive display systems using PC cameras and rods. The conversation method and interface can be applied to a digital museum. Recently, many museums have placed multimedia exhibits in the museum, such as video and three-dimensional graphic images. This means that these museums are equipped with multimedia computer systems. This suggests that we should allow existing museum visitors to handle computer systems installed in museums without knowledge of computers. The conversation method is used to provide a way for visitors to use the system simply and easily.

1 shows how the conversation method is applied to a digital museum. In other words, an application program such as an interactive exhibition system can be built using a 3D Graphical User Interface (GUI) toolkit.

First, the 3D input module 110 includes a gaze tracking module 111 and a behavior recognition module 112. The 3D input module 110 analyzes the information transmitted from the PC camera, finds the direction of the gaze of the object reflected on the screen, and calculates the three-dimensional position of the rod that the user is holding to track the rod movement.

The 3D GUI toolkit 120 provides an API for the application. These APIs derive three-dimensional positions of the gaze and rods from the 3D input module. Therefore, programmers can develop applications such as interactive exhibition systems used in museums using APIs to control 3D input modules. The toolkit is composed of a 3D widget 122 and a 3D window manager 121. Widget is the term used to refer to the names of the components that make up the GUI in Xtoolkit on X-Windows systems. For example, widgets are those in which parts such as buttons and scroll bars are independent as windows.

Finally, application 130 is developed using 3D GUI toolkit 120. As mentioned above, an example would be the development of an interactive exhibition system where museum visitors can use their eyes and rods to view and manipulate exhibits on the screen.

3D input module

The 3D input module 110 includes a gaze tracking module 111 and a behavior recognition module 112 as shown in FIG. 1.

The eye tracking module 111 uses a PC camera as an input sensor. When the PC camera provides an image of the face, the gaze tracking module analyzes the position and contrast of the eye to determine the direction of the gaze.

2 illustrates an image processing process. First, the site of the RGB image from the camera is adjusted to reduce the computational process. The image is then converted into a black and white image, and the initial values derived from the conversion process are used to simplify the image. The eye is the darkest part of the simplified image. The gaze tracking method calculates the initial value including the ratio of the eye's aspect and position to the position value. The fact that the eyes are usually at the top of the face can be used to speed up the gaze tracking operation.

3 is a diagram illustrating the range of gaze tracking and the position of a parameter, in which the direction Dir and the position Pos of the eye are represented by x, y, and d. x is the horizontal distance and y is the vertical distance. Assuming that the center of the user's line of sight is at (x, y) and d has a z value, which is the distance between the camera and the user, Dir and POS can be defined as shown in FIG. Where FOV is the field of view of the camera.

Figure 4 shows the rod proposed in the present invention, it consists of a head called a red ball (410) and a pillar called a blue band (Blue Band: 420). The behavior recognition module 112 recognizes the position of the rod using a color pattern, and uses the red ball and the blue band in combination. This gives a two-dimensional position, let's call it x and y. The z value, which is information about the depth, can be obtained by calculating the size of the red ball.

In the present invention, a color-based method is used to recognize the red ball of the rod. First, the RGB values are normalized. This process shows that the normalized values of the red ball are calculated as R / (R + G + B), G / (R + G + B), and B / (R + G + B). it means. This value is then used to determine the color of the object. Since RGB values are not predefined, a measurement process is required. This is a kind of calculation process in which the mean and deflection values of the color are calculated and the normalized color distribution is calculated. This value is used when the Behavior Awareness module tracks the Red Ball.

5 shows the process by which the normalized distribution of colors is calculated by the red ball value. By calculating the probability within the normal distribution of colors for each pixel value of the input video image, only the pixel values having a predetermined threshold or more probability are filtered to filter out only the red ball image within the entire image. From the red ball image thus obtained, the size and position of the red ball are obtained.

Once the rod's red ball is recognized, the system calculates the ball's position, x, y, z. 6 illustrates an operating range in which the movement of the red ball is an area recognized by the interface, where movement outside the operating area is ignored. The values of x, y, and z representing information about the position are normalized and the ranges are set to −1 ≦ x ≦ 1, −1 ≦ y ≦ 1 and O ≦ z ≦ 1. The PC camera is centered on the screen, and it only detects the red ball's movement when it's in range. Equation 1 below is to calculate the x and z values of the red ball.

z ' : z value of the red ball in the operating range

r _b : Red ball radius (r _min ≤r _b ≤r _max )

r _min : minimum radius of the red ball (when z = 1)

r _max : Maximum radius of the red ball (when z = 0)

x ' : x value of the red ball in the operating range

x _b X value of the red ball on the screen (O≤x _b ≤1)

FOV : field of view of the camera

The red ball y value can be calculated similarly. In this way the movement of the rods within the operating range is tracked and projected onto the screen.

3D Graphic User Interface Toolkit

The 3D GUI toolkit 120 is provided for application programmers who wish to develop an interactive display system, the basic structure of which is shown in FIG. The toolkit provides an application program interface (API) for handling data obtained from 3D input means. The toolkit consists of a basic 3D widget, a basic widget and a visualization widget. The most basic widget is the Basic Widget (710), which provides the basic features of the other widgets as shown in FIG. 7, specifically, simple GUI components such as label 720 and button 730 are provided. . These components are presented in a three-dimensional format, and three-dimensional pointing such as selection and rotation is also possible.

The 3D window system can be extended from the basic 3D widget. The system provides a window window that can be viewed in three dimensions. As shown in FIG. 8, the window manager displays several windows and arranges the windows in a three-dimensional form. The window manager changes the operation window according to the movement of the user's eyes. If the user's eyes are looking at another window, the PC camera detects this movement and the previous window is inactive. This process is illustrated in FIG. Widgets such as window 750 and window manager 760 use a three-dimensional window system.

The Model Viewer widget 740 also tracks the movement of the eye to change the object on the screen. FIG. 9 illustrates a model viewer in which an object changes on a screen due to the focus of the user. If the user looks down or faces the bottom of an object, the model viewer changes the viewpoint to show the floor as shown in FIG. 9. give.

3D conversation method

Hereinafter, the 3D conversation method employed in the interactive exhibition system according to the present invention will be described.

Eye gaze is one of the basic conversation methods and is the basis of eye tracking system. For example, the model viewer widget changes the position and direction of objects according to the movement of the eyes. If the user wants to see the object from a different direction, the PC camera tracks the eye's movement and the system shows the object from the perspective.

 Another basic method of dialogue is to use rods. The user can express various actions using the rod. Behavior should be natural to reflect the user's behavior in a museum or exhibition hall. Below we will look at what three-dimensional operations can be performed on the screen and list the results of the operations.

① Select

This action corresponds to a mouse click, and when the user places the pointer on the object and waits about one second, the object is selected.

② Cancel

This action cancels the selection action and is like releasing the clicked mouse. If the user quickly releases the pointer from an object, the previous selection is canceled.

③ Run

This operation corresponds to a double click of the mouse, which means to perform a command or to operate a selected object. The user shakes the rod vertically. Raising and lowering the rod quickly changes the red ball's y value.

④ rotation

This action corresponds to dragging the mouse and rotating the selected object on the screen. Rotation is performed by calculating the position of the rod with respect to the center of the operating range.

⑤ Zoom in and out

This action enlarges or reduces the image on the screen. The user can see the object in more detail through the magnification operation, or can see the overall shape of the object by executing the reduction operation. Push the rod to perform the enlargement operation and pull the rod to perform the reduction operation. That is, the user only needs to move the rod along the z axis for the enlargement and reduction operation.

This new conversational method can be designed and implemented differently each time a new application is created, using the functions in the toolkit to implement this method.

Experiment

In order to measure the performance of the system proposed in the present invention, the following experiment was performed. There are two cones on the screen, yellow and red, respectively. As shown in FIG. 10, the user was allowed to move the yellow cone toward the red cone. This process consists of selecting, moving, and rotating things.

The results of measuring the time taken to complete the operation are shown in FIG. 11 and Table 1 below. 11A and 11B are graphs showing execution time for finishing work using a 3D mouse and a rod, respectively, and Table 1 shows average execution time. Here, the time taken to select the yellow cone is T + S, the time taken to move this yellow cone to the position of the red cone is T, and the time it takes to rotate the yellow cone to match the yellow cone and the red cone is R. Displayed. This operation was performed 20 times for each of the three-dimensional mouse and the system according to the present invention.

T + S T R 3D mouse 3.55 2.20 2.57 Rod 4.14 2.34 3.40

The results of this experiment show that the time it takes to move and rotate an object is almost as much as a 3D mouse when using the rod according to the invention. When rods are used, the run time is somewhat irregular and the average run takes slightly longer. However, the process of moving an object using a rod has an advantage that the execution time is similar but simple to execute, compared to a 3D mouse.

Application examples of the present invention

The interactive exhibition system according to the present invention can be applied to many fields. As many museums are digitized, many exhibits are displayed using computers. On the other hand, visitors to the museum do not use computers efficiently, so the interactive exhibition system according to the present invention is well suited for digital museums.

12 illustrates an embodiment in which the interactive exhibition system according to the present invention is applied to a digital museum. Museum visitor 1210 stands in front of screen 1240 to view the exhibits and find related information. PC camera 1230 is mounted on top of the screen. Visitors grab rod 1220 and move it while looking at objects on the screen. The rod can be used to select, rotate and execute objects on the screen, and to zoom in and out.

As another example, Yangdowi, a musical instrument developed in ancient Korea, can be reproduced using an interactive exhibition system. This instrument is not present and we cannot deal with it directly. However, based on the records, the instrument can be reproduced in the form of a three-dimensional digital model.

By using the interactive exhibition system according to the present invention, it can be viewed and operated. As the PC camera follows the movement of the line of sight, the 3D model is transformed and shown along with the line of sight. You can use the rod to rotate and manipulate the three-dimensional model of the instrument. You can also use the rods to find more information about the instrument. The user's actions may be select, cancel, execute, rotate, zoom in, zoom out. The system can also show cartoon sessions that explain the basic knowledge of the assignments, as well as the structure and operation of the instrument. Thus, the user can find more detailed information through the hyperlinks and menus provided by the three-dimensional model.

According to the present invention, it is possible to implement a simple interactive display system by presenting a human-computer interaction in a simple and intuitive manner, unlike a system using a 3D mouse, the user can easily use the computer without prior knowledge of the computer. Can be.

In addition, no special equipment is required to display three-dimensional images on the screen. The GUI operation required to display and manipulate 3D images can be performed using only the movement of the eyes and the rods of the user.

Furthermore, the rod according to the present invention is cheaper than a 3D mouse or a method using a magnetic sensor for accurate positioning of the rod, and is efficient for use in an interactive display system since it does not lag much behind the 3D mouse in terms of speed.

1 is a view showing the structure of an exhibition system according to the present invention;

2 is a diagram illustrating an image processing process for eye tracking;

3 is a diagram illustrating a range of eye tracking;

4 is a view showing a baton according to the present invention;

5 is a diagram showing the peak, the standard supply calculation of the head color, head certification, respectively,

6 is a diagram illustrating an operation range of behavior recognition;

7 is a view showing the basic structure of the 3D GUI toolkit,

8 is a diagram illustrating management of a window manager in a 3D window system;

9 illustrates a model viewer in which an object is changed on a screen by a user's focus;

10 shows an example of an operation practice for measuring the performance of the present invention;

11 is a graph showing the working time using a 3D mouse and a rod, respectively;

12 is a diagram illustrating a case where the present invention is applied to a digital museum.

Claims (5)

delete In an interactive display system for displaying an exhibit on a computer screen, Rods with Red Ball and Blue Band; And A camera mounted on an upper portion of the screen to photograph a user holding the rod; A gaze tracking module that calculates the direction of the gaze by analyzing the position and contrast of the user's eyes in the image photographed and provided by the camera, and calculates the horizontal and vertical positions of the rods in the image, A 3D input module including a behavior recognition module for tracking the movement of the rod by calculating a distance between the screen and the rod using a size; And Interactive display system using a multi-modal user interface, characterized in that it comprises a; 3D GUI toolkit for receiving the direction of the gaze and the movement of the rod from the 3D input module to change the display state of the exhibit accordingly . The method of claim 2, The 3D GUI toolkit includes a window manager widget for changing an active window according to the movement of the gaze, and a model viewer widget for changing and displaying the view point of the exhibit according to the movement of the gaze. Interactive exhibition system using multi-modal user interface. The method of claim 2, The gaze tracking module adjusts the image to reduce a calculation process, converts the image into a black and white image, and simplifies the image by using an initial value derived from the conversion process, thereby simulating the image of the simplified image. Interactive display system using a multi-modal user interface, characterized in that the dark part is determined by the position of the user's eyes. The method of claim 2, The distance between the screen and the rod is expressed by a minimum value of 0 and a maximum value of 1 R _b is the radius of the red ball shown in the image, r _min is the minimum radius of the red ball when the distance is the maximum value, and r _max is the maximum radius of the red ball when the distance is the minimum value. Interactive display system using multi-modal user interface.