KR100611680B1 - Multi modal interaction-based remote meeting system - Google Patents

Abstract

The present invention relates to a multimodal interaction based remote meeting system using DVM-ARPN and network handshake, and more particularly, to a first interaction device, at least one first computer connected to the first interaction device, and a first computer. A first digital video transmission device to be connected, at least one second computer connected to a network with the first computer, a second interaction device connected to the second computer and corresponding to the first interaction device, and to the second computer. A second digital video transmission device to be connected; augment a virtual object by using a digital image provided from the first and second digital video transmission devices as a background of augmented reality, and use the virtual object as a first and second interaction device. It provides a multi-modal interaction based remote meeting system that applies interaction.

Augmented Reality, Haptic, Interaction, Virtual Objects, DV-ARPN, Network Handshake

Description

Multi Modal Interaction-based Remote Meeting System

1 is a schematic diagram showing a multi-modal interaction-based remote meeting system according to a preferred embodiment of the present invention.

FIG. 2 is a classification diagram illustrating mixed reality according to a definition of a millgram applied to the embodiment of FIG. 1.

3 is a flowchart of augmented reality using ARToolKit applied to the embodiment of FIG. 1.

4 is a structural diagram showing a relationship with the prior art applied to the DV-ARPN of FIG.

5 is a flowchart of a remote meeting using the remote meeting system of FIG. 1.

6 is an exemplary diagram illustrating an embodiment in which the remote meeting system of FIG. 1 is applied in a multicast environment.

7 is a conceptual diagram comparing the remote meeting system proposed in the present invention and the conventional remote meeting system.

8A is an exemplary diagram showing an image of a DV-ARPN according to the present invention implemented remotely.

8b is an exemplary view showing a local implementation of DV-ARPN according to the present invention.

The present invention relates to a remote meeting system, and more particularly, to a multi-modal interaction based remote meeting system using DV-ARPN and network handshake to provide interaction in a realistic and network environment.

Remote meeting system technology is an area that has been developed and studied day by day due to the rapid development of network environment and compression technology. This is a technology that allows distant people to meet on a single topic over the network, eliminating obstacles such as wasted travel time and delayed decision making that can occur in face-to-face meetings. Elimination can increase productivity and make effective use of time.

However, conventional remote meeting research has focused only on improving the quality of audio and video (A / V). In other words, the conventional remote meeting system has a limitation that the A / V-based service is the main. For example, a representative commercial program of the related art is Microsoft's NetMeeting, and an open project is an Access Grid project.

On the other hand, researches are being actively conducted to construct a remote meeting system in which a user can feel a sense of reality, and thus, various remote meeting systems using mixed reality have been introduced. However, these remote meeting systems have clear limitations compared to face-to-face meetings that are still face-to-face, and the only application of the user to user interaction is the haptic-based Networked SPIDAR. . Here, the network spider adds a network concept to the spider, which is one of the haptic devices that can feel tactile.

Introducing Augmented Reality ToolKit (ARToolKit), an open source software for implementing Augmented Reality, is a library that makes it easy for programmers to configure augmented reality. ARToolKit is a simple tool that uses computer vision technology to calculate the location of a real camera (single camera) that corresponds to the location of a marker in the real environment, and overlays the virtual object on top of the marker in real time. to be. Because of this convenience, it is used universally to implement augmented reality.

This conventional ARToolKit augmented reality technology is a real world used as the background image is obtained from the input of the local camera (local camera) The background that the user can configure becomes limited. That is, augmented reality technology does not include a network concept. In addition, when SXGA (1280x1024) monitors are becoming more common, the use of a general camera that provides only QVGA (320x240) or VGA (640x480) images has a problem of degrading the reality of augmented reality. In addition, there is no network concept in the conventional augmented reality technology.

On the other hand, the user can move the virtual object by moving the marker of the real world, but this is only a movement of the virtual object is a clear limitation that can not be seen as a substantial interaction with the virtual object.

An object of the present invention is to solve the above problems, by combining the digital image transmission technology to augmented reality and by using the interaction device to implement the interaction with the virtual object, multi-modal that can interact in the realistic and network environment Provides an interaction-based remote meeting system.

In order to achieve the above object, the multimodal interaction-based remote meeting system according to the present invention includes a first interaction device, at least one first computer connected to the first interaction device, and a first interaction device. A first digital video transmission device, at least one second computer connected to the first computer and a network, a second interaction device connected to the second computer and corresponding to the first interaction device, and the second And a second digital video transmission device connected to a computer, augmenting a virtual object using a digital image provided from the first and second digital video transmission devices as a background of augmented reality, and using the virtual object as the first image. And interact with the second interaction device.

In a multimodal interaction based remote meeting system, the first or second interaction device is a haptic device. The first and second haptic devices corresponding to each other provide a network handshake.

Hereinafter, exemplary embodiments of the present invention will be described in detail with reference to the accompanying drawings.

1 is a schematic diagram showing a multi-modal interaction-based remote meeting system according to a preferred embodiment of the present invention.

1, a first interaction device 116, a first computer 112 connected to the first interaction device 116, and a first digital video transmission device 114 connected to the first computer 112. ), At least one second computer 152 networked with the first computer 112, and a second interaction device connected to the second computer 152 and corresponding to the first interaction device 116 ( 156, and a second digital video transmission device 154 connected to the second computer 152, the digital image provided from the first and second digital video transmission devices (114, 154) via a network of augmented reality A multimodal interaction based remote meeting system 100 is shown that augments a virtual object using the background and applies the virtual object to the first and second interaction devices 116 and 156. The first and second interaction devices 116 and 156 may include a mouse, a tracker, a joystick, a haptic device, and the like. In this embodiment, the first interaction device may include a PHANToM 116 and a second interaction device. The furnace illustrates a force-feedback joystick 156. Phantom and force-feedback joysticks are included in haptic devices.

Here, the haptic means 'touch' or 'touch related', and the haptic device refers to various kinds of interface devices that provide a force-feedback function so that the user can feel the touch. Representative haptic devices include PHANToM, Cyberberg, or Spider, and use them to implement interaction between a user and a computer. However, in the present embodiment, not the interaction between the user and the computer, but the interaction between the user and the user connected via the network.

In other words, the phantom 116 and the force-feedback joystick 156 are connected to the network to sense the power of each other and at the same time transmit the power that they move to the other. Through this, users using the system of the present embodiment may feel immersive by making realistic contacts with each other.

The first and second digital video transmission devices 114 and 154 are digital video transport systems (DVTSs) capable of streaming digital video.

In order to implement the Augmented Reality Peripheral Network (DV-ARPN) proposed in the present invention, it was configured based on Visual C ++ on the Windows platform using ARToolKit, an open software. That is, in the present embodiment, the first or second computer connected to the first or second digital video transmission devices 114 and 154, respectively, by modifying the part receiving the image input from the first and second digital video transmission devices 114 and 154. DV streaming transmitted from (112, 152) to a second or first computer (152, 112) connected via a network is used as the background of augmented reality. Meanwhile, the transmitted DV streaming may be displayed through the first display device 110 or the second display device 150.

ARToolKit, which is used to augment virtual objects in the background of augmented reality, accepts input only from locally located digital video transmission devices (e.g. USB cameras), but in this embodiment streaming from a digital video transmission device connected to a network. By combining DVTS with ARToolKit, a DV-class (720 * 480) image located at a remote location can be used as the background of augmented reality. In this way, the user can install a digital video transmission device such as a DV camera anywhere in the network to overcome the limitations of configuring the background of augmented reality into the real world in which the user is located. Can be used as a background.

As illustrated, one remote (REMOTE) to be used as the background of augmented reality for configuring DV-ARPN and one local (LOCAL) where augmented reality is actually implemented will be networked unicast environment. Suppose, at least one DV camera 114, 154, a digital video transmission device, one marker (not shown), at least one computer 112, 152, respectively, for use at remote and local (LOCAL), and at least for interaction. One interaction device 116, 156 is minimally required. Preferably, at least one interaction device 156 is required locally (LOCAL).

Here, the computer 112 arranged with the DV camera 114 at the remote REMOTE acts as a DVTS server to transmit the real world of the remote REMOTE over the network. Computer 152, located locally (LOCAL), acts as a DVTS client to display DV streaming sent from the DVTS server to screen 150. FIG.

Accordingly, a marker is placed at a desired location of a remote place to be used as a background of augmented reality, the first digital video transmission device 114 is connected to the first computer 112, and a DVTS server program is executed to execute a local. When transmitting to LOCAL, LOCAL drives DV-ARPN to augment a virtual object (not shown) at the position of the marker in the DV streaming including the transmitted marker.

Here, although the left side of the network is shown as a remote (REMOTE) and the right side as a local (LOCAL), the configuration of the remote (REMOTE) and the local (LCOAL) is substantially the same, of course, it does not distinguish.

Now, by allowing augmented virtual objects to perform interactions using a virtual reality peripheral network (VRPN) irrespective of the network environment, it is possible to move away from the limitation of moving a virtual object only by moving a marker directly. Various interaction devices can be used to construct arbitrary interactions. In this embodiment, the phantom 116 and the joystick 156 are used to interact with the virtual object. Interaction with the virtual object is performed according to the movement or click of the joystick 156 as defined in the program. Here, the second computer 152 located in the local (LOCAL) is a VRPN server because the joystick 156 is located and is a VRPN client since the result of the interaction is reflected at the same time. In conclusion, every computer on the network running DV-ARPN is a DVTS client and a VRPN client at the same time. A computer with a digital video transmission device becomes a DVTS server and a computer with an interaction device becomes a VRPN server.

On the other hand, if only augmented results are received through DV streaming without directly augmenting locally (LOCAL), interaction is not possible, but streaming only augmented results may be more effective.

Furthermore, when multiple users form a network by using multicast, the results of the interaction may be shared with each other. For a description of a specific example thereof, see FIG. 6 below.

Thus, users can move away from the limitation of having to set up the real world where they are located in the background of augmented reality and install digital video transmission devices such as DV cameras wherever they are connected to the network. Can be used to bring background.

Figure 2 is a classification diagram showing the mixed reality according to the definition of the millgram applied to an embodiment of the present invention.

Referring to FIG. 2, according to Milgram's reality-virtuality Continuum, the surroundings of the virtual reality and the virtual world are virtual while the surroundings of the augmented reality are real, the virtual reality, the virtual world, and the augmentation. Realities are all defined as belonging to the category of mixed reality. Meanwhile, in order to construct augmented reality, an image of the real world as a background and virtual objects to be added to the image are needed. Here, the DV-ARPN proposed in the present invention is the development of augmented reality technology, augmented reality technology is a virtual reality (augmented virtuality), virtual environment (virtual environment) and the real world (real environment) without distinguishing one To integrate into the world. That is, augmented reality is expressed as if virtual objects exist in the same space in the real world, and virtual reality is expressed as if real objects exist in the same space in the virtual world, all of which are called mixed reality.

In addition, as mentioned above, a marker, which is a reference mark, is required to position the virtual object at a predetermined place in the real world, and the marker is placed directly in the real world to augment the virtual object at that location. However, since the marker is merely used as a reference marker to easily find a position and direction for augmenting a virtual object, it is not necessary and researches for implementing augmented reality without markers are also in progress.

3 is a flowchart of augmented reality using ARToolKit applied to the embodiment of FIG. 1.

Referring to FIG. 3, in step a (input image), an image is input through a camera. In step b, a thresholding image is binarized to find a marker in the input image. Here, binarization generates an image having only two values with a threshold level and a pixel value larger than that value in white and a small value in black (or vice versa). In step c (extraction of rectangle region), the rectangular region of the marker is found in the binarized image. In step d (pose and position estimation), the position of the marker is estimated at a certain position in the shape of the found rectangle. In step e (virtual image overlay), the virtual object is augmented according to its position and orientation, as estimated in step d.

4 is a structural diagram showing a relationship with the prior art applied to the DV-ARPN of FIG.

4, the technical relationship between DV-ARPN and DVTS, ARToolKit, and VRPN for implementing the remote meeting system according to the present invention is shown. In other words, by combining DVTS with ARToolKit, DV streaming from any digital video transmission device connected through a network can be implemented as an augmented reality background image, and by combining VRPN with ARToolKit, ARToolKit enhances augmented reality in the background. Interaction with objects can be implemented.

Therefore, since the DV-ARPN is implemented by organic coupling with DVTS, ARToolKit, and VRPN, it is not easily implemented by those skilled in the art because it requires a high technical knowledge.

5 is a flowchart of a remote meeting using the remote meeting system of FIG. 1.

Referring to FIG. 5, a preferred stage of remote meeting using a multimodal interaction based remote meeting system includes a greeting stage, a main meeting stage, and a farewell stage.

In the greeting stage or the fairwell stage, a network handshake using an interaction device, such as a haptic device, is applied to the network. The network handshake is a technology for virtually shaking hands with a partner using a haptic device, which combines the functions of the network with a force-feedback technique using a haptic interface.

In this example, the network handshake is implemented using the Ghost (GHOST) library, which enables you to control PHANToM based on Visual C ++ on the Windows platform, and directX for the Force-Feedback Joystick. By using DVTS, the other party's image is sent and his / her own image is input, and the other party's IP address is input and the 3DOF (Degree Of Freedom) information of each haptic device is transmitted to the other party in UDP (User Datagram Protocol) packets in real time. Here, in order to prevent a collision between information generated by the user's movement and information from the other party, the smaller information is ignored by comparing two force values.

The configuration of the network handshake is not particularly limited, and the network handshake may be configured by using two PHANToMs or using two force-feedback joysticks.

The main meeting stage, which covers the full-fledged meeting topic, uses the DV-ARPN described earlier.

Therefore, even in a remote meeting, the participants can participate in the meeting while recognizing the presence of the other party as in the actual meeting, thereby providing immersion to the party.

6 is an exemplary diagram illustrating an embodiment in which the remote meeting system of FIG. 1 is applied in a multicast environment.

Referring to FIG. 6, in an embodiment in a multicast environment, unlike a unicast environment, there may be a plurality of VRPN clients 760 sharing an interaction result. In other words, except for sharing the interaction result, if only the remote (REMOTE) and the local 1 (LOCAL1) exist in this figure, the unicast environment and communication method described in FIG. 1 will be substantially the same.

A computer 712 deployed with a DV camera 714 at REMOTE acts as a DVTS server to view the real world of the REMOTE over the network as DV streaming marked with marker 730, respectively. LOCAL2, LOCAL3).

Accordingly, the virtual object is overlaid on the position of the marker 730 in the background of the real world transmitted by the DV-ARPN proposed in the present invention, and the interaction device 740 is used to apply the interaction to the virtual object 710. Can be.

Here, the virtual object being augmented is a Buddha image 710, and the actual background where the DV streaming is transmitted is a mural 720 located at a remote location. The Buddha image 710 is augmented at the location of the marker 730 transmitted together with the mural 720 and displayed on the local (LOCAL1, LOCAL2, LOCAL3), and the interaction with the Buddha image 710 is performed by the joystick 740 as an interaction device. It is possible. On the other hand, the local (LOCAL2, LOCAL3) may only share the results of the interaction. In addition, at least two virtual objects may be created, and interactions with them may be performed independently.

7 is a conceptual diagram comparing the remote meeting system proposed in the present invention and the conventional remote meeting system.

Referring to FIG. 7, in the related art, a remote audio system (RAT), a videoconferencing tool (VIC), a VLC (VideoLAN Client), and a DVTS are applied to a remote meeting system 800 to improve the quality of voice and video of a remote meeting. the surface was placed, in the present invention is possible to give remote meetings, and realized by implementing a novel technique known as immersion-DV ARPN introduced the augmented reality and interaction. Here, RAT and VIC are open tools for teleconferencing developed by the University of London's Network and Multimedia Research Group, where RAT is for audio transmission and VIC is for video transmission. VLC is a software released under the GNU license under the VideoLAN project, which is capable of transmitting video and audio in various formats over a network.

8A and 8B are exemplary diagrams illustrating an image of remotely and locally implementing DV-ARPN according to the present invention, respectively.

Referring to FIG. 8A, a marker 930, a DV camera 920, and a DVTS server 910 for transmitting a DV streaming image are illustrated, and the DV streaming image being transmitted is displayed at a remote location.

Referring to FIG. 8B, an example of a DV-ARPN that implements augmented reality locally using a received DV streaming image is an interaction device, a joystick 940, which is an interaction device. The virtual image 950, a virtual object, is augmented and displayed in the marker region.

The present invention is not limited to the above embodiments, and many variations are possible by those skilled in the art within the spirit of the present invention.

As described above, according to the present invention, the network handshake and DV-ARPN by overcoming the limitation that the conventional A / V-based remote meeting system focuses only on voice and video, and does not give the user a realistic environment and interaction functions. With this new technology, you can create realistic remote meetings that support multi-modal interactions. This implementation is not limited to remote meetings, but can be applied to various fields such as more distance education and remote collaboration.

In addition, in the network handshake, the network concept is introduced to the haptic interface technology using the haptic device, so that the interaction between the user and the user can be realized by using a representative means of shaking hands, not the interaction between the user and the computer. have.

In addition, the DV-ARPN overcomes the limitation that virtual reality technology has to rely on the local camera input to set up the local real world in the background. It can be taken in the background and used. Therefore, there is an effect that can be the background of augmented reality at any time even in the real world where the distance is too far or dangerous so that the user can not visit directly.

Furthermore, by using a variety of interaction devices on augmented virtual objects, users can directly interact with each other to compose a variety of interactions, not just the movement of objects, and share the results of these interactions. There is an effect that can be comprehensively used in a variety of applications, such as education, remote meetings.

Claims (3)

A first interaction device, At least one first computer connected to the first interaction device, A first digital video transmission device connected to said first computer, At least one second computer connected to the first computer in a network; A second interaction device connected to the second computer and corresponding to the first interaction device; A second digital video transmission device connected to said second computer, Multi-modal interaction-based remote to augment virtual objects by using digital images provided from the first and second digital video transmission devices as a background of augmented reality and to interact with the virtual objects to the first and second interaction devices. Meeting system. The method of claim 1, And wherein the first or second interaction device is a haptic device. The method of claim 2, And the first and second haptic devices corresponding to each other provide a network handshake.

Images