KR20100064708A - Client terminal apparatus, collaboration service apparatus, real-time streaming apparatus, collaboration service system and its method - Google Patents

Abstract

PURPOSE: A client terminal apparatus, a collaboration service apparatus, a real-time streaming apparatus, a collaboration service system and a method thereof are provided to display modeling information in a virtual reality space to simultaneously display the modeling information on connected remote terminals, thereby cooperatively performing a modeling work through the virtual reality space. CONSTITUTION: An operation manager(204) receives communication/frame information with a collaboration service apparatus or a real-time streaming apparatus. The operation manager registers a molecular modeling simulation in the collaboration service apparatus through a connection manager(202). The operation manager performs a molecule modeling work for the molecular modeling simulation using a molecule renderer(210). The operation manager transmits the modeling information to the collaboration service apparatus through the connection manager.

Description

Client terminal device, collaboration service device, real-time streaming device, collaboration service system and method thereof

The present invention relates to a virtual reality-based collaboration system, and more particularly, to a client terminal device, a collaboration service device, a real-time streaming device, a collaboration service system, and a method for performing modeling work jointly through a plurality of terminals. It is about.

As is well known, molecular modeling simulation is one of the very important areas of information biology, and it is necessary to develop a pathogen therapeutic material such as AIDS virus, E. coli virus, mad cow disease and the like.

There are three major types of molecular modeling simulations. The first is to calculate the dynamic change in molecular structure. In general, the structure of protein changes in accordance with the change in the ambient temperature and the concentration of the aqueous solution, the properties also change this phenomenon is called protein folding (protein folding). Because these calculations occur in a very short time overall, they are difficult to calculate for large proteins, so we use a similar molecular model to solve the problem at the numerical level. This is called molecular dynamics.

The second is a calculation of the binding of two or more molecular structures that are static. In other words, docking is a process in which a candidate binds to a receptor such as a virus and changes to a stable protein. At the heart of the docking process is the process of finding the active site where the two molecular models combine to provide the lowest energy level.

The third is the process of protein matching that searches for similarities between proteins. This protein matching calculates the geometric similarity as well as the chemical similarity of the molecular structure model.

The characteristic of these molecular simulation tasks is that they use complex formulas to calculate the information of three-dimensional molecular structure. Therefore, there is a need for a task that can speed up a large amount of computation, and a task that can help experimenters to perform experiments more easily by visualizing in three dimensions.

On the other hand, it is also very important that a large number of experimenters participate in the experiment and collaborate in the experiment. Particularly, the performance of the experiment is judged to be improved according to the experimenter's experience. The help and advice of other experts is essential.

These molecular simulations are used to develop new materials and drugs, and they are used to simulate the biochemical experiments necessary through computers. Researchers in the field of biochemistry manipulate and combine molecular models modeled on computers. Based on these results, the energy calculation results of molecules are determined to determine whether the binding of molecules is stable.

In particular, molecular modeling is an experiment that finds a location where two molecules can be combined by calculating three-dimensional information and chemical characteristics of a molecular structure, which requires a large amount of computation, and according to the user's skill in the experiment, Is characterized by the performance of In order to support such molecular modeling work, a case has been reported using a real-time collaboration system in which many users can collaborate on molecular modeling work.

In the modeling work, in which many experimenters observe simulation results and collaborate with each other on the molecular modeling work as described above, a collaboration environment for observing modeling experiments in the web environment and providing feedback on the simulation results is provided. There is an urgent need, and the results of the modeling work are provided in real time to a large number of experimenters who are separated in space, and a collaborative system is being developed to cope with the modeling work.

Accordingly, the present invention establishes a virtual place for performing collaboration in a virtual reality space of a web environment, and is constructed by connecting a first client terminal device, a collaboration service device, and a second client terminal device connected through a communication network. The present invention provides a client terminal device, a collaboration service device, a real time streaming device, a collaboration service system, and a method of performing collaboration through a collaboration system.

In addition, the present invention provides modeling information to a plurality of second client terminal devices through the information storage device in the collaboration service device, to perform modeling work through a collaboration system that receives the modeling information in real time and performs a feedback function thereto. The present invention provides a client terminal device, a collaboration service device, a real time streaming device, a collaboration service system, and a method thereof.

In accordance with a first aspect, the present invention provides a connection manager for supporting wired and wireless communication with a collaboration service device or a real time streaming device, communication with the collaboration service device or a real time streaming device, and receiving frame information. Registers a molecular modeling simulation with the collaboration service device through the device, performs a molecular modeling operation using a molecular modeling tool on the registered molecular modeling simulation, and sends corresponding modeling information to the collaboration service device through the connection manager. A client terminal device including an operation manager for transmitting is provided.

According to a second aspect of the present invention, after registering a molecular modeling simulation according to a request from a first client terminal device and a communication unit in charge of various data communication through a wired / wireless communication module, corresponding modeling information is transmitted. When the modeling information is transmitted in real time to the connected second client terminal device at the same time through the communication unit, and additional modeling information corresponding to the molecular modeling simulation is transmitted from the second client terminal device. And a cooperative control unit which transmits the additional modeling information to the first client terminal device in real time, while transmitting to the real time streaming device through the communication unit, and devices of the first client terminal device and the second client terminal device. Information and this Provides a collaboration service apparatus including a collaboration information database for storing and managing the modeling information and the additional modeling information corresponding to each.

In a third aspect, the present invention provides a connection manager in communication with a plurality of client terminal devices, a collaboration service device, and a virtual reality device, and supports collaboration on molecular modeling simulation through the connection manager, and the molecular modeling simulation. And a session manager for real-time collaboration and collaboration over time according to a collaboration operation, and modeling information from any client terminal device through the collaboration service device. The present invention provides a real-time streaming device including a simulation exporter that stores corresponding to a molecular modeling simulation and transmits the same to another client terminal device and simultaneously transmits the same to another client terminal device.

In a fourth aspect, the present invention provides a first client terminal device for transmitting modeling information corresponding to a registered molecular modeling simulation, a second client terminal device for transmitting additional modeling information corresponding to the molecular modeling simulation, and A collaboration service device which registers a molecular modeling simulation and requests information transmission according to a real time collaboration or time-based collaboration of the modeling information or additional modeling information, and the first client terminal in response to a request from the collaboration service device; A real time streaming device which provides a real time streaming service between a device and a second client terminal device or provides the streaming service according to time, and transmits the modeling information or additional modeling information, and the modeling information transmitted from the real time streaming device. Or it provides a collaboration service system including a virtual reality device for displaying additional modeling information in the virtual reality space.

According to a fifth aspect of the present invention, there is provided a method of registering molecular modeling simulations in a cooperative service apparatus according to a request from a first client terminal apparatus, and modeling information corresponding to the registered molecular modeling simulations from the first client terminal apparatus. Receiving the modeling information in real time from the collaboration service device, and transmitting the modeling information in real time to a second client terminal device connected in response to the molecular modeling simulation in a real time streaming device according to a request from the collaboration service device; If additional modeling information corresponding to the molecular modeling simulation is received from the second client terminal device at the collaboration service device, transmitting the modeling information to the first client terminal device through the real-time streaming device. Provided.

According to a sixth aspect of the present invention, there is provided a method of registering molecular modeling simulations in a cooperative service apparatus according to a request from a first client terminal apparatus, and modeling information corresponding to the registered molecular modeling simulations from the first client terminal apparatus. Receiving at the collaboration service device and storing in the real-time streaming device, and when the second client terminal device corresponding to the molecular modeling simulation is connected, in response to a request from the collaboration service device is connected to the live streaming device; And providing the modeling information to a second client terminal device.

The present invention transmits the modeling information on which the modeling work is performed in real time, displays it on the virtual reality space, simultaneously displays the connected remote terminal, and additional modeling corresponding to the additional modeling work performed on the remote terminal. By transmitting the information in real time and displaying it on the virtual reality space, and simultaneously displaying it on the first client terminal device, the modeling work can be jointly performed using the virtual reality space.

According to the present invention, after activating the modeling tool of the first client terminal device, the modeling information according to the modeling work is transmitted to the cooperative service device, and the corresponding modeling information is stored to correspond to the molecular modeling simulation registered in the cooperative service device. The modeling information is transmitted to the real time streaming device, and the modeling information is transmitted to the virtual reality device, stored and displayed in the virtual reality space, and then the modeling information is transmitted to the connected second client terminal device for display and additional modeling work is performed. In this case, the corresponding modeling information is displayed in the virtual reality space and transmitted to the first client terminal device to be displayed, thereby providing a cooperative service. The technical means can solve the problems in the related art.

Hereinafter, exemplary embodiments of the present invention will be described in detail with reference to the accompanying drawings.

1 is a block diagram of a virtual reality-based collaboration system suitable for performing a collaboration task between a plurality of client terminals, according to an embodiment of the present invention, the first client terminal device 102, the communication network 104, the collaboration service device 106, a real time streaming device 108, a virtual reality device 110, and a second client terminal device 112.

Referring to FIG. 1, the first client terminal device 102 includes a terminal equipped with a virtual reality based molecular management system (VRMMS) for performing molecular modeling, and the like. , Personal work, time lag collaborative work, and the like, the collaboration service device 106, the real time streaming device 108, the virtual reality device 110, and the second client terminal device 112 connected through the communication network 104. ) To perform collaboration mode, and supports multiple interfaces including mouse, haptic interface, etc. for users who use virtual reality devices such as cave, haptic interface, or access through web browser. . Here, VRMMS is a virtual molecular modeling system that uses 3D information on molecular structure to express it as a 3D object through an open graphics library (OpenGL), manipulate a molecular structure using an input device, It means a system that can collaborate with users.

In addition, the first client terminal device 102 provides a stable connection with the collaboration service device 106, the real time streaming device 108, the virtual reality device 110, the second client terminal device 112, and the like, In the case of a job, it supports real-time streaming to receive various modeling information transmitted from the real-time streaming device 108. Here, the VRMMS mounted on the first client terminal device 102 for real time streaming may include, for example, a buffer using a sliding window.

Meanwhile, the first client terminal device 102 should perform 3D rendering, network transmission, and the like smoothly in order to perform collaboration work on various platforms. For this, for example, a Java-based applet Techniques used, techniques using active X, etc. are supported. Here, the technique using Java-based applets can express 3D rendering based on J3D, and has the advantage that it works on any platform with a JVM. The technique using ActiveX is written in C ++, OpenGL and Direct X. You can represent 3D representations with (direct X).

In addition, the first client terminal device 102 supports play modes such as play, pause, stop, and skip play for real-time streaming data, and rotates, zooms in and out, and moves the simulated molecular structure. It supports the operation mode of, and can support the representation mode (representation mode) to represent various aspects of the 3D molecules to be simulated.

On the other hand, the first client terminal device 102 includes a variety of configurations and functions as described above, to perform a collaboration mode, first, to connect to the collaboration service device 106 through the communication network 104 to experiment After registering the simulation work of molecular modeling and activating a molecular modeling tool such as VRMMS, various molecular modeling work (e.g., HIV virus modeling, E. coli virus modeling) in accordance with a manipulation signal input by a haptic interface method Etc.).

In addition, the first client terminal device 102 transmits modeling information (eg, rendering data, experimental data, image data, etc.) generated in the course of performing various molecular modeling operations through the communication network 104 ( And transmits the corresponding modeling information to a virtual place (for example, a virtual laboratory) provided in the virtual reality space of the virtual reality device 110 using a virtual reality viewer.

Meanwhile, the first client terminal device 102 performs additional modeling information performed by the second client terminal device 112 from the real-time streaming device 108 connected through the communication network 104 while performing various molecular modeling operations. For example, additional rendering data, additional experimental data, additional image data, etc.) are received and displayed. Here, when the additional modeling information is transmitted, the user can be notified of the reception using, for example, vibration or the like through the haptic interface, and the modeling work using the first client terminal device 102 can be paused. After the additional modeling information is transmitted and the additional modeling work of the second client terminal device 112 is completed, the modeling work using the first client terminal device 102 may be performed again.

The communication network 104 includes a wired / wireless communication network and includes a first client terminal device 102, a collaboration service device 106, a real time streaming device 108, a virtual reality device 110, and a second client terminal device 112. Are connected to each other and are responsible for transmitting various signals (data).

Next, the collaboration service device 106 provides various types of information in real time, including a server providing a real time relay service, a chat service, and the like, in order to support virtual reality-based collaboration. After registering the simulation according to the registration request for the simulation of the molecular modeling, and storing the modeling information transmitted from the first client terminal device 102 via the communication network 104 corresponding to the first client terminal device 102 In addition, the corresponding modeling information is transmitted to the real time streaming device 108 for its real-time collaboration work, and the first client terminal device 102 receives additional modeling information transmitted from the second client terminal device 112 through the communication network 104. ), And add it to the real-time streaming device 108 for its real-time collaboration. And transmits the model information.

That is, the collaboration service device 106 authenticates the second client terminal device 112 to participate in the joint work, stores the terminal information, and stores the first client terminal device 102 and the plurality of participating second devices. Each additional modeling information transmitted from the client terminal device 112 is transmitted to the real time streaming device 108 via the communication network 104, and the processed result is again the first client terminal device 102 and the plurality of second ones. Is transmitted to the client terminal device 112. The modeling information and the additional modeling information are transmitted to the first client terminal device 102 or the second client terminal device 112 by performing a real-time energy calculation and distributed processing.

Here, the collaboration service device 106 uses a protocol such as UDP (user datagram protocol, hereinafter referred to as 'UDP') for real-time relay service between the first client terminal device 102 and the second client terminal device 112. , Chat services and more.

The real-time streaming device 108 stores modeling information or additional modeling information transmitted from the collaboration service device 106 through the communication network 104 according to a log corresponding to the simulation, and stores the first client for the real-time collaboration work. The terminal device 102 and the second client terminal device 112 support real time simulation. Here, in the first client terminal device 102 and the second client terminal device 112, a user ID authenticated through the cooperative service device 106, a simulation ID registered for each user, and a time for requesting a simulation registration to execute a simulation After registering the information and the like with the real-time streaming device 108, the modeling information or additional modeling information transmitted from the cooperative service device 106 may be stored correspondingly.

In addition, if the real-time streaming device 108 is requested to transmit modeling information corresponding to the registered simulation from the first client terminal device 102 or the second client terminal device 112 through the communication network 104, the corresponding modeling information. Is extracted and transmitted in real time to the first client terminal device 102 or the second client terminal device 112 in a streaming manner.

In addition, the virtual reality device 110 includes a server that provides a virtual reality service through a virtual reality space and an avatar using the same, and the communication service device 106 or the real time streaming device 108 through the communication network 104. The modeling information or additional modeling information to be transmitted is displayed in a specific area (for example, a monitor provided in the virtual place) of the virtual place (for example, a virtual laboratory) provided in the virtual reality space, and the virtual reality data is displayed. A virtual reality service is provided to the connected first client terminal device 102 and the second client terminal device 112 in real time.

On the other hand, the second client terminal device 112 is a communication terminal including the configuration and function of the first client terminal device 102 as described above, connected to the collaboration service device 106 to simulate the modeling work. Requests for registration, receives and displays modeling information for jointly performing modeling tasks corresponding to registered simulations, and accesses the virtual reality device 110 to access a virtual place provided in the virtual reality space (for example, , Virtual laboratories, etc.), perform additional modeling operations using the displayed modeling information, and transmit the corresponding additional modeling information to the collaboration service apparatus 106 through the communication network 104.

Here, the second client terminal device 112 has been described as one terminal, but may be configured to include two or more terminals, each terminal information, such as terminal number, CPU, memory, graphics card performance, etc. According to the present invention, the modeling information whose data amount is adjusted can be received and displayed.

Next, the molecular modeling simulation is registered in the collaborative system having the above-described configuration, the modeling operation for the registered molecular modeling simulation is performed, the corresponding modeling information is transmitted, and further modeling is transmitted from the real-time streaming device. A client terminal device that receives and displays information and accesses a virtual reality device to display an image of a virtual reality space will be described.

FIG. 2 is a block diagram of a client terminal device performing a collaborative operation on a molecular modeling simulation according to an embodiment of the present invention. The client terminal device includes a first connection manager 202, an operation manager, and an operation manager. 204, a streaming manager 206, an IO parser 208, and a molecular renderer 210. Here, the client terminal device will be described as a device which is the first client terminal device 102 or the second client terminal device 112.

Referring to FIG. 2, the first connection manager 202 supports wired and wireless communication between the collaboration service device 106, the real time streaming device 108, and the client terminal device. For example, the first connection manager 202 may support, for example, a vrmms streaming service protocol. Application-based protocols such as) are used, and various web environments are supported by using a hyper-text transfer protocol (HTTP), UDP, etc. as a lower communication protocol.

The operation manager 204 is in charge of overall operation control and management of the client terminal device, and communicates with the collaboration service device 106 and the real time streaming device 108 through the first connection manager 202, and frame information. Control the streaming manager 206 to perform processing for the streaming service, control the IO parser 208 to handle parsing the 3D molecular structure, render and display the 3D molecular structure Control the molecular renderer 210 to perform the function.

In addition, the operation manager 204 accesses the collaboration service device 106 through the first connection manager 202 and registers the simulation work of the molecular modeling to be experimented with, and activates a molecular modeling tool such as VRMMS. And various molecular modeling operations (eg, HIV virus modeling, E. coli virus modeling, etc.) according to the input operation signals input by a haptic interface, and the like, and are generated in the process of performing various molecular modeling operations. The modeling information (for example, rendering data, experimental data, image data, etc.) is transmitted to the collaboration service device 106 through the first connection manager 202, and the virtual reality device (via the first connection manager 202). 110) to display the corresponding modeling information in a virtual place (for example, a virtual laboratory) provided in the virtual reality space. Such as word can be found by.

Next, the streaming manager 206 provides a transmission service using, for example, a buffer or the like in order to smoothly receive various modeling information received through the first connection manager 202. The information of one frame received through is stored in the buffer, and when the number of frames stored in the buffer exceeds a preset threshold, the corresponding value is read from the buffer and transmitted to the molecular renderer 210 to provide a transmission service. Here, the buffer may use a welcome queue or the like, for example.

On the other hand, the IO parser 208 performs a function of parsing the three-dimensional molecular structure file of the modeling information transmitted, parsing the three-dimensional molecular structure based on the chemical information of the molecular structure, the prototype of the three-dimensional protein structure It supports the PDB format which contains all information as general data and the PSF format which contains only chemical information of molecules, and supports to download and perform streaming service before streaming service.

In addition, the molecular renderer 210 renders and displays the molecular structure in three dimensions based on the frame information transmitted from the streaming manager 206. For the molecular structure, WF, Ball & Stick, Cartoon, Helix, Mesh, It can be rendered and displayed in the form of VDW.

Next, in the collaboration system as described above, the corresponding simulation is registered according to the registration request for the molecular modeling simulation from the first client terminal device, the collaboration service for the registered simulation is provided, and the corresponding modeling information is transmitted. When the second client terminal device is connected, it receives and stores it in the real-time streaming device. When the second client terminal device is connected, the real-time streaming device requests the real-time streaming device to transmit the modeling information for the registered simulation. A collaboration service device that requests a real time streaming device to transmit the additional modeling information to the first client terminal device when the first client terminal device connects with the request to be stored in the streaming device will be described.

3 is a block diagram of a collaboration service device suitable for providing a collaboration service between a plurality of clients according to an embodiment of the present invention, the collaboration service device 106 is a communication unit 302, collaboration control unit 304 and collaboration information database 306.

Referring to FIG. 3, the collaboration service device 106 will be described in detail. The communication unit 302 can communicate with the first client terminal device 102, the distributed processing device 108, and the virtual reality device 110 through the communication network 104. And it is connected to the second client terminal device 112 is responsible for transmitting various signals (data).

The collaboration control unit 304 receives modeling information transmitted from the first client terminal device 102 through the communication unit 302 and uses the modeling information corresponding to the first client terminal device 102 as the collaboration information database 306. ) And transmits the modeling information to the real-time streaming device 108 through the communication unit 302 for distributed transmission thereof.

In addition, the collaboration control unit 304, if additional modeling information is received from the second client terminal device 112 through the communication unit 302, the collaboration information database as additional modeling information corresponding to the second client terminal device 112. In operation 306, the additional modeling information is transmitted to the real-time streaming device 108 through the communication unit 302 for distributed transmission thereof.

Next, the collaboration control unit 304 receives the modeling information or the additional modeling information on which the predetermined distributed processing is performed through the communication unit 302 and receives the modeling information from the second client terminal device 112 and the virtual reality device 110. Or to transmit additional modeling information to the first client terminal device 102 and the virtual reality device 110.

On the other hand, the collaboration control unit 304 reduces the data amount of the modeling information according to the terminal information (for example, terminal number, CPU, memory, graphics card performance, etc.) of the second client terminal device 112 (for example, In the case of the low-specity terminal, the rendering data is simplified and transmitted), and the real-time relay service and the chat service between the first client terminal device 102 and the second client terminal device 112 are performed through the UDP protocol. To control.

Here, the collaboration information database 306 includes modeling information corresponding to the first client terminal device 102, additional modeling information corresponding to the second client terminal device 112, and device information of the first client terminal device 102. (For example, device number, CPU, memory, graphics card performance, etc.), terminal information of the second client terminal device 112, distributed result information, and the like are stored. 304).

Next, the modeling information transmitted from the collaboration service device is stored in correspondence with the registered molecular modeling simulation in the collaboration system as described above. Then, when the second client terminal device is connected, the modeling information corresponding to the simulation is extracted. The following describes a real-time streaming device that transmits to the second client terminal device, stores additional modeling information from the second client terminal device, and then transmits to the first client terminal device.

FIG. 4 is a block diagram of a real-time streaming apparatus that stores modeling information and performs real-time streaming transmission according to an embodiment of the present invention. The real-time streaming apparatus 108 includes a second connection manager 402 and a session manager. (session manager 404), simulation exporter (406) and session pool (408).

Referring to FIG. 4, the second connection manager 402 is responsible for connecting to the first client terminal device 102, the collaboration service device 106, the second client terminal device 112, and the like through the communication network 104. When connected to any one device through the communication network 104, a session is allocated according to the connection information to perform data communication. In this case, the allocated session may include a user ID, a simulation ID, time information, and the like.

In addition, the session manager 404 supports collaboration on real-time molecular modeling work according to the assigned session, and connects various devices participating in the simulation of molecular modeling for simulation through each session, and according to the collaboration work. It is divided into real-time collaboration and time-based collaboration. In the case of real-time collaboration, molecular modeling of each device can be transmitted to each other in real time. After being performed in one device, the modeling information is stored, when the other device is connected, the modeling information is transmitted, and the modeling information is transmitted to the virtual reality device 110. In this case, the collision occurring when manipulating multiple objects at the same time when transmitting to each other in real time may be controlled using, for example, a floor control mechanism.

In addition, when the modeling task or the additional modeling task is completed, the simulation exporter 406 stores modeling information or additional modeling information for the corresponding simulation, transmits it to another client terminal device connected to the communication network 104, and the communication network 104. ) Through the second connection manager 402 to the connected virtual reality device 110.

On the other hand, since the session pool 408 dynamically creates and deletes session information according to the situation of the connected device, the session pool 408 manages session resources by checking time information and state of the session.

Next, after activating the modeling tool of the first client terminal device having the above-described configuration, the modeling information according to the modeling work is transmitted to the collaboration service device, and stored in correspondence with the molecular modeling simulation registered in the collaboration service device. The modeling information is transmitted to the real-time streaming device so that the modeling information is transmitted to the virtual reality device, stored and displayed in the virtual reality space, and then the modeling information is transmitted to the connected second client terminal device for display. When the additional modeling operation is performed, the process of displaying the additional modeling information corresponding to the same in the virtual reality space and transmitting the same to the first client terminal device is described.

5 is a flowchart illustrating a process of jointly performing a modeling work in real time according to the first embodiment of the present invention. Hereinafter, the first client terminal device 102 will be described as the terminal device of the main experimenter and the second client terminal device 112 will be described as the terminal device of the secondary experimenter.

Referring to FIG. 5, in the standby mode of the virtual reality-based collaboration system (step 502), the collaboration service device 106 may perform a collaborative operation from the first client terminal device 102 through the communication network 104. It is checked if registration of the modeling simulation is requested (step 504).

As a result of the check in the step 504, if the registration of the molecular modeling simulation is requested, the cooperative service device 106 registers the corresponding molecular modeling simulation (step 506).

In addition, the collaboration service device 106 transmits the registration information of the registered molecular modeling simulation to the real time streaming device 108, and performs the molecular modeling simulation on the real time streaming device 108 according to the session information corresponding to the registration information. By registering and storing, the real time collaboration service mode is performed between the first client terminal device 102 and the second client terminal device 112 using the collaboration service device 106, the real time streaming device 108, and the virtual reality device 110. (Step 508).

Next, when the first client terminal device 102 inputs a three-dimensional position of the haptic interface for activating the modeling tool using an input means, a haptic interface, and the like, which are not shown in the cooperative service mode, the modeling tool that is generated accordingly. After activating the corresponding modeling tool according to the activation manipulation signal, various modeling operations are performed through a molecular modeling tool such as VRMMS according to the manipulation signal corresponding to the 3D position input of the haptic interface (step 510). Here, the first client terminal device 102 is equipped with a molecular modeling tool such as a VRMMS, and has a virtual reality viewer for displaying the virtual reality space by accessing the virtual reality device 110.

In addition, the first client terminal device 102 configures modeling information including rendering data, experimental data, image data, etc. according to the molecular modeling work, and transmits the modeling information to the cooperative service device 106 through the communication network 104 ( Step 512).

In addition, the collaboration service apparatus 106 receives modeling information transmitted from the first client terminal apparatus 102 through the communication network 104, stores the corresponding modeling information corresponding to the first client terminal apparatus 102, and stores the real-time streaming service thereof. Modeling information is transmitted to the real-time streaming device 108 to provide, the real-time streaming device 108 stores the modeling information corresponding to the registered molecular modeling simulation (step 514).

The real-time streaming device 108 transmits this modeling information to the virtual reality device 110 through the communication network 104 (step 516). Thereafter, the virtual reality device 110 stores such modeling information in its own database and at the same time a predetermined region (for example, a monitor provided in the laboratory) of a virtual place (for example, a virtual laboratory) provided in the virtual reality space. The virtual reality data displayed in the virtual reality space is transmitted to the first client terminal device 102 or the second client terminal device 112 connected through the communication network 104 in real time to be provided. Will be provided.

Meanwhile, the real-time streaming device 108 checks the connection of the second client terminal device 112 that registered the molecular modeling simulation corresponding to the modeling information, and then connects the modeling information through the communication network 104 to the second client terminal device. Transmit to step 112 (step 518). Here, the modeling information decreases the data amount according to the terminal information (for example, terminal number, CPU, memory, graphics card performance, etc.) of the second client terminal device 112 (for example, a low-specified terminal). In this case, the mesh of rendering data can be simplified and transmitted.

Next, the second client terminal device 112 receives modeling information transmitted from the real-time streaming device 108 through the communication network 104 and displays it through a molecular modeling tool (eg, VRMMS, etc.) (step) 520). Of course, the second client terminal device 112 may access the virtual reality device 110 and simultaneously display the corresponding 3D image displayed in the virtual place provided in the virtual reality space through the virtual reality viewer.

Therefore, the first client terminal device, the collaboration service device, the real-time streaming device, the virtual reality device performs a modeling operation for the molecular modeling simulation registered in the collaboration service mode connected to the second client terminal device and transmits to another connected client terminal device. By displaying this, the collaboration work between the first client terminal device and the second client terminal device can be performed.

Next, a process of transmitting and displaying additional modeling information about the additional modeling work performed through the second client terminal device while performing the cooperative service mode as described above is displayed to the connected first client terminal device.

6 is a flowchart illustrating a process of jointly performing a modeling work in real time according to a second embodiment of the present invention.

Referring to FIG. 6, a real-time collaboration operation between the first client terminal device 102 and the second client terminal device 112 is performed through the collaboration service device 106, the real time streaming device 108, and the virtual reality device 110. In the collaborative service mode (step 602) to be performed, the second client terminal device 112 displays additional modeling information through a modeling tool such as a VRMMS, for example, while displaying the modeling information worked on the first client terminal device 102. Perform the operation (step 604).

In operation 606, the second client terminal device 112 transmits additional modeling information corresponding to the additional modeling job to the collaboration service device 106 through the communication network 104.

Next, the collaborative service device 106 stores additional modeling information transmitted from the second client terminal device 112, and then transmits the additional modeling information to the real time streaming device 108 to provide a real time streaming service thereof. In 108, the additional modeling information is stored corresponding to the registered molecular modeling simulation (step 608).

In addition, the real-time streaming device 108 transmits the additional modeling information to the virtual reality device 110 through the communication network 104 (step 516). Thereafter, the virtual reality device 110 stores such additional modeling information in its own database and at the same time a predetermined area (for example, a monitor provided in the laboratory) of a virtual place (for example, a virtual laboratory) provided in the virtual reality space. The virtual reality data displayed in the virtual reality space is transmitted to the first client terminal device 102 or the second client terminal device 112 connected through the communication network 104 in real time to be displayed. To provide services.

Next, the real-time streaming device 108 checks the connection of the first client terminal device 102 that registered the molecular modeling simulation, and then connects the additional modeling information through the communication network 104 to the first client terminal device 102. (Step 610). Here, the modeling information decreases the data amount according to the terminal information (for example, terminal number, CPU, memory, graphics card performance, etc.) of the first client terminal device 112 (for example, a low-specified terminal). In this case, the mesh of rendering data can be simplified and transmitted.

Subsequently, the first client terminal device 102 receives the additional modeling information transmitted and displays it through a modeling tool such as a VRMMS (step 614). Of course, the first client terminal device 102 may access the virtual reality device 110 and simultaneously display the corresponding 3D image displayed in the virtual place provided in the virtual reality space through the virtual reality viewer.

Meanwhile, in the first and second embodiments of the present invention, the modeling information or the additional modeling information is transmitted and displayed on the corresponding terminal device. However, when the modeling work or the additional modeling work is performed, the corresponding models correspond to each other. The work signal is transmitted to the second client terminal device 102 or the first client terminal device 112 performing the collaborative work, and the modeling work is performed through the haptic interface through the first client terminal device 102 or the second client terminal device. Alternatively, with the vibration output indicating that an additional modeling operation is in progress, the modeling operation or the additional modeling operation may be stopped.

Accordingly, the first client terminal is transmitted by displaying additional modeling information corresponding to the additional modeling work performed by the second client terminal device in the collaborative service mode for the molecular modeling simulation in real time to the first client terminal device in collaboration. Collaboration between the device and the second client terminal device may be performed.

Next, when performing the cooperative service as described above, after performing the modeling operation in the first client terminal device, and transmits the corresponding modeling information to the cooperative service device, and in response to the registered molecular modeling simulation and the real-time streaming device and Next, a description will be given of a collaboration service based on a time difference performed by a method of transmitting the modeling information to the second client terminal device when the second client terminal device, which is stored in the virtual reality device and subsequently registers the molecular modeling simulation, is connected.

7 is a flowchart illustrating a cooperative service based on a time difference according to a third embodiment of the present invention. Hereinafter, the first client terminal device 102 completes the modeling work and transmits the modeling information, and after the connection of the first client terminal device 102 is terminated and a predetermined time elapses, the second client terminal device 112 ) Will be described as connecting to the collaboration service apparatus 106.

Referring to FIG. 7, after the collaboration service device 106 registers the registration information according to the molecular modeling simulation registration request from the first client terminal device 102, the registration information of the registered molecular modeling simulation is transferred to the real-time streaming device 108. By transmitting and registering the molecular modeling simulation in the real-time streaming device 108 according to the session information corresponding to the registration information, the collaboration service device 106, the real-time streaming device 108 and the virtual reality device 110 is stored. The cooperative service mode of the first client terminal device 102 used is performed (step 702).

The first client terminal device 102 activates a modeling tool generated according to a 3D position of the haptic interface that activates the modeling tool using an input unit, a haptic interface, and the like, which are not shown in the cooperative service mode. After activating the corresponding modeling tool according to the manipulation signal, various modeling operations are performed through a molecular modeling tool such as VRMMS according to the manipulation signal corresponding to the 3D position input of the haptic interface (step 704). Here, the first client terminal device 102 is equipped with a molecular modeling tool such as a VRMMS, and has a virtual reality viewer for displaying the virtual reality space by accessing the virtual reality device 110.

In addition, the first client terminal device 102 configures modeling information including rendering data, experimental data, image data, etc. according to the molecular modeling work, and transmits the modeling information to the cooperative service device 106 through the communication network 104 ( Step 706).

In addition, the collaboration service apparatus 106 receives modeling information transmitted from the first client terminal apparatus 102 through the communication network 104, stores the corresponding modeling information corresponding to the first client terminal apparatus 102, and provides a streaming service thereof. In order to transmit the modeling information to the real-time streaming device 108, the real-time streaming device 108 stores the modeling information corresponding to the registered molecular modeling simulation (step 708).

In operation 710, the modeling information is transmitted to the virtual reality device 110 through the communication network 104. Thereafter, the virtual reality device 110 stores such modeling information in its own database and at the same time a predetermined region (for example, a monitor provided in the laboratory) of a virtual place (for example, a virtual laboratory) provided in the virtual reality space. The virtual reality data displayed in the virtual reality space is transmitted to the corresponding terminal device in real time when the first client terminal device 102 or the second client terminal device 112 is connected through the communication network 104. It provides a virtual reality service.

On the other hand, the collaboration service device 106 checks whether the second client terminal device 112 that has registered the molecular modeling simulation corresponding to the modeling information is connected (step 712).

As a result of the check in the step 712, when the second client terminal device 112 is connected, the collaboration service device 106 requests to send the modeling information corresponding to the registered molecular modeling simulation, accordingly the real-time streaming device In operation 108, the modeling information is transmitted to the second client terminal apparatus 112 connected through the communication network 104. Here, the modeling information decreases the data amount according to the terminal information (for example, terminal number, CPU, memory, graphics card performance, etc.) of the second client terminal device 112 (for example, a low-specified terminal). In this case, the mesh of rendering data can be simplified and transmitted.

Next, the second client terminal device 112 receives modeling information transmitted from the real-time streaming device 108 through the communication network 104 and displays it through a molecular modeling tool (eg, VRMMS, etc.) (step) 716). Of course, the second client terminal device 112 may access the virtual reality device 110 and simultaneously display a 3D image including corresponding molecular modeling information displayed in a virtual place provided in the virtual reality space through the virtual reality viewer. have.

In addition, the second client terminal device 112 performs additional modeling through a modeling tool such as VRMMS, for example, while displaying modeling information worked through the first client terminal device 102. The information is transmitted to the collaboration service apparatus 106 via the communication network 104 (step 718).

Next, the collaboration service device 106 stores the additional modeling information transmitted from the second client terminal device 112, and then transmits the modeling information to the live streaming device 108 to provide a streaming service thereof. In operation 108, the additional modeling information is stored corresponding to the registered molecular modeling simulation.

In addition, the real-time streaming device 108 transmits the additional modeling information to the virtual reality device 110 through the communication network 104 (step 722). Thereafter, the virtual reality device 110 stores such additional modeling information in its own database and at the same time a predetermined area (for example, a monitor provided in the laboratory) of a virtual place (for example, a virtual laboratory) provided in the virtual reality space. The virtual reality data displayed in the virtual reality space may be displayed in real time on the corresponding terminal device when the first client terminal device 102 or the second client terminal device 112 is connected through the communication network 104. Transmitted to provide a virtual reality service.

Meanwhile, in the third embodiment as described above, the additional modeling information worked through the second client terminal device 112 has been described as being stored in the real-time streaming device and the virtual reality device. When the client terminal device 102 connects, the same process may be performed on the first client terminal device 102 as the second client terminal device 112 connects to receive and display modeling information.

Therefore, the molecular modeling information worked through the first client terminal device is transmitted to the collaboration service device, and stored in correspondence with the molecular modeling simulation registered in the real-time streaming device. Then, when the second client terminal device accesses the corresponding modeling information. By transmitting to the second client terminal device to display it, it is possible to effectively provide a cooperative service between the client terminal devices having a time difference.

In the foregoing description, the present invention has been described with reference to preferred embodiments, but the present invention is not necessarily limited thereto. Those skilled in the art will appreciate that the present invention may be modified without departing from the spirit of the present invention. It will be readily appreciated that branch substitutions, modifications and variations are possible.

1 is a block diagram of a virtual reality-based collaboration system suitable for performing collaboration between a plurality of client terminals according to an embodiment of the present invention;

2 is a block diagram of a client terminal device performing a collaborative operation on a molecular modeling simulation according to an embodiment of the present invention;

3 is a block diagram of a collaboration service apparatus suitable for providing a collaboration service between a plurality of clients according to an embodiment of the present invention;

4 is a block diagram of a real-time streaming device for storing modeling information and performing real-time streaming transmission according to an embodiment of the present invention;

5 is a flowchart illustrating a process of jointly performing a modeling work in real time according to the first embodiment of the present invention;

6 is a flowchart illustrating a process of jointly performing a modeling work in real time according to a second embodiment of the present invention;

7 is a flowchart illustrating a collaboration service based on a time difference according to a third embodiment of the present invention.

<Description of the symbols for the main parts of the drawings>

102: first client terminal device 104: communication network

106: collaboration service device 108: live streaming device

110: virtual reality device 112: second client terminal device

202: first connection manager 204: operation manager

206: Streaming Manager 208: IO Parser

210: molecular renderer 302: communication unit

304: collaboration control unit 306: collaboration information database

402: second connection manager 404: session manager

406 Simulation Exporter 408 Session Pool

Claims (34)

A connection manager that supports wired and wireless communication with a collaboration service device or a live streaming device; Communication with the collaboration service device or real-time streaming device and reception of frame information are performed, a molecular modeling simulation is registered with the collaboration service device through the connection manager, and a molecular modeling tool is applied to the registered molecular modeling simulation. An operation manager for performing molecular modeling by using the same, and transmitting corresponding modeling information to the collaboration service device through the connection manager; Client terminal device comprising a. The method of claim 1, The client terminal device, A streaming manager for providing a streaming service using a buffer for the modeling information received through the connection manager under the control of the operation manager; An IO parser for parsing the 3D molecular structure file of the modeling information; Molecular renderer that renders and displays the 3D molecular structure of the modeling information Client terminal device further comprising. The method of claim 2, The streaming manager, the client terminal device including the buffer of the sliding window method for providing the streaming service. The method of claim 3, wherein The connection manager includes a vrmms streaming service protocol (VSSP) corresponding to a virtual reality based molecular management system (VRMMS) used as the molecular modeling tool, and a hyper-text transfer protocol (HTTP) or a lower communication protocol. A client terminal device having a user datagram protocol (UDP). The method of claim 2, The operation manager is connected to the virtual reality device through the connection manager to control to receive and display the virtual reality data including the modeling information. The method of claim 2, And the operation manager is configured to receive and display additional modeling information of another client terminal device in real time from the real time streaming device connected through the connection manager. The method of claim 2, The operation manager is connected to the virtual reality device through the connection manager to receive and display virtual reality data including the additional modeling information. The method according to any one of claims 2 to 7, The modeling information includes rendering data, experimental data, and image data. The method according to any one of claims 2 to 7, The client terminal device supports a play mode of play, pause, stop, and skip play for the streaming service, and an operation mode of rotation, zoom in and out, and movement of the 3D molecular structure. And a representation mode for representing various aspects of the 3D molecular structure. Communication unit in charge of various data communication through wired and wireless communication module, After registering the corresponding molecular modeling simulation according to a request from the first client terminal device, if corresponding modeling information is transmitted, the modeling information is transmitted to the real-time streaming device through the communication unit and simultaneously connected to the second client terminal device. When the modeling information is transmitted in real time and additional modeling information corresponding to the molecular modeling simulation is transmitted from the second client terminal device, the first client terminal device is simultaneously transmitted to the real time streaming device through the communication unit. A collaboration control unit requesting to transmit the additional modeling information in real time; A collaboration information database for storing and managing device information of the first client terminal device and the second client terminal device, and the modeling information and the additional modeling information corresponding to the device information. Collaboration service device comprising a. The method of claim 10, The collaboration control unit transmits the modeling information corresponding to the molecular modeling simulation from the first client terminal device to the real time streaming device in case of a time difference collaboration service between the first client terminal device and the second client terminal device. And storing the modeling information when the second client terminal device is connected to the real-time streaming device. The method of claim 10, The collaboration controller, when the time difference collaboration service between the first client terminal device and the second client terminal device, the additional modeling information corresponding to the molecular modeling simulation is transmitted from the second client terminal device, to the real-time streaming device And requesting to transmit and store, and when the first client terminal device is connected, requesting to the real-time streaming device to transmit the additional modeling information. The method according to any one of claims 10 to 12, And the collaboration controller requests the real time streaming device to transmit the modeling information or the additional modeling information to the virtual reality device in which the virtual reality space corresponding to the molecular modeling simulation is implemented. The method according to any one of claims 10 to 12, The modeling information and the additional modeling information include rendering data, experimental data, and image data corresponding to the modeling task and the additional modeling task, respectively. The method according to any one of claims 10 to 12, And the device information includes CPU, memory, and graphics card capabilities corresponding to the first client terminal device and the second client terminal device, respectively. A connection manager for communicating with a plurality of client terminal devices, collaboration service devices, and virtual reality devices; A session manager to support collaboration on molecular modeling simulation through the connection manager, to connect various devices participating in the molecular modeling simulation, and to support real-time collaboration and collaboration over time according to a collaboration task; Simulation exporter that receives modeling information from one client terminal device through the collaboration service device, stores corresponding to the molecular modeling simulation, and transmits the same to another client terminal device to the virtual reality device. Live streaming device comprising a. The method of claim 16, The live streaming device checks time information and state of a session corresponding to a connected device to manage session resources. Live streaming device further comprising. 18. The method according to claim 16 or 17, When the connection manager is connected to any one device through a communication network, the connection manager performs data communication according to session information allocated according to a user ID, a simulation ID, and time information according to the connection information. 18. The method according to claim 16 or 17, The session manager supports the modeling information to be mutually transmitted to each other in real time in the case of the real-time collaborative collaboration, and stores the modeling information from any one client terminal device in the case of the collaborative collaboration over time. And transmitting the modeling information when the other client terminal device is connected. The method of claim 19, The session manager stores the additional modeling information from the other client terminal device in case of the collaborative collaboration over time, and transmits the additional modeling information when the one client terminal device is connected. The method of claim 20, The session manager, in the case of the real-time collaborative collaboration to control the collision occurring when operating a plurality of objects at the same time using a floor control mechanism (floor control mechanism). A first client terminal device for transmitting modeling information corresponding to the registered molecular modeling simulation; A second client terminal device for transmitting additional modeling information corresponding to the molecular modeling simulation; A collaboration service device that registers the molecular modeling simulation and requests information transmission based on real-time collaboration or time-based collaboration of the modeling information or additional modeling information; A real time streaming device for transmitting the modeling information or the additional modeling information by providing a real time streaming service between the first client terminal device and the second client terminal device or a streaming service according to the time according to a request from the collaboration service device. Wow, The virtual reality device displaying the modeling information or additional modeling information transmitted from the real time streaming device in a virtual reality space. Collaboration service system comprising a. The method of claim 22, And the modeling information and the additional modeling information include rendering data, experimental data, and image data corresponding to the modeling task and the additional modeling task, respectively.  Registering a molecular modeling simulation at the collaboration service device according to a request from the first client terminal device; Receiving modeling information corresponding to the registered molecular modeling simulation from the first client terminal device in the collaboration service device; Transmitting the modeling information in real time to a second client terminal device connected in response to the molecular modeling simulation in a real time streaming device according to a request from the collaboration service device; If additional modeling information corresponding to the molecular modeling simulation is received from the second client terminal device at the collaboration service device, transmitting the additional modeling information to the first client terminal device through the real-time streaming device. Collaboration service method comprising a. The method of claim 24, The collaboration service method, Transmitting the modeling information to the virtual reality device in the step of transmitting the modeling information in real time. Collaboration service method further comprising. The method of claim 24, The collaboration service method, Transmitting the additional modeling information to the virtual reality device at the same time as transmitting to the first client terminal device; Collaboration service method further comprising. The method according to any one of claims 24 to 26, The modeling information and the additional modeling information include rendering data, experimental data, and image data corresponding to the modeling task and the additional modeling task, respectively. The method according to any one of claims 24 to 26, And the modeling information and the additional modeling information are transmitted according to CPU, memory, and graphics card capabilities respectively corresponding to the first client terminal device or the second client terminal device. Registering a molecular modeling simulation at the collaboration service device according to a request from the first client terminal device; Receiving modeling information corresponding to the registered molecular modeling simulation from the first client terminal device at the collaboration service device and storing the modeling information in a real time streaming device; When the second client terminal device corresponding to the molecular modeling simulation is connected, transmitting the modeling information from the real-time streaming device to the connected second client terminal device according to a request from the collaboration service device. Collaboration service method comprising a. 30. The method of claim 29, The collaboration service method, Storing additional modeling information corresponding to the molecular modeling simulation from the second client terminal device in the collaboration service device, and storing the additional modeling information in the real-time streaming device; Transmitting the additional modeling information to the first client terminal device in the real time streaming device when the first client is connected to the collaboration service device; Collaboration service method further comprising. 31. The method of claim 30, The collaboration service method, Transmitting the modeling information to the virtual reality device at the same time as storing in the real time streaming device. Collaboration service method further comprising. 31. The method of claim 30, The collaboration service method, Transmitting the additional modeling information to the virtual reality device at the same time as transmitting to the first client terminal device; Collaboration service method further comprising. The method according to any one of claims 29 to 32, The modeling information and the additional modeling information include rendering data, experimental data, and image data corresponding to the modeling task and the additional modeling task, respectively. The method according to any one of claims 29 to 32, And the modeling information and the additional modeling information are transmitted according to CPU, memory, and graphics card capabilities corresponding to the first client terminal device or the second client terminal device, respectively.

Images