KR101265123B1 - 3D VR Controlling Middleware System interconnecting Voice Engine and interactive 3D Application in real time - Google Patents

Abstract

The present invention provides a 3D virtual reality control middleware system for interworking a voice engine and an interactive 3D application in real time, comprising: a voice input unit 10 for receiving a voice signal from a user; A voice engine processor 20 which receives a voice signal input from the voice input unit and transmits the voice signal to a voice engine and receives character string data corresponding to the voice signal retrieved and returned from the voice database 21 by the voice engine; A message processor (30) for receiving the string data from the voice engine processor and searching for the message data corresponding to the string data in a message database (22) storing message data recognizable by the 3D data; An operation control unit (40) for receiving the message data from the message processing unit and controlling execution and state change of a 3D object according to the message data; And a 3D renderer 50 which updates and displays the execution and the state change of the 3D object whose execution and state change are controlled by the operation control unit in real time.

Description

 3D VR Controlling Middleware System interconnecting Voice Engine and interactive 3D Application in real time}

The present invention relates to a 3D virtual reality control middleware system and a method for real-time control of 3D virtual reality content using the same, and in particular, a 3D virtual reality control middleware system for real-time interworking an interactive 3D application with a voice engine that recognizes a user's voice. It relates to a real-time control method of the 3D content used.

With the rapid development of information and communication technology, a new digital user environment is applied that applies future content technologies such as ubiquitous computing technology, network infrastructure, 3D technology, and virtual reality technology. The market is expanding to various fields. In addition, based on the PC, 3D content, that is, 3D applications are utilized for the platform of various devices such as digital devices and mobile devices such as PMP, navigation, MP3, smart phones.

As a result, the interface of 3D applications that can cope with various fields and platforms becomes a very important factor. Therefore, various and interactive results beyond the interface based mainly on keyboard input, touch input, and mouse input of existing 3D applications can be obtained. There is a need for an advanced interface based on voice recognition.

On the other hand, speech recognition technology refers to a technology that a computer interprets a speech language spoken by a person and converts its contents into text data, which is a character string, and processes the text by using a speech engine. It is currently used in a limited number of English education systems, navigation, and mobile search systems.

Therefore, in the development of an interface in which voice recognition technology and 3D content (application) are combined, there is an urgent need to develop a middleware technology in which a user's command, etc., are input in the form of a voice signal and an interactive 3D content. .

However, there is a technical problem that the text data such as commands recognized by the voice engine is not recognized in the 3D data of the 3D content (application) when the voice engine is combined with the 3D content (application).

Accordingly, the present invention provides a 3D virtual reality control middleware system for real-time interworking a voice engine and interactive 3D contents, and a method for real-time control of 3D contents using the same, which solves the problems of the related art as described above. It is aimed at making it possible to provide an advanced interface based.

In addition, by converting the voice data processed by the voice engine that recognizes the user's voice into message data that can be recognized by the 3D data, the 3D virtual reality control middleware system for real-time interworking of the voice engine and the interactive 3D content and the 3D content using the same It is an object to provide a real-time control method.

Furthermore, in the 3D virtual reality control middleware system and real-time control method of 3D content using the same by converting the voice data processed by the voice engine into message data that can be recognized by the 3D data, The purpose is to improve the performance of real-time control.

In order to achieve the above object, the present invention provides a 3D virtual reality control middleware system for interworking a voice engine and an interactive 3D application in real time, the voice input unit 10 for receiving a voice signal from a user; A voice engine processor 20 which receives a voice signal input from the voice input unit and transmits the voice signal to a voice engine and receives character string data corresponding to the voice signal retrieved and returned from the voice database 21 by the voice engine; A message processor (30) for receiving the string data from the voice engine processor and searching for the message data corresponding to the string data in a message database (22) storing message data recognizable by the 3D data; An operation control unit (40) for receiving the message data from the message processing unit and controlling execution and state change of a 3D object according to the message data; And a 3D renderer 50 which updates and displays the execution and the state change of the 3D object whose execution and state change are controlled by the operation control unit in real time.

Preferably, the voice input unit 10 is configured to check the connection state of the input device for voice input, to generate and output a status message.

In addition, when the message data corresponding to the string data is not found, the message processing unit 30 performs a process of deleting the first one character of the string data and searching for the message data corresponding to the deleted string data of one character. The process may be configured to be performed until the corresponding message data is searched or until one character remains in the string data.

Further, preferably, the message database has a data structure including a voice string, which is string data received from a user, and a message string, which is message data to be transmitted to an operation controller, wherein the string data is a Korean character string of 2 bytes, and the message data is Characterized as a 1-byte English string containing the action name.

       In addition, the operation control unit 40 is preferably configured to include a script including a message receiving unit, a message processing unit, and the termination processing unit generated for the object.

In another aspect, the present invention provides a method for real-time control of 3D content using a 3D virtual reality control middleware system, the method comprising: receiving a voice signal from a user through a voice input device connected to the middleware system (S10); Accessing a voice database and retrieving and storing string data corresponding to the received voice signal (S20); Retrieving message data corresponding to the stored string data from a message database storing message data recognizable by 3D data (S30); controlling a behavior of a 3D object according to the retrieved message data (S40); And updating and displaying the behavior of the 3D object whose behavior is controlled in real time (S50).

Preferably, the step (S10) of receiving a voice signal from the user includes an accuracy determination step (C5) for determining the accuracy of the voice based on the size of the voice, the accuracy of the voice, the speed of the voice.

Also, preferably, the step S30 of retrieving the message data may include: a first step of retrieving message data corresponding to the character string data from the message database; And a second step of deleting the first letter of the character string data and retrieving message data corresponding to the deleted character string data from the message database, by performing a first step on the stored character string data. If the corresponding message data is found, the searched message data is returned. If the corresponding message data is not found, the second step is performed until one character remains in the string data.

Further, the step (S40) of controlling the behavior of the 3D object may include: a message waiting step (S4) waiting for reception of message data; A message receiving step (S5) of receiving message data; An action execution step S6 of executing an action of a related object according to the received message data; And an action execution end determination step (S7) of determining whether execution of the action of the related object is completed according to the received message data, and if it is determined that execution of the action is completed (S8), In this case, it is preferable to perform a notification to receive an additional voice input.

Further, the step of updating and displaying the behavior of the 3D object in real time (S50) may include generating an application user interface (C1) and generating an application window view (C1) and every frame of the loaded screen. By performing the screen load step (C3) that is linked to the application window view to be updated and displayed, the behavior of the 3D object is updated and displayed in real time.

According to the present invention as described above, it is possible to provide an advanced interface based on voice recognition by providing a 3D virtual reality control middleware system for real-time interworking a voice engine and interactive 3D content and a real-time control method of the 3D content using the same. Done.

In addition, by converting the voice data processed by the voice engine that recognizes the user's voice into message data that can be recognized by the 3D data, it is possible to efficiently link the voice engine and interactive 3D content in real time.

Furthermore, it is possible to improve the performance of real-time control of 3D content by Hangul voice recognition by efficiently converting the voice data processed by the voice engine into message data that can be recognized by the 3D data.

1 is a block diagram of a 3D virtual reality control middleware system according to the present invention
2 is a detailed configuration diagram of a 3D virtual reality control middleware system according to an embodiment of the present invention.
3 is a flowchart of a method for real-time control of 3D content using a 3D virtual reality control middleware system according to the present invention.
4 is a detailed flowchart illustrating a real-time control method of 3D content using a 3D virtual reality control middleware according to an embodiment of the present invention.
5 is a flowchart illustrating a play loop in a server of an operation controller according to an embodiment of the present invention.
6 is a function code constituting a message module according to an embodiment of the present invention.
7 illustrates a data structure of a message database according to an embodiment of the present invention.
8 is a data structure diagram of 3D data according to an embodiment of the present invention;
9 is an exemplary screen of play of a CMO file in a server according to an embodiment of the present invention.

The present invention is a middleware technology that converts text data processed by voice recognition technology into message data that can be recognized by 3D data. Command to enable state change. This message data is made into DB and processed by the conversion middleware. Through the middleware of the present invention as described above, the voice is converted into string data and the string data is converted into message data. Through this message data, it is possible to control the execution of actions of the object of the 3D virtual reality, that is, movement execution and state change. do.

The 3D virtual reality control middleware system according to an embodiment of the present invention is implemented to be compatible with the Virtools ^TM SDK, which is a commercial software development kit (SDK), which is a function form of C ⁺⁺ . Use the Virtools SDK engine. For reference, this is possible because the speech engine, that is, the speech recognition library, is built as a C code callback function library, and 3D content is processed in the Virtools SDK C ⁺⁺ language. Therefore, the middleware system according to an embodiment of the present invention is an example of a case in which a C ⁺⁺ code form is created for compatibility between a voice engine and Virtools' 3D content production SDK.

The structure of the middleware system according to an embodiment of the present invention is largely divided into a server and a client based on its functional characteristics, and an executable file created by 3DVia Virtools serves as a server, and a part accessed by a program stage is a client's part. It is defined and described as playing a role. The main role of the server is summarized in the creation, configuration, and management of 3D content, waiting for data from the client. The main role of the client is summarized by converting the voice signal into a message data value and delivering it to the server.

The implementation of the middleware system of an embodiment of the present invention has its essence in the production of a message processing function, which receives a user's voice signal and receives desired string data from a voice database through a voice engine, that is, a voice recognition library. To get The processing method in the voice library uses the CallBack function. In other words, the voice data is input to call back the string data. The returned string data is converted from string data to message data through a middleware function. The converting process retrieves the corresponding message from the message database with the string data returned by the CallBack function. The retrieved message is delivered to the server side. The server controls the execution and change of state of the 3D object according to the received message name. Accordingly, the action occurs on the object, which is delivered to the user in real time.

In the following description of the specific contents for carrying out the invention, the concepts and implementation examples of the main terms used are as follows.

The voice engine is implemented in the form of software or embedded software that detects the phoneme of the input voice signal and recognizes the character string. It is a voice recognition engine that includes a voice database necessary for voice recognition or is used as a meaning except for the voice database. do. For reference, in an embodiment of the present invention, the voice engine including the voice database 21 is composed of an API DLL file and a dictionary database file of .Dict. In the .Dict file, a voice database 21 is formed which includes a voice signal and a character string data file corresponding thereto, and defines and stores character string data corresponding to the voice signal. The message data is a message that can be recognized by 3D data, which will be described later, and the message data serves as an operation command that enables an action to be performed for each object in the 3D content (application). For example, it has the same structure as 3D data of a server, that is, a 1-byte English character string of M_ (action name) that an object can recognize. Here, M represents a message.

In an embodiment of the present invention, the message database 22 may be implemented using an SQLDB, and is composed of tables in the SQL DB. The structure of the table is briefly expressed as shown in FIG.

Referring to Fig. 7, a table of message data is composed of columns consisting of an index, a voice string, and a message string. Here, the index represents the number of rows, the voice string represents the voice string data received from the client, the message string represents the message data to be sent to the server, and the word number represents the message string, that is, the number of words in the message data. For reference, in the case of FIG. 6, when the string data received from the client is "sound", the message string of column 2 at the same index as the voice string value of column 1 is returned.

    In the description according to an embodiment of the present invention, the 3D data refers to all objects in the 3D virtual reality, and the 3D data collectively refers to a behavior object executed in the virtual reality. 8 may have a structure including an input value (3D entity), such as the 3D data structure of FIG. 8. In the data structure diagram of FIG. 8, each input value (3D entity) is defined and used as follows.

      3D Entity: an object that has both width, height and depth

      Light: Object that provides lighting

      Camera: The object defining the viewpoint

      Character: An intelligent object that can be manipulated by users or software

      3D Sprite: Image object with width, height and depth

      Grid: A set of 2D entities whose values are determined by 3D coordinates.

     Curve: A set of 3D entities that define a curve in 3D space (including Curve Point).

    Place: collection of spatially related objects

  2D Entity: An object with a width but no depth (including Sprite and Sprite Text)

Level: An element that contains all the objects in the composition, and is the root of all elements in the composition.

Scene: Manages a scene. The scene and the concept of the movie are the same.

Group: A data structure for storing information and collecting objects.

Array: A table containing cells listed in rows and columns.

Sound: Sound Object

Mesh: A set of points that determine the shape of an object

Material: characteristics, color of the mesh surface

Texture: An image used for realism on the surface of an object

Hereinafter, a specific embodiment according to an embodiment of the present invention will be described with reference to the accompanying drawings. In the following description, detailed descriptions of well-known functions or configurations will be omitted when it is determined that the detailed descriptions of the known functions or configurations may unnecessarily obscure the subject matter of the present invention.

1 is a configuration diagram of a 3D virtual reality control middleware system according to the present invention, Figure 2 is a detailed configuration diagram of a 3D virtual reality control middleware system according to an embodiment of the present invention, 3D according to an embodiment of the present invention The VR control middleware system is designed to be compatible with the Virtools ^TM SDK, a commercially available 3D content development software development kit (SDK).

Referring to FIG. 1, the 3D virtual reality control middleware system according to the present invention includes a voice input unit 10, a voice engine processor 20, a message processor 30, an operation controller 40, and a 3D renderer 50. do.

By the above configuration, by converting the voice data processed by the voice engine that recognizes the user's voice into message data that can be recognized by the 3D data, it becomes possible to interlock the voice engine and the interactive 3D in real time, and executed by the voice input and The execution of the 3D object whose state change is controlled and the state change can be visually transmitted to the user in real time.

Hereinafter, functions of the voice input unit 10, the voice engine processor 20, the message processor 30, the operation controller 40, and the 3D renderer 50 will be described in detail with reference to FIGS. 1 and 2.

      The voice input unit 10 performs a function of receiving a voice signal from the user and is controlled to input a voice signal through a voice input device such as a microphone as a first step for giving a signal (input value) to the 3D content to the user. This function is implemented as part of the function of the Play Module in the Client in one embodiment of FIG. 2 and will be described in more detail with respect to FIG. Preferably, the voice input unit 10 is configured to check the connection state of the input device for the voice input, to generate and output a status message, the state of whether the device required for voice recognition is correctly connected or available It may be configured to output a message, for example, "Ready" or "Not Ready" to notify the user through the client. In one embodiment of Figure 2, as part of the function of the DB Set Module. Can be implemented.

The voice engine processor 20 receives a voice signal input from the voice input unit, transmits the voice signal to the voice engine, and receives string data corresponding to the voice signal retrieved and returned from the voice database 21 by the voice engine. And transmits the received string data to the message processor 30. The search for the string data returns the string corresponding to the character signal to the client if the string data matches the speech signal associated with the word signal in the speech database of the speech engine. In one embodiment of Figure 2, it may be implemented as a function of a message module.

The message processing unit 30 receives the string data from the voice engine processing unit and performs a function of searching for the message data corresponding to the string data in the message database 22 storing the message data recognizable by the 3D data. The message data is transmitted to the operation controller 40. In one embodiment of Figure 2, it can be implemented as a function of a message module, and performs the core function of the intermediate route role of the 3D application and the voice engine in terms of the entire middleware system of the present invention.

In the case of Hangul, since the string data searched and returned in the voice database 21 by the voice engine is basically composed of 2 bytes, if it is sent directly to the 3D content (application), the 3D data of the 3D content is recognized. The program contains unknown data. Therefore, it is necessary to convert 2 byte Korean character string data into message data which is 1 byte English message that 3D data can recognize. In this case, the message database 22 has a data structure including a voice string, which is string data received from a user, and a message string, which is message data to be transmitted to an operation controller, and the message data corresponding to a 2 byte Korean character string indicates an action name. For example, it is preferable that it consists of 1-byte English character strings, such as "M_Sound" of FIG.

Preferably, the message processing unit 30 may be configured to perform a method of processing and searching the received string data in a form that can increase the word search probability. In the case of Hangul, the string data is processed and searched so that the string data is received and processed by removing the irradiation from the string data to correspond to the processed string data that can be recognized by the 3D data in the message database 22. It is preferable to perform the process of retrieving the message data or the message data corresponding to the deleted string data by the first letter of the received string data until the corresponding message data is retrieved or one character remains in the string data. When the message data is not found as a result of performing the search process, the voice input unit 10 is a middleware system for receiving a new voice signal from the user. In the embodiment of FIG. 2, the play module in the client may be used. To the Play Module, It is.

The operation controller 40 receives the message data from the message processor and controls the execution and state change of the 3D object according to the message data, thereby executing the movement of the 3D object. As in the embodiment of Figure 2, the function of the Play Module (Play Module) in the server is implemented in the form of a script assigned to each object. Preferably, the operation controller 40 includes a script including a message receiver, a message processor, and a termination processor generated for the object described with reference to FIG. 5 to perform a message loop. do.

 The 3D renderer 50 updates and displays, in real time, the execution and state change of the 3D object whose execution and state change are controlled by the operation control unit. In the embodiment of FIG. 2, the connection module in the client is shown. It is implemented as part of the functionality of the (Connect Module), and as will be described below with reference to FIG. 2, the 3D renderer shows all the objects in which the operation occurs and is delivered to the user in real time. The Virtools Render view is mapped onto the application's WindowView, which runs every frame, provides continuous motion in real time, and can be configured by the RenderContext object of the 3DVia Virtools engine. The 3D renderer is configured to continuously update the view and show 3D data through interworking of the client and the server view, and to provide the user with functions such as play, stop, and pause, depending on the embodiment.

Hereinafter, the detailed configuration of the 3D virtual reality control middleware system according to an embodiment of the present invention will be described in detail with reference to FIG. 2.

As described above, the embodiment of FIG. 2 is implemented to be compatible with the Virtools SDK, which is a commercially available 3D content development software development kit (SDK). Described separately by client.

Referring to Figure 2, the server (Sever) is composed of a 3D creation module (Create Module) and a play module (Play Module).

The 3D Create Module in the server composes a scene from the 3D content to be created and creates objects, cameras, lights, etc. in the scene. Each created object has logic that can wait for a message and movement logic. When you have finished placing and commanding the 3D virtual world, you can save it as Save (create CMO file) or export (Create VMO file) to create a composition file. Here, CMO and VMO files are composition files of 3DVia Virtools. That is, it is a 3D data storage file consisting of 3D input values (Entity) and various actions (moving execution and / or state change). Shape 3D data on 3D Via Virtools. In other words, 3D data created by Max, Maya, Catia, etc. are placed and combined in Virtools' virtual reality space to create a virtual world. 3D input values (Entity) are placed in space with (X, Y, Z) coordinates, and 2D entities are placed in space with (X, Y) coordinates. In addition, it defines effects such as light, material, and texture to decorate the virtual world. When the settings for the virtual world are completed, the file is saved (exported) and produced in a format (CMO / VMO file) that can be used in middleware. CMO (VMO) files are composed of scenes, 3D, 2D objects, and various entity and scene scripts. 8 is an exemplary screen of play of a CMO file in a server.

The Play Module in the server performs an Init and a Play Loop. The motion controller 40 is configured in the 3D VR control middleware system of the present invention.

Init stores and stores the initial value of the object. This stored value is needed to return to the initial value when the user moves the object. The values required for storage are the coordinate values of the object, the scale value of the object, and the rotation value of the object. In addition, it implements the navigation of the camera required for camera walking, and initializes the external device to be input to the user.

The play loop in the server is implemented as script code in an object that needs an action (moving execution and / or state change) after the setting for 3D virtual reality is completed by Init. Each script consists of three parts: a message receiver which receives a message (that is, message data), a message processor that processes a message (action part), and a termination processor that determines the end of an action. (1) Message receiver (Wait) Message) routines run as soon as the server is running. The main role waits for a message from the client. This routine starts with the first server startup and waits indefinitely for the time the server is running and the time the message arrives. When the message arrives, it signals to the next routine. (2) The message processing (Play Behavior) routine consists of the definition of the behavior (moving execution and / or state change), and receives and executes the message. Actions play a character's animation and move objects. This part runs in a loop until the action ends. (3) The end processing unit (Test: MotionEnd?) Routine is a test routine that informs the end of the operation. The object is checked every frame to see if it has finished. When the operation is finished, we return to the Wait message routine and wait for a message from the client again. If the action has not ended, just pass the signal and continue the action.

The client is composed of an Init Module, a Connect Module, a Play Module, a Message Module, and a DB Set Module.

The Init Module in the client is the part that performs the initialization work before the Virtools Engine is used. Since the middleware of this embodiment of the present invention uses the Virtools SDK, an initialization operation for the engine is required. Initializes the path of Plug-ins (Render, Manager, Behavior, Media) DLL files. DLL files are installed together in the folder installed by the user, and the path is set to the corresponding path. Functions and codes provided by Virtools SDK are used and composed. Processing in the initialization module can proceed as follows.

(1) :: CKGetPluginManager () function to get CKPluginManager pointer.

(2) :: CKPluginManager :: ParsePlugins () registers the path.

(3) :: CKCreateContext () creates a CKContext object that manages all the objects in the Virtools Composition.

      (4) Create a CKRenderContext object that handles 3D Rendering related tasks. (5) Get a CKRenderManager pointer through CKContext :: GetRenderManager () and create a RenderContext through CKRenderManager :: CreateRenderContext ().

Connect Module is a part that loads CMO / VMO file which is Server File. In order to link voice engine and application, middle engine needs to connect voice engine part and CMO file. This is because you have to use both libraries to link to middle middleware code. It is also a module in charge of 3D rendering, and connects Virtools View created in server to application view. In this module, server and voice engine are linked to client. In the 3D virtual reality control middleware system of the present invention, the 3D renderer 50 is configured.

Functions and codes provided in the Virtools SDK are used and composed. The processing in the connect module can be performed as follows.

(1) Load Virtools CMO / VMO file to client through CKContext :: Load () function.

(2) Get CLLevel point through CKContext :: GetCurrentLevel () function.

(3) Get and register camera installed in server through CKRenderContext :: AttachViewpointToCamera () function.

(4) Register CKRenderContext to Level through CKLevel :: addRenderContext () function.

(5) Register the scene to execute through CKLevel :: LaunchScene () function.

The play module in the client plays a role of driving the middleware. Once the necessary code and files are initialized on the middleware, they can be actually executed to prepare for message processing by driving the middleware. The module receives a voice signal through a microphone, and a play module in the client constitutes the voice input unit 10 of the present invention.

      Functions and codes provided in the Virtools SDK are used and composed. The processing in the play module can be performed as follows.

(1) Processing BehaviorProcess through CKContext :: Process ()

(2) Process the BehaviorProcess through the CKContext :: Process () function. (3) Rendering process is processed through CKRenderContext :: Process () function, and it communicates with server file (CMO) in real time by looping every frame.

Message Module is a module needed to deliver 2Byte Korean character string obtained from voice engine as 1Byte message on the server side. It performs core functions of middleware. The message module performs interworking and processing of voice DB and message DB. In addition, it performs an optimization function to improve the accuracy of the Hangul string received through the callback function of the voice engine and transmits the retrieved message data. Voice DB file is in Dict format. Message DB file uses SQL method. The message module constitutes the voice engine processing unit 20 and the message processing unit 30 of the present invention.

Functions and codes provided by the Virtools SDK are used and configured, message processing is performed in the message module, and search optimization may be preferably performed.

        Message processing can be performed as follows.

      1. Obtain the objects needed to deliver the message to the CMO file produced by the Server. Objects utilize the library provided by the Virtools SDK.

(1) CKContext * pContex = (Virtools SDK Integration Object)-> m_CKContext;

      Create a context that manages the object. The pContex object manages the management of all objects used by Virtools.

      (2) CK3dEntity * p3DObject = (CK3dEntity *) pContext->

GetObjectBuNameAndClass (Object Name, Object ID)

Since we need to know where to send the message, we get and initialize the object of the object that needs action. This function is responsible for creating and initializing the object that should receive the message.

(3) CKMessageManager * pMsgManager = pContext-> GetMessageManager ();

Create and initialize a message object for sending messages. pMsgManager is in charge of all management of messages.

2. Deliver voice signal to voice engine. The callback function returns string data matching the voice signal from the voice engine.

(1) CString wdmessage = Callback_VS_VSRResult ()

It executes Callback_VS_VSRResult () function to access the voice library, and searches the DB to return a string file corresponding to the voice signal.

3. Get DB object to access message DB. Since message DB uses SQL DB, access to tables is required. Create an SQL class in the application to provide a connection with SQL DB. After creation and connection is completed, retrieve the desired message name from the message DB.

(1) CMessageTable * m_pSetWord;

Create an object to access the SQL DB interworking class.

(2) CDBPropSet propset (DBPROPSET_ROWSET)...

Set the properties of SQL DB. Property grants the right to insert, delete, update, change etc through SQL statement.

(3) vsMessage.Format (“SELECT * FROM MessageTable WHERE WorldList = '% s'' wdMessage)

m_pSetWord-> RunSQL (vsMessage, &propset);

Retrieve desired data from SQL DB through SQL statement. The value obtained is the message data to send as a string value.

4. After checking and checking if there is a message name, if there is a message name, send the message to Server. The message is passed as a variable where it will be sent (it will be the object specified in paragraph 1 (2)) and what message will be (the message name retrieved from SQLDB).

(1) CKMessageType nMsgType = pMsgManager-> AddMessageType (vsMessage);

Set some message type before sending. The message type is transmitted by setting the message name retrieved from SQL DB as a variable.

(2) pMsgManager-> SendMessageSingle (nMsgType, p3DObject)

This is the part that delivers the message to the server-side (CMO) file. The message type and which object to deliver are set as variables.

When the search optimization is performed, the search of the message data can be optimized by the following process.

(1) Send the delivered voice string to the message DB to search for the string. Search uses SQL statement. If the string exists, the corresponding message is returned.

(2) If the string does not exist, delete the first letter of the string extracted from the voice DB. After deleting, search the message DB again. If there is a string, the corresponding message is returned.

(3) Repeat step 2 until the last letter is left. If the DB search fails by one character, it returns to receive voice message input again.

The DB Set Module obtains system status information to distinguish whether a microphone device is connected to a computer. When the microphone recognition device is ready, initialize the voice recognition library. Proceed with voice engine initialization. Recognize the path and access the voice recognition database. Set the word list in the database. The word list is in dict file format. Once you have registered the words, you are ready to receive voice recognition.

3 is a flowchart of a method for real-time control of 3D virtual reality content according to an embodiment of the present invention, and FIG. 4 is a flowchart illustrating a detailed flow of a method for real-time control of 3D content using a 3D virtual reality control middleware according to an embodiment of the present invention. It is a flowchart. Hereinafter, with reference to FIGS. 3 and 4 in a range not overlapping with the description related to the above-described 3D virtual reality control middleware system, a real-time control method of 3D virtual reality content of the present invention and 3D content according to an embodiment of the present invention. The detailed flow of the real time control method will be described in detail.

Referring to FIG. 3, the present invention provides a method for real-time control of 3D content using a 3D virtual reality control middleware system, the method comprising: receiving a voice signal from a user through a voice input device connected to the middleware system (S10); Accessing a voice database and retrieving and storing string data corresponding to the received voice signal (S20); Retrieving message data corresponding to the stored string data from a message database storing message data recognizable by 3D data (S30); controlling a behavior of a 3D object according to the retrieved message data (S40); And updating and displaying the behavior of the 3D object whose behavior is controlled in real time (S50).

Preferably, the step of receiving a voice signal from the user (S10) is an accuracy determination step (C5) for determining the accuracy of the voice based on the size of the voice, the accuracy of the voice, the speed of the voice and the like described with reference to FIG. It is carried out including.

In addition, the step of retrieving the message data (S30), a first step of retrieving the message data corresponding to the character string data in the message database; And a second step of deleting the first letter of the character string data and retrieving message data corresponding to the deleted character string data from the message database, by performing a first step on the stored character string data. If the corresponding message data is retrieved, the retrieved message data may be returned, and if the corresponding message data is not retrieved, the second step may be performed until one character remains in the string data.

Further, the step (S40) of controlling the behavior of the 3D object may include: a message waiting step (S4) waiting for reception of message data, which will be described later with reference to the embodiment of FIG. 4; A message receiving step (S5) of receiving message data; An action execution step S6 of executing an action of a related object according to the received message data; And an action execution end determination step (S7) of determining whether execution of the action of the related object is completed according to the received message data, and if it is determined that execution of the action is completed (S8), In this case, it is preferable to perform a notification to receive an additional voice input.

Further, the step of updating and displaying the behavior of the 3D object in real time (S50) may include generating an application user interface (C1) and generating an application window view (C1) and every frame of the loaded screen. By performing the screen load step (C3) that is linked to the application window view to be updated and displayed, the behavior of the 3D object is updated and displayed in real time.

4, a detailed flow of a real-time control method of 3D content using the 3D virtual reality control middleware will be described.

First, the detailed flow of the server side will be described. In the step S1 of creating 3D data, 3D data is shaped on 3D Via Virtools. Create virtual reality spaces by placing and combining 3D data from Max, Maya, Catia, etc. in Virtools' virtual reality space. 3D entities are placed in space with (X, Y, Z) coordinates and 2D entities are placed in space with (X, Y) coordinates. In addition, it defines effects such as light, material, and texture to decorate the virtual world. When the settings for the virtual world are completed, the file is saved (exported) and produced in a format (CMO / VMO file) that can be used in middleware. It is processed in the 3D data creation module (Create Module) of the 3D virtual reality control middleware of FIG.

      In the behavior construct step S2, when an object is initialized, an object that is to receive a signal from a client is identified. For objects that require action (moving execution and / or state change) events, create a script, and wait message waiting for the message, and configure the PlayBehavior logic to act when the message comes in. It is processed in the 3D data creation module (Create Module) of the 3D virtual reality control middleware of FIG.

     The Initialize step (S3) stores initial values of 3D objects, materials, textures, lights, and cameras that are essential to the virtual world. Coordinate, rotation and scale values are stored for each entity. It is necessary to return to the original state later. Coordinate, rotation and scale values are stored for each entity. It is necessary to return to the original state later. Processing is performed in the server side Play Module (Init) of the 3D VR control middleware of FIG.

     Wait Message In step S4, the server always waits for a message from the client. The Wait message is always executed in a loop state every frame. It is processed in a server-side Play Module (Playing Loop) of the 3D VR control middleware of FIG.

Receive Massage step S5 receives message data corresponding to a voice signal from a client. In addition, it is determined whether the extraction of the message data has succeeded or failed from the voice signal, and if it is determined to fail, a failure message is transmitted to the client through MSG_ERROR, and the message data is waited to be received again. If successful, proceed to the next procedure. It is processed in a server-side Play Module (Playing Loop) of the 3D VR control middleware of FIG.

The play behavior rendering step S6 executes the behavior (moving execution and / or state change) of the related object according to the received message data. In the case of a character, movement is performed through animation play, and movement of a 3D object such as movement, rotation, and resizing is performed. Color changes can be made for materials and image changes can be done for textures.

      The action execution end determination step (S7) determines whether the execution of the action of the related object is completed according to the received message data, and when it is determined that the action is completed, terminates the routine (S8), and inputs a voice (C4). Notify the client to receive additional voice input.

Next, the detailed flow of the client side will be described with reference to FIG.

In the initializing step (C1) of the client side, an initialization operation for the 3DVia Virtools SDK engine, that is, initialization of a CMO (VMO) file produced by the server is performed. Register DLL file to use engine, initialize DLL file path and create CKContext object that manages all objects in Virtools Composition. After creating CKContext, create CKRenderContext object that manages 3D Rendering related tasks. When the DLL file and 3D object declaration are completed, the application UI is created to create the frame and window view. It is handled by the client-side Init Module of the 3D VR control middleware.

In the DB set step C2, the system state information is obtained and a voice input device such as a microphone is connected to a computer to determine whether the voice input device is in a ready state. A voice input device such as a microphone is ready, and a voice recognition library, that is, a voice engine initialization operation is performed. It also recognizes and accesses the path of the voice database, and sets up a word list in the voice database. The word list is in dict file format. When you finish the word registration, you are ready to receive voice input. It is handled by the client-side DB Set Module of the 3D VR control middleware.

      Scene load step (C3), the CMO (VMO) file produced on the server is linked to the Window View produced during the Initialize process. Initializes the components of all entities such as 3D entity, 2D entity, light, and camera in the scene of CMO file. When the scene is loaded, the application window view is mapped to the view of the CMO and updated every frame, so the user perceives it as if it were 3D rendered in real time. do.

In the voice input step C4, voice is input through a voice input device such as a microphone. In addition, if the correct voice is input when the voice is input by performing the accuracy judgment step (C5) of determining the accuracy of the voice based on the size of the voice, the accuracy of the voice, the speed of the voice, etc., proceed to the next step. If a voice is not input and voice discrimination is impossible, the voice may be executed to receive a re-input. If there is no voice input, it may be configured to terminate the event (C10). It is performed in the client side Play Module of the 3D VR control middleware.

The get voice signal step C6 is a step of accessing the voice DB to obtain string data matching the voice signal. The list registered in the voice DB and the most optimized (precise) string data of the voice signal input by the user are searched and returned to the client. The client stores this string data in a global variable and prepares it for processing within the client. It is handled by the client-side Message Module of the 3D VR control middleware.

The get message step C7 is a step of retrieving the string data stored in the global variable from the message DB. DB search is executed through SQL query statement. After retrieving the message list from the string data corresponding to the input voice, the message data of the uppermost (highest accuracy) message name is fetched. If it is determined whether there is a correct message name (C8), if there is no, it returns to the voice input step (C4), and if the message name is found, it can be configured to proceed to the next step. It is handled in the client side Message Module of the 3D VR control middleware.

Send message step C9 transfers the message data to the Server (CMO file) when the message name is retrieved from the message database. If the message name of the object waiting on the server side matches, the operation is executed on the server side. The delivered message is stored until the next voice input, and if there is a voice input again, the message data retrieved in the previous step is deleted. It is handled in the client side Message Module of the 3D VR control middleware.

The present invention has been described above with reference to one embodiment shown in the drawings according to the embodiments, but these are exemplary and the scope of the present invention is not limited thereto, and includes various modifications and equivalent embodiments of the present invention. The scope of the invention is determined by the matter set forth in the claims.

10: voice input unit 20: voice engine processing unit
30: message processing unit 40: operation control unit
50: 3D render unit

Claims (10)

In the 3D virtual reality control middleware system that interoperates a voice engine and an interactive 3D application in real time,
A voice input unit 10 for receiving a voice signal from a user;
A voice engine processor 20 which receives a voice signal input from the voice input unit, transmits the voice signal to a voice engine, and receives character string data corresponding to the voice signal that is retrieved and returned in the voice database 21 by the voice engine;
A message processor (30) which receives the string data from the voice engine processor and searches for the message data corresponding to the string data in a message database (22) storing message data recognizable by 3D data;
An operation control unit (40) for receiving the message data from the message processing unit and controlling execution and state change of a 3D object according to the message data; And
And a 3D renderer 50 which updates and displays the execution and the state change of the 3D object whose execution and state change are controlled by the operation controller in real time. The message processor 30 corresponds to the character string data. If the message data is not found, the first character of the string data is deleted and the message data corresponding to the character string data of which the first character is deleted is searched. 3D virtual reality control middleware system, characterized in that until the remaining one character. In the 3D virtual reality control middleware system that interoperates a voice engine and an interactive 3D application in real time,
A voice input unit 10 for receiving a voice signal from a user;
A voice engine processor 20 which receives a voice signal input from the voice input unit, transmits the voice signal to a voice engine, and receives character string data corresponding to the voice signal that is retrieved and returned in the voice database 21 by the voice engine;
A message processor (30) which receives the string data from the voice engine processor and searches for the message data corresponding to the string data in a message database (22) storing message data recognizable by 3D data;
An operation control unit (40) for receiving the message data from the message processing unit and controlling execution and state change of a 3D object according to the message data; And
And a 3D renderer 50 which updates and displays the execution and the state change of the 3D object whose execution and state change are controlled by the operation controller in real time. The message database is a string data input from the user. 3D virtual reality having a data structure including a voice string and a message string which is message data to be transmitted to the operation controller, wherein the string data is a 2-byte Korean character string and the message data is a 1-byte English character string including an action name. Control middleware system. In the 3D virtual reality control middleware system that interoperates a voice engine and an interactive 3D application in real time,
A voice input unit 10 for receiving a voice signal from a user;
A voice engine processor 20 which receives a voice signal input from the voice input unit, transmits the voice signal to a voice engine, and receives character string data corresponding to the voice signal that is retrieved and returned in the voice database 21 by the voice engine;
A message processor (30) which receives the string data from the voice engine processor and searches for the message data corresponding to the string data in a message database (22) storing message data recognizable by 3D data;
An operation control unit (40) for receiving the message data from the message processing unit and controlling execution and state change of a 3D object according to the message data; And
And a 3D renderer 50 for updating and displaying in real time the execution and state change of the 3D object whose execution and state change are controlled by the operation control unit. The operation control unit 40 is generated for the object. 3D virtual reality control middleware system comprising a script including a message receiving unit, a message processing unit and a termination processing unit. 4. The method according to any one of claims 1 to 3,
The voice input unit 10 checks the connection state of the input device for voice input, generates and outputs a status message 3D virtual reality control middleware system. delete delete delete delete delete delete

Images