KR20080010564A - System for multimedia naration using 3d virtual agent and method thereof - Google Patents

Abstract

A system for producing multimedia narration using a 3D(Dimensional) virtual agent and a providing method thereof are provided to reduce manpower, expense, and time for performing various kinds of presentations by making the 3D virtual agent narration, converting sentences into voice through a TTS(Text-To-Speech) function, and providing a text presentation. A source managing module(10) provides functions for collecting information from sources having various formats and converting the format, and a result generating module(20) stores project files compiled to play the multimedia narration using the time-synchronized 3D virtual agent. A project managing module(30) processes detail work according to generation, storage, and modification of the project files. A library managing module(50) manages individual libraries for a 3D avatar, background, motion, and A/V(Audio/Video) effect. A timeline compiling module(40) compiles each library and source according to a timeline. A lip-sync compiling module(60) is connected to each module of a TTS server(200), and compiles a script received from the source or the sentence inputted from a user. A keyword managing module(70) matches each keyword of the script with the motion or expression of the 3D avatar.

Description

Multimedia narration authoring system using 3D virtual agent and method for providing same

1 is a block diagram of a multimedia narration system using a 3D virtual agent according to the present invention.

2 is a block diagram of a 3D engine module of a multimedia narration authoring system (Wiz-Pub main program) using a 3D virtual agent according to an embodiment of the present invention.

FIG. 3 is a 3D image of a 3D avatar that selects various expressions, actions, backgrounds, and sound effects, and edits a lip sync according to a timeline to output a sentence in a corresponding language (Korean, English, Chinese, Japanese, etc.) using the TTS function. Diagram of Wiz-Pub main program providing narration through virtual agent.

4A is a diagram for explaining a timeline editing function of a multimedia narration authoring system (Wiz-Pub main program) using a 3D virtual agent.

4B is a view for explaining a keyword management function of the Wiz-Pub main program.

4C is a diagram for explaining a lip sync editing function of a Wiz-Pub main program;

4D is a diagram for explaining the function of a multimedia narration player (Wiz-Pub Player) using the 3D virtual agent.

FIG. 5 is a diagram illustrating a method of producing web content as a usage example of a multimedia narration (hereinafter referred to as Wiz-Pub) using a 3D virtual agent. FIG.

FIG. 6 is a diagram showing a help assistant as a use example of Wiz-Pub. FIG.

Figure 7 is a Wiz-Pub use case as a tutor (Tutor), voice guidance, face to face communication, experience learning through a virtual space, showing a variety of multimedia learning.

8 is a view showing a sales agent (Sales Agent) for product description as a use case of Wiz-Pub.

9 is a view showing a mascot production (Entertainer) screen of the company as a use case of Wiz-Pub.

FIG. 10 is a diagram illustrating an example of applying a 3D virtual agent to a large PDP and a kiosk offline as a use case of Wiz-Pub.

11A to 11U are diagrams for explaining the functions of a multimedia narration authoring system (Wiz-Pub main program) through a 3D virtual agent.

12A to 12M are diagrams for explaining the functions of the Wiz-Pub main program.

<Explanation of symbols for the main parts of the drawings>

10: source management module 20: output generation module

30: Project Management Module 40: Timeline Editing Module

50: Library Management Module 60: LipThink Editing Module

70: keyword management module 80: output module

90: Wiz-Pub Player

100: The main multimedia authoring tool (WizPub) using 3D Virtual Agent

     program

200: TTS server 210: TTS module

220: MP3 conversion module 300: Update Server

310: Update configuration module 320: Update Active X

The present invention relates to a multimedia narration authoring system using a 3D virtual agent and a method of providing the same. In particular, the narration is performed by a 3D virtual agent including a background, motion, facial expression, effect, sound effect, wallpaper, video, flash, and image of an avatar. , Text-to-speech (TTS) feature to convert language-specific characters (Hangul, English, Japanese, Chinese) into speech and text presentations to provide product descriptions, guides, in-house training, and marketing materials via 3D avatars. The present invention relates to a multimedia narration authoring system using a 3D virtual agent and a method of providing the same, which provide a corporate promotional material, a product catalog production, and a presentation material.

In general, as the company's in-house training market grows, the cost of investing in employee training in 2003 is estimated to be around 2.3 trillion, with at least 5% of the corporate training market going to Internet. It is expected.

The main reason why domestic SME e-learning education has not been activated is due to lack of cost, money and manpower to build the system and use contents. There are few operators and specialists who are in charge of in-house education in the company, and the Internet is changing according to the change of expression technology such as Text, Picture, Movie, Stream, 3D, etc. There is a need for service improvement.

However, the conventional Internet in-house education is a narration in the form of multimedia including Face to Face communication through 3D virtual agents with various actions, expressions, backgrounds, and sound effects. There was a problem that was not provided.

The present invention has been proposed to solve the problems of the prior art, and an object of the present invention is to use a multimedia authoring tool (Wiz-Pub main program) using a 3D virtual agent. Provides 3D Virtual Agent (3D Virtual Agent) narration including, desktop, video, flash, and image, and is written by language (Korean, English, Japanese, Chinese) by TTS function according to interactive multimedia content platform To speech, output text, and provide text presentations through 3D virtual agents for product manuals, guidance, in-house training, marketing materials, corporate promotional materials, product catalog production, presentations, Multimedia narration using 3D virtual agents to reduce training, manpower, cost, and time To provide a system and method for providing.

In order to achieve the object of the present invention, a multimedia narration authoring system using a 3D virtual agent according to the present invention, a source management module for providing information collection function and format conversion function for the source of various formats; A result generation module for storing the edited project for playing and viewing a multimedia narration using a synchronized 3D virtual agent according to a timeline; A project management module for performing detailed operations according to the project file generation, storage, and change; A library management module for managing individual libraries relating to 3D avatars, backgrounds, movements, and audiovisual effects; A timeline editing module for editing each library and the source according to a timeline; A lip sync editing module connected to a TTS module and an MP3P conversion module of a TTS server, for editing a script or a text input by a user; And a keyword management module for applying a predetermined action or facial expression of the 3D avatar to each keyword of the script.

In order to achieve another object of the present invention, in the method of providing a multimedia narration using a 3D virtual agent according to the present invention, (a) generating a new project on the main screen of the multimedia narration authoring system using a 3D virtual agent ; (b) converting the presentation material into an image file for each page by the image conversion module and setting and loading the script and the title; (c) obtaining a playback time of the flash file by the SWF information collecting module; (d) previewing a video or obtaining a playing time by the video file information collecting module; (e) selecting the 3D avatar for providing the multimedia narration by the library management module and loading the presentation data in the window view; (f) Select the background, motion, and audio-visual effects of the selected 3D avatar from the library list, link them to the timeline cells in the Timeline Editor by drag-and-drop, and then select each library and source (movie or image or Flash file). Editing; (g) editing the script input from the presentation or the user input from the presentation by the LipThink Editor, transmitting each sentence of the script to a TTS server, and receiving the voice-converted data; (h) generating an authoring output of each edited project for viewing in the multimedia narration player; (i) reproducing the authoring result in a preview in the multimedia narration player, and trial outputting the multimedia narration; And (j) storing the project.

In order to achieve another object of the present invention, the present invention provides a computer, (a) the function of creating a new project in the main screen of the multimedia narration authoring system using the 3D virtual agent; (b) converting the presentation material into an image file for each page by the image conversion module, and setting and loading the script and the title; (c) a function of obtaining a playing time of the flash file by the SWF information collecting module; (d) a function of previewing a video or obtaining a playing time by the video file information collecting module; (e) a function of selecting a 3D avatar for providing a multimedia narration by the library management module and loading the presentation data in a window view; (f) Select the background, motion, and audio-visual effects of the selected 3D avatar from the library list, link them to the timeline cells in the Timeline Editor by drag-and-drop, and then select each library and source (movie or image or Flash file). The ability to edit; (g) editing a script input from the presentation or a user input from the presentation by the LipThink Editor, transmitting each sentence of the script to a TTS server, and receiving the voice-converted data; (h) the ability to generate authoring output for each edited project for viewing in a multimedia narration player; (i) reproducing the authoring result in a preview in the multimedia narration player, and trial outputting the multimedia narration; (j) storing the project; (k) a function of inserting a watermark in the case of illegal copying by applying DRM (Digital Rights Management) to the multimedia narration player or the source to prevent illegal copying of contents produced from the multimedia narration authoring tool; (l) a function of changing a voice or a tone according to the replaced 3D avatar when the 3D avatar is replaced at a desired position on the timeline, and outputting the voice; (m) a function of virtually experiencing the movement of the object according to a timeline setting when adding a moving object to a background in 3D space; (n) controlling the movement of each 3D avatar in a timeline editing function such that a plurality of 3D avatars can interactively react and communicate with each other; And (o) a computer-readable recording medium for realizing a function of executing multimedia data on an object existing in a 3D space under control by setting of a timeline.

Hereinafter, exemplary embodiments of the present invention will be described in detail with reference to the accompanying drawings. Prior to this, terms or words used in the specification and claims should not be construed as having a conventional or dictionary meaning, and the inventors should properly explain the concept of terms in order to best explain their own invention. Based on the principle that can be defined, it should be interpreted as meaning and concept corresponding to the technical idea of the present invention. Therefore, the embodiments described in the specification and the drawings shown in the drawings are only the most preferred embodiment of the present invention and do not represent all of the technical idea of the present invention, various modifications that can be replaced at the time of the present application It should be understood that there may be equivalents and variations.

1 is a block diagram of a multimedia narration (WizPub) system using a 3D virtual agent according to the present invention.

As shown in FIG. 1, a multimedia narration (Wiz-Pub) system using a 3D virtual agent according to the present invention is a multimedia authoring system using a 3D virtual agent (hereinafter referred to as a WizPub main program) 100, a TTS server ( 200, and update server 300.

The multimedia authoring system (WizPub main program) 100 using the 3D virtual agent 100 includes a source management module 10, a result generation module 20, a project management module 30, a timeline editing module 40, and a library. A management module 50, a lip sync editing module 60, a keyword management module 70, and an output module 80.

The multimedia narration player (hereinafter referred to as Wiz-Pub Player) 90 using the 3D virtual agent includes the output module 80.

The 3D virtual agent includes a motion, facial expression, background, effect, sound effect, video, flash, and image data of the 3D avatar.

The source management module 10 includes a PPT image conversion module, a SWF information collection module, and a video file information collection module, and includes a source of various formats (including an image conversion file of a PPT presentation, a video file, and a flash file). Provides information gathering and format conversion.

The PPT image conversion module converts a PowerPoint (PPT) presentation file into an image file for each page, and imports a script and a title from the presentation.

The SWF information collection module obtains information about a play time of a flash file (SWF).

The video file information collection module previews a video or brings information regarding a play time.

The result generation module 20 includes a compression module, an encryption module, and a watermark module, and viewed as a multimedia narration player (Wiz-Pub Player) 90 using the 3D virtual agent synchronized according to a timeline. Save your edited project.

The compression module provides a function of compressing 3D avatar data and multimedia data (background image, flash data, voice data, sound data, video data).

The encryption module provides a function of encrypting a work result using a predetermined encryption algorithm (eg, RSA 128 bit method) so that it can be opened only by a dedicated player (multimedia narration player).

The watermark module applies digital rights management (DRM) to prevent illegal copying and tampering of multimedia narration data, and applies watermark to multimedia data (image, audio data, sound data (effect sound), video data). Provide the function to apply.

The project management module 30 performs detailed tasks according to the generation, storage and change of the project file.

The timeline editing module 40 displays the background, motion (motion, facial expression), audiovisual effect (effect, sound effect), each individual library of the background, source (video, image, SWF), etc. of the 3D avatar. Provides the ability to edit according to the timeline.

The library management module 50 manages each individual library of 3D avatar, background, motion (motion, facial expression), audiovisual effect (effect, sound effect), and background screen.

The lip sync editing module 60 is connected to the TTS module 210 and the MP3P conversion module 220 of the TTS server 200, and inputs and pastes a sentence or a user imported from a source such as PowerPoint (PPT). Edit the sentence.

The keyword management module 70 corresponds to each keyword (word of the script) of the script in correspondence with a certain motion or expression of the 3D avatar.

The output module 80 includes a mouth generating module, a sound management module, a 3D engine module, and a web control module.

The mouth generation module automatically generates a corresponding mouth shape when the 3D avatar providing the multimedia narration speaks the script.

The sound output module outputs an effect sound and a voice file according to a preset timeline.

The 3D engine module provides a function of representing a 3D avatar, a background, an image, and the like in a virtual space.

The web control module connects the 3D virtual agent to the web page.

The preview / multimedia narration player (Wiz-Pub Player) is a Play control module for controlling the playback results generated by the multimedia narration authoring system (Wiz-Pub main program) using the 3D virtual agent; And a paint module that the user expresses in the image with a tool such as a pencil.

The TTS server 200 is a multimedia narration authoring system (Wiz-Pub main program) using a 3D virtual agent that provides various movements (motions, facial expressions), backgrounds, audiovisual effects (effects, sound effects), videos, and flashes of 3D avatars. Or a TTS module 210 connected to a multimedia narration player (Wiz-Pub Player) using the 3D virtual agent and converting a sentence into a voice file, and an MP3 conversion module 220 to convert the voice file into an MP3 file. It includes.

The update server 300 is a version of the files to be managed included in the multimedia narration authoring system (Wiz-Pub main program) and the multimedia narration player (Wiz-Pub Player) using the 3D virtual agent. An update setting module 320 for updating and setting, and an update Active X module 310 for keeping files up to date in the multimedia narration authoring system with a set file list.

2 is a block diagram of a 3D engine module of a multimedia narration authoring system (Wiz-Pub main program) using a 3D virtual agent according to an embodiment of the present invention.

The 3D engine module may include a camera control module 81, an effect module 82, a character module 83, a background module 84, a texture management module 85, a motion management module 86, A mesh management module 87, a timeline management module 88, a renderer 89, and DirectX.

The view point control module 81 is responsible for view point control for the 3D virtual agent.

The effects module 82 provides various 3D animation effects for the source.

The character module 83 is a module that receives character data to configure and control the character of the 3D avatar.

The background module 84 loads the 3D background to position and control the mesh data.

The texture management module 85 is a module that manages textures of 3D avatars and backgrounds.

The motion management module 86 is a module for managing the motion (motion) of the 3D avatar.

The mesh management module 87 manages mesh objects for each body part (head, mouth, upper body, left hand, right hand, waist, lower body, left, right leg, etc.) of the 3D avatar representing the multimedia narration. Module.

The timeline management module 88 is a module for editing and managing a timeline of a timeline cell in a predetermined time unit (for example, 10 ms).

The renderer 89 is a module that performs final rendering on the screen using Direct 3D. The renderer 89 standardizes a video encoder / decoder 89a for encoding / decoding a video file, a Flash filter 89b for outputting a flash file on the screen, and standardizes a file format into a memory. And a file filter 89c.

FIG. 3 is a 3D image of a 3D avatar that selects various expressions, actions, backgrounds, and sound effects, and edits a lip sync according to a timeline to output a sentence in a corresponding language (Korean, English, Chinese, Japanese, etc.) using the TTS function. Wiz-Pub main program provides multimedia narration through virtual agent.

The multimedia narration authoring system (Wiz-Pub main program: Wiz-Pub Studio) through the 3D virtual agent includes a main menu providing basic menus of files, edits, controls, tools, and help menus of Wiz-Pub; Shortcut icons that display new projects, open projects, save projects, import, preview, convert full lip sync, auto-generate events, and keyword action management icons for quick and easy access to frequently used functions; A project window (①) which manages functions such as moving and deleting files that have been converted and "taken"; A window view (2) showing 3D avatars and background slides according to the baseline selection and play of the timeline, and supporting preview by clicking a list of avatars and backgrounds in the library; A library (③) for adding a 3D avatar's background, motion, facial expressions, effects, sound effects, and wallpaper to the timeline and window view using simple drag &drop; Registration information (4) for showing information of cells of each timeline including start time, playback time, and end time according to the timeline information; Lip think editor (⑤) for converting and outputting a sentence (Korean, English, Japanese, Chinese) written by the TTS server 200 to speech by a text to speech (TTS) function with a simple text input; And a timeline editor (6).

FIG. 4A is a diagram for describing a timeline editing function of a multimedia narration authoring system (Wiz-Pub main program) using a 3D virtual agent.

The timeline editor (⑥) lip syncs, actions, and matches the timeline cell of the 3D virtual agent based on image, video, and flash sources according to a unit time (eg, 10 msec). It provides easy and convenient editing of facial expressions, source transitions, effects and sound effects.

4B is a diagram for explaining a keyword management function of the Wiz-Pub main program. Actions that match the characteristics of 3D Virtual agents, such as Help Assistant, Tutor, Sales Agent, Entertainer, grouping of facial expressions, and defined keywords Automatic creation of timeline cells according to action groups and keywords is provided.

For example, if the keyword management function is set to the keyword "hello", and set to "greeting back" in the action, "blink left eye" in the expression, the greeting with the greeting "hello" of the 3D avatar It is output in conjunction with the action, facial expression, and sound effects.

4C is a view for explaining a lip sync editing function of a Wiz-Pub main program.

The lip sync editing function of the Wiz-Pub main program provides a function of automatically converting text according to a document to speech by a text to speech (TTS) function, provides a voice recording function using a microphone, and a lip sync editor It provides a function for converting a sentence of a foreign language expressed in any one of Korean, English, Chinese, or Japanese input into a voice, and provides a volume control, speed, and height control function.

4D is a diagram for explaining the function of a multimedia narration player (Wiz-Pub Player) using a 3D virtual agent.

The function of the multimedia narration player (Wiz-Pub Player) using the 3D virtual agent is played in the narration mode, the presentation (PT) mode, the avatar mode for the web, the size adjustment, slide, subtitles, 3D agent, background, sound You can play.

The narration mode is a mode in which the presentation material is included and played inside the 3D virtual agent, and provides a caption, a slide active inactivation function, and a slide position setting function.

The presentation mode is a mode in which the 3D virtual agent and the presentation material are reproduced at different positions by dividing the display screen.

The avatar mode for the web is a mode for playing a 3D virtual agent at a predetermined position of a web page, and provides an avatar view and a slide view user resizing function, a slide view active deactivation function, and a slide position setting function.

The multimedia narration authoring tool (Wiz-Pub main program: Wiz-Pub Studio) using the 3D virtual agent is executed when DirectX 9.X or higher is installed in an operating system of Windows 2000 or higher, and is used in Internet Explorer 5.5 or higher.

The multimedia narration player (Wiz-Pub Player) using the 3D virtual agent is executed when DirectX 9.X or higher is installed in an operating system of Windows 98 or higher, and is used in Internet Explorer 5.5 or higher.

FIG. 5 is a diagram illustrating a web content creation method as an example of using a multimedia narration (hereinafter referred to as Wiz-Pub) using a 3D virtual agent.

FIG. 6 is a diagram illustrating a help assistant as a usage example of Wiz-Pub.

FIG. 7 is a diagram illustrating an example of using Wiz-Pub as a tutor, showing voice guidance, face to face communication, hands-on learning through virtual space, and various multimedia learning.

The instructor tutor is a 3D assistant agent (virtual world) and the interaction of the educational content to enjoy the learning experience by the experience, focus on the problem and helps to learn effectively.

8 is a diagram illustrating a sales agent for product description as a use case of Wiz-Pub. The sales agent provides voice information such as customer benefits, face to face communication, customer service procedure guide, and multimedia information.

9 is a view showing a mascot production (Entertainer) screen of the company as a use case of Wiz-Pub. By customizing unique mascots, companies can provide their images and information to customers in a fun and entertaining way, increase awareness of the site, and support voice information to attract ongoing visitors.

FIG. 10 is a diagram illustrating an example of applying a 3D virtual agent to a large PDP and a kiosk offline as a use case of Wiz-Pub.

FIG. 11A illustrates a main window of a multimedia narration authoring system (Wiz-Pub main program) through a 3D virtual agent.

Through the 3D virtual agent, the multimedia narration authoring system (Wiz-Pub main program (Wiz-Pub Studio)) is connected to the main menu (①), shortcut icon (②), project window (③), window view (④), library ( ⑤), registration information (⑥), Lip Think Editor (⑦), and Timeline Editor (⑧).

The main menu (①) provides a function to sequentially find and enter basic functions of Wiz-Pub such as file, edit, control, tool, and help menu.

The icon menu ② helps to use frequently used functions quickly and easily. In one embodiment, the icon menu displayed a new project, open project, save project, import, preview, convert entire lip sync, auto-generate event, and keyword action management icon.

The project window ③ is responsible for managing functions such as moving and deleting files that have been converted and "taken".

The window view (4) shows the 3D avatar and the background slide according to the baseline selection and play of the timeline. It supports previewing by clicking the list of avatars and backgrounds in the library.

The library ⑤ shows various assets such as an avatar, a background, an action, an expression, an effect, a sound effect, and a wallpaper. These assets can be added to the timeline and window view using simple drag & drop.

The registration information (⑥) shows information of each cell of the timeline.

The Lip Think Editor (⑦) is a function of converting a sentence written in a predetermined language (Korean, English, Japanese, Chinese, etc.) using a TTS (Text To Speech) engine by accessing the TTS server with a simple text change. To provide.

The timeline editor (⑧) can edit the avatar source, lip sync, motion, facial expressions, effects, sound effects, and wallpaper in detail at intervals of a predetermined time (eg, 10 ms) on the timeline.

11B is a diagram illustrating a project creation screen of a Wiz-Pub main program.

In the project generation of the Wiz-Pub main program, when a new project button of the icon menu or a file => new project of the text main menu is selected, a new project name is inputted into the project name. The generation of the project confirms whether to save the existing working project, and then clears the main window.

11C is a diagram illustrating a project opening of a Wiz-Pub main program.

When the Wiz-Pub main program is selected to open a project, a file search window pops up, a file is selected and opened, and the selected project file is created in the main window.

If the Wiz-Pub main program is selected to save the project, it saves the project, or selects the save icon in the icon menu or the file => save project in the text main menu, and saves it as a * .wpp file in the designated project folder. do.

FIG. 11D is a diagram illustrating Save As under the Wiz-Pub main program. FIG.

Save as another name of the Wiz-Pub main program selects a new storage location and saves the project file.

11E is a diagram illustrating a file import of a Wiz-Pub main program.

Importing files from the main Wiz-Pub program includes documents (* .ppt), images (* .jpg, * .bmp, * .gif, * .tga), movies (* .avi, * .mov, * .mpg, * .asf, * .wma) and PowerPoint (PPT) presentation files are converted to * .jpg files and displayed as closed captions in empty voice cells when you enter notes. The files are converted into source cells with empty voice cells and inserted into the timeline in order.

11F is a diagram illustrating avatar insertion of a Wiz-Pub main program.

Inserting a 3D avatar is displayed in a window view by clicking on "Avatar" in the library and selecting the desired avatar from the avatar list.

11G is a diagram illustrating a background insertion of a Wiz-Pub main program.

To insert a background, click on "Background" in the library, select the background you want from the background list, provide a desktop click 2D or 3D background, and click on the desktop to insert the entire background.

Drag and drop different images from the list onto the background layer of the timeline to provide an insertion slide effect.

11H is a diagram illustrating cell insertion in a source layer of a Wiz-Pub main program.

Insert Image inserts a regular image separately via import (the image is inserted at the image size you set in preferences).

Insert Video inserts the video through import (insert the full size of the video).

Insert blank source cell is mainly used to show only 3D avatar in avatar mode for web. ① When clicking icon, blank voice cell is created in ② project window and ③ blank source cell of timeline and copy text of web to blank voice cell. Can be used by copying & pasting.

When adding cells to the source layer, the cells are placed at the back in the order in which they are added. You can change the order by using the ① up and down buttons.

11i is a diagram illustrating initial position setting of an avatar of a Wiz-Pub main program.

To set the initial position of the avatar, select the desired camera view of the avatar in the window view, and then click Main Menu => Set Initial Position to set the initial position of the avatar. You must set the initial position of the avatar to see the exact movement and movement of the avatar.

FIG. 11J is a diagram illustrating mapping of facial expressions to avatars of the Wiz-Pub main program.

Mapping motion expressions to 3D avatars shows motions and expressions of avatars in the library list (②) when clicking on motions in the library, and uses simple drag & drop to show motions matching the narration of 3D avatars. Insert it into the action line of the timeline editor (③). The mapped action expressions can be previewed using the play, pause, and stop functions in the timeline.

11K is a diagram illustrating lip sync editing of a Wiz-Pub main program.

The entire lip sync conversion function selects the entire lip sync conversion icon (①) and pops up when the content inserted in the note of PowerPoint (PPT) is present in an empty voice cell of the timeline editor (③) (②). ) To convert the subtitles to TTS voice.

The full lip sync conversion function can be used to change the size, margin, language, pitch, and speed settings of the voice cell.

The following describes the precautions for editing lip sync.

Cautions for entering narration scripts or sentences should be written so that the sentence can be read effectively by using a comma (,) when using long sentences, and be prepared with caution. When writing, it is used for spacing, but when it is actually read, it can be pronounced continuously without stopping. However, if you must keep the spacing, you must write it according to the spacing rules. For example, in the case of "must understand", the actual pronunciation is pronounced "must understand." This happens because the voice actor recorded the pronunciation according to the natural rhythm when recording the data of the voice program. Therefore, in order to derive "understand" into clear pronunciation, it should be written as "understand". As such, the tips for dividing the difference between the notation and the pronunciation are divided into // grammar // pronounced //.

In addition, attention should be paid to the pronunciation of the numbers. Numbers 1, 2, 3 ... etc. are pronounced as 1, 2, 3, etc., so be careful with the pronunciation of the numbers. In the case of numbers, it is necessary to judge the situation well and whether it should be expressed in ordinal or radix.

Also, question marks (?), Commas (,), periods (.), And surveys should be pasted with the previous word.

When the question mark (?) Is used, the accent increases accordingly, and when the comma (,) is used, it is paused and pronounced. When the period (.) Is used, it is paused. It should be written with words.

When entering the script, the main grammar uses // caption script // pronounced script //.

The first part enclosed in // is not converted to speech but is shown only for subtitles, and the latter part is converted to speech and used only for actual pronunciation. Even with words that the speech synthesis engine can't express, you can use this grammar to refine the script's voice (e.g. //wisenwide.Com//Wis-Pub//Wizpub/ /).

When you need room to sync to the front and back of the cell, as shown in Fig. 11M, enter // // {3000 (desired time)} // before and after the sentence to create an empty voice cell. Sophisticated editing of the avatar action is possible (the desired unit of time is msec).

For finer phonetic expressions and the sophistication of avatar actions, press Enter when possible, based on (,) or (.). These line breaks are generated by dividing each cell into voices during voice conversion.

11n is a diagram illustrating voice conversion after cell selection of a Wiz-Pub main program.

After selecting a specific cell (①) in the lip sync layer, click the Sync to Subtitles button in the lip sync editor (②), and the voice conversion setting window (③) pops up to confirm after user setting.

11o is a diagram illustrating voice cell correction and editing of the Wiz-Pub main program.

To edit voice cell, select the cell you want to edit, edit it as you want in the Lip Sync Editor, click Align Sync, and enter the maximum text that can be input without Enter key is 126 characters in English and 63 characters in Korean. When the sentence is separated by popping up the poem message or by pressing the Enter key, the cell is also identified in the timeline editor.

To make it easier to edit in detail, it is easy to edit by pressing the Enter key based on a comma or period of a sentence.

FIG. 11p is a diagram illustrating the division of voice cells of a Wiz-Pub main program.

To distinguish the voice cell, select the specific cell to be separated and press the Enter key at the location of the desired position and click the Think Fit button. (The maximum text that can be input without the enter key is 126 characters in English and 63 in Korean. character).

In the case of sorting the voice cell, when the cell is divided into the additional content, the user enters the end of the existing subtitle, inputs the additional content, and clicks Think Sync.

In the case of the voice cell, the source cell such as image, video, flash, etc. of the source part is automatically reduced and expanded according to whether to delete the cell (when the lip sync cell of the video is smaller than the original, it is cut by the lip sync size, From this size onwards, it will play repeatedly: if you delete the last lip sync cell in one source image, the source cell will disappear.

Figure 11p is a diagram showing the insertion of a blank lip sync cell of the Wiz-Pub main program.

Inserting blank lip sync cell inserts blank space by using a blank cell before and after the desired lip sync cell, and inserts a blank voice cell before and after the cell in the following format (=> // // {the desired number of seconds (In msec)} //).

11r is a diagram illustrating lip sync recording of a Wiz-Pub main program.

For lip sync recording, select the cell you want to record, click the right mouse button to select lip sync recording, select the cell and select Tools => Lip Sync Recording. Click to create a rip file in the list. If you click the Finish button after confirming with the Start and Stop buttons, the Rip file is created in the timeline editor and the original TTS file is deleted.

11s is a diagram illustrating keyword action management of a Wiz-Pub main program.

Keyword action management provides a function that a user can manage so that the user can preset and automatically generate an avatar action for a word used in TTS speech conversion. Click Tools => Manage Keyword Actions in the Icon menu and the Text menu => When the popup Keyword Action Management window pops up, select Add Group to set the group according to the characteristics of each avatar, enter the keyword action group name, and enter Create a group name in the list (③).

As shown in FIG. 11T, the keyword action management clicks the add keyword action button ④, pops up the keyword action setting window, inputs the keyword name ⑤, and then displays the action ⑥ and the expression ⑦. Select it and insert it into the keyword action list (8).

After selecting a keyword action group, right-click and click Auto Generate Keyword, or select one of the group list and click the Auto Generate button below to display different actions and facial expression cells. Is generated.

12A is a diagram illustrating a timeline editing tool of the Wiz-Pub main program.

The timeline editing tools include cell select buttons, hand buttons, zoom in buttons, zoom out buttons, trash can buttons, and play / stop / pause buttons.

The cell selection button provides a function of selecting a cell so that the cell can be selected, moved, edited, and deleted.

The hand button provides a full movement of the timeline (left, right).

The enlargement button enlarges the size of each cell and enables detailed editing.

The zoom out button collapses a cell in the timeline to view the overall composition of the cell.

The trash can button is used to delete the selected cell.

The play / stop / pause button can be previewed for confirmation when editing the timeline (displayed in the window view of the main screen).

As shown in FIG. 12B, the timeline editing tool generates a timeline baseline when the timeline of the timeline is selected by using a mouse after using the play, stop, and pause functions, and can view all window view functions by timeframe. Detailed editing is possible.

12C is a diagram illustrating timeline cell editing.

Editing timeline cells can be edited using the simple drag & drop of the mouse. After selecting a cell, you can set the margin between cells when you right-click. You can use the left and right arrows to move in detail after selecting the cell, and delete it using the Del key after selecting the cell.

FIG. 12D provides a preview through the Wiz-Pub Player according to one of 3D avatar narration mode including a slide or subtitle, a presentation (lecture style) mode, and an avatar mode for web.

12E is a diagram illustrating a 3D avatar narration.

3D Avatar Narration enables and disables (fixes size) optional slides and subtitles in the 3D Avatar view, provides control of play / pause / stop, and displays the full time Duration .

12F is a diagram illustrating a presentation (lecture style).

As shown in Fig. 12G, the presentation (lecture style) provides control functions of play, pause, and stop, and provides screen configuration functions such as full screen, avatar, subtitle, full zoom, and screen switch.

12H is a diagram illustrating a whiteboard function.

The whiteboard tool function converts all contents to a pause state when the whiteboard function is used.

12i is a diagram illustrating an avatar for the web.

Avatar for web can be resized independently of avatar window and slide window, slide window can be enable & disable as option, control play, pause, stop, slide It displays Duration when using window.

FIG. 12J is a diagram illustrating an example of storing a multimedia narration player (Wiz-Pub Player) using a 3D virtual agent.

Export (Save as Wiz-Pub Player) can be saved in 3 ways as shown above. It is saved as a wpf file.

*. You can download the simple player by linking with the web without clicking the wpf file even if the Wiz-Pub main program is not installed.

*. wpf files can be uploaded directly to the web to enable web services (distributing CDs and web services at the same time).

12K is a diagram illustrating an Asset library screen of a Wiz-Pub main program. Asset libraries can only be updated when avatars, backgrounds, behaviors, facial expressions, and effects are upgraded or when the customer creates them on request.

The avatar and background are applied to the window view immediately upon click.

Sound effects can be added and managed by selecting the sound you want.

The desktop can be used by adding an image to the library list, and when the list is clicked, it is selected as the image of the entire background, but the slide background can be applied to the avatar background by using multiple images in the timeline.

12L is a diagram for explaining audio setting.

Audio setup provides a user-friendly feature for simple mic and speaker setup for lip sync recording or playback of multimedia narration players (Wiz-Pub Player) using 3D Virtual Agent.

12m is a diagram illustrating an environment setting screen of the Wiz-Pub main program.

Preference settings include margin settings, including initial start margin (in msec) and final end margin (msec) for the entire project, PT image margin (msec), PT image size (%), and PT image transparency (%) , Initial default time (msec), etc. can be set, and the position of the avatar in the window view is possible by simple mouse movement.

In addition, the automatic update function checks the latest version (version) when the Wiz-Pub main program is executed, and updates are automatically executed to always maintain the latest version of the product.

If the Wiz-Pub main program is not the latest version or the multimedia narration player ((Wiz-Pub Player) using the 3D virtual agent is an older version, the Wiz-Pub player also works with the web to update server (Update Server) (300). ) To automatically update the version.

Method for providing a multimedia narration using the 3D virtual agent according to the present invention, generating a new project in the main screen of the multimedia narration authoring tool (Wiz-Pub main program) using the 3D virtual agent; Converting the presentation material (such as a PPT file) into an image file for each page by the PPT image conversion module, and setting and loading the script and the title; Obtaining a play time of a flash file (SWF) by the SWF information collecting module; Previewing a video or obtaining a play time by the video file information collecting module; After loading the presentation material, selecting and displaying a 3D avatar providing a multimedia narration in a window view by a library management module; Select the background, movement (action, facial expression), audiovisual effect (effect, sound effect), and desktop of the selected 3D avatar from the library list, and then drag and drop them to the timeline cell of the timeline editor. Editing each library, source (movie or image or flash file); Edit the script or user input script from the presentation by LipThink Editor, transmit each sentence of the script written in each language to TTS server, and receive the data converted from the TTS server into voice. Doing; Generating authoring results of each edited project for viewing in a multimedia narration player (Wiz-Pub Player) using a 3D virtual agent; Controlling the generated result of the authoring by playing, stopping, and stopping the function in the multimedia narration player by previewing a test output of a multimedia narration including a background, an action, an expression, an effect, an effect sound, and a wallpaper of a 3D avatar; ; And storing the generated project.

The 3D avatar's library list includes head movements (e.g. lean forward, shake left and right, nod twice, etc.), arm motions (e.g. right-handed waist), and full body motions (e.g. one-sided flank, lower back) Greetings), facial expressions (e.g. mouth smiles) and eye expressions (e.g. blinks of the left eye, etc.), and audiovisual effects can be used to remember what you want to convey. It provides a variety of effects that make it easy to 3D animate your source, such as flipping through the form of bulletin boards and flipping through the form of revolving doors.

Actions published in the library provide head movements, arm movements, and overall body movements.

In one embodiment, the head movements are forward lean, shake left and right, nod twice, lift 45 degrees to the right, turn 45 degrees to the right, turn 90 degrees to the right, spout to the right, lift 45 degrees to the left, and to the left Rotate 90 degrees and include left and right movements in the library list.

Arm movements include clapping, folding arms, rested postures, and placing your right hand on your waist.

Full body movements include: Basic posture, pointing left with 135 degrees high, pointing with left hand with 135 degrees lower, raising index finger with right hand, upward growth mark, downward growth mark, pointing up and looking up, hand facing Include in the library list: catching, counting (1, 2, 3), walking, placing both hands on the chest, clasping both hands, holding both hands, explaining with both hands, using the left hand, one-handed sides, bowing .

Expressions include eye expressions (such as blinking both eyes, left eye blinks, raising both eyes) and facial expressions (such as smiling faces) in the library list.

The method of providing a multimedia narration using the 3D virtual agent includes a function of printing a slide portion of a presentation material content, a function of using a video as a current image slide at a predetermined position of a background of the 3D avatar, and a content of the narration test output. According to the shape, the user can directly control the size of the 3D avatar and the view point of the 3D virtual agent.

The method can be applied to DRM on multimedia narration player (Wiz-Pub Player) or image slide, flash, video output to prevent illegal copying of contents produced from multimedia narration authoring system (Wiz-Pub main program) using 3D virtual agent. (Digital Rights Management) is applied to protect the contents by inserting a function to create a watermark in case of illegal copying.

The method may provide a function of replacing a 3D avatar at a desired position on the timeline, and changing a voice or a voice according to the replaced 3D avatar (when changing a TTS, changing a voice of a man and a woman or a voice of a woman according to the avatar voice). Change the content) by outputting another male avatar or female avatar.

The method provides a function of storing only a selected portion when selecting and storing a desired position in the timeline.

The method provides a function of adding a moving object to a background in 3D space, and provides a function of simulating a product simulation or an operation method by simulating the movement of the object of various avatars according to a timeline setting.

The method provides the ability to replace part of the body (face, hair, etc.), clothing, and accessories of the 3D avatar in the middle of the timeline.

The method provides a function of replacing the face of the 3D avatar with the face provided by the user by using the face picture provided by the user.

The method provides a function for controlling movement (movement, facial expression, and movement) of each 3D avatar in a timeline editing function so that a plurality of 3D avatars can come out and interact with each other interactively.

The method provides a moving line control function between avatars in 3D space that can control a plurality of 3D avatars to move along each moving line in 3D space.

In the above method, multimedia data (image slide, moving picture, flash) is executed on an object existing in the 3D space under the control by setting the timeline.

The method may be implemented by moving the 3D avatar in a 3D space so that an image slide, a video, a flash, a web, or an independent window may be displayed when the specific object is touched. .

As described above, the method of the present invention may be implemented as a program and stored in a recording medium (CD-ROM, RAM, ROM, floppy disk, hard disk, magneto-optical disk, etc.) in a computer-readable form.

As described above, although described with reference to a preferred embodiment of the present invention, those skilled in the art various modifications of the present invention without departing from the spirit and scope of the invention described in the claims below Or it may be modified.

As described above, the multimedia narration authoring system using the 3D virtual agent and the method of providing the same according to the present invention, using various authoring tools, background, motion, facial expressions, effects, sound effects, wallpaper, video, flash And 3D Virtual Agent Narration with various effects in presentation documents and images, written by language (Korean, English, Japanese, Chinese) by TTS function according to the interactive multimedia contents platform. Converts text to speech and outputs it, and provides text presentations through multimedia 3D avatars for product manuals, guidance, in-house training, corporate marketing materials, product catalog production, presentation materials, and training. Reduce manpower, cost, and time, and retrain employees It provides the effect of improving work efficiency.

It also uses a multimedia narration authoring system using 3D Virtual Agent to position displays in HTML documents to create web content, help assistants, educational tutors, sales agents, corporate mascodes. It is effective to use 3D virtual agent in large PDP or kiosk for entertainer and offline.

Claims (26)

In the multimedia narration authoring system using 3D virtual agent, A source management module that provides information collection and format conversion functions for sources in various formats; A result generation module for storing the edited project for playing and viewing a multimedia narration using a synchronized 3D virtual agent according to a timeline; A project management module for performing detailed operations according to the project file generation, storage, and change; A library management module for managing individual libraries relating to 3D avatars, backgrounds, movements, and audiovisual effects; A timeline editing module for editing each library and the source according to a timeline; A lip sync editing module connected to a TTS module and an MP3P conversion module of a TTS server, for editing a script or a text input by a user; And A keyword management module for applying a predetermined action or facial expression of the 3D avatar to each keyword of the script; Multimedia narration authoring system using a 3D virtual agent comprising a. The method of claim 1, The source includes an image conversion file, a movie file, and a flash file of the presentation, The source management module, An image conversion module for converting a presentation material file of a document format into an image file for each page and importing a script and a title from the presentation material; A SWF information collection module for obtaining information about a playback time of a flash file (SWF); And A video file information collection module for previewing a video or obtaining information regarding a play time; Multimedia narration authoring system using a 3D virtual agent comprising a. The method of claim 1, The result generation module, A compression module for compressing 3D avatar data, and multimedia data; An encryption module for encrypting the result of the authoring by a predetermined encryption algorithm (RSA 128bit method) so that it can be played only by a dedicated player; And A watermark module for applying a watermark to the multimedia data to prevent illegal copying and tampering; Multimedia narration authoring system using a 3D virtual agent comprising a. The method of claim 1, The result generated by the result generation module, Narration mode, in which the presentation is embedded and played inside the 3D virtual agent, Presentation (PT) mode by splitting the display screen so that the 3D virtual agent and the presentation are played at different positions, and Avatar mode for web to play 3D virtual agents at certain locations on web pages Multimedia narration authoring system using a 3D virtual agent comprising a. The method of claim 1, A mouth generation module for automatically generating a corresponding mouth shape when the 3D avatar speaks a sentence; A sound output module for outputting an effect sound and a voice file according to a preset timeline; A 3D engine module for representing the 3D avatar in a preset virtual space; And A web control module for connecting the 3D virtual agent to a web page; Multimedia narration authoring system using a 3D virtual agent further comprising an output module having a. The method of claim 5, wherein The 3D engine module, A view point control module for controlling the 3D virtual agent; An effect module for providing various 3D animation effects for the source; A character module configured to receive character data to configure and control the character of the 3D avatar; A background module for loading a 3D background to position and control the mesh data; A texture management module for managing textures of 3D avatars and backgrounds; A motion management module for managing movement of the 3D avatar; A mesh management module for managing a mesh object of data for each body part of the 3D avatar representing the multimedia narration; A timeline management module for managing a timeline of a timeline cell in a predetermined time unit; And A renderer including a video encoder / decoder, a Flash filter, and a file filter, for final rendering on the screen using Direct 3D; Multimedia narration authoring system using a 3D virtual agent comprising a. The method of claim 1, The 3D virtual agent, Multimedia narration authoring system using a 3D virtual agent including the motion, expression, background, effects, sound effects, wallpaper, video, flash, images of the 3D avatar. The method of claim 5, wherein The output module, A playback control module for controlling playback of the authoring result generated by the result generating module; And A paint module which allows a user to express a desired image; Multimedia narration authoring system using a 3D virtual agent comprising a. The method of claim 1, It is connected to the multimedia narration authoring system (Wiz-Pub main program) using 3D virtual agent that provides the motion, background, and audiovisual effects of 3D avatars, converts sentences to speech, and converts voice files to the multimedia narration authoring system. A TTS server for transmitting; And An update configuration module for updating and setting versions of files to be managed included in the multimedia narration authoring system using the 3D virtual agent, and an update Active X for keeping the files of the multimedia narration authoring system up to date with a set file list. An update server comprising a module; Multimedia narration authoring system using a 3D virtual agent further comprising. The method of claim 9, The TTS server, A TTS module for converting sentences into speech files for each preset language; And An MP3 conversion module for converting the voice file into an MP3 file; Multimedia narration authoring system using a 3D virtual agent comprising a. The method of claim 10, The preset language, Multimedia narration authoring system using 3D virtual agent, characterized in that the Korean, English, Japanese, Chinese. In the method of providing a multimedia narration using a 3D virtual agent, (a) generating a new project in the main screen of the multimedia narration authoring system using the 3D virtual agent; (b) converting the presentation material into an image file for each page by the image conversion module and setting and loading the script and the title; (c) obtaining a playback time of the flash file by the SWF information collecting module; (d) previewing a video or obtaining a playing time by the video file information collecting module; (e) selecting the 3D avatar for providing the multimedia narration by the library management module and loading the presentation data in the window view; (f) Select the background, motion, and audio-visual effects of the selected 3D avatar from the library list, link them to the timeline cells in the Timeline Editor by drag-and-drop, and then select each library and source (movie or image or Flash file). Editing; (g) editing the script input from the presentation or the user input from the presentation by the LipThink Editor, transmitting each sentence of the script to a TTS server, and receiving the voice-converted data; (h) generating an authoring output of each edited project for viewing in the multimedia narration player; (i) reproducing the authoring result in a preview in the multimedia narration player, and trial outputting the multimedia narration; And (j) storing the project; Multimedia narration providing method using a 3D virtual agent comprising a. The method of claim 12, In the step (f), the library list of the 3D avatar includes a head motion, an arm motion, an entire body motion, a facial expression and an eye expression, and the library list on the audiovisual effect makes it easy to store the content to be delivered. A method of providing a multimedia narration using a 3D virtual agent, characterized in that various effects for 3D animation are included. The method of claim 12, A method of providing a multimedia narration using a 3D virtual agent further comprising the step of printing a slide portion of the content of the presentation. The method of claim 12, And using the video obtained in the step (d) at a predetermined position of the background of the 3D avatar. The method of claim 12, In the step (i), when the multimedia narration test output, the method of providing a multimedia narration using the 3D virtual agent further comprises the step of the user directly controls the size of the 3D avatar and the view point for the 3D virtual agent according to the type of the content . The method of claim 12, In order to prevent illegal copying of contents produced from the multimedia narration authoring system, the multimedia narration player or the multimedia using the 3D virtual agent further includes inserting an illegal copy watermark by applying digital rights management (DRM) to the source. How to Provide Narration. The method of claim 12, When the 3D avatar is replaced at a desired position on the timeline, the multimedia narration using the 3D virtual agent further includes a step of explaining the content by outputting a voice or a tone according to the replaced 3D avatar in connection with the TTS server. Way. The method of claim 12, The method for providing a multimedia narration using a 3D virtual agent further comprises the step of storing only the selected portion when the desired location in the timeline is stored. The method of claim 12, When the moving object is added to the background in the 3D space, the method further comprises the step of the virtual experience of the movement of the object according to the timeline settings. The method of claim 12, A method of providing a multimedia narration using a 3D virtual agent further comprising the step of replacing a body part, a costume, and an accessory of a 3D avatar in the middle of a timeline. The method of claim 12, And replacing the face of the 3D avatar with the face picture provided by the user using the face picture provided by the user. The method of claim 12, A method of providing a multimedia narration using a 3D virtual agent further comprising a step of controlling each 3D avatar's movement in a timeline editing function so that a plurality of 3D avatars can interactively react and talk to each other. The method of claim 12, A method of providing a multimedia narration using a 3D virtual agent further comprising a moving line control step of controlling a plurality of 3D avatars to move along each moving line in a 3D space. The method of claim 12, A method of providing a multimedia narration using a 3D virtual agent, further comprising the step of executing multimedia data on an object existing in a 3D space under control of a timeline. On a computer, (a) creating a new project on the main screen of the multimedia narration authoring system using the 3D virtual agent; (b) converting the presentation material into an image file for each page by the image conversion module, and setting and loading the script and the title; (c) a function of obtaining a playing time of the flash file by the SWF information collecting module; (d) a function of previewing a video or obtaining a playback time by the video file information collection module; (e) a function of selecting a 3D avatar for providing a multimedia narration by the library management module and loading the presentation data in a window view; (f) Select the background, motion, and audio-visual effects of the selected 3D avatar from the library list, link them to the timeline cells in the Timeline Editor by drag-and-drop, and then select each library and source (movie or image or Flash file). The ability to edit; (g) editing a script input from the presentation or a user input from the presentation by the LipThink Editor, transmitting each sentence of the script to a TTS server, and receiving the voice-converted data; (h) the ability to generate authoring output for each edited project for viewing in a multimedia narration player; (i) reproducing the authoring result in a preview in the multimedia narration player, and trial outputting the multimedia narration; (j) storing the project; (k) a function of inserting a watermark in the case of illegal copying by applying DRM (Digital Rights Management) to the multimedia narration player or the source to prevent illegal copying of contents produced from the multimedia narration authoring tool; (l) a function of changing a voice or a tone according to the replaced 3D avatar when the 3D avatar is replaced at a desired position on the timeline, and outputting the voice; (m) a function of virtually experiencing the movement of the object according to a timeline setting when adding a moving object to a background in 3D space; (n) controlling the movement of each 3D avatar in a timeline editing function such that a plurality of 3D avatars can interactively react and communicate with each other; And (o) a function of executing multimedia data on an object present in 3D space under control by setting of a timeline; A computer-readable recording medium for realizing the

Images