KR100317299B1 - MPEG-4 Video Conference System And Multimedia Information Structure For MPEG-4 Video Conference System - Google Patents

Abstract

The present invention relates to a multimedia information structure for MPEG-4 videoconferencing, which allows a videoconferencing system to express (describe) moving pictures and audio using MPEG-4 technology and to perform real-time two-way video communication using the same. MPEG-4 video conferencing system.

The present invention (a). The information describing the scene describes the description information of the two-dimensional screen of the image and the voice, and (b). The information describing the scene describes information on a two-dimensional screen of image and sound in a node and tree structure, and (c). The information depicting the scene includes an object descriptor ID relating to information describing the object, an initial object descriptor initially delivered, and information relating to a codec of a stream of video and audio, (d ). MPEG-4 video conferencing system, characterized in that the information is binary encoded via the ASCII form and represented in BIFS to achieve MPEG-4 video communication.

Description

MPEG-4 Video Conference System And Multimedia Information Structure For MPEG-4 Video Conference System

The present invention relates to a multimedia information structure for MPEG-4 videoconferencing, which allows a videoconferencing system to express (describe) moving pictures and audio using MPEG-4 technology and to perform real-time two-way video communication using the same. MPEG-4 video conferencing system.

MPEG-4 is the standard for the next generation multimedia standard proposed and standardized by SG29 WG11 under ISO / IEC. Its application fields are broadcasting, home shopping using broadcasting, real-time moving picture, animation and voice transmission on the Internet. It can be used for various purposes such as home shopping, information shopping, etc.

An important advantage of MPEG-4 is that it is an object-oriented system that can view all the media used in the multimedia communication as a single object and compose the whole object. Is the point.

Independent processing of multimedia objects allows content creators to record each object independently and then choose the compression method with the maximum efficiency for each object. You can also develop.

And because it can contain information about the relationship between objects and the user's access to the object, it is possible to provide a scenario that can change dynamically according to the user's intention, not unilateral and fixed content. It also provides access control under copyright to address the issue of copyright in multimedia content that will flood in the future.

The development of video conferencing system using MPEG-4 with such features and advantages is also considered as a possible application field.

That is, for example, an image (video) of a speaker photographed with a camera based on a PC and a voice (audio) signal of a speaker collected by a microphone are expressed in MPEG-4, and video communication (bidirectional real-time communication) is used using this. In this case, the efficiency and effectiveness of the above MPEG-4 features and benefits will be maximized.

However, in the development and dissemination of the video communication system up to now, the so-called video phone is the mainstream, and the video phone compresses and transmits audio and video signals in real time to transmit and receive the video and audio of the other party. It is a technology for mutually displaying and listening to images and audio.

According to the present invention, a video conferencing system using MPEG-4 technology can be implemented by describing a moving picture and a voice of a speaker using MPEG-4 technology, and by using this description information structure to enable two-way real-time communication with the other party. , MPEG-4 videoconferencing information structure and MPEG-4 videoconferencing system.

In particular, the present invention transforms the reference models (IM1 Player2D, IM1 Player3D) provided by the MPEG-4 system, and ASCII forms of scenes and object descriptors (ODs) to be considered when developing a videoconferencing system. And a method of generating a BIFS (Binary Format for Scene Description) and a method of transmitting and receiving the data bidirectionally in real time according to a video conference system.

1 is a diagram showing an example of an MPEG-4 scene for explaining the present invention.

2 is a diagram showing an example of an MPEG-4 information structure (DS and stream) for explaining the present invention.

3 is a view showing an example of the screen configuration of a videoconferencing system for explaining the present invention;

4 is a diagram showing the structure of an MPEG-4 video conference system according to the present invention.

5 is a flowchart showing a transmission control procedure of the MPEG-4 video conference system according to the present invention.

FIG. 6 is a view showing an information structure in ASCII format representing a scene in the present invention. FIG.

7 illustrates the structure of an Initial Object Descriptor in the present invention.

8 illustrates the structure of an object descriptor of a media stream in the invention;

9 is a diagram illustrating the type of codec to be used in video conferencing for the purpose of illustrating the present invention.

FIG. 10 is a diagram for explaining and comparing an IM1 Player and a video conference system in the present invention. FIG.

Multimedia information structure for MPEG-4 video conferencing of the present invention;

In order to express multimedia information using MPEG-4 technology when the multimedia is a moving picture, information describing a scene, information describing a multimedia object corresponding to the scene, and audio and video streams corresponding to the scene and the object are displayed. When multimedia information is represented with indicating information, for MPEG-4 video communication;

(a). The information describing the scene describes the description of the two-dimensional screen of the image and sound,

(b). The information describing the scene describes the information on the two-dimensional screen of the image and sound in a node and tree structure,

(c). The information depicting the scene includes an object descriptor ID relating to information depicting an object, initial object descriptors initially delivered, and information relating to codecs of streams of video and audio,

(d). A multimedia information structure for MPEG-4 video conferencing, characterized in that each piece of information is encoded in binary through ASCII format and represented in BIFS to achieve MPEG-4 video communication.

Also, in the present invention, the information describing the scene describes the information of the node and the tree structure for the two-dimensional screen of the voice and the image in a Scene.bif file, and the object descriptor ID, the initial object descriptor, the image and the voice A multimedia information structure for MPEG-4 video conferencing, wherein information related to a codec of a stream is described in a Scene.od file.

Also, in the present invention, the Scene.bif and Scene.od files are multimedia information structures for MPEG-4 video conferencing, which are described as Scene.txt.

Also in the present invention, when the information describing the scene describes a node and a tree structure for the two-dimensional lower surface of the voice and the image, as the node; MPEG-, characterized by specifying the relative position, shape, shape, sound object, etc. of the object with a Group, a Translation2D, a Shape, an Appearance, and a Sound2D. 4 It is a multimedia information structure for video conference.

Also, in the present invention, the Image Texture node of the Appearance and the Sound2D node have url as a field indicating information (object descriptor ID) indicating a stream of video and audio on an MPEG-4 system. It is a multimedia information structure for MPEG-4 video conferencing.

Also, in the present invention, the initial object descriptor may specify a profile for compatibility between MPEG-4 terminals, reference to a Scene.od and a Scene.bif, and a decoder value. It is a multimedia information structure.

Also, in the present invention, the media stream object description information as the information related to the codec of the video and audio streams is information indicating the media stream buffer for MPEG-4 video conferencing and audio as the object type ID having different values for each codec. It is a multimedia information structure for MPEG-4 video conferencing that indicates when to use and when to use audio and video together.

On the other hand, the MPEG-4 video conference system of the present invention; Audio / video processing means for inputting and outputting audio and video signals of a user and the other party in real time, means for storing the multimedia information (Scene.bif, Scene.od, Initial.od), the audio input and output, And means for multiplexing and transmitting the information structure stored in the video signal in real time.

In addition, the MPEG-4 video conference system control method of the present invention; Configuring multimedia information (Scene.bif, Scene.od, Initial.od) for MPEG-4 video communication, setting communication conditions with the other party, Initial.od file to be transmitted and n used in online multiplexing program MPEG-4 video conferencing system characterized by setting .od, transmitting and receiving Initial.od and Scene.bif and Scene.od, and transmitting and receiving audio and video streams according to the codec to transmit and receive Control method.

According to the present invention made as described above, as a first feature of the present invention, a method of configuring in ASCII form according to various kinds of audio and video codecs (CODECs) used for scenes to be constructed in a videoconferencing system will be described.

That is, the BIFS for the scene and the BIFS for the object descriptor OD for each media stream are expressed in ASCII form.

In addition, as a second aspect of the present invention, after generating BIFS using variously generated ASCII forms according to various kinds of audio and video codecs used in the videoconferencing system, a method of using the same in the videoconferencing system will be described. do.

That is, by using the MPEG-4 video conferencing information structure of the present invention mentioned above, MPEG-4 content is generated and transmitted in real time with the input of voice and camera through a PC based on a PC, and also received by MPEG-4. The configuration of a videoconferencing system for achieving basic video communication and the control procedure thereof will be described.

First, for a videoconferencing system, a part for generating and transmitting MPEG-4 content with voice and camera input through a PC in real time is required.

In order to generate MPEG-4 contents in addition to the compressed and transmitted parts of moving images and audio, information of a BIFS (Binary Format for Scene Description) section describing a scene defined in the MPEG-4 Systems section is provided. need.

In other words, it is necessary to create a BIFS that describes the screen of the user (the speaker) and the counterpart screen as one scene during a video conference.

In addition, additional information is required for the definition of the initial stream and for the transmission of the characteristics of each stream. This information is called an object descriptor (OD), which is also expressed in BIFS, and becomes the content of the stream.

FIG. 1 shows a lecture scene as an example of an MPEG-4 scene, and FIG. 2 illustrates an MPEG-4 stream delivery method (FrameWork), and the scene (Scene), the object descriptor (OD), BIFS, etc. This is to explain the meaning of.

Since the example of FIG. 1 is composed of each object of the lecture scene, the object may be the instructor 101, the blackboard 102, the desk 103, the earth 104, the voice of the lecturer (audio), etc. The binary representation of a scene made up of these objects is BIFs.

1 shows how each object is placed and viewed on one plane.

That is, depending on the viewer's viewpoint, when the scene is a three-dimensional image, the scene is displayed in various shapes.

Meanwhile, an object representing voice or music may also sound differently depending on a user's viewpoint according to the location and characteristics of the object.

In addition, the scene may be configured to rotate when the user clicks on the earth using a mouse.

In the previous lecture scene of FIG. 1, audio information as an object of MPEG-4 is displayed by an audio compositor 105 and video information by a video compositor 106 is displayed 107. Listening 108 may be performed by a speaker or the like.

2 shows how MPEG-4 actually delivers information in MPEG-4 format.

Referring to FIG. 2, the MPEG-4 information structure is roughly divided into initial object description information 201, scene description schemes 202 and 203 depicting scenes, and object description schemes describing objects. 204, 205, and a video stream (Visual Stream) 206, an audio stream (Audio Stream) 207, and the like, which are section information on which the object indicated by the scene is performed.

The object descriptor in the MPEG-4 information structure serves as a pointer to each object in the MPEG-4 structure.

Initially in MPEG-4, an Initial Object Descriptor (Initial OD) 201 must be delivered for initial MPEG-4 information.

This value refers to two streams 202 describing the scene and two streams 204 describing the object descriptor (OD) 205, and the size of the decoder's buffer for these two streams. To convey specific information.

In general, stream 204 of object descriptor (OD) 205 points to streams corresponding to each media object, and also contains information for the decoder of each object.

Tree-type information about the scene is transmitted using the stream 202 describing the scene, and the receiver can generate the scene using the stream 202.

In the stream 204 describing the object descriptor (OD), information about each object pointed to by the scene 203 is transmitted, and the stream of each object can be connected to a corresponding node in the screen. It may be added or deleted as needed.

That is, logical information of the scene described in FIG. 1 is transmitted through the streams 202 and 203 describing the scene, and information indicating the object of each scene is transmitted through the object streams 204 and 205.

And the MPEG-4 system supports this structure.

Referring to FIG. 2, the MPEG-4 information structure is a structure including a new initial object descriptor (OD) in case of an InLine node including another scene in one scene, and only an additional respective media object stream. In this case, the MPEG-4 information structure is composed of a tree structure (208, ...).

Using this InLine, it can be structurally simple to include an already created MPEG-4 file.

Therefore, based on the above MPEG-4 information structure, for MPEG-4 videoconferencing, information corresponding to scenes and object descriptors (OD) describing scenes of videoconferencing in accordance with the MPEG-4 standard should be provided. do.

Therefore, the present invention provides such an MPEG-4 video conference information structure, and also provides a system for performing MPEG-4 video communication using this information structure.

3 shows a screen configuration of a general video conference system.

The features of the video conferencing system are video and audio, two-dimensional screens, and the user expects the video and audio to be synchronized from the moment the call is initiated, so there is no need to start and stop the video. .

In addition, the picture and sound used herein generally belong to the coding of the natural image and audio, and thus do not use the synthesized image or sound.

The above features belong to the Simple2D Profile according to the profile defining the stage of the scene of ISO / IEC 14496-1 Chapter 13 MPEG-4.

That is, it can be expressed with only the audio 301 and the image 302 defined in the two-dimensional plane.

FIG. 4 illustrates the structure of an MPEG-4 video conferencing system capable of performing video communication with an MPEG-4 information structure for video communication, represented only with audio and pictures, as shown in FIG.

The network interface 401, the IM1 Player 402, the audio / video device 403, the MPEG-4 video communication information 404, the multiplexer 405 and the like are shown.

Audio and video signals from the other party received via the network interface 401 are generated as audio (A) and video (V) signals via the IM1 Player 402, and displayed and output through the audio / video device 403. The audio and video signals of the user (speaker) collected by the audio / video device 403 are processed to be suitable for MPEG-4 video communication using the MPEG-4 video communication information 404, and the multiplexer 405 is used. The network interface 401 is sent to the other party.

The detailed structure and meaning of the MPEG-4 video communication information 404 will be described later.

The procedure of video communication control by the MPEG-4 video conference system of FIG. 4 is shown in FIG.

First, as a pre-communication step; A Scene.bif file, an Initial.od file, and a Scene.od file to be used for MPEG-4 video communication are created (MPEG-4 video communication information 404).

And as a communication start step; The communication module used in the videoconferencing system and the DMIF used in MPEG-4 are requested to exchange 'capability' of data transmission capability information with the other party, and set a common 'capability'.

After setting 'Capability', it decides the Initial.od file to send and the n.od to use in Online Mux program.

Here, Initia.od and Scene.bif use fixed ones and select Scene.od according to common Capability.

Next, as a screen transfer step; Once the codec to transmit / receive is determined, it starts to transmit using Online Mux. First, it transmits and receives Initial.od, Scene.bif and Scene.od, and then transmits and receives audio and video streams.

At this time, Initial OD is exchanged at the beginning of communication, n.od is transmitted as OD Update Command, and Scene.bif is transmitted as BIFS Update Commmand.

At the end of the communication, the meeting ends with a normal termination.

As described above, the MPEG-4 video conferencing system, ASCII information representing a scene, an Initial Object Descriptor, and a media stream for MPEG-4 video communication made by the system. It is necessary to define and use an MPEG-4 video conference information structure such as object description information of a stream.

The following describes how the MPEG-4 videoconferencing information structure of the present invention is made and what it means.

[One]. Configure ASCII Forms for BIFS for Use in Video Conferencing Systems

(One). As described above, the screen configuration (including audio) in video communication as shown in FIG. 3 is composed of video and audio (two-dimensional screen), and the user expects the video and audio to be synchronized from the start of the call. Therefore, there is no need for the function of starting and stopping images in particular, and the image and sound used do not use the synthesized image or sound.

Therefore, only audio and images defined in a two-dimensional plane can be represented.

(2). ASCII composition to represent the scene

Fig. 6 shows an ASCII form for representing Fig. 3 as a scene of MPEG-4.

This ASCII format is identical to that described in the VRML97 Spec of ISO / IEC 14772-1.

MPEG-4 adds new nodes and semantics, and defines binary coding of ASCII.

In short, an MPEG-4 scene consists of a tree of nodes.

Corresponding curly braces indicate the boundaries of a node, and the nodes contained within it become child nodes.

These types of nodes include Group, Translation2D, Shape, Appearance, and Sound2D, respectively, which means Root, and the relative of objects. It is responsible for specifying the location, specifying the shape, specifying the shape, and specifying the sound object.

The node has a field, which has a name and a value corresponding to that name. This value can also be a node.

In particular, the 'Appearance' Image Texture node and the 'Sound2D' node have a url field, which is an object descriptor (OD) that indicates a stream of video and audio on an MPEG-4 system. Has the value of ID.

In conclusion, the scene itself for videoconferencing can be fixed and used regardless of the type of audio video codec.

In FIG. 6, 'REPLACE BY SCENE' and 'UPDATE' are the same as commands for instructing the next node or information.

Therefore, not only the scene information is generated by the initial scene starting with the 'Group' node, but also the information about the object descriptor OD is made into the delivery stream.

The above file is actually encoded by MPEG-4 IM1-BifsEnc to 'Scene.bif' file under 'REPLACE BY SCENE', and separated by 'Scene.od' under 'UPDATE'.

The ASCII information above is usually stored as Scene.txt.

In more detail, Scene.bif is a constant scene description according to audio and video codec, but Scene.od is a change of information of mux.scr, that is, information of each object descriptor (OD). It varies according to.

That is, it varies depending on the type of audio and video codec used for video conferencing and the value thereof.

This information is separated into the 'muxScript Mux.scr' file.

The format used by IM1, which makes the information in the object descriptor (OD) in ASCII form, and then encodes it in binary, which is used by the Initial Object Descriptor (Initial OD) and the commands associated with each object descriptor (OD). It is used to refer to the information of the object descriptor (OD).

The contents of 'muxScript' are largely divided into contents of an initial object descriptor (FIG. 7) and an object descriptor for a media stream (FIG. 8).

(3). ASCII construct to represent object descriptors (OD) in Mux.scr

The file of 'Mux.scr' expresses MPEG-4 information called Initial Object Descriptor (Initial OD) and information related to the codec of each image and audio stream in BIFS.

This is called BIFS in relation to the object descriptor (OD).

The initial object descriptor exchanges a profile value for compatibility between MPEG-4 terminals, and specifies a reference to a Scene.od and a Scene.bif and a decoder value.

The initial object descriptor of Fig. 7 is the same as the general case of the videoconferencing, and thus does not need to be changed specifically for the videoconferencing.

In the case of video conferencing, the object descriptor of FIG. 8 may use audio only and audio and video together, and codecs may vary for each audio and video.

In FIG. 8, the square thick dotted portion 801 is a special type used in IM1-Mux, and refers to a structure in which a media stream to be used in an MPEG-4 file is referred to as a file.

However, since videoconferencing systems are bidirectional communication in real time, a structure is needed that points to an internally accessible buffer rather than a file.

In addition, the value of the thick thick dotted line 802 means an object type ID (Objcet Type Indication), each having a different value for each codec used.

For example, a combination of codecs to be used in a videoconferencing system can be considered as shown in FIG.

In Fig. 9, * 1 and * 2 are typical cases in the case of MPEG-4.

Codecs such as H.263 and G.723 are likely to be used because they are widely used in existing video conferencing systems.

The ObjectTypeindication values corresponding to each codec are listed in the standard and may be user defined.

In the example of FIG. 8, although the same value 'ox21' is used for audio and video, the stream type is different because it is different from the video stream and the audio stream.

In conclusion, in order to support various codecs of the video conference system, the object descriptor OD corresponding to the media of the mux.scr file of FIG. 8 may be modified.

In addition, the representation of the initial object descriptor corresponding to FIG. 7 and the scene corresponding to FIG. 6 may be fixed.

[2]. Changes to IM1 for Video Conferencing Systems

(One). Condition

* The video conferencing system uses a scene composed of Simple2D audio and video. In other words, you have a fixed scene.

Use a fixed initial object descriptor (Initial OD).

* Change the content of the Media OD you need based on the combination of codecs you want to use. That is, prepare several Mux.scr files containing the descriptor (OD) information according to the codec. File Mux1.scr that supports only MPEG-4 audio, or File Mux3.scr that supports MPEG-4 audio and video (see Figure 9 for number).

Has a module for encoding / decoding user voices and moving images of videoconferencing systems.

It has a module that can send and receive video conference system information in real time.

(2). What to transform in IM1

Fig. 10 shows the contents implemented in the reference model and the parts to be modified for application to video conferencing.

That is, in the reference model (IM1 Player), it is a structure that is read from an arbitrary mp4 file and then decoded through a decoder for each data and then composed, and not from a local file for MPEG-4 video conferencing. Real-time communication is performed through the network.

Also, in the reference model (IM1 Player), BifsEnc and Mux, which generate mp4 files, are separated, and a media stream, which is a Mux input, is provided in the form of a file.However, for MPEG-4 video conferencing, audio and video are encoded in real time. Each packet of Mux must be included in the IM1 Player.

In the reference model (IM1 Player), the operation of the IM1 Player was provided in the form of 'push senario', which is a broadcast and client-server type. You must change it so that it follows.

As described above, for MPEG-4 videoconferencing, scene descriptors, initial object descriptors, and object descriptors of media streams are defined. Initial.od and Scene.bif use fixed ones, and according to 'Capability' Select Scene.od to transmit and receive Initial.od, Scene.bif, Scene.od, and then transmit and receive the corresponding audio and video stream to enable MPEG-4 video communication (i.e. real-time two-way audio / video transmission and reception). The procedure of the communication system and the control is as described above with reference to Figs.

In particular, in FIG. 4, the BIFS streams 404 are generated in advance according to the MPEG-4 stream type, and can be used as needed.

According to the MPEG-4 videoconferencing system of the present invention, it is possible to develop a videoconferencing system compatible with MPEG-4 by a simple structure (modification) using IM1 under development as an MPEG-4 reference model.

In other words, the MPEG-4 video conferencing system can be created by using a mux operating offline in real time and using a previously generated BIFS stream and an audio and video stream generated in real time instead of generating an mp4 file. Can be.

In addition, the information required to make the videoconferencing system MPEG-4 is fixed except for the codec of the audio and video stream, and the types of * .bif and * .od files that are MPEG-4 elements are limited. To simplify the system.

Claims (9)

In order to express multimedia information using MPEG-4 technology when the multimedia is a moving picture, information describing a scene, information describing a multimedia object corresponding to the scene, and audio and video streams corresponding to the scene and the object are displayed. When multimedia information is represented with indicating information, for MPEG-4 video communication; (a). The information describing the scene describes the description of the two-dimensional screen of the image and sound, (b). The information describing the scene describes the information on the two-dimensional screen of the image and sound in a node and tree structure, (c). The information depicting the scene includes an object descriptor ID relating to information depicting an object, initial object descriptors initially delivered, and information relating to codecs of streams of video and audio, (d). And each of said information is encoded in binary via ASCII format and represented in BIFS to achieve MPEG-4 video communication. The method of claim 1, wherein the information describing the scene describes a node and a tree structure information of a voice and a two-dimensional screen of a picture in a Scene.bif file, an object descriptor ID, an initial object descriptor, and a picture and voice. And information related to the codec of the stream of the video stream is described in the Scene.od file. 3. The multimedia information structure of claim 2, wherein the Scene.bif and Scene.od files are described as Scene.txt. 2. The device of claim 1, wherein the information depicting the scene describes a node and a tree structure for a two-dimensional bottom surface of voice and image; MPEG-, characterized by specifying the relative position, shape, shape, sound object, etc. of the object with a Group, a Translation2D, a Shape, an Appearance, and a Sound2D. 4 Multimedia Information Structure for Video Conferencing. 5. The method of claim 4, wherein the Image Texture node of the Appearance and the Sound2D node have a url as a field indicating information (object descriptor ID) indicating a stream of video and audio on an MPEG-4 system. Multimedia information structure for MPEG-4 video conferencing. 2. The MPEG-4 video conferencing of claim 1, wherein the initial object descriptor designates a profile for compatibility between MPEG-4 terminals, a reference to Scene.od and Scene.bif, and a decoder value. Multimedia Information Structure. The media stream object description information as information related to the codec of the video and audio streams is information indicating the media stream buffer for MPEG-4 video conferencing and audio as an object type ID having a different value for each codec. A multimedia information structure for MPEG-4 video conferencing, which indicates only when to use and when to use together audio and video. In order to express multimedia information using MPEG-4 technology when the multimedia is a moving picture, information describing a scene, information describing a multimedia object corresponding to the scene, and audio and video streams corresponding to the scene and the object are displayed. When multimedia information is represented with indicating information, for MPEG-4 video communication; Audio / video processing means for inputting and outputting audio and video signals of a user and the other party in real time, and means for storing multimedia information (Scene.bif, Scene.od, Initial.od) according to claim 1 to claim 7. And means for multiplexing and transmitting the information structure stored in the input and output audio and video signals in real time. In order to express multimedia information by MPEG-4 technology when the multimedia is a moving picture, information describing a scene, information describing a multimedia object corresponding to the scene, and audio and video streams corresponding to the scene and the object are displayed. When multimedia information is represented with indicating information, for MPEG-4 video communication; Constructing the multimedia information (Scene.bif, Scene.od, Initial.od) according to claim 1 for MPEG-4 video communication, setting the communication conditions with the other party, Initial. It is controlled to set n.od to be used in od file and online multiplexing program, to send and receive Initial.od and Scene.bif and Scene.od, and to send and receive audio and video streams depending on the codec to send and receive. MPEG-4 videoconferencing system control method.