KR100580437B1 - Method and system for client-server interaction in interactive communications - Google Patents

Abstract

In an interactive communication system based on MPEG-4, a command descriptor along with a command route node or a server route in a screen description is provided. Can be used to support application-specific interactivity. Content selection can be supported by specifying an indication in a common parameter, where the command identification number indicates that the command is a content selection command. The initial screen may be generated in text and in various pictures describing the display associated with the picture. Each picture and corresponding text associated with it is a content selection descriptor. When the user clicks on the picture, the client sends a command holding the selected presentation and the server starts a new display. As HTTP and CGI are commonly used to implement any server-based application behavior, this technique can be used in any application environment.

Mpeg-4, interactivity, client, server,

Description

Method and system for client-server interaction in interactive communication {METHOD AND SYSTEM FOR CLIENT-SERVER INTERACTION IN INTERACTIVE COMMUNICATIONS}

The present invention relates to a communication system, and more particularly to a technique for performing client-server interaction in a communication system based on the MPEG-4 standard.

Interactivity is an important concern for the development of the MPEG-4 International Standard (ISO / IEC 14496 Parts 1-6, Committee Draft, October 31, 1997, Friborg, Switzerland). The back channel is defined for supporting interactive messages. However, the syntax and semantics of the message to be carried over that channel are still undefined, and so is the mechanism for triggering the transmission of such a message. Existing standards such as DSM-CC (ISO / IEC International Standard 13818-6) and RTSP (RFC 2326) have traditional VCR-format interactions to reposition the media stream during playback. , But it is not suitable for MPEG-4 applications that require more complex interactivity control.

The interaction message may be generated by some user action or system event. The interaction message is then returned to the server, which can in turn change the stream delivered by adding or separating objects, or the stream switching to the whole new screen. User actions may include clicks on objects, input of text streams, and the like. System events include timers, conditional tests, and the like.

Interactivity is application specific and you cannot define fully interactive activities for user events. To support application-specific interactivity, CGI-style methods must be adopted. Certain user events allow application-specific command data to be returned to the server. The server may then respond by typically sending a screen description update command. This allows for full interactivity as required by the application.

MPEG-4 uses essentially two modes of interactivity: local and remote. Local interactivity is based on the VRML 2.0 ROUTE design (see "The VRML Handbook" published in 1996 by J.Hartman and J.Wernecke at www.vrml.org and Addison-wesley publishers) This can be fully implemented using the original event structure of MPEG-4 Binary Format for Scenes (BIFS), which is included in Part 1 of the System. If the MPEG-4 receiver is a host in another application, events that need to be communicated by the application to the MPEG-4 receiver may be transferred to a BIFS update command as defined in part 1 of the MPEG-4. have.

Remote interactivity is done with the current URL. As defined in the MPEG-4 System Committee Draft, these remote interactivity can only be used to gain access to content. After all, they cannot be used to initiate a command.

Server interactions that fully integrate with the local interactivity model from the fact that the MPEG-4 system already supports local interactions using event source / sink routes that are part of the screen description (BIFS). It is more preferable to have an action treatment.

It is a primary object of the present invention to provide a technique for communicating messages between two entities, such as "client" and "server," using the MPEG-4 international standard.

It is a second object of the present invention to provide a technique for a user or a system to generate messages of the context of an MPEG-4 player or a client.

A third object of the present invention is to provide a technique for generating a message conforming to a local interactivity model defined in MPEG-4 based on the VRML 2.0 specification.

Another object of the present invention is to provide a technique for encoding a message in an MPEG-4 bitstream, as well as a technique for linking the encoded message with a scene description.
It is another object of the present invention to provide a technique for encoding a message in a manner that allows the server to easily change the message before sending the message for use by the client. This is important for interactive applications. An example is " cookie " management in which the server must be able to quickly update the content of a command having a codeword that stores status information about the user's activity at a particular site.

In order to achieve these and other objects, which will become apparent with reference to the additionally disclosed description below, the present invention provides a wide range of techniques for combining server commands with MPEG-4 clients. The technique involves using a special type of descriptor that is sent with a command descriptor, i.e., screen description information, that holds the command returned to the server at the start of the associated event. The desired event source in the screen description is associated with these command descriptors.

In one embodiment, the association is performed using a server root. They work similar to a typical MPEG-4 BIFS root, but link the source field with the sink command descriptor instead of linking the source field with the sink field. The server root requires an expanded MPEG-4 BIFS ROUTE statement.

In another embodiment, the association is performed using a command node. These nodes hold sink fields and are associated with command descriptors. This technique involves the addition of one or more node types to a set of MPEG-4 BIFS nodes.

In all cases, the normal interaction model defined by MPEG-4 is that local interactivity, i.e. events generated and processed on the local client, as well as server interactivity, i.e. events generated on the client, are returned to the server. Can be used in all cases that produce. When initiating an event associated with a command descriptor via a server root or regular route to a command node, the client obtains the command information stored in the command descriptor, binds the command information into a command message (syntax provided in the preferred embodiment). (preferably using syntax), and send this command message to the server using the appropriate return channel.

Data carried by the generated command returned to the server is included in the command descriptor. Since the command descriptors are part of the full descriptor framework of MPEG-4, they can be dynamically updated using a timed object descriptor update. This provides a great deal of flexibility in the customized commands, for example to perform "cookies" management.

In order to further assist the server processing the generated command, additional information is also provided in the client's message, such as when the event was generated, source node, and the like.

1 is a block diagram illustrating the overall structure of an MPEG-4 client or terminal.

2 shows an MPEG-4 system decoder model.

3 illustrates a method used in MPEG-4 for associating an audiovisual object with encoded data in another stream through object descriptors and elementary stream descriptors.

4 shows a general configuration of a client / server MPEG-4 based communication system.

FIG. 5 illustrates a method of associating a scene description node, in particular a sensor node, with a command descriptor using (a) a server root, and (b) a command root node.

FIG. 6 illustrates the binary syntax of the command descriptor as described in the preferred embodiment.

Figure 7 illustrates the binary syntax of the command descriptor removal command as described in the preferred embodiment.

Figure 8 illustrates the binary syntax of the server root structure, and how it is added to the main MPEG-4 BIFS event syntax.

9 illustrates node syntax and semantics of the command root structure.

Fig. 10 shows a display list of preset command identification numbers and associated interpretations.

Figure 11 illustrates the binary syntax of the commands included in the command descriptor in the preferred embodiment.

12 is a flowchart illustrating a process of generating a server command in an MPEG-4 client when a server root is used.

FIG. 13 is a flow chart illustrating a process of collecting data to be placed in a command returned to the server when the server root is used.

14 is a flowchart illustrating a process of generating a server command in an MPEG-4 client when a command root node is used.

15 is a flow chart illustrating a process of collecting data to be placed within a command returned to the server when a command root node is used.

Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the drawings.

MPEG-4 is an international standard developed under the auspices of the International Organization for Standardization (ISO). Its official name is ISO / IEC 14476. The fundamental difference from previous ISO or ITU standards such as MPEG-1, MPEG-2, H.261, or H.263 is in dealing with the presentation of audiovisual objects. Thus, different components, including the audiovisual picture, are first encoded separately using the techniques defined in part 2 (video) and part 3 (audio) of the above provision. These objects are sent to the receiver or read from the mass storage device with screen description information describing how these objects are placed spatially and temporally so that they can be communicated to the user.

In addition to the screen description information, encoded data for each audiovisual object is transmitted in its own "channel" or elementary stream. Additional control information is also transmitted so that the receiver can correctly associate the audio stream object referenced in the picture with the elementary stream holding its encoded data as described below.

In order to fully describe the structure of MPEG-4, reference is made to FIG. 1. In the lower part of FIG. 1, several possible delivery systems are shown, including but not limited to ATM, IP, MPEG-2 Transport Stream (TS), DVB, over a communications link or mass storage. Compared to MPEG-2, MPEG-4 does not define its own transport layer facility to be able to be delivered for more diverse communication environments. A simple multiplexing tool called FlexMux is defined for a delivery system that may lack the appropriate multiplexing capabilities required, eg, GSM wireless data channels, or low delay.

These infrastructures are used to deliver a set of elementary streams to the client. The stream holds either screen description information, audiovisual object data (e.g., video encoded as an MPEG-2 or MPEG-4 video stream), or control information (i.e., an object descriptor). Each elementary stream can only hold one type of data.

The data retained in the elementary stream is packaged according to the MPEG-4 Sync Layer (SL), which packages the access unit of the underlying medium (e.g., video or audio frame, or picture description command). Add timing information (clock reference value and time display), serial number, and so on. The SL is shown in the middle of FIG. 1.

The encoding of the individual audiovisual object data is performed according to parts 2 and 3 of the MPEG-4 specification. Other encodings such as MPEG-1 or MPEG-2 may be used in addition. The screen description as well as the control information (object descriptor) are encoded as specified in part 1 of MPEG-4.

The receiver processes and composes the screen description information as well as the decoded audiovisual object data, i.e. the process of combining the objects together into a single unit, and the rendering, i.e. displays the result on the user monitor. Or in the case of audio, the process of reproducing the result in the user speaker. This is shown at the top of FIG. 1.

Depending on the information held in the screen description, the user may have the opportunity to interact with the screen. In addition, the screen description may carry information that enables dynamic activity. In other words, the screen itself can generate events without user intervention.

The object-based structure of MPEG-4 required a more general system decoder model to be compared to MPEG-2 or other systems. In particular, as shown in FIG. 2, the receiver is considered to be provided with a set of decoders, one for each object. Each decoder has a decoding buffer as well as a synthesis buffer. The decoding buffer is managed by the sender using a technique similar to MPEG-2, i.e., using a clock reference value for clock recovery, and a decoding time format for separating data from the decoding buffer which is theoretically immediately decoded. The data placed in the synthesis buffer can be used by the synthesizer and overlaps any previously placed data. The decoding buffer is filled with a demultiplexer that is summarized within the Digital Media Integration Framework (EMF-4 Part 6) application interface. The interface is a conceptual interface that requires no further explanation herein.

MPEG-4 Screen Description

Screen description information in MPEG-4 is in the extended Virtual Reality Modeling Language (VRML 2.0) specification. VRML uses a tree structured approach to screen selection. Each node in the picture performs a synthesis and / or grouping operation with leaves that hold actual visual or auditory information. In addition, the node holds fields that affect its activity. For example, a transform node holds a rotation field to determine the angle of rotation.

Because VRML is a pure three-dimensional picture description language, MPEG-4 defines some additional nodes that deal with two-dimensional compositing. This addresses the needs of applications focused on low cost systems. Compared to VRML, MPEG-4 does not use text-based screen descriptions, but instead defines an efficient, bandwidth-compressed representation called BIFS (binary format for screens).

The BIFS encoding closely follows the text specification of the VRML screen. In particular, node coding is performed in a depth-first fashion similar to text-based VRML files. As in VRML, the fields of each type of node assume default values. Therefore, only those fields that do not have a default value need to be defined within each node. Field coding in a node is performed using a simple index-based method followed by coded field values. Node-type coding is more complicated. In order to further increase the bandwidth efficiency, the parent field back and forth relationship (if any) is considered. Each field that accepts child nodes is associated with a specific node data type. The nodes are then coded using indices specific to these node data types in a well known manner.

Each encoded node may also be assigned a node identifier (typically an integer). The assignment of this node identifier allows the node to be reused elsewhere in the screen. This is the same as VRML's USE / DEF mechanism. However, it is even more important that it participates in the interaction process.

The interaction model used in MPEG-4 is the same as in VRML. In particular, the field of nodes may act like an event source, an event sink, or both. Event sources are associated with specific user actions or system events. An example of a user event is a sensor node that can be detected when the mouse is clicked. An example of a system event is a timer (TimeSensor node) that is started according to the time of the system.

Dynamic screen activity and interactivity is validated by linking the event source field to the event sink field. The actual link is done using the mechanism of ROUTE. The MPEG-4 root mechanism, if any, is given immediately after the screen node description. The encoding is based on the field identifiers of the source and sink fields as well as the node identifiers for the source and sink nodes.

An important difference between VRML and MPEG-4 is that later in the description the screen description can be updated by dynamically using time-displayed commands. In contrast, VRML operates on a static "world." After the world is loaded, there is no mechanism to change it. In MPEG-4, objects may be added or deleted, and part (or entire screen) of the screen may be replaced.

Object descriptor

To have a very flexible structure (easy editing, etc.), the actual content of the audiovisual object is not held in the screen description itself. In other words, BIFS only provides information for the screen structure, as well as the red rectangle constructed using the objects to be integrated, for example BIFS / VRML nodes. Audiovisual objects that require encoded information are displayed on the screen by leaf nodes that point to URLs or object descriptors.                 

The object descriptor is hierarchically organized information describing an elementary stream containing an encoded representation of a single audiovisual object. One or more streams may be required, for example, for stereo or multi-language descriptors, or hierarchically coded video. The top descriptor is called an object descriptor and is a shell used to associate an object descriptor identifier (OD-ID) with an elementary stream descriptor set. The elementary stream descriptor set holds the ES-ID as well as information about the data format held in the elementary stream associated with a particular ES-ID. This information tells the receiver, for example, that the stream holds MPEG-2 video data, followed by a mail profile at the main level.

Mapping the ES-ID into the actual elementary stream is performed using a stream map table. For example, ES-ID 10 may be associated with support number 1025. This table can be used at the receiver during setup work. Using multiple indirect levels facilitates manipulation of MPEG-4 content. For example, remultiplexing may only require different stream map tables. No other information is changed within the MPEG-4 content.
The object descriptor is sent in its own elementary stream and packaged into commands according to the sync layer syntax. This allows the object descriptor to be updated, added, and removed.

3 illustrates a process by which an object descriptor and an elementary stream descriptor are used to associate an audiovisual object in a screen description with its elementary stream. First, a particular initial object descriptor is used to bootstrap the MPEG-4 receiver by indicating the object descriptor stream and the screen descriptor stream associated with the selected content. This descriptor is communicated to the receiver during setup.                 

The screen description in this example holds an audio source node and represents one of the object descriptors. The descriptor holds an elementary stream descriptor that provides the associated stream to the ES-ID. The screen also has a Movie Texture node that uses video that can be accumulated with two streams in this case. As a result, two elementary stream descriptors are retained, each pointing to a suitable stream (base and enhancement layer).

MPEG-4 Client / Server Interaction

From the foregoing description, it is clear that considering the MPEG-4 / VRML screen description structure, there is no convenience in affecting server-based interactions as long as the local interaction structure is sufficiently provided. In particular, there is no mechanism to describe the messages returned to the server or to initiate such message generation.

4 illustrates an example of a client / server environment. On the left side of the figure is an MPEG-4 server, which includes a pump containing an in-time transfer of data read into an SL-packed stream, and in this case, a DMIF service provider using the MPEG-4 FlexMux multiplexing tool. Include an example.

Using FlexMux is optional. Other server structures known in the art may be used. For example, the data source may be an MPEG-4 file instead of an SL-packaged stream.

The information generated by the server is transmitted to the receiver via a network (eg IP or ATM). At the receiver side, there is an example similar to a DMIF service provider delivering a demultiplexed elementary stream to a player. DMIF-to-DMIF signaling is one way of performing the setup work and is described in part 6 of MPEG-4. Other methods are possible, including Internet-based protocols such as SIP, SDP, RTSP, etc., as known in the art.

One of the main objectives of the present invention is to provide a process wherein a server command is first described and sent from the server to the client, then initiated at the client at the appropriate time, and eventually returned to the server for further operation.

Command descriptor

The command descriptor structure provides a means of associating a command with an event source in a node of the screen graph. When the user interacts with the screen, the associated event is triggered and the command is processed continuously and returned to the server.

The actual interaction itself is defined by the content creator and may be mouse clicks or mouse-overlaps or some other form of interaction (eg, system event).

The command descriptor structure consists of three components: a command descriptor, a server root or command root node, and a command.

The command descriptor contains an identification number (an integer identifier) as well as the actual command that eventually returns to the server when the associated event is initiated.

The identification number is used to refer to this command descriptor from the screen description information. By separating the held command descriptor and command from the screen itself, one or more events are allowed to use the same command. This allows you to modify the command without changing the screen description in any way.

Associating a command descriptor with an event source field node can be performed in different ways.

First, server roots can be added to the normal root installation of BIFS. The difference from the conventional route is that the object of the route is a command descriptor rather than another field. This structure is shown in FIG. 5A. In this example, we have two nodes in the screen tree field event with two command descriptors. The screen node initiates an event to an event source, event-output, or event node / sink, another node with exposed fields in the MPEG-4 area, and is distributed in turn through the server root to command descriptor 1. For command descriptor 2, the server root is directed from the node to the descriptor.

The second method involves adding a new command root node type to the list of nodes supported by MPEG-4. This node has a "execute" event sink field as well as a field holding the command descriptor identification number. Whenever the "Execute" field receives an event using a regular MPEG-4 / VRML route, the command descriptor associated with that identification number is used to issue a command to return to the server. This structure is shown in Figure 5b. In comparison with FIG. 5A, the server root is replaced with a command root node. The operations in both cases are essentially the same.

The command descriptor syntax is shown in FIG. Flavor media representation language is used to be described as a bit stream, which is also used in part 1 of the MPEG-4 specification (see www.ee.columbia.edu/flavor or part 1 of MPEG-4). The command descriptor begins with a specific tag that identifies itself with this particular form of descriptor. It then has a descriptor identification number, followed by a command identification number. The command identification number is "start". It is used to signal a preset server command, such as "pause" or "stop". Next, the length indication of the remaining data in the descriptor counted in bytes is included. Then, the count of the number of ES_IDs provided to send the message is returned to the server. A value of 1 or greater is given if one wants to affect one-to-many communication, i. This is followed by a series of predetermined ES_IDs. Finally, a set of application-specific parameters is included. They return to the server when the command is initiated. Depending on the value of the command identification number, the meaning of these parameters can be predetermined.

The byte-oriented structure of the command descriptor can be easily generated in contact with the server. This is an important characteristic of an application that includes "cookie" management, especially if the command parameters need to be continuously updated by the server after each user event processing.

To update the command descriptor, the server or content creator only needs to submit a new command descriptor with the same identification number.

In order to remove the command generator, a specific command is provided. The syntax is shown in FIG. The command begins with a tag identifying this descriptor as a command descriptor removal command. This is followed by the identification number of the command descriptor to be removed.

The command descriptor and command descriptor removal structure may be carried in an object descriptor stream. Because of the object descriptor structure that uses tags to identify descriptors, they can be interspersed with other descriptors.

As mentioned above, there are two ways to link the command descriptor to the screen. The first method depends on the server root. These require the bitstream syntax of the screen description of expanded MPEG-4 as shown in FIG. In particular, the main BIFS screen structure is extended to include the server root structure. The only difference between a normal root and a server root is that the identification number of the target command descriptor is defined instead of the target node / field. It is well known to those skilled in the art how to modify other root commands (insert, delete, etc.) to accommodate the server root. In all cases, the target node / field pair needs to be changed to point to the target command descriptor.

Using the command root node method, a new node type needs to be determined. Node definitions are provided in FIG. 9 using the standard node definition table used in part 1 of the MPEG-4 specification. The node contains only two fields: the 'execute' field, which acts as an event sink, and the 'command descriptor' field, which holds the URL or command descriptor identifying the command descriptor to be associated with the command root node. Include. As the SFUrl field, selection between the identification number and the URL is performed using a 1-bit flag (SFUrl field encoding is specified in part 1 of MPEG-4).

Using the command root node approach, the change of the event to association with the command descriptor can be performed using standard MPEG-4 BIFS commands that change nodes and fields.

The command descriptor may be updated using a command descriptor update that allows the node to support different interaction activities at different times according to the commands in the command descriptor. The command is sent to the server using the DAI user command primitives supported by DMIF.

It is also possible to extend the command descriptor to include the protocol used to send the command. For example, different tags can be used to indicate a proposal using HTTP POST / GET rather than a standard MPEG-4 facility.

Although command descriptor structures are specific and supported for application-specific user interactions, some standard commands are required to support matching application activity for applications and servers. This is particularly suitable for common commands such as stream control. Finally, a set of common command controls is defined. The set of standard commands will interact with its server. Fig. 10 shows a command set with a command identification number for this embodiment. Other assignments are also possible, as will be apparent to those skilled in the art.

Fig. 11 shows the syntax of the command when it is sent from the client to the server. It is essential that a copy of the command descriptor be left out of the descriptor identification number. In particular, it includes a command identification number, a set of ES_IDs to which this command is sent, as well as a set of parameters defined within the command descriptor. These commands are sent in an SL-packaged stream, so that full timing and subsequent information is available to the server.

Event handling for sending server commands

Hereinafter, a process of generating a command based on a user or system event started using the server root will be described in detail.

Referring to FIG. 12, upon generation of a user or system event, the receiver propagates the event over a network of routes and server routes. For event propagation, the format of the route does not matter and the same algorithm can be used. When an event propagates through the server root, the system checks whether the event corresponds to a condition associated with a logical true value. If no, server command processing ends; If yes, then the sending process is executed.

For the server root case, this process is shown in FIG. The process obtains a command descriptor identification number from the server root. Then, the information including the command descriptor available in the tie scene is correlated with the command descriptor identification number. If it does not match, this is an error and no further action is taken. If so, the system then checks the command identification number to see if it corresponds to a known meaning (predetermined command identification number). If it corresponds to a known meaning, then the system can process the command parameter according to the predetermined meaning. If it does not correspond to a known meaning, the system skips this state, packages the indicated command directly, and sends it to the server.

Referring to FIG. 14, in the case of a command root node, upon generation of a user or system event, the receiver propagates the event through the network of routes. When the event reaches the 'run' field of the command root node, the system checks whether the event corresponds to a condition associated with a logical true value. If no, the server command processing ends; If yes, the dispatch process is executed.

Processing for the command root node case is shown in FIG. The order of the steps is essentially the same as the order of FIG. 13, except that the criteria for the command descriptor identification number are now within the command root node, not the command server root.

delete

Claims (14)

A method of communicating command information between a server and a client in an interactive communication system, Generating a command message comprising a command, a command descriptor, and one of a server route and a command node, and Transmitting the command message when a trigger event occurs Communication method comprising a. In claim 1, And wherein the interactive communication system communicates command information based on MPEG-4. In claim 2, Generating the command message is consistent with a local interactive model defined in MPEG-4. In claim 1, And the trigger event is a mouse click. In claim 1, And the trigger event is a timer signal. In claim 1, And the command information is sent from the server to a client. In claim 1, And the command information is sent from the client to a server. An interactive communication system comprising means for communicating command information between a server and a client, the method comprising: Means for communicating command information between the server and client Means for generating a command, a command descriptor, and a command message having one of a server root and a command node, and Means for sending the command message upon occurrence of a trigger event Interactive communication system comprising a. In claim 8, And wherein the interactive communication system is based on MPEG-4. In claim 9, And the means for generating the command message is consistent with a local interactive model defined in MPEG-4. In claim 8, And wherein the trigger event is a mouse click. In claim 8, And wherein said trigger event is a timer signal. In claim 8, And wherein said command information is sent from said server to a client. In claim 8, And wherein said command information is sent from said client to a server.

Images