KR100339374B1 - multimedia middleware in distributed process system - Google Patents

Abstract

The present invention relates to multimedia middleware in a distributed processing system, and more particularly, to implement middleware to systematically and efficiently process and transmit stream media such as audio and video, which are commonly handled by various types of multimedia applications. Objects for creating applications in the processing system, implementations of the interfaces defined in the applications, and an object for transferring control data between the application object and the application component object. It consists of a request repeater.

Description

Multimedia middleware in distributed process system

The present invention relates to a distributed processing system, and more particularly, to multimedia middleware in a distributed processing system.

Middleware generally refers to software located between an application and an operating system.

The role of this middleware is to provide application developers with a programming abstraction one step further than the services provided by the operating system.

Here, abstraction is a term commonly used in the operating system, meaning literally something abstract.

Hereinafter, the middleware in the distributed processing system according to the prior art will be described with reference to the accompanying drawings.

First of all, a computer system consists of hardware such as CPU, memory, disk, Ethernet controller, etc., but when general application developers write application for specific purpose, they do not write the program to control these hardware by themselves. Do not.

The operating system of the computer system abstracts this hardware into the form of processes, virtual memory, files, sockets, and the like, respectively, and provides high-level applications that can control these abstracted objects. Provide it to the developer.

Application developers then use these abstractions to write programs that implement the desired functionality.

The middleware provides application developers with a programming environment that is suitable for implementing a specific application service to be developed by defining high-level abstractions that are not defined at the operating system level.

In addition, middleware suggests a new programming model to applications that use it according to its type and characteristics. In particular, middleware used in a network programming environment is a stable programming base for writing applications on various computer systems and communication networks. Can be.

This middleware makes it easy to write network programs without detailed knowledge of application program interfaces.

In addition, low-level network services provided by the operating system are used indirectly through middleware.

In recent years, many application programs have been modeled and written in object-oriented languages as the object-oriented approach has been widely spread as a software modeling technique, and as the object-oriented programming languages supporting it have become popular.

This object-oriented programming technique not only facilitated the modularization and reuse of software components, but also greatly influenced the formation of basic concepts for distributed processing in a network environment.

Research continues to extend object-oriented programming techniques to distributed processing environments, especially in the software industry, with several specifications already defined or under review.

A typical example is the Common Object Request Broker Architecture (COBA) specification from the Object Management Group (OMG), a consortium of related software companies, and products that comply with this specification have already been developed by various companies.

Microsoft also offers its own object-oriented distributed programming environment called DCOM (Data Communication System).

These products are the middleware needed to write object-oriented distributed programs. With this, multiple components of the software defined as objects can exist distributed on the network, which is used to access each distributed component on the network. This means that transparency of the location is guaranteed.

However, since the existing middleware supporting object-oriented distributed programming in the multimedia middleware in the distributed processing system according to the prior art is based on the request / reply type interaction according to the client / server model, There is a problem that it is inappropriate to write an application that handles continuous media in the form of streams such as audio or video.

Accordingly, the present invention has been made to solve the above-mentioned problems, and the present invention provides a systematic and efficient processing and transmission of continuous media data in a stream format, such as audio and video, which are commonly handled by various types of multimedia applications in a distributed processing environment. The purpose is to provide a multimedia middleware in a distributed processing system.

1 is a view showing the overall configuration of a distributed multimedia system according to the present invention

2 is a diagram illustrating an application software operating state of the same type in each system by applying multimedia middleware in a distributed processing system according to the present invention.

3 is a diagram illustrating a detailed configuration of a virtual device of FIG. 1.

FIG. 4 is a diagram illustrating a stream connection negotiation process of FIG. 1.

5 is a diagram illustrating a multicast stream connection negotiation process of FIG. 1.

6 is a diagram illustrating a process of processing an event in the port of FIG. 3;

7 is a diagram illustrating a processing process when an event of FIG. 1 occurs;

Explanation of symbols for main parts of the drawings

101: application component

102: virtual device

103: stream connection

104: Group

105: Multicast Stream Connection

106, 107: Interface implementation part defined by Apptraction

108: Object Request Broker

A feature of A Multimedia-supporting Architecture with High-level Abstraction (MAHA) in a distributed processing system according to the present invention for achieving the above object is an application for creating an application program in a distributed processing system ( Abstraction), an implementation part of the interfaces defined in the application pieces, and an object request repeater for transmitting control data between the application piece object and the application component object.

Another feature of the multimedia middleware in the distributed processing system according to the present invention for achieving the above object is a virtual device for the application to abstract the actual hardware as an object, and between the virtual devices, Stream connection for performing the connection, Group for controlling the grouping of the virtual devices, and Multicast Stream Connection for performing the multi-connection between the virtual devices. ), Etc.

Another feature of the multimedia middleware in the distributed processing system according to the present invention for achieving the above object is a processing element for the virtual device to perform a unique operation of each virtual device (Virtual Device) (Processing Element) and the media data generated by the virtual device (Virtual Device) to pass to the stream connection (Stream Connection) or the media data accumulated in the stream connection internal buffer from the virtual device (Virtual Device) A port controller and a virtual device that are connected to the plurality of ports and the plurality of ports, respectively, to determine location information of the media stream or control the flow of the media stream, and the virtual device. A stream controller in units of) and the plurality of ports, respectively, Is separate from the control of the process and the media stream of the media data may makin made in a format (Format) abstraction of the specific details relating to the format of the media data.

Another feature of the multimedia middleware in the distributed processing system according to the present invention for achieving the above object is to generate a callback function for processing an event, and to generate a specific MAHA object Registering a callback function to be called by using register function in event registration server when a specific event is received, and event occurs to the MAHA object that will actually generate the event. Requesting to call a send event member function of an event registration server, and when the event occurs, an event related to the event in the object that generated the event ( event) calling a send event function, which is a member function of a registration server, and an application developer based on the information received through the send event function. There through interaction that calls the lock that the callback (callback) function step.

Hereinafter, preferred embodiments of the multimedia middleware in the distributed processing system according to the present invention will be described with reference to the accompanying drawings.

First, the name of the multimedia middleware described in the present specification is referred to as a multimedia-supporting architecture with high-level abstractions (MAHA).

1 is a diagram showing the overall configuration of a distributed multimedia system according to the present invention, wherein a plurality of application components 101, a virtual device 102, and a stream connection ( MAHA consisting of 103, Group 104, Multicast Stream Connection 105, and Interface Implementation Units 106 and 107 defined in the application, and Object Request Repeater Request Broker) 108, Camera 109, Disk 110, 119, Microphone 111, Operating System 112, 116, Device Manager 113, 118, RTP ( Real Time Transport Protocol) 114, 117, a network 115, a speaker 120, and a screen 121.

The virtual device 102, the stream connection 103, the group 104, and the multicast stream connection 105 are stored as virtual resources. This is called.

The operating systems 112 and 116 comprise a device driver 11, a User Datagram Protocol (UDP) 12, TCP 13, and IP 14, respectively.

The virtual device 102 means to abstract various physical devices that generate, process, and finally consume various media data, and do different things according to each virtual device. .

For example, a microphone functions to capture analog voice, and a speaker functions to output analog voice through a reverse process.

In addition, there may be devices for encoding and decoding various media data, and there may be a mixing device for synthesizing voices from various sound sources.

The stream connection 103 serves as a connection path for media data transmission between the virtual devices 102 but does not actually transmit media data.

The group 104 controls the media stream and allocates resources by grouping the virtual devices 102.

That is, the group 104 controls the media streams flowing through the virtual device 102 and the stream connection 103 as a whole, and an arbitrary media stream is requested by an application. Various resources of the computer system are optimally allocated to each virtual device 102 and stream connection 103 through which the media stream passes so as to flow while satisfying QoS.

The multicast stream connection 105 serves to allocate a multicast address to enable one-to-many connection between the virtual devices 102.

The interface implementation unit 106 and 107 defined in the application section include a virtual device 102, a stream connection 103, a group 104, and a multicast stream connection. It implements the interface defined in (Multicast Stream Connection) 105 and the like to control hardware devices that produce or consume continuous media data such as audio or video and transmit the continuous media data.

The object request broker 108 includes the plurality of application components 101, a virtual device 102, a stream connection 103, and a group. (Group) 104 and Multicast Stream Connection 105 are in charge of communication.

The plurality of application components 101, the virtual device 102, the stream connection 103, the group 104, and the multicast stream connection Communication between the Multicast StreamConnections 105 is represented in the form of Interface Definition Language (IDL) required by the object request repeater 108.

2 is a diagram illustrating an application software operating state of the same type applied to each system by applying multimedia middleware in a distributed processing system according to the present invention. 209 to 212, the microphones 213 to 216, and the first to fourth devices 221 to 224, which are connected to the cameras 217 to 220, respectively, are connected through a network, and a computer system having various hardware platforms and operating systems. In this network environment, the same object request repeater and MAHA are installed on each system, and the application written on the basis of them is consistent with the uniform, uniform (regardless of the location of the network of the system in which it is installed). Indicates that it can be accessed in a uniform form.

FIG. 3 is a diagram illustrating a detailed configuration of the virtual device of FIG. 1, and includes a processing element 301 for performing a unique operation of each virtual device, and the processing element 301. A plurality of ports 302 to 304 for delivering the generated media data to a stream connection or reading media data accumulated in a stream connection internal buffer from a processing element, and the plurality of ports ( Ports 302 to 304 are connected to each of the stream controller (305 ~ 307) and the virtual device (Virtual device) unit of the port unit to determine the location information of the media stream or to control the flow of the media stream A stream controller 308 and the plurality of ports 302 to 304 are respectively connected to process the media data and control the media stream. Separated is configured in a format (Format) (309 ~ 311) abstraction of the specific details relating to the format of the media data.

There is a parent / child relationship between the stream controller 308 in the virtual device unit and the stream controllers 305 ˜ 307 in the port unit.

Each of the ports 302 to 304 transfers the media data generated by the processing element to a stream connection 103 or the media data accumulated in an internal buffer of the stream connection 103. It reads from device processing elements.

4 is a diagram illustrating a stream connection negotiation process of FIG. 1, FIG. 5 is a diagram illustrating a multicast stream connection negotiation process of FIG. 1, and FIG. 6 is an event at a port of FIG. 3. FIG. 7 is a diagram illustrating a processing process when an event occurs and FIG. 7 is a diagram illustrating a processing process when an event occurs in FIG.

The multimedia middleware in the distributed processing system according to the present invention configured as described above will be described in detail with reference to the accompanying drawings.

First, an application developer may transmit audio data input from a microphone 213 of a first device 221 through a network in a network environment including computer systems having various hardware platforms and operating systems as shown in FIG. 2. When writing an application program that transmits to the speaker 210 of the device 222, the stream connection 103 as shown in FIG. 1 has the first device 221 and the second device 222 as shown in FIG. Format negotiation is used to connect between networks.

That is, the stream connection 103 searches whether the media data to be transmitted is in a form that can be recognized by both the first and second virtual devices corresponding to the first and second devices 221 and 222.

As a result of the search, if the media data can be recognized by the first and second virtual devices, the stream connection 103 opens a transmission path.

The format negotiation process can be classified into three types according to the preference of the application developer.

First, if the application predetermines all the formats of media data to be transferred between the two virtual devices, the stream connection 103 does not perform a format negotiation and detects whether the formats of the predefined media data for the two virtual devices are in conflict. If there is a conflict, an error is returned in the transmission path connection, otherwise the connection path connection is established.

Second, if the application predetermines the media data type of only one virtual device of the two virtual devices, the stream connection 103 determines the format of the remaining virtual device by which the application defines the media data type of one virtual device that has not been determined. To match.

Third, if the application program does not determine the media data format of both virtual devices, the stream connection 103 finds and determines the best media data format that can be agreed between the two virtual devices.

On the other hand, since the two virtual devices connected to the stream connection 103 may exist in the same system, or may exist in different systems, the stream connection 103 internally determines the actual location of the two virtual devices, and sometimes the shared memory. An area or network connection channel should be allocated to create a media data transmission path between the first and second devices 221 and 222.

That is, as shown in FIG. 3, a plurality of ports 302 to 304 in the virtual device deliver the generated media data to a stream connection or a stream connection. Read the media data stored in the internal buffer from the virtual device.

Then, the stream controllers 305 to 307 in the port unit and the stream controller 308 in the virtual device unit are connected to the plurality of ports 302 to 304, respectively. The location information of the media stream or control the flow of the media stream.

In addition, formats 309 to 311 are connected to the plurality of ports 302 to 304, respectively, and are separated from the processing of the media data and the control of the media stream, thereby abstracting detailed matters related to the format of the media data. To control.

Accordingly, the processing element 301 transmits through the ports 303 and 304 by performing an element's own operation.

The format of the media data may be related to a quality of service (QoS). For example, a virtual device generating a WAVE format and a virtual device playing a WAVE format may be sampled during a format negotiation process by a stream connection. Sample rate, resolution, and channel (mono or stereo sound) will be determined.

Since these three factors ultimately determine the amount of data to be transmitted per second, the stream connection can implicitly obtain information about bandwidth during QoS. You can determine the network bandwidth, the size of the internal buffer, and so on.

Here, other QoS factors (eg jitter, delay, etc.) are ignored until directly specified by the application.

On the other hand, Stream Connection basically performs one-to-one connection between virtual devices.

Therefore, when attempting to connect from one source virtual device to two destination virtual devices, two stream connections must be allocated.

If the media data delivered through two stream connections are the same format, one to many connections may be made in one stream connection.

This concept of stream connection is extended to enable one-to-many connection as well as one-to-many connection between the virtual devices, so that an application called multicast stream connection is called an abstraction. Define a new).

The reason for distinguishing a stream connection from a multicast stream connection is to assign a multicast address to a network connection unless it distinguishes between a connection to be multicasted and a connection that is not. This creates a burden.

Therefore, when connecting virtual devices on a network through a general stream connection, as shown in FIG. 5, the MAHA is assigned a unicast Transport Service Address Pair (TSAP), but does not include a multicast stream. When connecting (Multicast Stream Connections) in a network, a multicast address is allocated from a multicast address allocation server.

Here, the method of connecting a stream connection uses a member function of a stream connection called connect ().

From the application's point of view, the connect () only creates a virtual data path between two virtual devices, and is actually included in a stream connection to transmit media data through the virtual data path. You must use the stream class named Stream Controller.

Here, the media data path between the virtual devices connected through the connect () may be invalidated by disconnect ().

In the case of the multicast stream connection, a media data path between two virtual devices is established using connect (), but once the connection () is successfully performed, another destination virtual device ( To add a Destination Virtual Device, attach () must be used.

Since the stream controller is included as a member of the virtual resource class, the stream controller becomes a member of all classes that inherit the virtual resource class.

The stream controller does not actually transmit the media stream, even though a connect () is called between two virtual devices, even though a channel for transmitting the media data is created.

In order to transmit a media stream through the passage, a function called start () of a stream controller included in the stream connection must be called.

However, if the stream controller of the virtual device connected to the stream connection does not start (), only the stream controller of the stream connection starts (). The stream connection initiates data transfer of the internal buffer but the virtual device does not fill the internal buffer and eventually nothing is transmitted.

Therefore, in order for a media stream to be transmitted through a virtual transmission path set by a stream connection, not only a stream controller of a stream connection, but also a virtual device connected by the stream connection. The stream controllers of all of them must also be start ().

Application protocol interfaces provided by the stream controllers (hereinafter, referred to as APIs) include stop (), resume (), pause (), and play scale () in addition to start (). It stops, retransmits, fast forwards, and rewinds a media stream, similar to the functions on the control panel of a VCR.

The final goal of the MAHA is to support all kinds of applications for recording, converting, transmitting and playing media streams.

Therefore, video-conferencing and computer-telephony integration, as well as unidirectional recording, playback and transmission of stored media data such as video-on-demand and audio-on-demand (AoD). Applications such as computer-telephony integration and the like to record and play back and transmit live stream data in both directions are also supported by the MAHA.

However, some of the functions of the control panel of the VCR are not applicable in applications that use live stream data, so some of the APIs of the Stream Controller can be applied to live stream data. Will not work.

In addition, the stream controller provides various operations related to one media stream for writing an application program.

Meanwhile, the port serves to deliver media data generated by the virtual device to a stream connection, or the media data accumulated in an internal buffer of the stream connection is stored in the virtual device. Device) to read from.

The port is an object inaccessible to the application program and is defined for the modularity and portability of the virtual device.

Since the port has information on the format of the media data referred to when the stream connection negotiates a connection, the fact that the stream connection negotiates a format means that the virtual connection of both ends is connected. This means that you can find out whether there are any ports that support each other's media data formats among the ports of the virtual device.

In general, the format of the media data that a port has is not a strong type but a week type.

That is, a media data format that a port can support is not a specific value but a range of values or a list of values.

For example, for a port that supports WAVE format, the media data format information that this port has is

Sample Rate: 8000㎐, 11025㎐, 2205㎐

Channel: mono or stereo

Resolution: 8 bit per sample or 16 bit per sample

The main purpose of the port is to provide a uniform interface to a virtual device.

That is, the virtual device uses an interface provided by the port to write the generated media data to the stream connection or to read the media data to be played from the stream connection. ), port_read (), etc.

Although the port is an object included in the virtual device, the port is not related to the virtual device syntactically, and the port is used for the audio virtual device. Ports used for the video and the virtual device are objects instantiated from the same class.

FIG. 6 is a diagram illustrating a process of generating an event in the port of FIG. 3, wherein a port may generate an event in the virtual device.

That is, by sending an event to a virtual device from time to time according to a state of a local buffer of a stream connection or a state of a stream connection, the virtual device may have a specific situation. Provide an opportunity to cope.

FIG. 7 is a diagram illustrating a processing process when an event occurs in FIG. 1. As shown in FIG. 1, a function called process_event () is called by a MAHA event registration server when an event occurs. callback) function that the application developer specifies.

The event occurs, whereby a callback function is called.

First, the application developer creates a callback function to handle the event.

In addition, the application developer registers a callback function, which is called when a specific MAHA object generates a specific event, by using register ().

Then, the event registration server calls register_event () to request the MAHA object that will generate the actual event to call the server's own send_event () member function when the event occurs.

Then, when an event occurs, the MAHA object that generated the event calls send_event (), which is a member function of an event registration server related to the event.

Then, the event registration server calls a callback function registered by the application developer based on the information received as the argument in the send_event ().

Meanwhile, there is a group as a main application provided by MAHA along with a virtual device, a stream connection, a multicast stream connection, and the like.

The group controls virtual media streams and allocates resources by grouping virtual devices.

That is, the group controls the media stream transmitted through the virtual device and the stream connection as a whole, and the media stream is transmitted so that any media stream is transmitted while satisfying the QoS required by the application. Optimally allocate various resources of the computer system for each virtual device and stream connection that passes.

For example, if no group is defined, and audio data is generated from the microphone and transmitted to the speaker, the microphone and speaker may be controlled to control the media stream from the media data source to the sink. A separate control command should be given to the respective Stream Controllers of the corresponding Virtual Devices and the Stream Connections connecting them.

However, if you combine these virtual devices and stream connections into one group, you can totally control them even if you give only one control command to the stream controller defined in the group. Will be.

As a result, the intention of defining a group is to control a single media stream generated from a source device, reaching a sink device, and destroyed.

Here, the media stream is also a unit that must satisfy the QoS desired by the user.

The QoS factors may include GuranteeLevel, Reliable, DelayBounds, BandwidthBounds, JitterBounds, and the like, which are defined in a virtual resource class.

For example, if a user gives DelayBounds 2 seconds to a group, the group allocates 2 seconds to the microphone, 1 second to the Stream Connection, and 0.5 seconds to the speaker. Allocate resources.

As described above, the multimedia middleware in the distributed processing system according to the present invention configures the middleware to systematically and efficiently process and transmit continuous media in a stream format, such as audio and video, which are commonly handled by various types of multimedia applications. By doing so, there is an effect that can provide convenience to the user when writing the application.

Claims (5)

A virtual device that abstracts real hardware into objects for creating a multimedia application program in a distributed processing system; A stream connection for negotiating a connection between the virtual devices for the transmission of media data between the virtual devices; A group for controlling a media stream and allocating resources by grouping the virtual devices; A multicast stream connection for performing multiple connections between the virtual devices; An interface implementation unit for controlling hardware for producing or consuming continuous media data by implementing an interface defined in the application units such as the virtual device, stream connection, group, and multicast stream connection; Multimedia middleware in a distributed processing system comprising an object request repeater for performing communication between the virtual device, stream connection, group, multicast stream connection. The method of claim 1, wherein the virtual device (Virtual Device) A processing element for performing a unique operation of each virtual device; By providing a uniform interface to the processing element, the media data generated in the processing element to pass to the stream connection (Stream Connection) or a plurality of media data accumulated in the stream connection internal buffer from the processing element A port; A stream controller for each port connected to each of the plurality of ports to determine the location information of the media stream or to control the flow of the media stream, and a stream controller for the virtual device. )Wow; A distributed processing system comprising a format connected to each of the plurality of ports, each of which is separated from the processing of the media data and the control of the media stream to abstract specific details related to the format of the media data Middleware in the world. Generating a callback function for handling an event; Registering a callback function to be called using a register function in an event registration server when a specific event generated by a specific MAHA object is received; Requesting a MAHA object to generate an actual event to call a send event member function of an event registration server when an event occurs; Invoking a send event function, which is a member function of an event registration server related to the event, from the object generating the event when the event occurs; Multimedia middleware in a distributed processing system comprising the step of calling a callback (callback) function registered by the application developer based on the information received through the send event function. The method of claim 1, Distributed connection, characterized in that the stream connection is assigned a unicast transport service address pair (TSAP) when the virtual devices are connected in a network through the stream connection. Multimedia middleware in the system. The method of claim 1, When the virtual devices are connected in a network using the multicast stream connection, the multicast stream connection is a multicast address from a multicast address allocation server. Multimedia middleware in a distributed processing system, characterized in that the address is assigned.