JP2002209196A - Multimedia communication system and method for mounting device - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a multimedia communication system with which voice delay time is shortened and the mounting of a multiplexing demultiplexing part can also be simplified, and a method for mounting a device. SOLUTION: In a video conference system, a communication controlling part 11 in which a real time property is not required is mounted on a general purpose OS, the multiplexing demultiplexing part 14 and a voice processing part 12 in which a real time operation has to be guaranteed are mounted on a different resource having a real time property, the processing of the multiplexing demultiplexing part 14 is started after finishing the voice processing part 12, and the processing of the voice processing part 12 is started after finishing the multiplexing demultiplexing part 14.

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to multimedia communication executed by an information processing apparatus such as a personal computer or a portable information terminal operating on various communication lines, for example, communication between applications for realizing videophone communication. The present invention relates to a method for mounting a multimedia communication system and apparatus.

[0002]

2. Description of the Related Art There has been developed a video conference system in which a participant in a remote place has a conference through video and audio shown on a video conference terminal. In a video conference system, it is important to clearly convey the voices and images of all speakers. Protocols for videoconferencing include ITU-T Recommendation H.264. 320
There are standard specifications such as

FIG. 6 is a diagram showing an ITU-T recommendation H.264. FIG. 324 is a diagram for describing an overview of a video conference system indicated by reference numeral 324. In FIG. 6, a communication control unit 11 is a processing unit that performs negotiation with a partner terminal, and exchanges information such as an exchange of a voice / image coding method and a media multiplexing pattern.
Information exchange of the multiplexing pattern is performed by a multiplexing / demultiplexing unit 1 described later.
4 is the information required.

[0004] The audio processing unit 12 is a processing unit for encoding / decoding audio. The image processing unit 13 is a processing unit that encodes / decodes an image. The multiplexing / demultiplexing unit 14 multiplexes data from the communication control unit 11, the audio processing unit 12, and the image processing unit 13 and transmits the multiplexed data to the communication line 15, and decomposes the multiplexed data from the communication line 15 to perform communication control. Unit 11, audio processing unit 12, and image processing unit 1
3 is a processing unit to be handed over.

Next, the problem of the audio delay time when a video conference system or the like is mounted on a general-purpose OS (Operating Systems) that does not guarantee real-time operation will be described.
FIG. 7 is a timing chart when the videophone system is realized by the general-purpose OS. FIG. 7A is a timing chart on the transmission side, and FIG. 7B is a timing chart on the reception side. As shown in FIG. 7A, when data is transferred from the audio processing unit 12 to the demultiplexing unit during transmission,
Also, as shown in FIG. 7B, when the multiplexing / demultiplexing unit receives data from the communication line 15, the multiplexing / demultiplexing unit activates each processing unit, such as passing data to the audio processing unit 12 when multiplexing / decomposing. In the case of OS scheduling, jitter
Occurs. Normally, a buffer for absorbing jitter is provided to compensate for the interruption of audio by the amount of jitter, which causes an increase in audio delay.

FIG. 8 is a diagram showing an example of implementation of a conventional video conference system. As shown in FIG. 8, when the communication control unit 11 and the demultiplexing unit 14 are left in the general-purpose OS and the audio processing unit 12 and the image processing unit 13 are mounted on other resources for the purpose of improving the performance of the general-purpose OS. There is. FIG.
Then, the audio processing unit 12 and the image processing unit 13 are replaced with a general-purpose OS.
Is implemented by DSP (Digital Signal Processor). In this case, the jitter of the audio processing unit 12 is improved, but the jitter due to the audio processing, the scheduling of the OS of the demultiplexing unit, and the jitter from the communication line 15 shown in FIG.

For example, as shown in FIG. 1 of Japanese Patent Application Laid-Open No. 9-317551, a communication control unit 11 and a data I / O unit are provided in a system, and image / audio data transfer is performed from the communication control unit. There is disclosed a technology for generating a periodic signal of the above and directly performing image / audio data transfer by the signal to realize real-time multimedia communication.

[0008]

However, such a conventional multimedia communication system has the following problems. ITU-T Recommendation H. In a system that requires real-time performance, such as a video conference system 324, when a general-purpose OS that does not guarantee real-time operation is used, audio delay time increases due to jitter in OS scheduling.

Further, as shown in FIG. 8, when a multiplexing / demultiplexing unit for exchanging with the communication line 15 is mounted on another resource capable of guaranteeing real-time performance, in order to improve the voice delay, the communication control unit After the negotiation with the partner terminal, the multiplexing / demultiplexing unit needs to pass a large amount of information parameters such as voice, image coding method, media multiplexing pattern, etc., which increases the complexity of implementation. I will.

Although it is possible to mount all the processing units on resources that can guarantee real-time processing, the processing efficiency of a processing unit relatively close to an application, such as a communication control unit, is lower than that of a general-purpose OS. Conceivable. In addition, implementation is difficult due to the limitation of memory resources, and the cost increases.

In the device disclosed in Japanese Patent Application Laid-Open No. 9-321751, when a communication control unit and a data I / O unit are provided in a system to realize real-time processing, an image / video signal is periodically received from the communication control unit. In order to transfer audio data, the implementation must be realized by hardware. Moreover, the transfer of a large number of multiplexing parameters, which must be transferred from the communication terminal management unit to the demultiplexing unit shown in FIG. 1 of the publication, must be controlled from the communication control unit, and the implementation is complicated. Increase.

SUMMARY OF THE INVENTION The present invention has been made in view of the above-mentioned problems, and is directed to a multimedia communication system capable of reducing a voice delay time and realizing simplification of mounting of a demultiplexing unit. An object of the present invention is to provide a method for mounting a device. Another object of the present invention is to provide a multimedia communication system and a mounting method of an apparatus capable of realizing all processing units by software, reducing costs and improving development efficiency. .

[0013]

SUMMARY OF THE INVENTION A multimedia communication system according to the present invention is a multimedia communication system comprising a plurality of communication terminals for communicating audio / video media data, which are interconnected by a communication line.
The communication terminal device includes a communication control unit that performs communication control using a general-purpose OS that does not guarantee real-time operation, and a multiplexing unit that is mounted on a resource capable of real-time operation and performs demultiplexing processing of audio / image media data. It is characterized by comprising a demultiplexing unit and an audio processing unit for encoding / decoding audio. In the multimedia communication system of the present invention configured as described above, the multiplexing / demultiplexing unit is mounted on the same real-time guaranteed resource as the audio processing unit, and the audio delay time can be reduced.

[0014] More preferably, the demultiplexing unit has the same structure as the structure of the demultiplexing unit parameters of the communication control unit and the order and size of the memory area in which the structure is expanded. Configuration, comprising a storage means for storing the parameters of the order and size of the memory area, the communication control unit, by passing a parameter for the demultiplexing unit that the communication control unit has, to the storage means by data transfer,
Diverting the interface part used to transfer the audio / video media data using the contents of the memory area where the structure of the parameter part of the communication control unit is expanded,
By transferring the data from the start address to the parameter memory area of the demultiplexing / demultiplexing unit at once, it is not necessary to set a large number of individual parameters, and the parameters can be easily transferred. For this reason, the demultiplexing unit can be easily mounted on a resource that can operate in real time.

A method for mounting a device according to the present invention is a method for mounting processing units of a plurality of communication terminals for communicating audio / image media data, wherein the audio processing unit performs audio encoding / decoding and the audio / image media. A multiplexing / demultiplexing unit for performing multiplexing / demultiplexing of data is mounted on a resource capable of real-time operation.

The method of mounting the device according to the present invention includes the steps of: performing audio encoding by the audio processing unit started in synchronization with a network synchronization signal; and, after the audio encoding processing by the audio processing unit is completed, the demultiplexing unit. Activating, multiplexing audio / image media data by the activated multiplexing / demultiplexing unit, and transmitting multiplexed data to a communication line; receiving multiplexed data from the communication line; After decomposing the multiplexed data by the demultiplexing unit, activating the audio processing unit;
By sequentially executing the steps of performing the audio decoding process by the activated audio processing unit, it is possible to eliminate the jitter of the OS scheduling and to shorten the audio delay time.

Further, as a specific mode, before realizing the multimedia communication system of the present invention, the system is realized by a general-purpose OS whose mounting can be easily corrected, and operation verification is performed. It is preferable to use a message box having the same configuration as the data format of the parameter transfer when the demultiplexing unit is mounted on another resource so that the demultiplexing unit can be mounted on difficult resources.

[0018]

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS Preferred embodiments of a multimedia communication system and an apparatus mounting method according to the present invention will be described below in detail with reference to the accompanying drawings. The multimedia communication system according to the present embodiment uses the ITU-T
Recommendation H. 324 is an example applied to a video conference system.

FIG. 1 is a diagram showing a configuration of a video conference system according to an embodiment of the present invention. 6 and 8 are denoted by the same reference numerals. In FIG. 1, a video conference system (multimedia communication system) 10
Is a communication control unit 11 that performs communication control using a general-purpose OS that does not guarantee real-time operation, and an audio processing unit 12 that is mounted on a resource such as a DSP that enables real-time operation guarantee and performs audio encoding / decoding. The audio processing unit 12 is mounted on a resource including a multiplexing / demultiplexing unit 14 that performs multiplexing / demultiplexing processing of audio / video media data and a DSP that guarantees real-time operation and is mounted on a resource having real-time operation guarantee. , And an image processing unit 13 that encodes / decodes an image.

That is, in this video conference system, the communication control unit 11 is left in a general-purpose OS that does not guarantee real-time operation, and the multiplexing / demultiplexing unit 14 is mounted on the same resource that guarantees real-time operation. As a general-purpose OS that does not guarantee real-time operation, for example, Windows
There are 95/98, Windows NT and Linux. Although a DSP can be used as a resource for which real-time operation is guaranteed, an OS (for example, μ astron) capable of real-time operation may be used.

Hereinafter, a processing flow of the multimedia communication system configured as described above will be described. FIG. 2 is a timing chart showing a processing flow when the demultiplexing unit 14 is mounted on the same resource as the audio processing unit 12. As shown in FIG. 2, the audio processing unit 12 is activated in synchronization with the network synchronization signal, and performs an audio encoding process (encoding). Thereafter, the demultiplexing unit 14 is started. The multiplexing / demultiplexing unit 14 can be started immediately because of mounting on the same resource. The activated demultiplexing unit 14 acquires audio data / image data, performs multiplexing processing, and performs communication
To send the multiplexed data.

Since the operation is performed in synchronization with the network synchronization, if the line data is received after the transmission, the data must exist. Therefore, after that, the demultiplexing unit 14 receives the data from the communication line 15, decomposes the multiplexed data, and passes the data to each processing unit. After that, the audio processing unit 12 is activated to perform audio decoding processing (decoding). Also in this case, since the audio processing unit 12 is mounted on the same resource, the audio processing unit 12 can be started immediately.

As described above, the data transfer from the voice processing unit 12 to the multiplex / demultiplex unit 14 during transmission described in the conventional example is performed by the mounting method of mounting the multiplex / demultiplex unit 14 on the same resource as the voice processing unit 12. It is possible to eliminate jitter due to OS scheduling at the time. Also, it is possible to eliminate jitter when receiving data from the communication line 15 to the multiplex / demultiplex unit 14 and jitter when passing data from the multiplex / demultiplex unit 14 to the audio processing unit 12.

Further, as described in the above processing flow, the audio encoding processing (encoding), the multiplexing processing, the communication line 1
5, the transmission processing, reception processing, multiplexed data decomposition processing, and audio decoding processing (decoding) can be executed as a series of operations, so that the implementation itself is simplified. This allows
This eliminates the need for a buffer for guaranteeing audio interruption due to jitter, and saves memory resources.

By the way, it is possible to implement a means for mounting all the processing units on resources that can guarantee real-time properties. However, processing units relatively close to an application such as a control unit have a higher development efficiency than a case where they are mounted on a general-purpose OS. A decrease is considered. At present, the development environment for general-purpose OS is DS
Compared to resources such as P that can guarantee real-time performance,
Very rich.

Next, the multiplexing / demultiplexing unit 14 is connected to the communication control unit 11
A description will be given of a large number of parameters for the multiplexing / demultiplexing unit 14, which is a problem when mounting on a different resource. The communication control unit 11 exchanges a multiplexing pattern for decomposing the multiplexed data received from the partner terminal, but it is necessary to notify the multiplexing / demultiplexing unit 14 of the multiplexing pattern. FIG. 3 is a diagram showing a parameter structure for the demultiplexing unit 14 and a memory arrangement to be expanded.

The communication control unit 11 mounted on the general-purpose OS has various parameters in the form of a structure shown in FIG.
This parameter ensures that the size of the variable is a multiple of 16 bits and the alignment of the structure is 16 bits.
The data is expanded in the memory area as shown in FIG. The multiplexing / demultiplexing unit 14 mounted on another resource secures a parameter area based on the order and size of the same parameters as the memory area where the above-described structure is expanded.

The demultiplexing unit 1 of the communication control unit 11
From the beginning to the end of the parameters for 4 the interface used for the transfer of data used by the audio processing unit / image processing unit is diverted to perform simple data transfer (memory copy) and transfer. By adopting such a mounting method of the multiplexing / demultiplexing unit 14, it is possible to solve the complexity of processing by setting parameters individually in the control unit,
Easy parameter transfer.

Next, the demultiplexer 14 and the communication controller 1
The first operation sequence will be described. FIG. 4 is a diagram showing an operation sequence when the present video conference system is implemented. As shown in FIG. 4, the communication control unit 11 exchanges multiplexing patterns after the capability exchange. When a multiplex pattern notification (Multiplex Entry Send) is received from the partner terminal, the multiplex pattern is notified to the multiplex / demultiplex unit 14,
A response (Multiplex Entry Send Ack) is returned to the other terminal.
The multiplexing pattern notification from the own terminal to the other terminal performs the reverse.

Opening a logical channel from the partner terminal (Open L
In the case where the multiplexing has been performed, the logical channel information is set to the multiplexing / demultiplexing unit 14, and then a response (Open Logical Channel Ack) is returned to the partner terminal. The other terminal receives the response and transmits the media data.

By performing such an operation sequence, it is possible to prevent data from being received from a partner terminal before setting a multiplexing pattern in the multiplexing / demultiplexing unit 14. Multiplex Entry Send, Multiplex Entry Send Ac
k, Open Logical Channel, OpenLogical Channel Ack
Is based on ITU-T Recommendation H.264. 324 is a message used for negotiation between terminals. The transfer of the setting of the multiplexing pattern and the like to and from the multiplexing / demultiplexing unit 14 is considered so as not to increase the complexity by diverting the data format used for the audio / image data.

FIG. 5 is a diagram showing a data format when the video conference system is mounted, and shows a format for the demultiplexing unit 14. In FIG. 5, the data format 20 includes an ID part 21, a data length 22, and a data part 23. The type of voice / image / command is set in the ID section 21 and indicates what meaning the data section 23 after the data length 22 has. This ID
The unit 21 determines audio / video data and multiplexing parameters.

In addition, by implementing the entire system on a general-purpose OS using the same message box as the above-mentioned data format in such an operation sequence, first, a video conference is performed on a general-purpose OS which can be easily modified after mounting. After implementing a communication system such as a system and verifying the operation of the algorithm, the multiplexing / demultiplexing unit 14 is assigned to a resource that is relatively difficult to modify after implementation, in order to improve the performance such as the voice delay improvement described above. Enable transplantation. Also, as a great effect, the operation of the entire system can be verified before porting to other resources.

As described above, in the video conference system 10 according to the present embodiment, the communication control unit 11 that does not require real-time operation is mounted on a general-purpose OS, and the multiplexing / demultiplexing unit 14 and the audio processing unit that need to guarantee real-time operation. Unit 12 and another real-time resource, and the audio processing unit 1
After the end of 2, the multiplexing / demultiplexing unit 14 is activated, and after the multiplexing / demultiplexing unit 14 is terminated, the audio processing unit 12 is activated to start processing. Is generated, and there is no need to directly perform image / audio data transfer based on the signal, so that all the processing units can be realized by software. As a result, cost reduction and improvement in development efficiency can be expected, audio delay time can be shortened, and mounting of the multiplexing / demultiplexing unit 14 can be simplified.

Further, in the present embodiment, the memory of the demultiplexing unit which is mounted on another resource in the same structure configuration as the control unit mounted on the general-purpose OS, the order of the memory area where the structure is expanded, and the size. Secure as a parameter area in the area,
The interface part used for the transfer of audio / video media data uses the contents of the memory area of the separated part of the parameter part of the communication control unit implemented in the general-purpose OS, and multiplexes from the head address at once. Since the data is transferred to and passed to the memory area for the parameters of the demultiplexing unit, the complexity of the mounting due to the transfer of a large number of parameters which is a problem for mounting the demultiplexing unit 14 on the same resource as the voice processing unit 12 Can be improved.

The above embodiment is an example in which a system requiring real-time properties is applied to a video conference system. However, any system that is constituted by connecting a plurality of devices to each other via a communication line can be used. Applicable to any system. Further, the types, numbers and connection methods of the units constituting the multimedia communication system, for example, the communication control unit and the demultiplexing unit, and the types of resources are not limited to the above-described embodiments.

The multimedia communication system described above and the method for mounting the same are also realized by a program for causing the communication system and the method for mounting the same to function. This program is stored in a computer-readable recording medium. In the present invention, as the recording medium, the main memory itself may be a program medium, or a program reading device is provided as an external storage device, and the program medium can be read by inserting the recording medium into the program reading device. You may. In any case, the stored program may be configured to be accessed and executed by the CPU, or the program may be read in any case, and the read program may be stored in a program storage area (not shown). And the program may be executed. It is assumed that this download program is stored in the main unit in advance.

Here, the program medium is a recording medium configured to be separable from the main body, and is a tape system such as a magnetic tape or a cassette tape, a floppy (registered trademark).
Magnetic disk and CD-R such as disk and hard disk
Disk system of optical disk such as OM / MO / MD / DVD, card system such as IC card / optical card, or mask ROM, EPROM, EEPROM, flash RO
It may be a medium that carries a fixed program including a semiconductor memory such as an M.

Further, although not shown, when a device capable of connecting to an external communication network is provided, the program is fluidly downloaded from the communication network via the communication connection device. It may be a medium that carries the program. When the program is downloaded from the communication network, the download program may be stored in the main device in advance or may be installed from another recording medium. Note that the content stored in the recording medium is not limited to a program, but may be data.

[0040]

As described above, according to the present invention, the jitter of the scheduling of the general-purpose OS can be reduced, and the voice delay time can be reduced. In addition, since the individual parameters required for passing a large number of parameters of the demultiplexing / demultiplexing unit are increased, the complexity of the implementation is increased, and the implementation on another resource, which has been difficult, can be performed by a simple data transfer (data copy) between memories. Can be realized.

The processing section relatively close to the application, such as the communication control section, is implemented by a general-purpose OS, so that the development efficiency does not change, and the demultiplexing section is mounted on the originally required resources of the audio processing section. Cost increase is small. Therefore, the audio delay time can be reduced, and the implementation of the multiplexing / demultiplexing unit can be simplified, so that the cost can be reduced and the development efficiency can be improved.

[Brief description of the drawings]

FIG. 1 is a diagram showing a configuration of a video conference system according to an embodiment of the present invention.

FIG. 2 is a timing chart showing a processing flow when the demultiplexing unit of the video conference system of the present embodiment is mounted on the same resource as the audio processing unit.

FIG. 3 is a diagram showing a parameter structure for a demultiplexing unit and an expanded memory arrangement of the video conference system according to the embodiment;

FIG. 4 is a diagram showing an operation sequence when the video conference system of the present embodiment is mounted.

FIG. 5 is a diagram showing a data format when the video conference system according to the present embodiment is implemented.

FIG. 6 shows ITU-T Recommendation H. FIG. 324 is a diagram for describing an overview of a video conference system indicated by reference numeral 324.

FIG. 7 is a timing chart when a videophone system is realized by a general-purpose OS.

FIG. 8 is a diagram showing an implementation example of a conventional video conference system.

[Explanation of symbols]

Reference Signs List 10 Video conference system (multimedia communication system) 11 Communication control unit 12 Audio processing unit 13 Image processing unit 14 Demultiplexing unit 15 Communication line 20 Data format 21 ID unit 22 Data length 23 Data unit

Claims (5)

[Claims] 1. A multimedia communication system in which a plurality of communication terminal devices for communicating audio / image media data are interconnected by a communication line, wherein the communication terminal devices communicate using a general-purpose OS. A communication control unit that performs control, and a voice /
A multimedia communication system, comprising: a demultiplexing unit that performs demultiplexing processing of image media data; and an audio processing unit that performs audio encoding / decoding. 2. The multiplexing / demultiplexing unit includes: a structure configuration of a multiplexing / demultiplexing unit parameter included in the communication control unit; a structure configuration same as an order and a size of a memory area in which the structure is expanded; And a storage unit for storing parameters of the order and size of the communication control unit, wherein the communication control unit passes the parameter for the demultiplexing unit included in the communication control unit to the storage unit by data transfer. A multimedia communication system as described. 3. The communication control unit according to claim 2, wherein the communication control unit collectively transfers data for a demultiplexing unit included in the communication control unit from the head address to the storage unit. Multimedia communication system. 4. A method for mounting a processing unit of a plurality of communication terminal devices for communicating audio / image media data, comprising: an audio processing unit for encoding / decoding audio;
A method for mounting an apparatus, comprising: mounting a multiplexing / demultiplexing unit that performs multiplexing / demultiplexing processing of media data of an image on a resource capable of real-time operation. 5. A step of performing audio encoding by the audio processing unit started in synchronization with a network synchronization signal, and a step of starting the multiplexing / demultiplexing unit after the end of the audio encoding processing by the audio processing unit. Transmitting multiplexed data to a communication line by performing multiplexing processing of audio / image media data by the multiplexing / demultiplexing unit; receiving multiplexed data from the communication line; 5. The method according to claim 4, wherein after the decomposition processing of the coded data, the step of activating the audio processing unit and the step of performing the audio decoding process by the activated audio processing unit are sequentially performed.
A mounting method for the described device.