JP2000307637A - Multimedia terminal device and inter-network connecting device - Google Patents

Abstract

PROBLEM TO BE SOLVED: To simultaneously realize the efficiency and simplicity of adaptation processing such as the synchronization management, order control, abort processing, etc., of packets by providing an adaptation information processing means for processing adaptation information and executing synchronization control, abort control, etc. SOLUTION: A multiplexing/demultiplexing circuit 1 multiplexing/ demultiplexing the multiplexing bit stream of packet data received from a network. An adaptation information processing circuit 2 processes adaptation information to execute synchronization control, abort control, etc. A quantity statistic quantity holding circuit 6 holds network quality information obtained from the circuit 2. A media decoding circuit 3 decodes media information of sound and video among encoded data. A media output circuit 4 outputs decoded media information. A control information decoding circuit 5 decodes control information among encoded data. A control circuit 7 controls a terminal device based on the control information and the network quality information.

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

[0001] The present invention relates to the Internet (In)
The present invention relates to a multimedia terminal device connected to a variable delay network such as ternet) and an inter-network connecting device interconnecting various networks, and more particularly to an adaptation processing technology such as packet synchronization management, sequence control, and discard control.

[0002]

2. Description of the Related Art An international standard for real-time transfer of audio and video over a network is ITU-T (Internatio
nal Telecommunication Union Telecommunication
Standardization Sector, ISO / IEC (International
Organization for Standardization, International
al Electro-technical Commission), IETF (Interne
tEngineering Task Force). In particular, data multiplexing means is optimized for each network, and ATM (Asynchronous)
ous Transfer Mode) Used in networks and digital broadcasting networks
ISO / IEC MPEG-2 Systems standard / ITU-T H.222.0 recommendation, ITU-T H.223 recommendation for use in telephone networks and mobile communication networks, ITU-TH.225.0 recommendation for use in local area networks and the Internet, IETF RFC1889 RTP (Real-t
ime Transport Protocol) is known. MPEG-2 Systems and H.222.0 are joint standards of ISO / IEC and ITU-T, and H.225.0 is a recommendation that officially refers to and includes RTP.

FIG. 31 shows a protocol layer commonly used for these. FIG. 31 shows an example in which media information such as audio and video is transferred from terminal A to terminal B. First, the terminal A encodes the input media information, encodes the control information, adds adaptation information according to the respective quality requirements, multiplexes the information, and sends it to the network.

The terminal B receives the multiplexed bit stream from the network, performs demultiplexing, performs error processing, synchronization processing, and the like according to the added adaptation information, decodes and outputs media information, and performs control. The information is used for subsequent terminal control.

[0005] Fig. 32 is a table showing an example of adaptation information specified in major standards. In FIG. 32, for example, when an error occurs in the compressed data of the video information, the resynchronization information allows the receiving terminal to decode from the subsequent resynchronization point instead of discarding all the compressed data of one frame. This is a mechanism to suppress the influence locally. A typical example is a slice structure. Depending on the compression algorithm of the video information, there are cases where it is provided by a compression algorithm and cases where it is separately provided as adaptation information on the system side.

[0006] The sequence number is a mechanism for enabling a receiving terminal to perform appropriate order control and discard detection when, for example, the transfer order of packets is changed in a packet switching network or a packet is discarded due to network congestion. It is.

[0007] The time stamp is additional information for enabling a receiving terminal to perform appropriate synchronous reproduction in, for example, a network where a fixed transfer delay is not guaranteed.

[0008] The error detection / correction code is additional information for performing error detection at the receiving terminal and correcting if possible when a random error or a burst error occurs during transmission through the network. These are mainly added to media information requiring real-time properties. For control information that requires more reliability than real-time performance, an error correction code may be provided by another protocol, or a data retransmission mechanism may be applied.

FIG. 33 shows the protocol layer (ie, the multiplexing, adaptation, media / control layer) shown in FIG. 31 more specifically. ISO / IEC
MPEG-2 Systems standard transport stream (T
When using S: Transport Stream, PES (Packetized Ellipse) is first used from compressed audio or video data.
ementary stream).
This PES is audio data of several tens of milliseconds or video data of one frame. In addition, a time stamp is added to the header part of the PES.

Next, the PES is divided and multiplexed by a fixed-length packet called a 188-byte transport packet. A 4-bit sequence number is assigned to the header of the transport packet.

[0011] Regarding the transfer of control information and general application data, the MPEG-2 TS allows various forms, such as adding an error detection / correction code or repeating the same data in the case of a broadcast form. Or transfer.

[0012] The MPEG-2 TS system is presumed to be used in a high-speed network such as an ATM network or a digital broadcasting network, which generally has little fluctuation in transfer delay.

When the ITU-T H.223 recommendation is used, control information and application data, audio data, and video data are respectively passed to adaptation layers called AL1, AL2, and AL3, and are divided into appropriate sizes. Different types of adaptation information are added to each medium and multiplexed.

The H.223 recommendation has several levels, and the higher the adaptation function, the stronger the adaptation function. For voice, error detection and sequence number can be selected. For video, error detection, sequence number, and retransmission can be selected.

Regarding control information, when error detection, sequence number, and retransmission are performed by itself, LAPM (Link A
Various forms are assumed, for example, when using a higher-level protocol such as “Ccess Procedure for Modems”, or when the application itself performs error correction. The selection of the adaptation information to be used is performed via the control information. The H.223 scheme is generally intended for use in low-speed networks with fixed delays, such as telephone networks and mobile communication networks. Therefore, no time stamp is used.

When the ITU-T H.225.0 recommendation is used, audio and video data are packetized to an appropriate size, and RTP
UDP (User Datagram Protocol) after adding header
Multiplexed via the Internet Protocol (IP) layer.
Control information consists of TCP (Transport Control Protocol) layer, IP
Multiplexed through layers. TCP guarantees reliable data transfer, although the delay increases due to the retransmission mechanism.

On the other hand, UDP is a simplified version of TCP.
In this UDP, delay is reduced because retransmission is not performed, but packet discarding occurs. RTP provides a sequence number as adaptation information and adds a time stamp for synchronous reproduction.

As an aid to RTP, RTCP (Real-time
Transport Control Protocol).
Information such as transfer delay, jitter, and packet loss rate is exchanged between terminals using TCP. The H.225.0 scheme is intended for use in a packet switching network such as the Internet, where transfer delay generally fluctuates.

The ITU-T H.246 recommendation is known as an international standard for mutually exchanging multiplexed audio and video stream data between different networks. I
The TU-T recommends a multimedia terminal for each network called the H series, and H.246 recommends an inter-network connection device for enabling communication between different H series terminals or existing telephone terminals. The purpose is to make.

FIG. 34 shows a general protocol layer used in the network connection device. Here, a case is shown in which media information such as audio and video is transferred from terminal A to terminal B via the network connection device. The network connection device has an interface corresponding to the first network on the terminal A side, and an interface corresponding to the second network on the terminal B side. First, terminal A encodes media information and control information, and performs multiplexing by adding adaptation information suitable for the first network,
Send to the first network. The network connection device is the first
Receives the multiplexed bit stream from the network, performs demultiplexing, processes the added adaptation information, transcodes media information, control information, or both as needed, and Multiplexing is performed by adding adaptation information suitable for the second network, and transmitted to the second network.
The terminal B receives the multiplexed bit stream from the second network, performs demultiplexing, performs error processing, synchronization processing, and the like according to the added adaptation information, and decodes and outputs media information. The information is used for subsequent terminal control.

However, the above-mentioned international standard only defines the syntax for creating a multiplexed stream and the semantics of each field, but does not mention any detailed means for mounting a multimedia terminal device. Absent. In particular, regarding the implementation of adaptation means using a sequence number and a time stamp in an Internet terminal, at present, various systems are inconsistent.

Further, regarding the inter-network connecting device, ITU-T
The H.246 Recommendation is currently in the process of being enacted and is currently under ISDN (In
integrated Digital Services Network)
The interconnection procedure between the H.320 terminal and the H.323 terminal connected to the Internet is only recommended, and the connection procedure between the remaining H series is blank. Further, the conversion procedure of the control information is described only for the connection procedure.

For specific implementation means of adaptation information processing including a sequence number and a time stamp, reference is made to, for example, the description of Japanese Patent Application Laid-Open No. Heisei 4-319832 (Title of Invention: "Cell Information Security System"). In this method, packet discarding is detected for a packet sent via an ATM network by referring to a sequence number added to the packet, and when discarding is detected, dummy data is inserted at a discarding position. It is described that this eliminates the need for a retransmission request to the packet sender and realizes immediacy suitable for audio and video.

Also, for example, in Japanese Patent Application Laid-Open No. 10-84388 (“Packet Data Time Interval Correction Apparatus and Reproducing Method”), a receiving terminal first separates a time stamp from a packet and initializes a counter. A configuration is disclosed in which packet data is written to a memory at the time of matching with the separated time stamp value, time reproduction is performed, and the data is passed to the next processing, which aims at suppressing the memory size. I have.

Japanese Patent Application Laid-Open No. Hei 10-271482 (Title of Invention: "Synchronous playback control method and system for encoded video")
In multimedia data in which audio and video are multiplexed, a time stamp is added to each of the audio and video transfer packets, and the packet receiving terminal adds the time stamp to the video based on the time stamp added to the audio. If the time stamp is early, video data is sequentially decoded, and if the time stamp is late, buffering is performed and the time is adjusted, and then the video data is decoded. No reproduction is possible.

Next, as a configuration of an inter-network connecting device, for example, Japanese Patent Laid-Open No. 10-164143 (title of the invention: "Gateway device and communication system using the same") discloses a fixed delay network such as CATV. In an interconnection device located between variable delay networks such as the Internet, transcoding of encoded video data from a fixed delay network to a variable delay network is performed on the assumption that a bandwidth guarantee protocol is used on the Internet side. A configuration for converting the transfer rate to a transfer rate guaranteed by a bandwidth guarantee protocol is disclosed, which aims at performing stable video signal distribution.

For example, Japanese Patent Laid-Open Publication No. Hei 10-164129 (title of the invention: "communication system") discloses a structure of an inter-network connecting device for connecting a normal telephone network to the Internet. There is disclosed a procedure for receiving a call from a normal telephone terminal, receiving a call to the other telephone terminal via the Internet and a second network connection device, and making a voice call. This method measures the packet transfer delay between two network interconnecting devices and selectively discards the transmitted voice packet or the received voice packet according to the delay time, thereby reducing the overall transfer delay between telephone terminals. It is intended to be.

Further, for example, Japanese Patent Application Laid-Open No. 10-173696 (title of the invention: "Voice gateway based on wide area network") includes a telephone network, an ATM, a frame relay,
In an interworking device that connects networks using different transmission standards such as the Internet, an interworking device that performs a conversion of a call setting protocol and a conversion of an encoding format has been disclosed. It aims to establish a communication session carrying audio and video signals.

[0029]

However, it cannot be said that the techniques described in the above publications realize a sufficient adaptation process especially in a variable delay network such as the Internet.

More specifically, in the method described in Japanese Patent Laid-Open No. Hei 4-319832, a packet containing an error is discarded at a receiving terminal and discarding is detected by a sequence number. It is not assumed that is replaced. Therefore, even if the packet arrives within the timeout period, the packet is discarded if the order is reversed, and it is difficult to apply the packet to a variable delay network such as the Internet.

The system described in Japanese Patent Application Laid-Open No. H10-84388 enables stable synchronous reproduction in accordance with a time stamp. However, there is no mention of order control for controlling the switching of the arrival order of packets. Not. Therefore, it cannot be directly applied to a variable delay network such as the Internet.

The method described in Japanese Patent Laid-Open No. Hei 10-271482 enables stable synchronous reproduction of two media, audio and video, in accordance with a time stamp. Not. For this reason,
It cannot be applied directly to variable delay networks such as the Internet.

The method described in Japanese Patent Laid-Open No. Hei 10-164143 relies on the reliability of a bandwidth guarantee protocol in the Internet. However, in the Internet where high-speed to low-speed and high-reliability to low-reliability mixed with various quality environments according to regions, the above-mentioned bandwidth guarantee protocol does not always function stably. Conversely, in many cases this is not the case at the moment. For this reason, the application range is considerably limited.

The method described in the above-mentioned Japanese Patent Application Laid-Open No. 10-164129 assumes that the receiving terminal is a conventional telephone, and the adaptation processing is performed only by the network connection device. Further, although the network connection device selectively discards packets, it does not mention anything about packet discarding on the Internet or packet order control. Therefore, it cannot be directly applied to multimedia communication including video.

Although the method described in Japanese Patent Application Laid-Open No. 10-173696 describes conversion of an encoding format, the main purpose is conversion of a call setup protocol across various networks, and adaptation is performed. It does not provide any means for improving processing efficiency. For this reason, it does not contribute to efficient synchronous reproduction of media information such as audio and video in the multimedia terminal device and the network connection device.

In addition, Internet telephone terminals, Internet video terminals, and inter-network connecting devices for interconnecting Internet terminals with telephones connected to an existing telephone network or ISDN have already been realized in various places. Packet discard rate will increase due to excessive discard control, circuit will be complicated due to too complicated sequence control,
And have other problems. In order to diversify and increase the scale of video services in the future, and to deploy video services on various existing networks to the Internet, it is necessary to improve the efficiency of adaptation processing in terminal devices and inter-network connection devices. Parallel simplification is essential.

Accordingly, the present invention has been made in view of the above problems, and a main object of the present invention is to provide a multimedia terminal apparatus and an inter-network connection apparatus with respect to packet synchronization management, sequence control, discard processing, and the like. It is an object of the present invention to provide a device that simultaneously realizes efficiency and simplification of adaptation processing. Other objects, advantages and the like of the present invention will be easily clarified in the following description.

[0038]

According to the present invention, there is provided a multimedia terminal apparatus for receiving, decoding, and reproducing encoded data such as audio signals, video signals, and control information transmitted through a network. A multimedia terminal device, wherein adaptation information such as a time stamp and a sequence number is added to the encoded data, and the multimedia terminal device performs multiplexing and demultiplexing on a multiplexed bit stream of the encoded data. Processing means for processing the adaptation information, performing synchronization control, discarding control, etc .; a quality statistic holding means for holding network quality information obtained from the adaptation information processing means; Media decoding that decodes when data is media information such as audio and video Means, means for outputting the decoded media information, control information decoding means for decoding when the encoded data is control information, and holding in the decoded control information and the quality statistic holding means. Control means for controlling the terminal device according to the obtained network quality information. In the present invention, the adaptation information processing means includes a packet receiving means, an adaptation information separating means for separating the received packet into adaptation information and encoded data, an adaptation information holding means for holding the adaptation information, and an offset. Offset delay adding means for providing an attached time, adaptation processing means for performing an adaptation process using the adaptation information and the time with the offset, and output timing generation for providing an output timing using the adaptation information and the time with the offset Means for storing encoded data provided from the adaptation processing means and outputting the encoded data in accordance with the output timing.

In the present invention, preferably, the adaptation processing means selectively discards encoded data using the adaptation information and the time with offset, evaluates network quality, and stores the quality statistic. Time-out determining means for notifying the means, sequence number comparing means for evaluating the network quality using the adaptation information and notifying the quality statistic holding means, and encoding data to the storage means using the adaptation information. Writing means for writing.

In the present invention, preferably, the storage means further stores the adaptation information, information indicating the validity / invalidity of the encoded data, and information indicating the length of the encoded data as management information. A storage unit for holding data; and a read control unit for outputting encoded data according to the output timing and the management information.

[0041] In the multimedia terminal device of the present invention, in the adaptation information processing means in the multimedia terminal device, the offset delay adding means may further include network quality information held in the quality statistic holding means and the media decoding means. To adaptively change the offset delay.

The multimedia terminal device according to the present invention is the multimedia terminal device, further comprising: a unit for inputting media information such as audio and video; a media encoding unit for encoding the media information; Control information encoding means for encoding control information to be output, the media encoding means, and adaptation information adding means for adding new adaptation information to encoded data output from the control information encoding means, Multiplexing means for multiplexing the output of the adaptation information adding means and sending the multiplexed output to the network.

An inter-network connecting apparatus according to the present invention is located between a first network and a second network using different transmission systems, and is provided between a terminal connected to the first network and the second network. An inter-network connection device for interconnecting connected terminals and transferring encoded data such as audio signals, video signals, and control information, wherein said encoded data is added with adaptation information such as a time stamp and a sequence number. The network connection apparatus further includes network connection means for going from the first network to the second network, and the network connection means demultiplexes a multiplexed bit stream of the encoded data. Demultiplexing means, adaptation information processing means for processing the adaptation information and performing synchronization control, discard control, and the like; and the adaptation information processing means. Control information transcoding means for converting or newly generating encoded data of the first control information given from the information processing means, and outputting encoded data of the second control information; Media transcoding means for converting or passing the given encoded data of the first media information according to the control output of the control information transcoding means and outputting the encoded data of the second media information; An adaptation information adding unit for adding new adaptation information, and a multiplexing unit for multiplexing an output of the adaptation information adding unit and transmitting the multiplexed output to a network.

In the present invention, the control information transcoding means includes a control information decoding means for decoding encoded data of the first control information, and a quality statistic holding network quality information obtained from the adaptation information processing means. Control for converting the first control information using the network quality information held in the quality statistic holding unit, generating the new control information as needed, and outputting the second control information Information conversion / generation means;
And control information analyzing means for analyzing and accumulating the second control information, and control information encoding means for encoding the second control information.

[0045]

Next, a preferred embodiment of the present invention will be described. In a preferred embodiment of the multimedia terminal device according to the present invention, referring to FIG. 1, a time stamp and a time stamp are added to encoded data of audio and / or video signals and control information transmitted via a network. Demultiplexing means for receiving, decoding, and reproducing packet data to which adaptation information including at least a sequence number is added, and for demultiplexing a multiplexed bit stream of packet data received from a network. (1) an adaptation information processing means (2) for processing the adaptation information to perform synchronization control and discard control; and a quality statistic holding means for holding network quality information obtained from the adaptation information processing means (2). (6) and media of audio and video in the encoded data. Media decoding means (3) for decoding information; means (4) for outputting the decoded media information; control information decoding means (5) for decoding control information in the encoded data; A control unit (7) for controlling a terminal device based on the control information and the network quality information held in the quality statistic holding unit.

The multimedia terminal device according to the present invention comprises:
In its preferred embodiment, referring to FIG.
The adaptation information processing means (2) includes a receiving means (11) for receiving the packet data output from the demultiplexing means, and an adaptation information separation means for separating the received packet data into adaptation information and encoded data. Means (12), and adaptation information holding means (13) for holding the adaptation information
An offset delay adding means (15) for providing time information with offset based on the output from the clock generating means (14); and an adaptation processing means (16) for executing adaptation processing using the adaptation information and the time information with offset. ), Output timing generating means (17) for generating output timing using the adaptation information and the time with offset,
A storage unit (18) for holding the encoded data provided from the adaptation processing unit and outputting the encoded data according to the output timing.

The multimedia terminal device according to the present invention comprises:
In its preferred embodiment, referring to FIG.
The adaptation processing means (16) selectively discards encoded data using the adaptation information and the time with offset, evaluates network quality, and outputs a quality evaluation result including the number of data discards to the quality statistic. A timeout determination unit (23) for notifying the holding unit (6), and a network quality evaluation by checking a sequence number using the adaptation information, and notifying the quality statistic holding unit (6) of an evaluation result. An order number comparing unit (24) and a writing unit (25) for writing encoded data into the storage unit (18) using the adaptation information are provided.

The multimedia terminal device according to the present invention comprises:
In its preferred embodiment, referring to FIG.
The storage unit (18) stores the adaptation information, information indicating whether the encoded data is valid / invalid, and information indicating the length of the encoded data as management information, and a storage unit (27) that retains the encoded data. And read control means (26) for outputting encoded data according to the output timing and the management information.

The multimedia terminal device according to the present invention comprises:
In a preferred embodiment of the present invention, referring to FIG. 14, the offset delay adding means (75) is provided with the network quality information held in the quality statistic holding means (6) and the input from the media decoding means (3). May be used to adaptively change the offset delay.

Next, an interworking apparatus according to the present invention, in a preferred embodiment thereof, is arranged between a first network and a second network using different transmission systems, as shown in FIG. Interconnecting a terminal connected to the first network and a terminal connected to the second network, and transferring encoded data such as an audio signal, a video signal, and control information, Adapted information including at least a time stamp and a sequence number is added to the encoded data, and the internetwork connecting apparatus includes internetwork connecting means for going from the first network to the second network. Multiplexing / demultiplexing means (201) for multiplexing / demultiplexing a multiplexed bit stream of the encoded data; And an adaptation information processing means (202) for performing synchronization control, discard control, etc., and converting or newly generating encoded data of the first control information given from the adaptation information processing means. Control information transcoding means (203) for outputting encoded data of the second control information; and control information transcoding means (203) for encoding the encoded data of the first media information provided from the adaptation information processing means (202). Media transcoding means (204) for converting or passing according to the control output from the control unit and outputting encoded data of the second media information, control information transcoding means (203) and media transcoding Receiving encoded data output from the means (204) and adding new adaptation information That includes a adaptation information adding means (205), and a multiplexing means (206) for sending to the network by multiplexing the output of the adaptation information adding means.

The adaptation information processing means (20
2) has the same configuration as the adaptation information processing means (2) in the multimedia terminal device.

In a preferred embodiment of the interworking apparatus according to the present invention, referring to FIG. 19, the control information transcoding means comprises control information decoding means for decoding the encoded data of the first control information. (211) and a quality statistic holding unit (2) for holding network quality information obtained from the adaptation information processing unit (202).
12) and converting the first control information using the network quality information held by the quality statistic holding means (212), newly generating the control information as needed, and converting the second control information Output control information conversion generation means (213)
Control information analyzing means (215) for analyzing and accumulating the first control information and the second control information; and control information encoding means (214) for encoding the second control information.
And prepare.

In a preferred embodiment of the interworking apparatus according to the present invention, referring to FIG. 20, the media transcoding means comprises a media decoding means for decoding encoded data of the first media information. (222)
Media encoding means (223) for transcoding the output of the media decoding means, and switching means (221, 221) for controlling the execution of transcoding of the media information according to the control information given from the control information transcoding means (203). 224).

In order to describe the above-described embodiment of the present invention in more detail, various embodiments of the present invention will be described below in detail with reference to the drawings.

[0055]

FIG. 1 is a diagram showing a configuration of a multimedia terminal device according to a first embodiment of the present invention. Referring to FIG. 1, a first embodiment of the present invention includes a multiplexing / demultiplexing circuit 1,
Adaptation information processing circuit 2 and media decoding circuit 3
, A media output circuit 4, a control information decoding circuit 5, a quality statistic holding circuit 6, and a control circuit 7. The outline of these circuits is configured to operate as follows.

The demultiplexing circuit 1 demultiplexes a multiplexed bit stream of coded data such as audio, video signals and control information received from the network. For each of the audio information, video information, and control information,
It is assumed that adaptation information such as a time stamp, a sequence number, an error detection code, and an error correction code is added. In the following, description is made assuming that the encoded data is transferred as a packet.
In such a fixed delay network, each data may be arranged on a time-division slot according to a predetermined multiplexing pattern. Even in this case, the present invention can be applied by replacing the packet with the time division slot.

The adaptation information processing circuit 2
Adaptation processing such as error detection, error correction, synchronization management, and discard control is performed on each of the encoded data such as audio, video, and control information separated by the demultiplexing / demultiplexing circuit 1 in accordance with the added adaptation information. The detailed configuration and operation of the adaptation information processing circuit 2 will be described later.

The media decoding circuit 3 decodes the encoded audio and video data on which the appropriate synchronization management and error control have been performed by the adaptation information processing circuit 2. The coding algorithm is IT for audio signals.
U-TG Recommendation Series, ISO / IEC MPEG for audio signals
In the case of audio standards and video signals, ITU-T H.26x recommendation series and ISO / IEC MPEG video standards are known, and the media decoding circuit 3 performs a decoding operation according to these standards.

The media output circuit 4 is a media decoding circuit 3
Output, reproduce, and display media information such as audio and video provided as an output.

The control information decoding circuit 5 decodes the coded data of the control information to which appropriate error control or the like has been performed by the adaptation information processing circuit 2. As an encoding algorithm, ASN.1 PER used in the ITU-TH.245 recommendation is used.
A method and a method using bit-unit syntax used in the ITU-T H.221 recommendation and MPEG-2 Systems are known, and the control information decoding circuit 5 executes a decoding operation according to these.

The quality statistic holding circuit 6 calculates the quality statistic of the network from information such as the bit error rate, the packet discard rate, the packet transfer delay, and the fluctuation of the packet transfer delay provided by the adaptation information processing circuit 2. Hold.

The control circuit 7 uses the control information given from the control information decoding circuit 5 and the quality statistics of the network stored in the quality statistics holding circuit 6 to execute an appropriate control operation of the terminal.

FIG. 2 is a diagram showing a configuration of the adaptation information processing circuit 2 in the first embodiment of the present invention shown in FIG. Referring to FIG. 2, the adaptation information processing circuit 2 includes a packet reception circuit 11, an adaptation information separation circuit 12, and an adaptation information holding circuit 13
, An offset delay adding circuit 15, an adaptation processing circuit 16, an output timing generating circuit 17, and a storage circuit 18.
And Although not shown in FIG. 1, a clock generation circuit 14 is provided outside the adaptation information processing circuit 2 as shown in FIG.

Referring again to FIG. 2, the received packet includes adaptation information and encoded data.

Each of the above circuits constituting the adaptation information processing circuit 2 has the following functions.

The packet receiving circuit 11 includes a demultiplexing circuit 1
Receives the packet given by.

The adaptation information separating circuit 12 separates a packet supplied from the packet receiving circuit 11 into adaptation information and encoded data. The encoded data is encoded data itself such as audio, video, control information, and the like. In addition, the adaptation information is checked, and if the adaptation processing cannot be performed due to a data error or the like, the packet is discarded, and the discard of the packet is notified to the quality statistic holding circuit 6.

The adaptation information holding circuit 13 accumulates the adaptation information separated by the adaptation information separation circuit 12.

The offset delay adding circuit 15 adds a predetermined offset delay value to the time information supplied from the clock generating circuit 14 and outputs the result. The value of the offset delay is determined by, for example, setting the offset value to a predetermined offset value or setting the value according to the time stamp of the packet that has arrived first.

The adaptation processing circuit 16 performs error detection on the encoded data supplied from the adaptation information separating circuit 12 in accordance with the adaptation information supplied from the adaptation information holding circuit 13 and the time information supplied from the offset delay adding circuit 15. Adaptation processing such as error correction, delay control, and discard control is appropriately performed, and the encoded data is written to the storage circuit 18. Further, a statistic related to network quality obtained as a result of the adaptation processing is stored in a quality statistic holding circuit 6 provided outside the adaptation information processing circuit 2.
Pass to. The detailed configuration and operation of the adaptation processing circuit 16 will be described later.

The output timing generation circuit 17 converts the encoded data from the storage circuit 18 at an appropriate timing in accordance with the time stamp as the adaptation information given from the adaptation information holding circuit 13 and the time information given from the offset delay adding circuit 15. Output, and at the same time, adaptation information processing circuit 2
Is instructed to start a decoding operation to the media decoding circuit 3 or the control information decoding circuit 5 provided outside the device.

When the network is a fixed delay network and a time stamp is not provided as adaptation information, since the network guarantees a fixed delay, the timing information is obtained using only the time information provided from the offset delay adding circuit 15. May occur.

The storage circuit 18 is an adaptation processing circuit
Holding the encoded data given from 16 in a predetermined format,
Also, it outputs the encoded data held at the timing given from the output timing generation circuit 17 and passes the encoded data to an external decoding circuit. The detailed configuration and operation of the storage circuit 18 will be described later.

The clock generation circuit 14 existing outside the adaptation information processing circuit 2 supplies a clock to the offset delay addition circuit 15 and, although not described in the present embodiment, to each circuit of the entire terminal.

Depending on the network environment in which the terminal exists, there is a case where a network clock is supplied from a network like ISDN, and a case where a reference clock is embedded in a bit stream like MPEG-2 Systems and delivered. Each piece of clock information may be used as an input for controlling the output of the clock generation circuit 14.

The decoding circuit existing outside the adaptation information processing circuit 2 may be a media decoding circuit 3 or a control information decoding circuit 5 depending on the encoded data.

FIG. 3 is a diagram showing the configuration of the adaptation processing circuit 16 of one embodiment of the present invention shown in FIG. Referring to FIG. 3, the adaptation processing circuit 16 includes an error detection circuit 21, an error correction circuit 22, a timeout determination circuit 23, a sequence number comparison circuit 24, and a write circuit 25. The functions and operations of these circuits are as follows.

The error detection circuit 21 detects an error in the coded data using the error detection code given from the adaptation information holding circuit 13 and converts the coded data into an error correction circuit.
Pass to 22. Further, the obtained bit error rate is passed to the quality statistic holding circuit 6.

The error correction circuit 22 corrects an error in the coded data using the error correction code provided from the adaptation information holding circuit 13, and discards the coded data if the error cannot be corrected. The obtained packet discard number is passed to the quality statistic holding circuit 6. When there is no error in the encoded data, the error correction circuit 22 does nothing.

The time-out judgment circuit 23 compares the time stamp value given from the adaptation information holding circuit 13 with the time information given from the offset delay adding circuit 15, and if the time stamp value is larger than the time information, the It determines that the arrival time is too late and discards the encoded data. Further, the obtained packet discard number, transfer delay, and transfer delay fluctuation information (jitter) are passed to the quality statistic holding circuit 6.

The sequence number comparing circuit 24 compares the sequence number given from the adaptation information holding circuit 13 with the expected sequence number, and when the sequence number is skipped or reversed, a mechanism described later is used. The information of the number of discarded packets of the quality statistic holding circuit 6 is updated.

The writing circuit 25 writes the encoded data, which has undergone one kind of adaptation processing, to the storage circuit 18 in accordance with the time stamp value and the sequence number given from the adaptation information holding circuit 13. Note that not all of the above adaptation information is provided depending on the transmission method, and in that case, the corresponding circuit is omitted and the adaptation processing is executed.

FIG. 4 is a diagram showing an example of the configuration of the storage circuit 18 shown in FIG. Referring to FIG. 4, the storage circuit 18
Includes a read control circuit 26 and a storage circuit 27.

The read control circuit 26 controls the output of the encoded data stored in the storage circuit 27 according to the output timing given from the output timing generation circuit 17.

The storage circuit 27 is an adaptation processing circuit
It holds encoded data provided from 16 and adaptation information such as a sequence number and a time stamp, outputs encoded data in accordance with an instruction from the read control circuit 26, and passes it to a decoding circuit.

FIG. 5 is a diagram showing an example of encoded data stored and managed by the storage circuit 18 and associated management data. Referring to FIG. 5, the storage circuit 18 has a storage area for each sequence number or a degenerated number thereof, and the storage area further includes a time stamp, a valid flag of a data area, a delimiter flag, It consists of a data length and a data area that holds encoded data. The management information may be held by the read control circuit 26 or by the storage circuit 27.

In the storage circuit 18, a number of storage areas corresponding to the maximum value of the sequence number may be prepared. For example, when the sequence number is represented by 16 bits, 65
536 (= 2 to the 16th power) storage areas must be prepared, which is inefficient. Therefore, for example, if degeneration corresponding to the lower 4 bits of the sequence number is performed,
The number of storage areas can be 16 (= 2 to the fourth power).

The time stamp is effective for identifying encoded data constituting the same frame when information of one frame is divided into a plurality of packets and transmitted, such as a video signal. For example, the initial value of the valid flag of the data area is “0”, and is “1” when the encoded data has correctly arrived. As a result, when the encoded data is discarded, the valid flag remains “0”, and the encoded data of the corresponding sequence number is not passed to the decoding circuit.

When one frame of information is divided into a plurality of packets and transmitted, such as a video signal, the delimiter flag is used to indicate that the corresponding packet is the last information of the frame. This function is already provided, for example, as the M field of IETFRTP, and is used as an option in the present invention. In the case of a short data length such as voice, the data is generally divided into one packet unit. The plurality of effective data lengths are information indicating the length of the received encoded data, that is, the encoded data in packet units. Although there is a transmission method such as MPEG-2TS that imposes fixed-length packets, the length of encoded data generally differs from packet to packet, and is used to indicate this.

The data area is an area for storing encoded data.

FIG. 6 is a flowchart showing an example of the operation of the adaptation information processing circuit 2 according to the first embodiment of the present invention. The operation of the adaptation information processing circuit 2 will be described with reference to FIGS.

At step 31, the adaptation information processing is started.

If a packet has been received in step 32, the process proceeds to step 33, and if not, the process returns to the beginning and waits for reception of the packet. The above is the operation of the packet receiving circuit 11.

In step 33, the adaptation information and the encoded data are separated from the received packet.
Proceed to 34.

In step 34, the adaptation information is checked. If the adaptation processing is not possible, the process proceeds to step 35, and if the adaptation process is possible, the process proceeds to step 36.

In step 35, the packet is discarded, and the process returns to the packet waiting state. The above is the operation of the adaptation information separation circuit 12.

In step 36, an adaptation process is executed, the encoded data is accumulated, and the process returns to a packet reception waiting state. Step 36 is the adaptation processing circuit 16
The operation will be described later in detail.

At step 37, it is checked whether or not the time is the output time.
Proceed to 38 and return to the wait state for the output time if it is not the output time.

In step 38, the coded data is output to the external decoding circuit, and after completion, the process returns to the wait state for the output time.
The above is the operation of the storage circuit 18, and the details of step 38 will be described later.

FIG. 7 is a flowchart showing details of the operation of step 36 (adaptation processing) in FIG.
With reference to FIG. 7 and FIG. 3 showing the configuration of the adaptation processing circuit 16, the adaptation processing will be described below.

In step 41, it is checked whether there is a data error in the encoded data. If there is no data error, the process proceeds to step 45. If there is a data error, the process proceeds to step 45.
Proceed to 42. The above is the operation of the error detection circuit 21.

In step 42, if a data error is detected in step 41, it is checked whether the data error is corrected. If the data error can be corrected, the process proceeds to step 43, and if not, the process proceeds to step 44. move on. In step 43, a data error is corrected, and the process proceeds to step 45. In step 44, the encoded data is discarded, and the process returns to a packet reception waiting state. The above is the operation of the error correction circuit 22.

In step 45, the time stamp value is compared with the offset time. If the packet arrival exceeds the timeout time, the flow proceeds to step 44, and if not, the flow proceeds to step 46. The above is the operation of the timeout determination circuit 23.

In step 46, the actual sequence number is compared with the stored expected sequence number value. If the actual sequence number matches the expected sequence number value, the process proceeds to step 50, where If not, go to step 47.

FIG. 8 is a flowchart showing the details of the operation of step 38 (output processing) in FIG. 6, and shows the control operation of the write circuit 25 in FIG.

In step 51, the storage circuit 18
Gets the valid flag of the storage area indicated by the current pointer,
Proceed to step 52.

In step 52, it is determined whether the data area is valid according to the validity flag, that is, whether the encoded data has arrived correctly or has been discarded. If the data area is invalid, the process proceeds to step 53, where the data area is determined. If it is valid, go to step 54.

In step 53, the pointer indicating the storage area is advanced by one, and the process returns to step 52.

In step 54, the time stamp of the storage area indicated by the current pointer is obtained, and the flow advances to step 55.

In step 55, an output timing generation circuit
The value of the output time provided from 17 is compared with the value of the time stamp. If they do not match, the process proceeds to step 56.

In step 56, if the value of the time stamp is smaller than the value of the output time, the process proceeds to step 53,
If it is larger, go to step 60.

In step 57, the data in the data area is output according to the effective data length of the storage area indicated by the current pointer, that is, the encoded data is output.

In step 58, a delimiter flag of the storage area indicated by the current pointer is obtained, and the flow advances to step 59.

In step 59, it is determined whether or not the coded data in the storage area indicated by the current pointer is a delimiter for decoding. If it is a delimiter, the flow proceeds to step 60; otherwise, the flow proceeds to step 53. Alternatively, if the delimiter flag is not used, steps 58 and 59 do not exist, and the process proceeds from step 57 to step 53.

In step 60, the current pointer value is advanced by one and stored, and the data output process ends.

FIG. 9 shows a series of operations of encoding, packet transmission, network transfer, packet reception, and decoding realized by the first embodiment of the present invention in a time series using a concrete example. is there.

Referring to FIG. 9, at the transmitting terminal, time 10
In 000, 10100, 10200, 10300, 10400, 10500, input, encode, packetize media information,
Sending to the network. In the example shown in FIG. 9, the packet transmitted at time 10200 is discarded in the network.

At the receiving terminal, the packets are received, and the time 10,000, 10100, 1030
At 0 and 10500, the media information is decoded and reproduced, respectively. In the example shown in FIG. 9, the packet to be decoded at time 10400 is discarded in the receiving terminal because the transfer delay is too large.

FIGS. 10 and 11 show in detail a process from packet reception to decoding realized by the first embodiment of the present invention, using a specific example.

In the example shown in FIG. 10, the sequence number 10
The same time stamps 10000 and 10100 are added to the packets to which 0, 101, and 102 are added, and to the packets to which the sequence numbers 103 and 104 are added. However, the packet to which the sequence number 101 is added is discarded for some reason, and the sequence number is skipped.

In the storage circuit 18 shown in FIG. 2, encoded data is stored in a storage area according to the sequence number. Time 1000 on the time axis with offset delay
Decoding is performed at 0 and 10100. However, the packet corresponding to sequence number 101 has been discarded, and time 1
In 0000, only the encoded data of the sequence numbers 100 and 102 are provided to the decoding circuit. However, as described below,
By adding the resynchronization information to the encoded data, the effect of the error can be reduced.

Also in the example shown in FIG. 11, the same time stamps 10000 and 10100 are added to the packets to which sequence numbers 100, 101 and 102 are added, and to the packets to which sequence numbers 103 and 104 are added. .
However, unlike the example shown in FIG. 10, in the example shown in FIG. 11, the packet to which the sequence number 101 is added
Written in the storage circuit. For this reason, at the time of decoding at time 10000, all the encoded data of the sequence numbers 100, 101, and 102 are provided, and decoding is performed without being affected by discard.

FIG. 12 shows an example of a specific method of forming encoded data, particularly in the case of a video signal. FIG.
FIG. 12A shows a form in which resynchronization information is added to the head of compressed data, and FIG. 12B shows a form in which resynchronization information is inserted between compressed information. Regarding resynchronization information, it is an encoding algorithm for ISO / IEC video signals.
When provided by the encoding algorithm itself as a slice structure, such as MPEG-1, MPEG-2, MPEG-4, or a new version of H.263 of ITU-T;
In some cases, such as H.261, or older versions of H.263, are provided outside of the encoding algorithm. In any case, even when a packet is discarded, it is possible to start decoding again from this resynchronization information,
It is possible to suppress the influence of packet discard.

FIG. 13 is a flowchart showing steps in the flowchart of FIG.
The operation of the sequence number comparison circuit 24 indicated by 46, 47, 48, 49
This is shown using a specific example.

In the example shown in FIG. 13, the sequence number is
The packets of 99, 100, 103, 104, 102, and 105 are received in order. Further, it is assumed that the expected value of the sequence number of the next packet is set to 99 and the value of the discard counter indicating the number of discarded packets is set to 5 before receiving the packet having the sequence number of 99. First, when the value of the sequence number of the received packet matches the value of the expected value, one is added to the expected value of the sequence number of the next packet, and the value of the discard counter remains unchanged.

In the example shown in FIG.
If a sequence number skip occurs, as in the case of receiving a packet of the next packet, the expected value of the sequence number of the next packet is set to a value obtained by adding `` 1 '' to the value of the sequence number of the received packet. , A value obtained by subtracting the expected value of the sequence number from the value of the sequence number of the received packet is added to the discard counter.

Further, in the example shown in FIG.
When the sequence number is reversed, such as when the packet of 102 is received, the expected value of the sequence number of the next packet is left as it is, and the discard counter is decremented by one. By such a simple process, the packet discard rate can be efficiently evaluated.

As described above, according to the first embodiment of the present invention, in particular, when multimedia information is multiplexed and transmitted via a variable delay network such as the Internet, packet synchronization management, Efficiency and simplification of adaptation processing such as order control and discard processing are achieved, and in particular, since all processing is performed sequentially, an effective multimedia processing terminal is realized compared to the conventional method Have been.

In the first embodiment of the present invention, a reproduction delay time required from a media input at a transmitting terminal to a media output at a receiving terminal and an offset delay which has a great influence on packet discarding are fixedly assigned. For this reason, for example, even when the network is busy and discarding and delay hardly occur, the reproduction delay is not fixedly changed.

Therefore, the value of the offset delay may be adaptively controlled according to the state of the network. But,
If the value of the offset delay can be changed completely freely, the audio is interrupted or suddenly shortened in media information that is reproduced continuously in time, such as audio, and the subjective sound quality is significantly reduced. There is a problem of being damaged. A second embodiment of the present invention addresses this problem.

FIG. 14 is a diagram showing the configuration of the adaptation information processing circuit 2 according to the second embodiment of the present invention. The configuration of the terminal is the same as that shown in FIG.

Referring to FIG. 14, according to the second embodiment of the present invention, the adaptation information processing circuit 2 includes a packet receiving circuit 71, an adaptation information separating circuit 72, an adaptation information holding circuit 73, and an offset delay adding circuit. 75, an adaptation processing circuit 76, an output timing generation circuit 77, and a storage circuit 78. Further, similarly to the configuration shown in FIG. 2, a clock generation circuit 74 exists outside the adaptation information processing circuit 2.

A packet receiving circuit 71, an adaptation information separating circuit 72, an adaptation information holding circuit 73, an adaptation processing circuit 76, an output timing generating circuit 77,
The operations of the storage circuit 78 and the clock generation circuit 74 are the same as those in the first embodiment of the present invention.

The second embodiment of the present invention is different from the first embodiment in that inputs from the quality statistic circuit 6 and the decoding circuit 3 are added to the offset delay 75.

That is, the offset delay adding circuit 75
In addition to the clock input from the clock generation circuit 74, the network quality statistic given from the quality statistic circuit 6 and the decoding state of the media given from the decoding circuit 3 are input, and the value of the offset delay is adaptively updated. I do.

FIG. 15 is a flowchart showing an example of the operation of the offset delay adding circuit 75.

At step 81, the operation of updating the offset delay is started.

In step 82, an unvoiced section is detected based on, for example, an input from a speech decoding circuit. If the section is not a voiceless section, the process returns to the standby state for the next input.

In step 83, the quality statistic of the network is evaluated. For example, if the delay discard rate has changed significantly, the process proceeds to step 84, while if it has not changed, the process returns to the state of waiting for the next input.

In step 84, for example, if the delay discard rate has decreased, the process proceeds to step 85, while if it has increased, the process proceeds to step 86.

In step 85, the offset delay is reduced, and the process returns to the state of waiting for the next input. In step 86, the offset delay is increased, and the process returns to the state of waiting for the next input.

Depending on the speech encoding algorithm, there is an algorithm for notifying an unvoiced section as syntax, for example, as in ITU-T G.723.1. There is also a method of notifying an unvoiced section outside the encoding algorithm, that is, as a part of adaptation information. In this case, the offset delay can be changed without waiting for an input from the decoding circuit. Alternatively, it is also possible to detect a still area of the video signal and change the offset delay using this section. In the case of video, unlike audio, there is a characteristic that even if the reproduction delay time changes, the user is not so concerned. On the other hand, when the allowable maximum value of the reproduction delay time is determined, control is performed so that the maximum value of the offset delay does not exceed the allowable value. In this case, it is inevitable that a large amount of packets are temporarily discarded and the quality is deteriorated due to the temporary discarding.

As described above, in the second embodiment of the present invention, the reproduction delay time is efficiently reduced while adapting to the network conditions, and the case where the discard rate increases is adaptively dealt with. A multimedia terminal device can be realized.

In the above-described first and second embodiments of the present invention, the multimedia terminal device is dedicated to reception and does not have its own transmission means. Third of the present invention
The embodiment of the present invention has a transmitting means.

FIG. 16 is a diagram showing the configuration of the third embodiment of the present invention. Referring to FIG. 16, a third embodiment of the present invention provides a demultiplexing circuit 91, an adaptation information processing circuit 92, a media decoding circuit 93, a media output circuit
94, a control information decoding circuit 95, and a quality statistic holding circuit 96
, A control circuit 97, a control information encoding circuit 98, a media input circuit 99, a media encoding circuit 100, an adaptation information adding circuit 101, and a multiplexing circuit 102.

Here, the demultiplexing circuit 91, the adaptation information processing circuit 92, the media decoding circuit 93, the media output circuit 94, the control information decoding circuit 95, the quality statistic holding circuit 96
The operations of the multiplexing / demultiplexing circuit 1, the adaptation information processing circuit 2, the media decoding circuit 3, the media output circuit 4, the control information decoding circuit and the demultiplexing circuit 1 shown in the first embodiment or the second embodiment of the present invention described above. 5. The operation is the same as the operation of the quality statistic holding circuit 6.

The control circuit 97 includes the control information supplied from the control information decoding circuit 95 and the network stored in the quality statistic holding circuit 96, as in the above-described first or second embodiment of the present invention. Performs an appropriate control operation of the terminal using the quality statistic, and further outputs control information.

The control information encoding circuit 98 encodes control information output from the control circuit 97. Media input circuit 99
Inputs media information such as audio and video. The media encoding circuit 100 encodes the media information provided from the media input circuit 99. The adaptation information adding circuit 101 adds appropriate adaptation information to the encoded data provided from the control information encoding circuit 99 and the media encoding circuit 100. Multiplexing circuit
102 is obtained from the output of the adaptation information adding circuit 101,
It generates a multiplexed bit stream of coded data such as audio, video signals, control information, and outputs the multiplexed bit stream to a network.

FIG. 17 shows an adaptation information adding circuit.
FIG. 2 is a diagram showing an example of the configuration of 101. Referring to FIG. 17, the adaptation information adding circuit 101 includes a packetizing circuit 111, a sequence number adding circuit 112, a time stamp adding circuit 113, and an error detecting code / error correcting code adding circuit 114.
And

The packetizing circuit 111 divides or integrates the encoded data supplied from the encoding circuit into an appropriate size suitable for the multiplexing means, and forms a packet.

The sequence number adding circuit 112 is mainly applied to a packet of media information, and adds a sequence number to a packet given from the packetizing circuit 111.

The time stamp adding circuit 113 is mainly applied to media information packets, and generally adds the input time of media information.

Error detection code / error correction code adding circuit 114
Adds an error detection code and an error correction code to the composed packet, outputs the packet, and passes the packet to the multiplexing circuit 102. The packet given to the multiplexing circuit 102 may be transferred as it is, or may be packetized again by the multiplexing circuit 102. Also, like MPEG-2TS,
In the case where a reference clock value is added to a packet, FIG.
7, a reference clock adding circuit is further added.

As described above, in the third embodiment of the present invention, it is possible to realize a multimedia terminal device that performs both reception and transmission while maintaining the advantage of the efficient adaptation processing of the present invention. .

However, the advantage of the adaptation processing in the multimedia terminal device according to the present invention described in each of the above embodiments is also effective in an inter-network connecting device interconnecting different networks. Hereinafter, an embodiment of the network connection apparatus according to the present invention will be described.

FIG. 18 is a diagram showing the configuration of the fourth embodiment of the present invention. Referring to FIG. 18, a fourth embodiment of the present invention provides a demultiplexing circuit 201, an adaptation information processing circuit 202, a control information transcoding circuit 203, a media transcoding circuit 204, and an adaptation information adding circuit. 205 and a multiplexing circuit 206. The demultiplexing circuit 201 is connected to the first network, and the multiplexing circuit 206 is connected to the second network.

[0157] The demultiplexing circuit 201 demultiplexes and demultiplexes a multiplexed bit stream of coded data such as audio, video signals and control information received from the first network.

The adaptation information processing circuit 202 performs error detection, error correction, synchronization management, discarding on encoded data such as audio, video, and control information separated by the demultiplexing circuit 201 in accordance with the added adaptation information. Execute adaptation processing such as control.

The control information transcoding circuit 203 receives as input the encoded first control information and the first network quality information provided from the adaptation information processing circuit, and converts the first control information into the second control information. , Encoded and output. In addition, it controls the transcoding operation of the media transcoding circuit 204 according to the held control information. The detailed configuration and operation of this control information transcoding circuit will be described later.

The media transcoding circuit 204 transcodes the encoded first media information supplied from the adaptation information processing circuit into second media information in accordance with the control information supplied from the control information transcoding circuit 203. Output. If there is no prior negotiation between the terminals, or if there is no problem with the transmission band between the first network and the second network, the coded data may be passed without transcoding. Is the control information transcoding circuit
Determined by 203. The detailed configuration and operation of the media transcoding circuit 204 will be described later.

The adaptation information adding circuit 205 adds appropriate adaptation information suitable for the second network to the encoded data supplied from the control information transcoding circuit 203 and the media transcoding circuit 204.

The multiplexing circuit 206 generates, from the output of the adaptation information adding circuit 205, a multiplexed bit stream of coded data such as audio, video signals, control information and the like, and outputs the multiplexed bit stream to the second network.

The adaptation information processing circuit 202
Uses the same one as the adaptation information processing circuit 2 described in the first embodiment or the second embodiment of the present invention. Further, the multiplexing / demultiplexing circuit 201, the adaptation information adding circuit 205, and the multiplexing circuit 206 perform the same operations as the multiplexing / demultiplexing circuit 1, the adaptation information adding circuit 101, and the multiplexing circuit 102, respectively.

FIG. 19 shows a control information transcoding circuit 20.
FIG. 6 is a diagram illustrating an example of the configuration of FIG. Referring to FIG.
The control information transcoding circuit 203 includes a control information decoding circuit 2
11, a quality statistic holding circuit 212, a control information conversion / generation circuit 213, a control information encoding circuit 214, and a control information analysis circuit 215.

The control information decoding circuit 211 decodes and outputs the encoded data of the first control information supplied from the adaptation information processing circuit 202.

The quality statistic holding circuit 212 holds the quality information of the first network given from the adaptation information processing circuit 202.

The control information conversion / generation circuit 213 converts the input first control information into second control information and outputs it. Alternatively, new control information is created as needed. Regarding the conversion of control information, ITU-TH.24
6 Those described in the recommendation shall be followed, and those not described shall be converted according to their own rules. In particular, it is necessary to pay attention to the control information related to the network quality, and the operation is determined with reference to the network quality information held in the quality statistic holding circuit 212. The details will be described later.

The control information coding circuit 214 codes and outputs the second control information.

The control information analysis circuit 215 converts the control information
The first control information provided from the generation circuit 213 and the second control information
This control information is analyzed, accumulated as necessary, and can be referred to in the subsequent control information conversion operation. Also, the compatibility between the encoding capability of the transmitting terminal and the decoding capability of the receiving terminal determined by the control information, or the first network and the second network determined or estimated by the control information.
Of the transmission band of the network as a determination factor, and controls the operation of the media transcoding circuit 204. In general, transcoding of media information causes a large increase in the amount of processing, so it is better to avoid it if it can be avoided, and it is desirable to reduce the processing if it cannot be avoided. For this purpose, the network connection device monitors the encoding / decoding capability of media information exchanged as control information between terminals or between the terminal and the network connection device, and selects a combination that minimizes processing. Alternatively, the terminal may be made to select.

The control information decoding circuit 211, the quality statistic holding circuit 212, and the control information coding circuit 214 are the same as the control information decoding circuit 95, the quality statistic holding circuit 96, and the control information coding circuit 98, respectively. Perform the operation.

FIG. 20 shows a media transcoding circuit.
FIG. 2 is a diagram illustrating an example of the configuration of a 204. Referring to FIG. 20, the media transcoding circuit 204 includes a first switching circuit 221, a media decoding circuit 222, a media encoding circuit 223, and a second switching circuit 224.

Each of the first switching circuit 221 and the second switching circuit 224 transcodes the encoded data of the first media information or passes the encoded data as it is in accordance with an instruction from the control information transcoding circuit 203. To decide.

[0173] The media decoding circuit 222 decodes the encoded data of the first media information when transcoding is selected.

When transcoding is selected, the media encoding circuit 223 re-encodes the decoding result of the media decoding circuit 222 to generate encoded data of the second media information. Regarding the purpose of transcoding, there are cases where the same encoding algorithm is used and only the encoding rate is changed, and cases where algorithm conversion is performed using a different encoding algorithm. FIG.
Although not shown, these operations are determined by the control information transcode 203 and are notified to the media decoding circuit 222 and the media encoding circuit 223.

It is to be noted that the media decoding circuit 222 and the media encoding circuit 223 correspond to the media decoding circuit 93,
The same operation as the media encoding circuit 100 is performed.

As described above, in the fourth embodiment of the present invention, the configuration of the inter-network connecting apparatus which maintains the effectiveness of the adaptation processing described in the first and second embodiments of the present invention is described. It becomes possible.

In the fourth embodiment of the present invention, it is assumed that the encoded data of the media information is only a unidirectional network connection from the first network to the second network.
An embodiment of the bidirectional network connection will be described below.

FIG. 21 is a diagram showing the configuration of the fifth embodiment of the present invention. Referring to FIG. 21, a fifth embodiment of the present invention provides a demultiplexing circuit 301, an adaptation information processing circuit 302, a control information transcoding circuit 303, a media transcoding circuit 304, and an adaptation information adding circuit. 305, multiplexing circuit 306, and demultiplexing circuit
307, an adaptation information processing circuit 308, a control information transcoding circuit 309, a media transcoding circuit 310, an adaptation information adding circuit 311 and a multiplexing circuit 312. The multiplexing / demultiplexing circuit 301 and the multiplexing circuit 312 are connected to the first network by the multiplexing circuit 306.
And the demultiplexing circuit 307 are connected to the second network.

A demultiplexing circuit 30 for realizing a first inter-network connection from the first network to the second network.
1, the adaptation information processing circuit 302,
A transcoding circuit 304, an adaptation information adding circuit 305, a multiplexing circuit 306, and a multiplexing / demultiplexing circuit 307 realizing a second network connection from the second network to the first network; The processing circuit 308, the media transcoding circuit 310, the adaptation information adding circuit 311 and the multiplexing circuit 312
Each of the multiplexing / demultiplexing circuit 201 and the adaptation information processing circuit 202 described in the fourth embodiment of the present invention,
The media transcoding circuit 204, the adaptation information adding circuit 205, and the multiplexing circuit 206 perform the same operation as each. However, information can be exchanged between the control information transcoding circuit 303 used in the first network connection and the control information transcoding circuit 309 used in the second network connection. I have.

FIG. 22 shows a control information transcoding circuit 30.
3 is a diagram illustrating an example of the configuration of a control information transcoding circuit 309. FIG. Referring to FIG. 22, the control information transcoding circuit 303 includes a control information decoding circuit 321, a quality statistic holding circuit 322, a control information conversion / generation circuit 323, a control information encoding circuit 324, and a control information analysis circuit. 325. Similarly, the control information transcoding circuit 309 includes a control information decoding circuit 331, a quality statistic holding circuit 332, a control information conversion / generation circuit 333, a control information encoding circuit 334, and a control information analysis circuit 335. Be composed.

The control information decoding circuit 321 constituting the control information transcoding circuit 303, the quality statistic holding circuit 322,
The control information encoding circuit 324, the operation of the control information analysis circuit 325, and the control information decoding circuit 331 constituting the control information transcoding circuit 309, the quality statistic holding circuit 332,
The operations of the control information coding circuit 334 and the control information analysis circuit 335 are respectively performed by the control information decoding circuit 211, the quality statistic holding circuit 212, and the control information coding circuit 214 shown in the fourth embodiment of the present invention. This is the same as the operation of the control information analysis circuit 215. However, information can be mutually exchanged between the control information conversion / generation circuit 323 and the control information conversion / generation circuit 333.

By allowing information to be exchanged between the control information conversion / generation circuit 323 and the control information conversion / generation circuit 333, several advantages can be obtained. The most notable of these is that the inter-network connection device spoofed two terminals connected to different networks, and the terminal connected to the first network and the terminal connected to the second network This means that communication can be separated and terminated as a different connection for each network. That is, the network connection device terminates communication with a terminal connected to the first network as another terminal connected to the first network, and terminates communication with a terminal connected to the second network. Then, the communication is terminated as another terminal connected to the second network. For media information, adaptation processing such as synchronization control, delay control, discard control, etc. is completed once, and adaptation information suitable for another network is added and transmitted to the partner terminal.

As for the control information, the terminal connected to the first network is returned with the control information as communication on the first network, and the terminal connected to the second network is responded to the control information. The control information as communication on the second network is returned. Then, for control information exchanged between terminals, a predetermined conversion rule is applied, and second control information suitable for each network is created and transferred.

If the control information transferred from the terminal is a message requesting improvement of the quality of the network, the terminal that transferred the control information may be caused by the first network or the second network. May not know what to do. For example, consider a case where the first terminal is connected to the mobile communication network, the second terminal is connected to the Internet, and a failure occurs in the mobile communication network as the first network for some reason. . In this case, media quality degradation occurs in the first terminal, and a quality improvement request message suitable for the mobile communication network is sent to the second terminal.

Next, the network connection device converts the received message into a quality improvement request message suitable for the Internet, and transfers it to the second terminal. The second terminal responds to the request message and performs a quality improvement operation in a form suitable for the Internet, such as reducing the coding rate or enhancing the error correction function in the case of IP connection on the mobile communication network. . However, the obstacles on the mobile communication network are not improved at all, and the above operation only further deteriorates the media quality. In this manner, a situation occurs in which the quality is deteriorated by the repetition of the quality improvement message.

In order to solve the above problem, it is sufficient that the network connection device performs appropriate message conversion in consideration of network quality.

FIG. 23 is a flowchart showing an example of the operation of the control information conversion / generation circuit 323 and the control information conversion / generation circuit 333. In this example, a terminal connected to the first network and a terminal connected to the second network are communicating with each other, and a control information message from the terminal connected to the first network is transmitted to the network connection device. It is assumed that there is an incoming call.

First, in step 341, a control information message is input from the first network, and the flow advances to step 342.

In step 342, it is checked whether or not the control information can be processed according to the normal conversion rule. If the control information can be processed, the process proceeds to step 343. If not, the process proceeds to step 344.

In step 343, normal control information conversion / generation processing is executed in accordance with, for example, the conversion rules described in ITU-TH.246, and the created control information is returned to the first network or 2 to the network.

In step 344, it is checked whether or not the input control information is a request for improving the network quality. If the request is an improvement request, the process proceeds to step 345. If not, the process proceeds to step 348.

In step 345, the cause of the deterioration of the network quality is investigated by referring to the quality statistic holding circuit. If the cause is in the first network, step 346 is performed. If the cause is in the second network, step 347 is performed. Proceed to.

At step 346, control information for response is created and returned to the first network.

In step 347, the control information is transferred to the second network as the quality improvement request control information for the second network.

In step 348, it is checked whether or not the input control information is a response to the request for improving the network quality. If the response is a response, the process proceeds to step 349. If not, the process proceeds to step 350.

At step 349, control information as a rejection message is created and transferred to the second network.
Note that this is an example of a measure to be taken when a quality improvement request is made in advance from a terminal connected to the second network.

At step 350, it is determined that the input control information is an unknown message, and control information for notifying the fact is created and returned to the first network.

FIGS. 24 and 25 are diagrams schematically showing the operation of the network connection apparatus realized by the fifth embodiment of the present invention using a specific example.

FIG. 24 shows a terminal A connected to the first network and a terminal B connected to the second network.
Is communicating via an inter-network connection device located between the first network and the second network, and particularly shows a case where encoded data is transmitted in one direction from terminal A to terminal B. ing.

[0200] Upon receiving the encoded data from the terminal A, the inter-network connecting apparatus executes adaptation information processing, and transmits the resulting quality information such as the delay, bit error rate, and packet discard rate of the first network to itself. And accumulates it in the quality statistics holding circuit.

Terminal B, which has received the encoded data from the network connection device, also performs the adaptation information processing, and obtains the resulting quality information such as the first network delay, bit error rate, and packet discard rate. It accumulates in its own quality statistic holding circuit.

In FIG. 24, it is assumed that communication between terminal A and terminal B is temporarily terminated by the inter-network multiplexer, and terminal B does not know any information on the first network. Therefore, the terminal B grasps and holds only the quality information of the second network. However, a method of transferring dummy data to the terminal B to notify an error is also possible, and the present invention is not limited to the example illustrated in FIG.

FIG. 25 shows the details of the operation of the network connection apparatus when a control information message requesting quality improvement is transmitted from terminal B, corresponding to FIG.

When a quality improvement request is received from terminal B,
The network connection device refers to its own quality statistic holding circuit,
Investigate the first network for any problems. If it is determined that there is a problem in the first network, the network connecting device notifies the terminal A of a quality improvement request message in a form suitable for the first network. Terminal A responds to this and returns a response message to the network connection device.

Upon receiving the response message from terminal A, the network connection device notifies terminal B of a rejection message notifying that the quality improvement operation from terminal B is rejected. When the terminal A rejects, the network connection device also creates a rejection message and notifies the terminal B. On the other hand, if it is determined that there is no problem in the first network, the internetwork connecting device notifies the terminal B of a response message,
And perform a quality improvement operation. Through the above operations, problems on each network are alleviated, and quality improvement is realized. Note that the above operation assumes that all processing is realized only by changing the adaptation information.However, if a change in the encoding level such as a change in the transmission rate is involved, a message to that effect is added. Correspond.

[0206] In the above example, only one-way media transfer is shown, but the present invention is applied to two-way media transfer as it is. Furthermore, when RTP is used on the Internet, since RTCP is used as a pair, it is possible to evaluate the delay and the drop rate using this.

FIG. 26 shows a specific example of ITU-T H.245.
2 shows a control information message used in the communication. The case of H.222, H.223 and H.225 is shown as a multiplexing method. The item described as “common” is a message that does not depend on the network and is used in common by all multiplexing means. Items described as “dependent” and “used” are items used depending on the network. Among these, messages related to the multiplexing mode and the multiplexing reconfiguration affect the network quality, and are the control information of which the present invention focuses.

As described above, in the fifth embodiment of the present invention, a bidirectional network connection is realized, and furthermore, it is possible to notify control information which does not contradict a request for quality improvement of the network. , Stable multimedia communication is realized.

In the fifth embodiment of the present invention, it is assumed that there is only one inter-network connecting device between the communicating terminals. Hereinafter, an example in which a plurality of inter-network connection devices exist between terminals will be described.

FIG. 27 is a diagram showing the configuration of the sixth embodiment of the present invention. Referring to FIG. 27, a sixth embodiment of the present invention relates to a terminal 401 connected to the first network and a third terminal.
A terminal 403 connected to the first network is an inter-network connection device 402 located between the first network and the second network, and an inter-network connection device 403 located between the second network and the third network. The network connection device 402 and the network connection device 403 are provided with means for mutually exchanging information on the quality statistic of the network.

Next, FIG. 28 is a diagram showing details of the operation sequence of the network connection device 402 and the network connection device 403. In FIG. 28, terminal A connected to the first network and terminal B connected to the third network
The communication is performed via a first inter-network connection device located between the first network and the second network, and a second inter-connection device located between the second network and the third network. 5 shows an example of an operation sequence of each interworking apparatus when a control information message requesting quality improvement is transmitted from terminal B. In addition, information exchange of quality statistics is performed in advance between the internetworking devices, and the internetworking devices are connected to the first, second, and third networks, respectively.
Know the communication status of the network.

When receiving a quality improvement request from terminal B,
The second network connection device refers to its own quality statistic holding circuit, and checks whether there is any problem in the second network. When it is determined that there is a problem in the second network, the second network device notifies the first network device of a quality improvement request message in a form suitable for the second network. Upon receiving the quality improvement request from the second network device, the first network device refers to its own quality statistic holding circuit and checks whether there is any problem in the first network. If it is determined that there is a problem in the first network, the first network connecting device notifies the terminal A of a quality improvement request message in a form suitable for the first network. Terminal A responds to this and returns a response message to the first network connection device. Upon receiving the response message from the terminal A, the first internetwork connecting device sends a rejection message to the second internetworking device, indicating that the quality improvement operation from the second internetworking device will be rejected. Notice. When the terminal A rejects the message, the first network device also creates a rejection message and notifies the second network device. Upon receiving the rejection message from the first network connection device, the second network connection device notifies terminal B of a rejection message notifying that the quality improvement operation from terminal B is rejected. Next, when the first network device determines that there is a problem with the second network, the first network device notifies the second network device of a response message, and Perform quality improvement operations. Upon receiving the response message from the first network connection device, the second network connection device notifies terminal B of a rejection message notifying that the quality improvement operation from terminal B is rejected.

Next, if the second network device determines that there is a problem in the third network, the second network device notifies the terminal B of a response message and improves the quality. Perform the action. Through the above operations, problems on each network are alleviated, and quality improvement is realized. Note that the above operation assumes that all processing is realized only by changing the adaptation information.However, if a change in the encoding level such as a change in the transmission rate is involved, a message to that effect is added. Correspond.

In the case of the Internet, the exchange of quality statistics is redundant because it is provided by RTCP, but is usually not provided by other networks.
Also, forcing that function to all terminals is inefficient. Therefore, it is efficient to add the above function only to the network connection device. In addition, specific means for information exchange between the network connection devices is, as in the case of RTCP, in-band on a channel shared with media information, and as in a normal telephone exchange, This may be done out-of-band separately from the media information channel,
In the present invention, this is not mentioned at all.

As described above, in the sixth embodiment of the present invention, even when a plurality of inter-network connecting devices are interposed, it is possible to notify control information that does not contradict a request for quality improvement of a network. Thus, stable multimedia communication is realized.

FIG. 29 is a diagram showing an example of a system configuration to which the multimedia terminal device according to the present invention and the internetwork connecting device according to the present invention are applied. The terminal A and the terminal B, which are the multimedia terminal devices according to the present invention, are provided with a mobile communication network,
Multimedia communication is performed via the Internet and an ATM network, and an inter-network connection device according to the present invention is arranged between a mobile communication network and the Internet, and between the Internet and an ATM network. .

The multimedia terminal device according to the present invention comprises:
Adaptation information is added to encoded data of audio information, video information, and control information, transmitted, and received, decoded, reproduced, and displayed. For the adaptation information processing of the present invention, stable synchronous reproduction and error control are realized.

On the other hand, the inter-network connecting apparatus according to the present invention applies the adaptation information processing of the present invention to encoded data received from a network, realizes stable synchronous reproduction and error control, and adapts to another network. Adaptation information is added and transferred. The control information processing according to the present invention realizes stable inter-network connection.

FIG. 30 is a diagram showing an example of a system configuration to which the multimedia terminal device according to the present invention and the internetwork connecting device according to the present invention are applied. A terminal A and a terminal B, which are multimedia terminal devices according to the present invention, are connected to the Internet,
The multimedia communication is performed via the ISDN / telephone network and the mobile communication network, and between the Internet and the ISDN / telephone network and between the ISDN / telephone network and the mobile communication network. An internetwork connection device is arranged.

As with the configuration shown in FIG. 29, the multimedia terminal apparatus according to the present invention transmits adaptation information to encoded data of audio information, video information, and control information, transmits the encoded data, and receives and transmits it. It performs decryption, playback, and display. In order to perform the above-described adaptation information processing in the multimedia terminal device according to the present invention, stable synchronous reproduction and error control are realized.

On the other hand, the interworking apparatus according to the present invention applies the above-described adaptation information processing in the interworking apparatus according to the present invention to the encoded data received from the network, and realizes stable synchronous reproduction and error control. Then, adaptation information suitable for another network is added and transferred, thereby achieving a stable inter-network connection.

[0222]

As described above, according to the present invention, the following effects can be obtained.

The first effect of the present invention is that, particularly when multimedia information is transmitted via a variable delay network such as the Internet, the efficiency of adaptation processing such as packet synchronization management, order control, and discard processing is improved. That is, an effective multimedia processing terminal can be realized.

The reason for this is that, in the present invention, in particular, there is a contrivance for the interaction between time management and discard control, and contrivance for a means for holding encoded data which simplifies order control.

The second effect of the present invention is to adapt adaptively to a variable delay network such as the Internet, in particular, to reduce the reproduction delay time while adapting to the network conditions and to increase the discard rate. This means that a multimedia terminal device can be realized. The reason for this is due to the ingenuity of flexible delay control utilizing the characteristics of media information and discard control.

A third effect of the present invention is that a multimedia terminal device that performs both reception and transmission more effectively than before can be realized. The reason is that the transmission unit is incorporated in the terminal device while maintaining the first and second effects.

A fourth effect of the present invention is that an inter-network connection device that is more effective than the conventional one can be realized. This is because the first and second effects are incorporated in the network connection device.

[0228] A fifth effect of the present invention is that it is possible to realize an inter-network connection device that performs stable multimedia transfer as compared with the related art. This is because the control information notification mechanism has been devised so as not to contradict the request for quality improvement of the network.

[0229] The sixth effect of the present invention is that, even when a plurality of inter-network connecting devices are interposed, an inter-network connecting device capable of performing more stable multimedia transfer than before can be realized.
That's what it means. This is because network quality information can be exchanged between the network connecting devices.

[Brief description of the drawings]

FIG. 1 is a diagram showing a configuration of a multimedia terminal device according to a first embodiment of the present invention.

FIG. 2 is a diagram illustrating a configuration of an adaptation information processing circuit according to the first embodiment of the present invention.

FIG. 3 is a diagram illustrating a configuration of an adaptation processing circuit according to the first embodiment of the present invention.

FIG. 4 is a diagram illustrating a configuration of a storage circuit according to the first embodiment of the present invention.

FIG. 5 is a diagram showing a data structure recorded in a storage circuit according to the first embodiment of the present invention.

FIG. 6 is a diagram illustrating an operation of an adaptation information processing circuit according to the first embodiment of the present invention.

FIG. 7 is a flowchart showing an operation of an adaptation process in the first embodiment of the present invention.

FIG. 8 is a flowchart illustrating an operation of an output process of an adaptation information processing circuit according to the first embodiment of the present invention.

FIG. 9 is a diagram showing packet discarding in the first embodiment of the multimedia terminal device according to the present invention.

FIG. 10 shows a first example of the multimedia terminal device according to the present invention.
FIG. 9 is a diagram showing an example of data storage in the embodiment.

FIG. 11 is a diagram showing an example of data storage in the multimedia terminal device of the first embodiment of the present invention.

FIG. 12 is a diagram showing resynchronization information.

FIG. 13 shows a first example of the multimedia terminal device according to the present invention.
FIG. 10 is a diagram for explaining handling of a sequence number in the embodiment of FIG.

FIG. 14 shows a second example of the multimedia terminal device according to the present invention.
FIG. 6 is a diagram illustrating a configuration of an adaptation information processing circuit in the example of FIG.

FIG. 15 shows a second example of the multimedia terminal device according to the present invention.
9 is a flowchart showing the operation of the offset delay circuit in the embodiment of FIG.

FIG. 16 shows a third example of the multimedia terminal device according to the present invention.
FIG. 3 is a diagram illustrating a configuration of an example of FIG.

FIG. 17 is a diagram illustrating a configuration of an adaptation information adding circuit according to a third embodiment of the present invention.

FIG. 18 is a diagram showing a configuration of an embodiment of an internetwork connecting apparatus according to the present invention.

FIG. 19 is a diagram showing a configuration of a control information transcoding circuit in one embodiment of the network connection apparatus according to the present invention.

FIG. 20 is a diagram showing a detailed configuration of a media transcoding circuit in an embodiment of the network connection apparatus according to the present invention.

FIG. 21 is a diagram showing the configuration of a second embodiment of the network connection apparatus according to the present invention.

FIG. 22 is a diagram showing a configuration of a control information transcoding circuit in a second embodiment of the network connection apparatus according to the present invention.

FIG. 23 is a flowchart showing the operation of the control information conversion / generation circuit in the second embodiment of the network connection apparatus according to the present invention.

FIG. 24 is a diagram showing a specific example of a second embodiment of the network connection apparatus of the present invention.

FIG. 25 is a flowchart showing the operation of the second embodiment of the network connection apparatus of the present invention.

FIG. 26 is a diagram showing specific control information.

FIG. 27 is a diagram showing the configuration of a third embodiment of the network connection apparatus of the present invention.

FIG. 28 is a flowchart showing the operation of the third embodiment of the network connection apparatus of the present invention.

FIG. 29 is a diagram showing an application example of the present invention.

FIG. 30 is a diagram showing an application example of the present invention.

FIG. 31 is a diagram illustrating a protocol hierarchy of a terminal that performs communication via a network.

FIG. 32 is a diagram illustrating a specific example of adaptation information.

FIG. 33 is a diagram illustrating an example of a protocol layer of a specific multiplexing scheme.

FIG. 34 is a diagram showing a protocol hierarchy of the network connection device.

[Explanation of symbols]

 DESCRIPTION OF SYMBOLS 1 Demultiplexing circuit 2 Adaptation information processing circuit 3 Media decoding circuit 4 Media output circuit 5 Control information decoding circuit 6 Quality statistic holding circuit 7 Control circuit 11, 71 Packet receiving circuit 12, 72 Adaptation information separating circuit 13, 73 Adaptation information Hold circuit 14, 74 Clock generation circuit 15, 75 Offset delay addition circuit 16, 76 Adaptation processing circuit 17, 77 Output timing generation circuit 18, 78 Storage circuit 21 Error detection circuit 22 Error correction circuit 23 Timeout determination circuit 24 Sequence number comparison circuit Reference Signs List 25 writing circuit 26 read control circuit 27 storage circuit 31 start step 32 packet reception branch 33 adaptation information separation 91 demultiplexing circuit 92 adaptation information processing circuit 93 media decoding Path 94 media output circuit 95 control information decoding circuit 96 quality statistic holding circuit 97 control circuit 98 control information encoding circuit 99 media input circuit 100 media encoding circuit 101 adaptation information adding circuit 102 multiplexing circuit 111 packetizing circuit 112 sequence number Addition circuit 113 Time stamp addition circuit 114 Error detection code correction addition circuit 201 Demultiplexing circuit 202 Adaptation information processing circuit 203 Control information transcode circuit 204 Media transcode circuit 205 Adaptation information addition circuit 206 Multiplexing circuit 211 Control information decoding circuit 212 Quality statistics holding circuit 213 Control information conversion / generation circuit 214 Control information coding circuit 215 Control information analysis circuit 221 First switching circuit 222 Media decoding circuit 2 3 Media encoding circuit 224 Second switching circuit 301 Demultiplexing circuit 302 Adaptation information processing circuit 303 Control information transcoding circuit 304 Media transcoding circuit 305 Adaptation information adding circuit 306 Multiplexing circuit 307 Demultiplexing circuit 308 Adaptation information Processing circuit 309 Control information transcoding circuit 310 Media transcoding circuit 311 Adaptation information adding circuit 312 Multiplexing circuit 321 Control information decoding circuit 322 Quality statistics holding circuit 323 Control information conversion / generation circuit 324 Control information encoding circuit 325 Control information Analysis circuit 331 Control information decoding circuit 332 Quality statistic holding circuit 333 Control information conversion / generation circuit 334 Control information encoding circuit 335 Control information analysis circuit

 ──────────────────────────────────────────────────続 き Continued on the front page F term (reference) 5K030 GA01 HA08 HB15 HB16 HB21 HD03 JA01 JT10 KA03 KA19 LA15 LB11 LB15 LC18 LE17 MB01 MB13 5K033 AA02 AA03 BA15 CB06 CB08 CB15 DA05 DB10 DB13 DB16 DB17 DB18 EA06 EA07 EA07 JJ13 5K101 RR05 SS08

Claims (14)

[Claims] 1. Receiving and decoding data which is transmitted via a network and is obtained by adding adaptation information including at least a time stamp and a sequence number to encoded data of audio and / or video signals and control information. And a multimedia terminal device for reproducing the data, comprising: demultiplexing means for demultiplexing a multiplexed bit stream of data sent and received from the network; and receiving the data output from the demultiplexing means, An adaptation information processing unit that processes information and performs synchronization control and discard control; a quality statistic holding unit that holds network quality information obtained by the adaptation information processing unit; and data output from the demultiplexing / demultiplexing unit. Receiving the audio and / or video Media decoding means for decoding media information of an image; means for outputting the decoded media information; control information decoding means for decoding control information of the encoded data; decoded control information and the quality Control means for performing required control of the own terminal device based on the network quality information held in the statistic holding means, and a multimedia terminal device. 2. The adaptation information processing means: receiving means for receiving packet data output from the demultiplexing means; and separating the packet data received by the receiving means into adaptation information and encoded data. Adaptation information separating means, adaptation information holding means for holding the adaptation information, offset delay adding means for giving time information with an offset, and adaptation for executing an adaptation process using the adaptation information and the time information with the offset Processing means; output timing generating means for generating output timing using the adaptation information and the time with offset; storing and holding encoded data output from the adaptation processing means; 2. The multimedia terminal device according to claim 1, further comprising: a storage unit that outputs the encoded data according to a power timing. 3. The adaptation processing means selectively discards encoded data using the adaptation information and the time with offset, evaluates network quality, and evaluates network quality including the number of encoded data discards. Timeout determination means for notifying the evaluation result to the quality statistic holding means, and evaluates network quality by checking a sequence number using the adaptation information,
Sequence number comparing means for notifying the network quality evaluation result to the quality statistic holding means, and writing means for writing the coded data after the adaptation processing to the storage means using the adaptation information. The multimedia terminal device according to claim 2, wherein: 4. A storage means for storing the adaptation information, information indicating validity / invalidity of the coded data, and information indicating the length of the coded data as management information, and storing the coded data. The multimedia terminal device according to claim 2, further comprising: a read control unit that outputs encoded data according to the output timing and the management information. 5. The apparatus according to claim 1, wherein the offset delay adding means adaptively changes the offset delay using the network quality information held in the quality statistic holding means and an input from the media decoding means. The multimedia terminal device according to claim 2. 6. A multimedia terminal device for receiving, decoding, and reproducing encoded data including an audio signal and / or a video signal and control information transmitted via a network, wherein the encoded data includes Is added with adaptation information such as a time stamp and a sequence number.The multimedia terminal device processes the adaptation information by multiplexing / demultiplexing means for multiplexing / demultiplexing a multiplexed bit stream of the encoded data. Adaptation information processing means for performing synchronization control and discard control; quality statistic storage means for storing network quality information obtained from the adaptation information processing means; medium for decoding audio and video media information in the encoded data Decoding means; means for outputting the decrypted media information; Control information decoding means for decoding control information in the encoded data, and control means for controlling a terminal device based on the decoded control information and the network quality information held in the quality statistic holding means, The adaptation information processing means, a packet receiving means, an adaptation information separating means for separating the received packet into adaptation information and coded data, an adaptation information holding means for holding the adaptation information, and a time with offset An offset delay adding means, an adaptation processing means for performing an adaptation process using the adaptation information and the time with an offset, and an output terminal for providing an output timing using the adaptation information and the time with the offset. And a storage unit for holding the encoded data provided from the adaptation processing unit and outputting the encoded data according to the output timing, wherein the adaptation processing unit includes the adaptation information and the offset. Time-out determining means for selectively discarding encoded data using time and evaluating network quality and notifying an external quality statistic holding means; evaluating the network quality using the adaptation information; and Order number comparing means for notifying the statistic holding means, and writing means for writing encoded data to the accumulating means using the adaptation information, further comprising: the accumulating means for storing the adaptation information and the encoded data. Information indicating valid / invalid and encoded data Holding information indicating the length of the data as management information, storage means for holding coded data, and read control means for outputting coded data according to the output timing and the management information, Multimedia terminal device. 7. A means for inputting audio and / or video media information, a media encoding means for encoding the media information, and a control information encoding means for encoding control information output from the control means. An adaptation information adding unit for adding new adaptation information to encoded data output from the media encoding unit and the control information encoding unit; and an output of the adaptation information adding unit is multiplexed and transmitted to a network. 7. The multimedia terminal device according to claim 1, further comprising a multiplexing unit. 8. A terminal provided between a first network and a second network that use different transmission schemes and connected to the first network and a terminal connected to the second network. An interconnecting apparatus for interconnecting and transferring encoded data such as audio signals, video signals, and control information, wherein the encoded data is added with adaptation information including at least a time stamp and a sequence number. Wherein the network connection device comprises network connection means for going from the first network to the second network, wherein the network connection means demultiplexes the multiplexed bit stream of the encoded data. A demultiplexing unit; an adaptation information processing unit that processes the adaptation information and performs synchronization control, discard control, and the like; The given from flop station information processing unit 1
Control information transcoding means for converting or newly generating the encoded data of the control information, and outputting encoded data of the second control information;
Media transcoding means for converting or passing encoded data of the media information according to the control output of the control information transcoding means, and outputting encoded data of the second media information; Adaptation information adding means for receiving encoded data output from the information transcoding means and media transcoding means and adding new adaptation information; and multiplexing means for multiplexing the output of the adaptation information adding means and sending the multiplexed output to a network. A network connection device, comprising: 9. An adaptation information processing unit, comprising: a receiving unit that receives packet data output from the demultiplexing unit; and an adaptation information separation unit that separates the received packet data into adaptation information and encoded data. Means, an adaptation information holding means for holding the adaptation information, an offset delay adding means for giving time information with offset, an adaptation processing means for performing an adaptation process using the adaptation information and the time information with offset, Output timing generation means for generating an output timing using the adaptation information and the time with offset, and holding encoded data provided from the adaptation processing means, according to the output timing The network connection device according to claim 8, further comprising: a storage unit that outputs the encoded data. 10. The adaptation processing means selectively discards encoded data using the adaptation information and the time with offset and evaluates network quality, and evaluates network quality including the number of encoded data discards. Timeout determination means for notifying the result to the quality statistic holding means, Evaluation of network quality by checking a sequence number using the adaptation information,
Sequence number comparing means for notifying the network quality evaluation result to the quality statistic holding means, and writing means for writing the coded data after the adaptation processing to the storage means using the adaptation information. 10. The network connection device according to claim 9, wherein: 11. The control information transcoding means, control information decoding means for decoding encoded data of the first control information, and quality statistic holding for holding network quality information obtained from the adaptation information processing means Means for converting the first control information using the network quality information held in the quality statistic holding means, newly generating the control information as necessary, and outputting the second control information Conversion generating means, control information analyzing means for analyzing and accumulating the first control information and the second control information, and control information encoding means for encoding the second control information. 9. The network connection device according to claim 8, wherein: 12. The media transcoding means,
Media decoding means for decoding encoded data of the first media information; media encoding means for transcoding the output of the media decoding means; transcoding of the media information according to control information given from the control information transcoding means. 9. The network connection device according to claim 8, further comprising: switching means for controlling execution of the code. 13. A method according to claim 1, wherein said first and second networks use different transmission schemes.
An inter-network connection device interconnecting a terminal connected to the network and a terminal connected to the second network, and transferring encoded data including an audio signal and / or a video signal and control information, Adaptation information including at least a time stamp and a sequence number is added to the encoded data. The inter-network connecting device includes a first inter-network connecting unit that goes from the first network to the second network. And a second inter-network connection unit that goes from the second network to the first network. At least one of the first inter-network connection unit and the second inter-network connection unit includes the code Multiplexing / demultiplexing means for multiplexing / demultiplexing a multiplexed bit stream of coded data, processing the adaptation information, And Cormorant adaptation information processing means, first given from the adaptation information processor 1
Control information transcoding means for converting or newly generating the encoded data of the control information, and outputting encoded data of the second control information;
Media transcoding means for converting or passing encoded data of the media information according to the control output of the control information transcoding means, and outputting encoded data of the second media information; Adaptation information adding means for receiving encoded data output from the information transcoding means and the media transcoding means and adding new adaptation information; and multiplexing means for multiplexing the output of the adaptation information adding means and sending it to the network. And the control information conversion and generation means included in the first network connection means and the second network connection means are provided with means for exchanging information with each other, and the control information received from the terminal. Is the information relating to the network quality, the first network and Using the network quality information of the second network to convert the control information, it performs a new product if necessary, and outputs the second control information, inter-network connection device, characterized in that. 14. The inter-network connecting device according to claim 8, wherein the inter-network connecting device is provided between networks using different transmission schemes, and is connected to a different network. Interconnecting and transferring the audio signal, the video signal, and the encoded data of the control information, and when there are a plurality of the internetwork connection devices between the terminals, furthermore, mutually between the internetwork connection devices An inter-network connection device comprising means for exchanging information.