JP2003179664A - Method and unit for processing data - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To dynamically change transmission format information. <P>SOLUTION: This unit for processing data is provided with a reception management section 11 that receives information including data and its transmission format information from a storage device and a communication channel, a separation section 12 that analyzes the received information and separates it, a transmission section 13 that sends information to the storage device and the communications channels, and an image expansion section 14 that expands image data. An image expansion management section 15 manages a processing state of the image expansion section 14 that expands at least one image or over. Then, an image composite device consisting of an image composite section 16 that composites images based on the expanded information, of an output section 17 that outputs the result of composite, and of a terminal control section 18 that controls and manages each means composites a plurality of images simultaneously and copes with dynamic changes in transmission format information. <P>COPYRIGHT: (C)2003,JPO

Description

Detailed Description of the Invention

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a data processing method and a data processing device in the fields of communication and broadcasting.

[0002]

2. Description of the Related Art Conventionally, for example, a person image is extracted from an image of a landscape of a space where oneself is present, and the image and the person image sent from the other party are commonly used together with the other party stored in advance. By superimposing and displaying the image in the virtual space to be displayed, it is possible to satisfy the presence of the other person in front of the person and aim for a realistic image communication (Japanese Patent Publication No. 4-24914). Gazette).

In particular, in the prior art, an invention relating to a method for speeding up image synthesis and reducing memory has been made (for example, Japanese Examined Patent Publication No. 5-46592: Image synthesizer).

[0004]

In such a conventional technique, there has been proposed a communication system utilizing image synthesis for synthesizing two-dimensional still images and three-dimensional CG data, but a plurality of moving images and voices have been proposed. There was no specific discussion from the following viewpoints on how to implement a system that simultaneously synthesizes and displays.

That is, (A1) data and control information (data is transmitted in a packet different from data) by using a plurality of logical transmission lines constructed by software on one or more actual transmission lines. Information (information for controlling processing on the terminal side) which is transmitted independently, and transmission of images and voice (communication and broadcasting) under the environment, and its control method, (A2) images to be transmitted, A method of dynamically changing header information (corresponding to the data management information of the present invention) added to voice data, (A
3) A method of dynamically changing header information (corresponding to the transmission management information of the present invention) added for transmission, (A4) Dynamic multiplexing and separation of a plurality of logical transmission paths, and transmission of information Method, (A5) program and data reading, image and audio transmission method considering startup time, and (A6)
There was a problem that no specific discussion was made from the viewpoint of image and audio transmission methods that consider zapping.

On the other hand, conventionally, as a method of dynamically adjusting the transmission amount to the network, a method of changing the encoding method or a method of discarding data in frame units according to the frame type of video has been proposed. (Hiroshi Jinzenji, Tetsuo Tajiri, A Study on Distributed Adaptive VOD System, D-81, IEICE System Society (1995)).

As a method of adjusting the processing amount on the encoder side, a dynamic calculation amount scalable algorithm capable of providing a high-quality image under the processing time constraint has been proposed (Fuminori Ohsako, Yoshiyuki Yajima, Hiroshi Kodera, Yu Watanabe, Shimamura
Kazunori: Software Image Coding with Dynamic Complexity Scalable Algorithm, IEICE Transactions D-2,
Vol.80-D-2, No.2, pp.444-458 (1997).).

As an example of realizing the synchronized reproduction of the moving image and the sound, there is an MPEG1 / MPEG2 system.

In such a conventional technique, (B1) in the conventional method of discarding video according to the frame type of the video, since the granularity of information that can be handled is within a single stream, a plurality of video streams Or handling of multiple audio streams and reflecting the editor's intention,
There was a problem that it was difficult to focus important scene cuts along with audio and perform synchronized playback.
(B2) In addition, since MPEG1 / MPEG2 is premised on realization by hardware, it is premised that the decoder can decode all the given bitstreams. Therefore, there is a problem that the method of dealing with the case where the processing capacity of the decoder is exceeded is undefined.

On the other hand, in the conventional transmission of moving images,
H. 261 (ITU-T Recommendation
n H. 261-Video codec for a
audiovisual services at px
64) and the like, which have been implemented by hardware. For this reason, when the hardware design was performed, the upper limit of the required performance was taken into consideration, so that the case where the decoding process could not be completed within the specified time did not occur.

Here, the designated time is the time required to transmit a bit stream obtained by encoding one image. If the decoding cannot be performed within this time, the excess time becomes a delay, and if it is accumulated and becomes large, the delay from the transmitting side to the receiving side becomes large and it becomes unsuitable for use as a videophone. This situation should be avoided.

Further, there is a problem that the moving image cannot be transmitted when the decoding process cannot be completed within the designated time because the communication partner generates a non-standard bit stream.

The above problem is a problem occurring not only in moving images but also in audio data.

However, in recent years, as a result of the development of the network environment for personal computers (PCs) in the form of the spread of the Internet and ISDN, the transmission speed has increased, and moving images can be transmitted using the PC and the network. It was The demand for moving image transmission from users is also rapidly increasing. Further, the improvement of the CPU performance has made it possible to sufficiently decode a moving image by software.

However, in a personal computer, the same software can be executed by computers having different device configurations such as a CPU, a bus width, and the presence or absence of an accelerator, so it is difficult to consider the upper limit of the required performance in advance. There are cases where the image cannot be decoded within the specified time.

Further, when the encoded data of a moving image having a length exceeding the processing capability of the receiving device is transmitted, the encoding within the designated time becomes impossible.

Problem (C1): The image is decoded within a designated time, and the delay is kept small.

Further, as a means for solving this problem C1, for example, when a moving image is input as waveform data, a part of the transmitted bit stream is not used, so that the actual use efficiency of the transmission path is poor. In some cases, the problem of. Also, depending on the encoding method, there is one that generates the decoded image of this time based on the decoded image of the previous time (P picture, etc.), but since the decoded image of the previous time may not be completely restored, the image quality may deteriorate. However, there is also the problem that it will spread out over time.

Problem (C2): The actual use efficiency of the transmission line is poor. In addition, the deterioration of image quality spreads.

In software implementation, since the frame rate of an image is determined by the time required for one encoding process, if the frame rate specified by the user exceeds the processing limit of the computer, the specification must be met. I couldn't.

Problem (C3): If the frame rate designated by the user exceeds the processing limit of the computer, the designation cannot be met.

The present invention relates to the above-mentioned first prior art (A1).
It is an object of the present invention to provide a data processing method and a data processing device that solve at least any one of these problems (A6).

[0023]

According to a first aspect of the present invention, (1) the actual transfer rate of image or audio information exceeds a target transfer rate, or (2) the elapsed time from the start of transfer processing. And when it is determined that the writing of the encoded information to the transmission buffer is delayed as a result of comparing the time added to the encoded information with the time to be decoded or output. , A data processing method in which the transmission of the information is thinned by using the priority added to the encoded information.

According to a second aspect of the present invention, (1) when the actual transfer rate of image or audio information exceeds the target transfer rate, or (2) the elapsed time from the start of transfer processing,
When it is determined that the writing of the encoded information to the transmission buffer is delayed as a result of comparison with the time to be decoded or output, which is added to the encoded information, the A data processing device that thins out the transmission of the information by using the priority added to the coded information.

According to a third aspect of the present invention, the divided time-series data and the priority information thereof are paired and sequentially input, and (1) the information of the divided time-series data is lost. Performs resend request processing to request resending of the damaged data, and (2) when the divided time series data is lost continuously or at high frequency,
This is a data processing method in which the retransmission request processing is performed only on data having a high priority.

According to a fourth aspect of the present invention, the divided time-series data and the priority information thereof are sequentially input as a pair, and (1) when the information of the divided time-series data is lost. Performs resend request processing to request resending of the damaged data, and (2) when the divided time series data is lost continuously or at high frequency,
This is a data processing device that performs the retransmission request process only for data having a high priority.

According to a fifth aspect of the present invention, the divided time-series data and the priority information thereof are paired and sequentially output, and the priority is determined according to the transmission amount of the divided time-series data. This is a data processing method for preferentially transmitting data having a high data rate.

According to a sixth aspect of the present invention, the time-series data that has been divided and the priority information thereof are paired and sequentially output, and the priority is determined according to the transmission amount of the time-series data that has been divided. It is a data processing device that preferentially transmits high data.

[0029]

BEST MODE FOR CARRYING OUT THE INVENTION Embodiments of the present invention will be described below with reference to the drawings.

The embodiment described here mainly solves any of the problems (A1) to (A6) described above.

The "image" used in the present invention includes both a still image and a moving image. Further, the target image may be three-dimensional image data such as a two-dimensional image such as computer graphics (CG) and a wire frame model.

FIG. 1 is a schematic configuration diagram of an image / voice transmitting / receiving apparatus according to an embodiment of the present invention.

In the figure, a reception management unit 11 for receiving information and a transmission unit 13 for transmitting information are coaxial cables, C
It is a means for transmitting information such as ATV, LAN, and modem. The communication environment may be a communication environment such as the Internet in which a plurality of logical transmission lines can be used without being aware of the multiplexing means, or an analog telephone or satellite broadcasting. It may be a communication environment that must be done.

As the connection form of the terminals, bidirectional video and audio transmission / reception between the terminals such as a TV telephone and a TV conference system, satellite broadcasting, CATV, broadcasting type video on the Internet, The form of audio broadcasting may be mentioned. The present invention considers the connection form of such a terminal.

The separating unit 12 shown in FIG. 1 is means for analyzing received information and separating data from control information. Specifically, it is a means for disassembling the transmission header information and data added to the data for transmission, or disassembling the data control header and data contents added to the data itself. . The image expansion unit 14 is a unit that expands the received image. For example, H.264. 261, H.264. 263, MPE
It may or may not be a standardized moving image such as G1 / 2 or JPEG or a compressed image of a still image.

The image expansion management section 15 shown in FIG. 1 is means for monitoring the expansion status of an image. For example, by monitoring the image decompression state, when the receiving buffer is about to overflow, when the image is decompressed, the receiving buffer is idle-read and the image is decompressed. From, it is possible to resume the expansion of the image.

Further, in the figure, the image synthesizing section 16 is means for synthesizing the expanded images. As for the synthesizing method, in a scripting language such as JAVA (registered trademark), VRML, or MHEG, the structure information of the images (the display position and the display time (the display period may be included)), the method of grouping the images, and the image The image synthesizing method can be defined by describing the relationship between these attributes and the display layer (depth), the object ID (SSRC described later), and the like. The script describing the composition method is input / output from / to a network or a local storage device.

The output unit 17 is a display, a printer or the like that outputs the image synthesis result. The terminal control unit 18 is means for controlling each of these units. It should be noted that even a configuration in which voice is expanded instead of an image (this can be achieved by changing the image expansion unit to the audio expansion unit, the image expansion management unit to the audio expansion management unit, and the image synthesis unit to the voice synthesis unit. ),
It is also possible to expand both the image and the sound and synthesize and display them while keeping the synchronization in time.

Further, by providing an image compression unit for compressing an image, an image compression management unit for managing the image compression unit, an audio compression unit for compressing audio, and an audio compression management unit for managing the audio compression unit, an image and an audio can be obtained. Can also be transmitted.

FIG. 2 is a diagram showing the reception management unit 11 and the separation unit 12.

A data receiving unit 101 for receiving data, a control information receiving unit 102 for receiving control information for controlling data, and a separating unit 12 are shown in the reception management unit 11 shown in FIG.
A transmission format storage unit 103 that stores a transmission structure (details will be described later) for interpreting the transmission content.
By configuring each unit with the transmission information interpretation unit 104 that interprets the transmission content based on the transmission structure stored in the transmission format storage unit 103, it becomes possible to receive data and control information independently, For example, it becomes easy to delete or move a received image or voice while receiving.

As described above, the communication environment targeted by the reception management unit 11 is a communication environment (Internet profile) in which a plurality of logical transmission lines can be used without being conscious of the multiplexing means like the Internet. Or a communication environment (raw profile) such as analog telephone or satellite broadcasting in which the multiplexing means must be taken into consideration. However, from the user's point of view, it is premised on a communication environment in which a plurality of logical transmission paths (logical channels) are prepared (for example, in a communication environment in which TCP / IP can be used, an expression called “communication port” is generally used). used).

Further, as shown in FIG. 2, the reception management unit 11
It is assumed that the information received by the device includes one or more types of data transmission lines and one or more types of control logical transmission lines for controlling the data to be transmitted. A plurality of transmission lines for data transmission may be prepared, and only one transmission line for data control may be prepared. In addition, H. R also used in 323
A transmission line for data control may be prepared for each data transmission like TP / RTCP. Further, in consideration of broadcasting using UDP, a communication mode using a single communication port (multicast address) may be used.

FIG. 3 is a diagram for explaining a method of transmitting and controlling images and sounds by using a plurality of logical transmission lines. The data itself to be transmitted is called an ES (elementary stream), and the ES may be image information for one frame, or image information for GOB units or macroblock units smaller than one frame.

The voice may have a fixed length determined by the user. The header information for data control added to the data to be transmitted is called AL (adaption layer information). Examples of the AL information include information indicating whether or not the data processing start position is possible, information indicating the reproduction time of the data, information indicating the priority of the data processing, and the like. The data management information of the present invention corresponds to AL information. The ES and AL used in the present invention are MPE.
It does not necessarily have to match the contents defined in G1 / 2.

As the information indicating whether or not the start position of the data can be processed, specifically, there are two types of information. One is a flag for random access, which is information indicating that an image can be independently read and reproduced regardless of preceding and following data such as an intra frame (I picture) for an image. Secondly,
An access flag can be defined as a flag that simply indicates that reading is possible. For example, in the case of an image, it is information indicating that it is the head of an image in GOB units or macroblock units. Therefore, if there is no access flag, it is in the middle of data. Both the random access flag and the access flag are not necessarily required as information indicating whether or not the data processing start position is possible.

In real-time communication such as a TV conference system, there may be no problem even if both flags are not added, and a random access flag is necessary to enable easy editing. Whether a flag is necessary or, if necessary, which flag is necessary may be determined before data transfer via the communication path.

The information indicating the reproduction time of the data indicates the information of the time synchronization when the image and the sound are reproduced.
In / 2, it is called PTS (Presentation Time Stamp). In real-time communication such as a TV conference system, time synchronization is usually not taken into consideration, and thus information that means a reproduction time is not necessarily required. The necessary information may be the time intervals of encoded frames.

By adjusting the time interval on the receiving side, a large fluctuation of the frame interval can be prevented, but there is a possibility of delay by adjusting the reproduction interval. Therefore, it may be determined that the time information indicating the encoding frame interval is not necessary.

Before the data transfer through the communication path, it is determined whether the information indicating the data reproduction time means PTS or frame interval, and whether the data reproduction time is not added to the data itself. The data may be transmitted together with the determined data management information by notifying the receiving terminal.

The information indicating the priority of data processing is such that when the processing or transmission cannot be performed due to the load of the receiving terminal or the load of the network, the processing of the data is stopped or the transmission is stopped to reduce the load of the receiving terminal or the network. The load can be reduced.

In the receiving terminal, the image expansion management unit 15 can be used, and in the network, the relay terminal or router can be used. A numerical expression or a flag may be used as the priority expression method. By transmitting the offset value of the information indicating the priority of data processing as control information or data management information (AL information) together with the data, it is possible to prevent sudden changes in the load of the receiving terminal or the load of the network. By adding an offset value to the priorities assigned to images and sounds in advance, it becomes possible to dynamically set priorities according to the operating status of the system.

Further, information for identifying the presence / absence of scramble, the presence / absence of copyright, and the original or the copy are transmitted as control information together with the data identifier (SSRC) separately from the data, so that the scrambling at the relay node is performed. It will be easier to cancel.

The information indicating the priority of data processing may be added in a stream unit composed of a set of a plurality of video or audio frames, or in a video or audio frame unit. Good.

H. 263 and G.H. A priority adding unit that determines the priority of the processing of the encoded information when the information is overloaded by a predetermined criterion by using an encoding method such as 723, and associates the encoded information with the determined priority. To the transmitting terminal device (see FIG. 54).

FIG. 54 is a view for explaining the priority adding means 5201 for adding priority to video and audio.

That is, as shown in the figure, the encoded video and audio data (each processed by the video encoding means 5202 and the audio encoding means 5203) is based on a predetermined rule. Add priority. The rules for adding the priority are stored in the priority addition rule 5204. The rule is that an I frame (intra-frame coded video frame) has a higher priority than a P frame (inter-frame coded video frame), or that a video has a lower priority than audio. The rule is to add. Further, this rule may be dynamically changed according to a user's instruction.

Objects to which the priority is added are, for example, a scene change for an image, an image frame or stream designated by an editor or a user, and a voiced section and a silent section for a voice.

The priority adding method for defining the priority of the processing at the time of overload is the method of adding the priority to the image or audio frame unit, and the method of adding to the communication header and the header of the encoded bit stream of video or audio at the time of encoding. A method of embedding in The former can obtain information about the priority without decoding, and the latter can be handled independently by the bitstream alone without depending on the system.

When priority information is added to the communication header,
When one image frame (for example, intraframe-encoded I frame, interframe-encoded P, B frame) is divided into a plurality of transmission packets, if it is an image, it can be accessed as independent information. Priority is added only to the communication header that transmits the top part of the image frame (If the priority is the same in the same image frame, the priority does not change until the beginning of the next accessible image frame appears. do it).

It is to be noted that the range of values in which the priority can be expressed (for example, time information is expressed in 16 bits or 32 bits) can be made variable according to the purpose of use, and can be configured by control information. You may

Further, in the decoding device, the receiving terminal device is provided with the priority determining means for determining the processing method in accordance with the priorities of the various kinds of received coded information at the time of overload (see FIG. 55). .

FIG. 55 is a priority determining means 5 for interpreting the priority added to video and audio and determining whether or not decoding processing is possible.
It is a figure explaining 301.

That is, as shown in the figure, the priority is the priority added to each video or audio stream, and the priority added to each video or audio frame. These priorities may be used independently, or the frame priority and the stream priority may be used in association with each other. The priority determining means 5301 determines a stream or frame to be decoded according to these priorities.

Decoding processing is performed using two types of priorities that determine the processing priority when the terminal is overloaded. That is, Stream Priority (Stream Priority) which defines the relative priority between bit streams such as video and audio.
y; inter-time-series priority) and a frame priority (Frame Priority; intra-time-series priority) that defines a relative priority between decoding processing units such as video frames in the same stream (FIG. 30).

The former stream priority makes it possible to handle a plurality of videos and audios. The latter frame priority makes it possible to add different priorities even to the same intra-coded video frame (I frame) depending on the scene change of the video or the intention of the editor.

The stream priority is managed in association with the allocation time or the processing priority of the operating system (OS) of the image or audio encoding or decoding process, and is managed at the OS level. The processing time can be managed. For example, Microsoft's Wind
With ows95 / NT, the priority can be defined at five levels of the OS level. When the encoding / decoding means is realized by software on a thread-by-thread basis, the OS-level priority assigned to each thread can be determined from the stream priority of the stream to be processed.

The frame priority and the stream priority described here can be applied to a transmission medium and a data recording medium. For example, the priority of a packet to be transmitted is set to access unit priority (Access Unit Priority).
ty) is defined as Access Unit Pri
ority = Stream Priority-Fra
frame priority, such as me Priority,
From the relational expression of the stream priority, it is possible to determine the priority relating to packet transmission or the priority of processing when the terminal is overloaded.

A floppy (registered trademark) disk, an optical disk or the like can be used as the data recording medium. The recording medium is not limited to this, but an IC card,
The same operation can be performed as long as the program can be recorded, such as a ROM cassette. Further, an image or voice relay device such as a router or gateway that relays data may be targeted.

As a specific method of utilizing the priority, when the receiving terminal is overloaded, a priority determining means for determining the threshold of the priority of the encoded information to be processed is used as the image expansion management unit 15. It is provided in the audio decompression management unit and compares the time to be displayed (PTS) with the elapsed time from the start of processing up to the present or the time to be decoded (DTS) with the elapsed time from the start of processing up to the present. The threshold value of the priority of the encoded information to be processed is changed according to the comparison result (I-frame insertion intervals and the granularity of the priority may be referred to as the information for changing the threshold value).

In the example shown in FIG. 25 (a), at the time of encoding, the captured image of the size of QCIF and CIF is encoded by the encoder (H.263).
Decoding time (DT
S), a time stamp (PT) indicating the time when the image is displayed
S), priority information (CG indicating the order of processing at the time of overload)
D, Computational Graceful
Degradation), frame type, and sequence number (SN) are output.

Further, in the example shown in FIG. 25 (b), voice is also recorded through the microphone, encoded by the encoder (G.721), and the time of decoding (DTS) and voice are reproduced together with the encoded information. A time stamp (PTS) indicating the time to perform, priority information (CGD),
The sequence number (SN) is output.

At the time of decoding, as shown in FIG. 26, the image and the sound are passed to different buffers, and the image and the sound are compared with each DTS (decoding time) and the elapsed time from the start of the current processing. If the DTS is not delayed, the video and audio are decoded by their respective decoders (H.263,
G. 721).

In the example of FIG. 27, the method of adding the priority when the encoder is overloaded is described. As for images, I-frames (intra-frame encoded image frames) are assigned high priorities with priorities “0” and “1” (the higher the number, the lower the priority). The P frame has a priority of “2” and is assigned a lower priority than the I frame. Since I-frames are assigned two levels of priority, it is possible to reproduce only I-frames with a priority of “0” when the load on the decoding terminal is high. Note that it is necessary to adjust the I-frame insertion interval according to the priority addition method.

The example of FIG. 28 is a diagram showing a method of determining the priority at the receiving terminal at the time of overload. Set the priority of discarded frames to be higher than CutOffPriority. That is, all image frames are processed. The maximum value of the priority added to the image frame can be known in advance by notifying the receiving side from the transmitting side when the terminal is connected (step 101).

When the elapsed time from the start of the current processing is compared with the DTS, and when the elapsed time is longer (when the decoding processing is not in time), the threshold CutOffPriority of the priority of the image and audio to be processed is set. If the elapsed time from the start of processing is shorter (when the decoding processing is in time), the threshold value of the priority is increased in order to increase the number of images and sounds to be processed. The CutOffPriority is raised (step 103).

If the previous image frame is skipped by P frames, no processing is performed. If not, the priority offset value is added to the priority of the image frame (or the audio frame), the threshold value of the priority is compared, and if the threshold value is not exceeded, the data to be decoded is passed to the decoder ( Step 104).

Note that the priority offset is used in advance by checking the performance of the machine and notifying the receiving terminal of the offset (the user may instruct the receiving terminal), and the offset of a plurality of video and sound streams can be used. It can be used to change the priority of each stream (for example, the background value at the back is increased by an offset value so as to thin out the processing).

When a multi-stream is targeted, the priority of each stream may be added and the skip determination of the decoding of the image or audio may be performed. In addition, even in real-time communication By handling and using the TR (temporary reference) of H.263 in the same manner as the DTS, it can be determined whether the decoding process in the terminal is advanced or delayed, and the same skip process described above can be realized. .

FIG. 29 is a diagram in which the algorithm of FIG. 28 is implemented and the change in priority with time is examined.

In the same figure, the change of the priority added to the video frame is shown. This priority is a priority for determining whether or not decoding is possible when the terminal is overloaded, and is added for each frame. The smaller the value of the priority, the higher the priority. In the example of the figure, 0 has the highest priority. When the priority threshold value is 3, frames with priority values greater than 3 are not decoded but discarded, and frames with priority values less than or equal to 3 are decoded. By selectively discarding frames according to priority, it is possible to reduce the load on the terminal. This priority threshold is the current processing time and the decoding processing time (DTS) added to each frame.
It may be dynamically determined from the relationship with. This method can be applied to not only video frames but also audio in a similar manner.

Considering a transmission line such as the Internet, when it is necessary to retransmit coded information that is lost during transmission, a retransmission request priority for determining a threshold value of the priority of coded information to be retransmitted The reception management unit 11 is provided with a degree determination unit, and the priority determined by the priority determination unit, the number of retransmissions, the information loss rate, the insertion interval of the intra-coded frame, and the granularity of the priority (for example, 5 levels). (Such as the priority of each), determine the priority threshold added to the coded information that should be retransmitted, and request only the image and audio that the receiving terminal needs. You can If the number of retransmissions and the loss rate of information are large, it is necessary to raise the priority of the information to be retransmitted and reduce the retransmission and loss rate. Further, by knowing the priority used in the priority determining unit, it is possible to eliminate the transmission of information that is not the processing target.

Regarding the transmitting side terminal, when the actual transfer rate exceeds the target transfer rate of the information of the transmitting terminal, or when the encoded information is written in the transmission buffer,
If the writing of information to the transmission buffer is delayed by comparing the elapsed time from the start of the transfer process up to now with the decoding or display time added to the encoded information, it is encoded. A terminal added to the information and used by the priority determining unit of the receiving terminal can thin out the transmission of the information by using the priority at the time of overload, and thus it is possible to transmit the image and the audio that meet the target rate. . Further, by introducing a skip function for processing at the time of overload, which is performed in the receiving side terminal, in the transmitting side terminal as well, it is possible to suppress failure due to overloading of the transmitting side terminal.

By enabling only the necessary information to be transmitted from the AL information described above, the amount of transmitted information can be adjusted in a narrow band communication path such as an analog telephone line, which is effective. Is. As an implementation method, the data management information to be added to the data itself at the transmitting terminal is determined in advance before data transmission, and the data management information to be used at the receiving terminal is used as control information (for example, using only the random access flag). At the same time as the notification, the receiving side terminal rewrites the information about the transmission structure stored in the transmission format storage unit 103 (which indicates which AL information is used) based on the obtained control information, thereby transmitting It is possible to rearrange the AL information (data management information) used on the side (see FIGS. 19 to 20).

FIG. 4 is a diagram for explaining a method of dynamically changing header information added to image or audio data to be transmitted. In the example of the figure, the data (ES) to be transmitted is decomposed into data pieces, and the obtained data pieces have identification information (sequence number) for indicating the order relation of the data and the start position where the data pieces can be processed. Information indicating whether or not it is (marker bit) and time information (time stamp) regarding the transfer of the data piece are added to the data piece in the form of a communication header as corresponding to the transmission management information of the present invention. There is.

As a concrete example, RTP (Realt
image Transfer Protocol, RFC
1889) uses information such as the above sequence number, marker bit, time stamp, object ID (called SSRC), and version number as a communication header. The header information items can be expanded, but the above items are always added as fixed items. However, multiple images and audios with different encodings,
In a simultaneous communication environment where real-time communication such as a TV phone and transmission of a storage medium such as video-on-demand are mixed, the meaning of the communication header is different and a means for identifying is required.

For example, the time stamp information is MPE.
In the case of G1 / 2, the playback time is PTS as described above.
, H. 261 and H.264. 263 represents the encoded time interval. However, H. When processing 263 in synchronization with voice, it is necessary to indicate that the time stamp is PTS information. Because H. Two
In the case of 63, the time stamp information indicates the time interval between encoded frames, and the time stamp of the first frame is defined by RTP as being random.

Therefore, (a) a flag indicating whether the time stamp is PTS or not is set in the communication header information (it is necessary to extend the communication header) or (b) H.264. 263 and H.264. It is necessary to add it as header information (that is, AL information) of the 261 payload (in this case, the payload information needs to be extended).

As the RTP header information, a marker bit, which is information indicating whether the start position of the data piece can be processed, is added, and as the AL information, the data is accessed as described above. There is a case where it is desired to have an access flag indicating that it is a start time point and a random access flag indicating that data can be randomly accessed. Since it is inefficient to duplicately give it to the communication header, it is conceivable to substitute the flag prepared for the AL for the flag of AL.

(C) A flag indicating that the AL flag is substituted by the header added to the communication header without adding the flag to AL is newly provided in the communication header, or the marker bit of the communication header is set to AL. Defining it to be the same thing solves the problem (expecting faster interpretation than having AL). That is, it is a flag indicating whether the marker bit has the same meaning as the flag of AL. In this case, improve the communication header, or
It is possible to describe it in the extension area.

Conversely, (d) the meaning of the marker bit of the communication header may be interpreted to mean that at least either the random access flag or the access flag is present in AL. In this case, the version number of the communication header can be used to know that the meaning of the interpretation has changed. In addition to this, as a simple method, the process is simple if an access flag or a random access flag is provided only in the communication header or the AL header (in the former case, both flags may be provided, Requires new extension of header).

Although it has been described that the information indicating the priority of the data processing is added as the information of the AL, the priority of the data processing is added to the communication header so that the determination of the processing of the priority of the data processing is performed by the network. Also in the above, it is possible to do without interpreting the contents of the data. IP
In the case of v6, it is possible to add at a layer lower than the RTP level.

By adding a timer or counter for indicating the valid period of data processing to the communication header of RTP, it is possible to judge how a certain state change of the transmitted packet is changing. . For example, if the required decoder software is stored in a storage device with a slow access speed, the information that the decoder is needed and a timer or counter can be used to determine when it is needed. Become.
In this case, depending on the application, the AL information does not require a timer, a counter, or data processing priority information.

5 (a) to 5 (b) and 6 (a) to
FIG. 6D is a diagram illustrating a method of adding AL information.

As shown in FIG. 5A, AL is added only at the beginning of the data to be transmitted, or
As shown in (b), by sending control information to the receiving terminal, which notifies whether or not the data (ES) to be transmitted is decomposed into one or more data pieces and is added to each of the data pieces. It becomes possible to select the handling granularity of the transmission information. Attaching the AL to the fragmented data is effective when access delay is a problem.

As described above, in order to notify the receiving side terminal in advance that the data management information on the receiving side is rearranged and the method of arranging the data management information in the data is changed, a flag, a counter, a timer are provided. By preparing as AL information or using as a communication header and notifying the receiving terminal by using the expression method as described above, the receiving terminal can be smoothly handled.

In the above examples, the RTP header (or,
Communication header) and method to avoid duplication of AL information, R
The method of extending the communication header of TP and the information of AL has been described. However, the present invention does not necessarily have to be RTP. For example, a unique communication header or AL information may be newly defined using UDP or TCP. Ra may be used in Internet profile, but Ra
The w profile does not define a multifunctional header such as RTP. There are four ways of thinking about AL information and communication headers (Fig. 6 (a)-
See FIG. 6D).

(1) The RTP header information or the AL information is corrected or expanded so that the header information already assigned to the RTP and the AL does not overlap (especially, the time stamp information is duplicated, the timer or counter, The data processing priority information is extended information). Alternatively,
The RTP header may not be expanded, and the AL information may be duplicated with that of the RTP and not considered. These are equivalent to the contents shown so far. RTP is already part of H.264. Since it has been put to practical use in H.323, the extension of RTP that maintains compatibility is effective (see FIG. 6A).

(2) Regardless of RTP, the communication header is simplified (for example, only the sequence number is used), and the rest is included in the AL information as multifunctional control information. Further, by making it possible to variably set the items used in the AL information before communication, a flexible transmission format can be defined (see FIG. 6B).

(3) Regardless of the RTP, the AL information is simplified (in the extreme example, no information is added to the AL), and the communication header has all the control information. Sequence numbers, which are often referenced as communication headers,
Time stamp, marker bit, payload type,
The object ID is set as fixed header information, the data processing priority information and the timer information are provided as extended information with an identifier indicating whether or not the extended information exists, and reference is made if the extended information is defined. You may do so (see FIG. 6C).

(4) Regardless of RTP, communication header,
By simplifying the AL information, the format is defined as a packet different from these communication header and AL information,
To transmit. For example, the AL information defines only marker bits, time stamps, and object IDs, the communication header defines only sequence numbers, and payload information, data processing as a transmission packet (second packet) different from these information. A method of defining priority information, timer information, etc., and transmitting the information can be considered (see FIG. 6D).

As described above, in consideration of the purpose of use and the header information already added to the image and voice, the packet (the first one) to be transmitted separately from the communication header, the AL information, and the data is used in accordance with the purpose. It is desirable to be able to freely define (2 packets) (customize).

FIG. 7 is a diagram for explaining a method of dynamically multiplexing and separating a plurality of logical transmission lines to transmit information. In order to save the number of logical transmission lines, multiplex information of the logical transmission line for transmitting a plurality of data or control information according to a user's instruction or the number of logical transmission lines This can be realized by providing the transmission unit 13 with an information multiplexing unit capable of starting and ending the transmission, and the reception management unit 11 with an information demultiplexing unit for demultiplexing the multiplexed information.

Note that, in FIG. 7, the information multiplexing unit is set to "Group".
MUX ", and specifically, a multiplexing method such as H.223 may be used. This Group MUX
May be provided at the transmission / reception terminal, or by being provided at a relay router or terminal, it is possible to support a narrow band communication path,
up MUX to H.264. H.223 if realized in H.223. Can be interconnected with 324.

In order to quickly take out the control information (multiplexing control information) about the information multiplex section, the control information of the information multiplex section is not multiplexed with the data in the information multiplex section and is transmitted, but is not multiplexed and is transmitted separately. By transmitting on a logical transmission path, the delay due to multiplexing can be reduced. Along with this, it is notified whether the control information related to the information multiplexing unit is multiplexed with data and transmitted, or whether it is not multiplexed and transmitted with data but is transmitted through another logical transmission line without being multiplexed. Then, the user can select whether to maintain consistency with the conventional multiplexing or reduce the delay due to the multiplexing. Here, the multiplexing control information regarding the information multiplexing unit is, for example, what kind of multiplexing is performed by the information multiplexing unit for each data,
This is information indicating the content of multiplexing.

As described above, similarly, at least the information for notifying the start and end of the multiplexing, the information for notifying the combination of the logical transmission lines to be multiplexed, the control information on the multiplexing (the multiplexing control information) ) The transmission method notification is transmitted as control information or as data management information together with data by a representation method such as a flag, a counter, or a timer, and is transmitted to the receiving side terminal, so that the setup time at the receiving side is reduced. Can be shortened. Further, as described above, items representing flags, counters, and timers may be provided in the RTP transmission header.

When there are a plurality of information multiplexing units or information demultiplexing units, if control information (multiplexing control information) is transmitted together with an identifier for identifying the information multiplexing units or information demultiplexing units,
It is possible to identify which information multiplexing unit is control information (multiplexing control information). Control information (multiplex control information)
Examples of such patterns include multiplexing patterns. Also,
It is possible to generate the identifier of the information multiplexing unit by determining the identifiers of the information multiplexing unit and the information separating unit between the terminals using random numbers. For example, a random number within a range determined between the transmitting and receiving terminals may be generated, and the larger value may be used as the identifier (identification number) of the information multiplexing unit.

Since the data multiplexed by the information multiplexing unit is different from the media type conventionally defined in RTP, it indicates that the payload type of RTP is the information multiplexed by the information multiplexing unit. Information (new media type, defining H.223) may be defined.

As a method for improving the access speed to the multiplexed data, it is expected that the information to be transmitted or recorded in the information multiplexing section will be arranged in the order of the control information and the data information so that the analysis of the multiplexed information can be speeded up. it can. Further, the items described in the data management information added to the control information are fixed, and an identifier (unique pattern) different from the data is added and multiplexed to speed up the analysis of the header information.

FIG. 8 is a diagram for explaining the transmission procedure of the broadcast program. The control information is transmitted using the correspondence between the logical transmission path identifier and the broadcast program identifier as the broadcast program information, or the broadcast program identifier is used as the data management information (AL
It is possible to identify for which program the data transmitted through a plurality of transmission paths is broadcast by adding the data as information and transmitting it. Further, the relation between the data identifier (SSRC in RTP) and the logical transmission line identifier (for example, LAN port number) is transmitted to the receiving side terminal as control information, and can be received by the receiving side terminal. After confirming (Ack / Reject), by transmitting the corresponding data, the correspondence between the data can be obtained even if the control information and the data are transmitted through independent transmission paths.

An identifier indicating the order of transmission of broadcast programs and data, and counter or timer information for indicating the expiration date when the broadcast programs and data can be used as information are combined and added to the broadcast programs and data. By doing so, the broadcast can be realized without the return channel (when the expiration date is about to expire, reproduction of information and data of the broadcast program is started even if there is insufficient information). It is also possible to use a single communication port address (multicast address) to broadcast without separating control information and data.

In the case of communication having no back channel, the control information needs to be transmitted well in advance of the data so that the receiving terminal can know the structure information of the data. In addition, control information should generally be transmitted on a highly reliable transmission channel without packet loss, but when using a less reliable transmission channel, control information with the same transmission sequence number is repeated periodically. Need to be transmitted. This is not limited to sending control information about setup time.

Further, items that can be added as data management information (for example, access flag, random access flag, data reproduction time (PTS), priority information of data processing, etc.) are selected, and data identifiers are used as control information. Before sending data, the sending side decides whether to send it along with (SSRC) through a logical transmission path different from data or as data management information (AL information) together with the data, and controls it to the receiving side. By notifying and transmitting as information, flexible management and transmission of data becomes possible.

As a result, data information can be transmitted without adding information to the AL. Therefore, when transmitting image or audio data using RTP, the definition of the payload that has been conventionally defined is used. Eliminates the need to expand

FIGS. 9A and 9B are diagrams showing a method of transmitting an image and a sound in consideration of the reading of programs and data and the start-up time. In particular, when there is no return channel such as satellite broadcasting or a mobile terminal, and there is limited one-way resource of the terminal, and the program or data exists and is used in the receiving side terminal, the necessary program (for example, , H.263, MPEG1 / 2, audio decoder software, etc.) and data (eg image data and audio data) take time to read (eg DVD, hard disk, file server on network, etc.) ), An identifier for identifying a program or data required in advance and an identifier of a stream to be transmitted (for example, SSRC or Logic).
al Channel Number), a flag for estimating the time required by the receiving terminal, a counter (count up, down), a representation method such as a timer,
By receiving the control information or the data as the data management information together with the data, it is possible to shorten the setup time of the necessary programs and data (FIG. 2).
2).

On the other hand, when a program or data is transmitted, the receiving destination of the program or data at the receiving terminal (for example, hard disk, memory), the time required for starting or reading, the type or storing destination of the terminal and starting or reading. By transmitting programs and data from the transmitting side together with such time correspondence (for example, CPU power, relationship between storage device and average response time), and information indicating the order of use, programs and data required by the receiving side terminal can be obtained. When data is actually needed, scheduling is possible with respect to the storage location of the program or data and the time of reading.

FIGS. 10A and 10B are views for explaining a method of dealing with zapping (TV channel switching).

Unlike the conventional satellite broadcasting which only receives a certain image, when the program has to be executed by the receiving terminal, there is a big problem in the setup time until the program is read and the program is started up. The same can be said when the use resources are limited like a mobile terminal.

As one of the solutions, (a) it takes a long time to read the necessary programs and data for the main viewing section for the user to watch and the programs other than the programs the user is watching. When present in the storage device, the receiving terminal is provided with a sub-viewing unit for allowing the receiving terminal to periodically watch a program other than the program being watched by the user, and an identifier for identifying a necessary program or data in advance and a receiving terminal Information such as flags, counters, and timers for estimating the required time point and the correspondence with programs are received as control information (information transmitted in a packet other than data and used for controlling terminal processing). Or, by receiving it together with the data as data management information (AL information) and preparing to read the program or data, the setup time at the receiving terminal can be shortened. There can be expected.

The second solution is to provide a broadcasting channel for broadcasting only index images of images broadcast on a plurality of channels so that the viewer can switch the viewing program so that necessary programs and data can be displayed. , If it exists in a storage device that takes a long time to read, it is necessary to temporarily select the heading image of the program to be viewed and present it to the viewer, or to indicate that the program is being read, and By reading the data and after the reading is completed, the viewer can restart the program he / she wants to watch, thereby preventing the screen from being stopped during the setup. The heading image here means a broadcast image obtained by sampling a program that is periodically broadcast on a plurality of channels.

A timer is a time expression, for example,
The program required to decode the data stream sent from the sender indicates when it is needed from the present. The counter may be information indicating the number of times in a basic time unit determined between the transmitting and receiving terminals. The flag is transmitted and notified together with data sent before the time required for setup or control information (information for controlling terminal processing, which is transmitted in a packet different from the data). Both the timer and the counter may be embedded in the data for transmission, or may be transmitted as control information.

Further, as a method for determining the setup time, for example, ISDN operating on a clock basis is used.
When a transmission line such as the above is used, a transmission serial number for identifying the order relation of transmission is used as transmission management information in order to notify the time when a program or data is needed from the transmission side terminal to the reception terminal. By notifying the receiving terminal with the data as the data management information or as the control information, the time when the setup is performed can be predicted. If the transmission time varies due to jitter or delay like the Internet, the propagation delay of the transmission is added from the jitter or delay time by means already realized by RTCP (Internet media transmission protocol). Then add it to the setup time.

11 to 24 are diagrams showing specific examples of protocols actually transmitted and received between terminals.

The transmission format and the transmission procedure are ASN.
Described in 1. This transmission format is ITU H.264 standard. Expansion was performed based on 245. As shown in FIG. 11, the image and audio objects may have a hierarchical structure. In this example, each object ID is a broadcast program identifier (ProgramID) and an object ID.
It has an attribute of (SSRC), and the structure information between images and the composition method are described in a script language such as Java or VRML.

FIG. 11 is a diagram showing an example of the relationship between objects.

In the figure, the object is a medium such as video, audio, CG, and text. In the example shown in the figure, the objects have a hierarchical structure. Each object has a program number (corresponding to a TV channel,
“Program ID”) and an object identifier “Object ID” for identifying an object. R
TP (Media transmission protocol used on the Internet, Realtime Transfer Pro
When each object is transmitted by means of tocol), the object identifier can be easily identified by associating the object identifier with the SSRC (synchronization source identifier).
The structure description between objects is JAVA, VRML.
It is possible to describe in a description language such as.

There are two possible methods of transmitting these objects. One is a broadcast type, which is a mode of unilaterally transmitting from a transmitting side terminal. The other is a mode (communication type) in which an object is transmitted between transmitting / receiving terminals (terminal A, terminal B).

For example, as the transmission method, RTP can be used in the case of the Internet. The control information is T
In the V telephone standard, transmission is performed using a transmission channel called LCNO. Although a plurality of transmission channels are used for transmission in the example of the figure, the same program channel (Program ID) is assigned to these channels.

FIG. 12 is a diagram for explaining a method for realizing a protocol for realizing the functions described in the present invention. Here, the videophone standard (H.324, H.32
This will be described using the transmission protocol (H.245) used in 3). H. The functions described in the present invention are realized by expanding 245.

The description method shown in the example of FIG. 1
Is a protocol description method called. "Termin
“Al Capability Set” expresses the performance of the terminal. In the example of the figure, “mpeg4 Capabi”
The function described as “light” is extended from the existing H.245.

In FIG. 13, "mpeg4 Capabi" is displayed.
“Litity” is the maximum number of videos (“Max Number Of Video”) that can be processed simultaneously by the terminal, and the maximum number of audio (“Max Number Of Sou”).
nds ”) and the maximum number of multiplexing functions that can be realized by the terminal (“ Max Number Of Mux ”).

In the figure, the maximum number of objects that can be processed (“Number of Pro
cess Object ”). Also, a flag is written to indicate whether the communication header (expressed as AL in the figure) can be changed. When this value is true, the communication header is changed. Yes, "MPEG4 C"
When notifying each other of the number of objects that can be processed between terminals using "ability", if the notified side can accept (process), "MEPG4 Capa"
"bility Ack", otherwise "MEPG
4 "Capability Reject"
PG4 Capability ”is returned to the sending terminal.

In FIG. 14, in order to share and use one transmission channel (LAN transmission channel in this example) with a plurality of logical channels, a plurality of logical channels are multiplexed into one transmission channel. Gro mentioned above
It shows how to describe the protocol for using up MUX. In the example of the figure, a transmission channel (“LAN Port”) of a LAN (local area network) is used.
t Number ”) to the multiplexing means (Group MU).
X) is associated. "Group MuxID"
Is an identifier for identifying the multiplexing means. "Cr
When notifying each other when using multiplexing means between terminals using "eate Group Mux", if the notified side can accept (use), "Create
e Group Mux Ack ", otherwise" Create Group Mux Reject "
Is returned to the terminal that has transmitted the "Create Group Mux". The demultiplexing means, which is a means for performing the reverse operation of the multiplexing means, can be realized by a similar method.

FIG. 15 describes the case of deleting the already generated multiplexing means.

FIG. 16 describes the relationship between the LAN transmission channel and a plurality of logical channels.

The transmission channel of the LAN is "LAN Po
"rt Number" and a plurality of logical channels are described by "Logical Port Number".

In the example of the figure, a maximum of 15 logical channels can be associated with one LAN transmission channel.

In the figure, when the number of MUXs that can be used is only one, the GroupMux ID is unnecessary. When a plurality of Muxes are used, the H.264 standard is used. 22
A Group Mux ID is required for each command of 3. Also, a flag may be provided for notifying the correspondence of the ports used between the multiplexing and the demultiplexing means. In addition, a command may be provided so that the control information may be multiplexed or transmitted via another logical transmission path.

Although the transmission channel is LAN in the description of FIGS. A method that does not use the Internet protocol, such as 223 and MPEG2, may be used.

In FIG. 17, "Open Logical"
"Channel" indicates a protocol description for defining the attributes of the transmission channel. In the example of FIG.
H. For the H.245 protocol, "MPEG4 Lo"
"gicalChannelParameters" is extendedly defined.

In FIG. 18, the program number (corresponding to the TV channel) for the LAN transmission channel,
Corresponds to the program name ("MPEG4
Logical Channel Parameter
s ″).

Further, in the figure, "Broadcast"
"Channel Program" is a description method in the case of transmitting the correspondence between the LAN transmission channel and the program number in a broadcast type. In the example of FIG.
It is possible to send the correspondence between 23 transmission channels and program numbers. In the case of broadcasting, since it is only unilaterally transmitted from the transmitting side to the receiving side, it is necessary to periodically transmit these pieces of information in consideration of the loss during transmission.

In FIG. 19, the attributes of objects (eg, video, audio, etc.) transmitted as a program are described (“MPEG4 Object Cla”).
ssdefinition ”). The information of the object (“ Object Structure Elem ”with respect to the program identifier (“ Program ID ”)).
ent ”) is associated with each other. A maximum of 1023 objects can be associated with each other. Information on the objects includes LAN transmission channels (“ LAN ”).
Port Number "), a flag indicating whether scrambling is used (" Scramble Fla ")
g ”), a field defining an offset value for changing the processing priority when the terminal is overloaded (“ CG
D Offset ") and an identifier (Me) for identifying the type of media (video, audio, etc.) to be transmitted.
dia type) is described.

In the example of FIG. 20, in order to manage the decoding process of ES (here, it is defined as a data string corresponding to one frame of video), it is necessary to manage AL (here, to decode one frame of video). Defined as additional information). The AL information includes (1) Random A
access flag (a flag indicating whether or not reproduction is possible independently, true in the case of an intra-frame coded video frame), (2) Presentation
Time Stamp (frame display time), (3)
CGD Priority (priority value for determining processing priority when the terminal is overloaded) is defined.
These one-frame data strings are RTP (protocol for transmitting continuous media on the Internet, R
ealtime Transfer Protocol
An example in the case of transmitting using l) is shown. "AL
"Reconfiguration" is a transmission expression for changing the maximum value that can be expressed by the AL.

In the example of the figure, "Random Acce"
A maximum of 2 bits can be expressed as “ss Flag Max Bit”. For example, if 0, Rando is used.
No mAccess Flag is used. If 2, the maximum value is 3.

It is also possible to represent the real part and the mantissa part (for example, 3 ^ 6). When not set, it may be operated in a state determined by default.

In FIG. 21, "Setup Requests"
"t" indicates a transmission expression for transmitting the setup time. Before transmitting the program, "Setup"
"Request" is a transmission channel number ("LogicalChannel Number") that is transmitted and transmitted.
r ”) and the program ID to be executed (“ excute ”
Program Number ”), data I to use
D (“data Number”), ID of the command to be executed (“execute CommandNumber”
r ") is sent to the receiving terminal in association with each other. As another expression method, the execution permission flag (" flag ") is associated with the transmission channel number, and how many more times the Setup is performed.
It may be a counter ("counter") indicating whether to execute the request when the Request is received, or a timer value ("timer") indicating how long the request is executed.

As an example of a request scheduled to be requested,
Examples include rewriting AL information and securing the rise time of GroupMux.

FIG. 22 is a diagram for explaining a transmission expression for notifying the use / non-use of the AL described in FIG. 20 from the transmitting terminal to the receiving terminal (“Control AL”).
definition ”).

In the same figure, "Random Acce"
Random A if ss Flag Use ”is true
The ccess Flag is used. Otherwise do not use. This AL change notification may be transmitted as control information on a transmission channel different from that of the data, or may be transmitted together with the data on the same transmission channel.

As the program to be executed, there is a decoder program or the like. Further, the setup request can be used for broadcasting or communication. Further, the reception terminal is instructed by the above request which item is used as the control information and which is used as the AL information. Similarly, it is possible to instruct the receiving terminal which item is used in the communication header, which item is used as AL information, and which item is used as control information.

In FIG. 23, the information framework identifier ("hea"
der ID ") is used to show an example of a transmission expression for changing the structure of header information (data management information, transmission management information, control information) to be transmitted between transmitting and receiving terminals according to the application.

In the figure, "class ES he"
“Ader” distinguishes the data management information transmitted on the same transmission channel as the data and the structure of information transmitted by the transmission management information between the transmitting and receiving terminals by the information framework identifier.

For example, if the value of "header ID" is 0, only the item of buffer Size ES is used,
If the value of "header ID" is 1, then "reserv
"ed" item is added and used.

Also, the default identifier ("use Hea"
der Extension ") is used to determine whether to use the default information framework or not." use Header Extension "
If n "is true, the item inside the if statement is used.
It is assumed that the structural information is preliminarily agreed between the transmitting and receiving terminals. The information framework identifier and the default identifier may be configured to use either one.

In FIG. 24, "AL configure"
“Tion” indicates an example in which the structure of control information transmitted on a transmission channel different from data is changed between the transmitting and receiving terminals according to the usage. The method of using the information framework identifier and the default identifier is shown in FIG. Same as the case.

In the present invention, a method of realizing a system for simultaneously synthesizing and displaying a plurality of moving images and sounds has been specifically described from the following viewpoints.

(1) Image and audio transmission (communication and broadcasting) using a plurality of logical transmission paths and a method of controlling them. In particular, the method of transmitting control information and data independently of the logical transmission path has been described.

(2) A method of dynamically changing header information (AL information) added to image or audio data to be transmitted.

(3) A method for dynamically changing header information for communication added for transmission.

Specifically, regarding (2) and (3),
The method of integrating and managing the information duplicated in the AL information and the communication header and the method of transmitting the AL information as the control information have been described.

(4) A method of dynamically multiplexing and separating a plurality of logical transmission lines to transmit information.

The method of saving the number of channels in the transmission path and the method of realizing efficient multiplexing have been described.

(5) A method of transmitting an image and sound in which the program and data are read and the startup time is taken into consideration. Described how to shorten the apparent setup time for various functions and applications.

(6) A method of transmitting images and sounds for zapping.

The present invention is not limited to two-dimensional image composition. An expression format in which a two-dimensional image and a three-dimensional image are combined may be used, or an image synthesizing method of synthesizing a plurality of images so that they are adjacent to each other such as a wide-field image (panoramic image) may be included.

Further, the communication mode targeted by the present invention is not limited to the wired bidirectional CATV and B-ISDN. For example, video and audio are transmitted from the center side terminal to the home side terminal by radio waves (for example, VHF band, UHF band) or satellite broadcasting, and information is transmitted from the home side terminal to the center side terminal by an analog telephone line or N. It may be ISDN (video, audio and data do not necessarily have to be multiplexed).

Further, a communication form using radio such as IrDA, PHS (Personal Handy Phone) or wireless LAN may be used. Furthermore, the target terminal is
It may be a portable terminal such as a personal digital assistant, or a tabletop terminal such as a set-top box or a personal computer. In addition, as an application field, TV
Examples include a telephone, a multipoint monitoring system, a multimedia database search system, a game, and the like. The present invention includes not only a receiving terminal, but also a server or a relay device connected to the receiving terminal.

Further, in the above examples, the method of avoiding the duplication of the RTP (communication) header and the AL information, and the RT
The method for expanding the P communication header and AL information has been described. However, the present invention does not necessarily have to be RTP. For example, a unique communication header or AL information may be newly defined using UDP or TCP. RTP may be used in Internet profile, but RAW
The profile does not define a multifunctional header such as RTP. There are four ways of thinking about AL information and communication headers, as described above.

As described above, the framework of each information of the data management information, the transmission management information, and the control information used by the transmitting terminal and the receiving terminal (for example, the first is a random access flag and is assigned as 1-bit flag information). It is possible to change the information framework according to the situation by dynamically determining the information framework with the order of the information to be added and the number of bits, such as allocating 16 bits for the second sequence number. It can be changed according to the application and transmission path.

The frame of each information is shown in FIG.
~ It may be the one already shown in Fig. 6 (d),
In the case of RTP, the data management information (AL) is header information for each medium (for example, H.263-specific video header information and payload header information in H.263), and the transmission management information is RTP header information. Control information is RTC
It may be information that controls RTP such as P.

In addition, whether information is transmitted / received and processed in a well-known information framework preset between transmission / reception terminals,
The default identifier for indicating whether or not the data management information,
By setting it in the transmission management information and control information (information that is transmitted in a packet different from the data and that controls terminal processing), it is possible to know whether the information framework has been changed. The default identifier is set and the change contents (for example, the time stamp information is changed from 32 bits to 16 bits) are notified by the method shown in FIGS. Thus, even if the information framework information is not changed, the configuration information need not be transmitted unnecessarily.

For example, when it is desired to change the information framework of the data management information, the following two methods can be considered.
First, when describing the method of changing the information framework of the data management information in the data itself, it is necessary to write in the default identifier (fixed area, position) of the information existing in the data described regarding the framework of the information of the data management information. Set) and then describe changes to the information framework.

As another method, when a method of changing the information framework of data is described in control information (information framework control information) and the framework of information in the data management information is modified, a default identifier provided in the control information. Set, describe the contents of the information framework of the data management information to be changed, notify the receiving terminal that the information framework of the data management information has been changed with ACK / Reject, and after confirming that the information framework Transmit modified data. Even when changing the information framework of the transmission management information and the control information itself, it can be realized by the above two methods (FIGS. 23 to 24).

As a more specific example, for example, MPE
The header information of G2 is fixed, but MPEG2-TS
A program that associates (transport stream) video and audio streams
The default identifier is set by defining a default identifier in the map table (defined by PSI) and defining the configuration stream that describes how to change the information framework of the video stream and audio stream. With this, it is possible to first interpret the configuration stream, and then interpret the headers of the video and audio streams according to the contents of the configuration stream. The configuration stream may have the contents shown in FIGS.

The contents (transmission format information) relating to the transmission method and / or the structure of data to be transmitted according to the present invention correspond to, for example, the information framework in the above embodiments.

Further, in the above-mentioned embodiment, the description has been centered on the case where the contents to be changed, the transmission method and / or the structure of the data to be transmitted are transmitted. However, the present invention is not limited to this. Of course, it may be configured to transmit only the data. In this case, for example, as shown in FIG. 52, the transmitting device transmits (1) the content related to the transmission method and / or the structure of the data to be transmitted, or the identifier indicating the content as the transmission format information. Transmission means 5001 for transmitting using the same transmission path as the data transmission path or a transmission path different from the transmission path;
(2) A storage unit 5002 that stores a plurality of types of contents relating to the transmission method and / or the structure of data to be transmitted and its identifier, wherein the identifier is data management information, transmission management information, or a terminal side The image / sound transmission device included in at least one of the information for controlling the process may be used. As the receiving device, for example, as shown in FIG. 53, receiving means 5101 for receiving the transmission format information transmitted from the image / audio transmitting device, and transmission information for interpreting the received transmission format information. It may be an image / sound receiving device including an interpreting unit 5102. Further, the image / sound receiving device is provided with a storage unit 5103 for storing a plurality of types of contents relating to the transmission method and / or the structure of data to be transmitted and its identifier, and receives the identifier as the transmission format information. in case of,
The content stored in the storage unit may be used when interpreting the content of the identifier.

More specifically, a plurality of information frameworks are prepared in advance by the transmitting and receiving terminals, and the plurality of kinds of information frameworks are identified and the plurality of kinds of data management information, transmission management information, control By transmitting an information framework identifier for identifying information (information framework control information) together with data or as control information, it is possible to identify each type of data management information, transmission management information, and control information. It becomes possible, and the information framework of each information can be freely selected according to the format of the medium to be transmitted and the thickness of the transmission path. The identifier of the present invention corresponds to the framework identifier of the above information.

The information frame identifier and the default identifier are added to a predetermined fixed length area or position of the information to be transmitted, so that the receiving side terminal can read the information frame even if the information frame is changed. , Can be interpreted.

Further, in addition to the configuration described in the above-mentioned embodiment, a broadcast channel for broadcasting only the index image of the images broadcast on a plurality of channels is provided so that the viewer can switch the viewing program. When it takes time to set up another program or data, the heading image of the program to be viewed may be once selected and presented to the viewer.

As described above, according to the present invention, by dynamically determining the framework of each information of the data management information, the transmission management information, and the control information used by the transmitting terminal and the receiving terminal, the information according to the situation can be obtained. It is possible to change the framework of, and it is possible to change according to the application and transmission path.

[0182] In addition, whether information is transmitted / received and processed in a known information framework preset between transmitting / receiving terminals,
The default identifier for indicating whether or not the data management information,
By providing the transmission management information and the control information respectively, it is possible to know whether the information framework has been changed. Only when the change has been made, the default identifier is set and the changed contents are notified. Thus, even if the information framework information is not changed, unnecessary configuration information need not be transmitted.

Further, a plurality of information frameworks are prepared in advance by the transmitting and receiving terminals, and an information framework identifier for identifying a plurality of types of data management information, transmission management information, and control information is provided together with the data or control. By transmitting it as information, it is possible to identify multiple types of data management information, transmission management information, and control information, and the information of each information according to the format of the medium to be transmitted and the thickness of the transmission path. The framework of can be freely selected.

These information framework identifiers and default identifiers are added to a predetermined fixed-length area or position of the information to be transmitted so that the receiving side terminal can read the information framework even if the information framework is changed. Can be interpreted.

Embodiments of the present invention will be described below with reference to the drawings.

Here, mainly, the above-mentioned problem (B1)
To (B2) are solved.

The term “image” used in the present invention includes both still images and moving images. Further, the target image may be three-dimensional image data such as a two-dimensional image such as computer graphics (CG) and a wire frame model.

FIG. 31 is a schematic configuration diagram of an image encoding / decoding device according to the embodiment of the present invention.

The transmission management unit 4011 for transmitting or recording various encoded information is a coaxial cable, CAT
A means for transmitting information such as V, LAN, and modem. The image coding device 4101 uses the H.264 standard. 263, MPEG1 /
2, an image encoding unit 4012 that encodes image information such as JPEG or Huffman encoding, and the transmission management unit 4011. The image decoding device 4102 also includes a reception management unit 4013 that receives various encoded information, an image decoding unit 4014 that decodes the various received image information, and one or more decoded images. This is a configuration including an image combining unit 4015 that combines images and an output unit 4016 that includes a display, a printer, and the like that outputs images.

FIG. 32 is a schematic configuration diagram of a speech encoding / decoding apparatus according to the embodiment of the present invention.

Speech encoding apparatus 4201 transmits and records various encoded information by transmission managing section 4021.
And G. 721, an audio encoding unit 4022 that encodes audio information such as MPEG1 audio. Also, the speech decoding apparatus 4202 synthesizes the reception management unit 4023 that receives various encoded information, the speech decoding unit 4024 that decodes the various speech information, and one or more decoded speeches. A voice synthesis unit 4025 that
It is the structure provided with the output part 4026 which outputs a sound.

The time-series data of voices and moving images are specifically coded or decoded by each of the above devices.

31 and 32, the communication environment may be a communication environment in which a plurality of logical transmission paths can be used without paying attention to the multiplexing means, such as the Internet.
It may be a communication environment such as an analog telephone or satellite broadcasting in which the multiplexing means must be taken into consideration. In addition, as a connection form of terminals, such as a TV phone and a TV conference system in which video and audio are bidirectionally transmitted and received between terminals, satellite broadcasting, CATV, broadcasting type video and audio broadcasting on the Internet. The form may be mentioned.

Similarly, regarding a method of synthesizing images and voices, a script language such as JAVA, VRML, and MHEG is used to determine the structure information (display position or display time) of images / voices and the grouping of images / voices. Method,
Image display layer (depth) and object I
By describing the relationship between D (an ID for identifying an individual object such as image and sound) and these attributes, a method of synthesizing an image and sound can be defined. The script describing the composition method can be obtained from a network or a local storage device.

Note that the transmission / reception terminal may be constituted by any number of image encoding devices, image decoding devices, audio encoding devices, and audio decoding devices in any combination.

FIG. 33 (a) is a diagram for explaining the priority addition unit and the priority determination unit for managing the priority of the processing at the time of overload. H. 263 and G.H. A priority adding unit that determines the priority of the processing of the coded information when the coded information is overloaded by a predetermined criterion by using a coding method such as 723, and associates the coded information with the determined priority. 4031 is provided in the image encoding device 4101 and the audio encoding device 4201.

The criteria for adding the priority are, for example, a scene change in the case of an image, an image frame or stream instructed by an editor or a user, and a voiced section and a silent section in the case of voice.

As a method of adding a priority which defines the priority of processing at the time of overload, a method of adding it to a communication header and a method of embedding it in the header of a bit stream of which video or audio is encoded at the time of encoding can be considered. The former can obtain information on the priority without decoding, and the latter can be handled independently by the bitstream independently of the system.

As shown in FIG. 33B, when priority information is added to the communication header, one image frame (for example, intra-frame coded I frame, inter-frame coded P, When a (B frame) is divided into a plurality of transmission packets, priority is added only to the communication header that transmits the head part of the accessible image frame as independent information in the case of an image (within the same image frame If the priorities are the same, the priorities should not change until the beginning of the next accessible image frame appears.)

Further, in the decoding apparatus, the priority determining unit 4032 for determining the processing method is provided in the image decoding apparatus 4102 and the audio decoding in accordance with the priority of various received encoded information at the time of overload. The device 4202 is provided.

34 to 36 are diagrams for explaining the granularity to which the priority is added. The decoding process is performed using two types of priorities that determine the priority of the process when the terminal is overloaded.

[0202] That is, stream priority (Stream Priority; priority between time series data) that defines the priority of processing at the time of overload in bit stream units such as video and audio, and frame units such as video frames in the same stream. Frame Priority (Frame Priority) that defines the priority of processing during overload in
y; priority within time-series data) is defined (see FIG. 34).

The former stream priority makes it possible to handle a plurality of videos and audios. The latter frame priority makes it possible to add different priorities even to the same intra-coded video frame (I frame) depending on the scene change of the video and the intention of the editor.

The value represented by the stream priority may be treated as a relative value or as an absolute value (see FIGS. 35 and 36).

The stream priority and the frame priority are handled on the network if they are relay terminals such as routers and gateways, and if they are terminals, they are transmission terminals and reception terminals.

There are two ways to express absolute values and relative values.
You can think of it. One is the method shown in FIG.
The other is the method shown in FIG.

In FIG. 35, the absolute value priority means that an editor, a mechanically added image stream or audio stream is (or should be) processed when it is overloaded.
It is a value that represents the order (it is not a value that takes into account actual network and terminal load fluctuations). The relative value priority is a value for changing the absolute priority value according to the load of the terminal or the network.

By managing the priority by separating it into a relative value and an absolute value, it is possible to change only the relative value on the transmitting side or the relay device according to the fluctuation of the network load. Then, the absolute priority that was originally added to the image or audio stream remains, and the hard disk or VT
Recording on R becomes possible. If the absolute priority value is recorded in this way, it is possible to reproduce video and audio without being affected by network load fluctuations. The relative priority and the absolute priority may be transmitted through the control channel independently of the data.

Similarly, in FIG. 35, the frame priority defining the processing priority of the frame at the time of overload is treated as a relative priority value by making the granularity finer than the stream priority. It is also possible to treat it as an absolute priority value. For example, the absolute frame priority is described in the information of the coded image, and the relative priority to the absolute priority added to the previous video frame is reflected in order to reflect the change due to the load of the network or the terminal. By describing the frame priority in the communication header of the communication packet for transmitting the encoded information, the priority is added according to the load of the network and the terminal at the frame level or while keeping the original priority. Is possible.

The relative priority may be transmitted by describing the correspondence with the frame on the control channel independently of the data rather than the communication header. By this,
It is possible to record on the hard disk or VTR with the absolute priority added to the image or audio stream remaining.

On the other hand, in FIG. 35, when recording is not performed at the receiving terminal and reproduction is performed at the receiving terminal while transmitting through the network, the receiving terminal separates and manages absolute values and relative values. Since it is not necessary to do so, even in the case of both the frame and stream levels, the sending side calculates the absolute priority value and the relative priority value before sending and sends only the absolute value. May be.

In FIG. 36, absolute value priority means Stream Priority and Frame.
It is a value that is uniquely determined between frames that is obtained from the Priority relationship. The relative priority of a value is a value that represents the order in which an editor or a mechanically added image stream or audio stream is (or should be) processed when it is overloaded. In the example of FIG. 36, frame priorities (relative values) of video and audio streams and stream priorities are added to each stream.

Absolute frame priority (absolute
e; absolute value) is calculated from the sum of relative frame priority and stream priority (that is, absolute frame priority = relative frame priority + stream priority).
Note that this calculation method may be a method of subtracting or multiplying by a constant.

The absolute frame priority is mainly used in the network. This is a relay device such as a router or gateway, and Stream Priority and Fra
This is because it is not necessary to determine the priority for each frame in consideration of the mePriority and the absolute value does not require it. By using this absolute frame priority, processing such as discarding frames in the relay device becomes easy.

On the other hand, the relative frame priority can be expected to be applied mainly to the storage system for recording and editing. In editing work, multiple video and audio streams may be handled simultaneously. In such a case, the number of video streams or frames that can be reproduced may be limited due to the load of the terminal or the network.

In such a case, Stream Pri
Priority and Frame Priority are separated and managed, for example, the editor prefers to display them preferentially, or the user changes the Stream Priority of the stream to be viewed,
FrameP is different from when absolute value is expressed.
There is no need to recalculate all rioritities. In this way, it is necessary to properly use the absolute expression and the relative expression according to the purpose.

Also, by describing whether the stream priority value is used as a relative value or an absolute value, it is possible to express an effective priority both during transmission and during storage. Become.

In the example of FIG. 35, a flag and an identifier that indicate whether the value represented by the stream priority is an absolute value or a relative value are provided in association with the stream priority to make a distinction. In the case of the frame priority, the relative value is described in the communication header and the absolute value is described in the encoded frame, so that the flag and the identifier are unnecessary.

In the example of FIG. 36, a flag or identifier for identifying whether the frame priority is an absolute value or a relative value is provided. If it is an absolute value, it is a priority calculated from the frame priority relative to the stream priority, and therefore the calculation process is not performed by the relay device or the terminal. Further, in the receiving terminal, when the calculation formula is known between the terminals, it is possible to calculate the relative frame priority backward from the absolute frame priority and the stream priority.
For example, the absolute priority (Acce
ss Unit Priority)
It may be obtained from a relational expression of “Unit Priority = Stream Priority−Frame Priority”. Here, the frame priority may be expressed as the subordination priority because the stream priority is subtracted.

Furthermore, one or more stream priorities may be set to T
The data processing may be managed in association with the processing priority of the data flowing through the CP / IP logical channel (LAN port number).

In addition, it can be expected that the need for retransmission processing can be reduced for images and voices by assigning a stream priority or frame priority lower than that of characters or control information. This is because there is often no problem even if a part of the image or sound is lost.

FIG. 37 is a diagram for explaining a method of assigning priorities to multiresolution image data.

When one stream is composed of two or more substreams, the stream priority is added to the substreams, and the logical sum or the logical product is described at the time of storage or transmission, so that the substreams are described. It is possible to define the processing method of.

In the case of wavelet, one video frame can be decomposed into a plurality of video frames having different resolutions. Further, even in the DCT-based coding method, it is possible to decompose into video frames of different resolutions by dividing the high-frequency component and the low-frequency component for coding.

In addition to the stream priority added to a plurality of video streams composed of a series of decomposed video frames, AND (logical product) and OR (logical Define the relationship with (sum). Specifically, the stream A has a stream priority of 5, and the stream B has a stream priority of 1.
If it is 0 (the smaller the number, the higher the priority), if the stream data is discarded due to the priority, the stream B
Is discarded, but by defining the relationship between streams, in the case of AND, even if the priority of stream B is lower than the priority of the threshold value, it is defined not to be discarded and to be transmitted and processed. I'll do it.

As a result, the related stream can be processed without being discarded. Conversely, in the case of OR,
Defined as disposable. As in the past, the discarding process may be performed by the transmission / reception terminal or the relay terminal.

As an operator for relational description, when the same video clip is encoded into another stream of 24 Kbps and 48 Kbps, either of them may be reproduced (exclusive OR as relational description). E
X-OR).

When the former priority is 10 and the latter is 5, the user may reproduce the latter based on the priority, or the user may select the latter without following the priority.

FIG. 38 is a diagram for explaining the method of constructing the communication payload.

In the case of being composed of a plurality of substreams, it is easy to discard at the transmission packet level by composing the transmission packets in the order of higher priority according to the stream priority added to the substreams. Further, even if the granularity is made fine and the information of the objects having the high frame priority is collected into one to form the communication packet, the discarding at the communication packet level becomes easy.

By associating the slice structure of an image with a communication packet, it is easy to recover when a packet is dropped. That is, by associating the slice structure of a moving image with the structure of a packet, a resync marker for resynchronization becomes unnecessary. If the slice structure does not match the structure of the communication packet, it is necessary to add a resync marker (a mark for notifying the return position) so that resynchronization can be performed when information is lost due to a packet drop or the like.

In addition to this, it is conceivable that high error protection is applied to a communication packet having a high priority. The image slice structure is a unit of image information such as GOB and MB.

FIG. 39 is a diagram for explaining a method of associating data with a communication payload. By transmitting a method of associating a stream or an object with a communication packet together with control information or data, it is possible to generate an arbitrary data format according to the communication situation or application. For example, RTP (Real time Transfer)
r Protocol), an RTP payload is defined for each encoding to be handled. The current RTP format is fixed. H. In the case of H.263, as shown in FIG.
Three data formats from ode A to mode C are defined. H. In H.263, a communication payload for a multi-resolution video format is not defined.

In the example of the figure, the Layer No. And the above-mentioned relational description (AND, OR) are added to the data format of Mode A and defined.

FIG. 40 is a diagram for explaining the correspondence between frame priority, stream priority and communication packet priority.

[0236] Further, the figure shows an example in which the priority added to the communication packet on the transmission path is the communication packet priority, and the stream priority and the frame priority are associated with the communication packet priority.

Normally, in communications using IP, it is necessary to transmit data by associating the frame priority and stream priority added to image and audio data with the priority of the lower IP packet and the packet. Image and audio data are divided and transmitted by being divided into IP packets for transmission, so it is necessary to associate priorities. In the illustrated example, the stream priority takes values from 0 to 3, and the frame priority is 0.
Since it takes a value from 5 to 5, it is 0 to 1 in the upper data.
It can take up to 5 priorities.

In IPv6, 0 out of priority (4 bits)
Nos. 1 to 7 are reserved for congestion-controlled traffic, and Nos. 8 to 15 of the priorities are reserved for real-time communication traffic or non-congestion-controlled traffic. Priority 15 has the highest priority, and priority 8 has the lowest priority. This becomes the priority at the IP packet level.

In data transmission using IP, it is necessary to associate the higher priority levels 0 to 15 with the lower level IP priority levels 8 to 15. The association may be performed by clipping a part of the higher priority, or by providing an evaluation function. The correspondence between the higher-order data and the lower-order IP priority is managed by a relay node (router, gateway, etc.) and a transmission / reception terminal.

The transmission means is not limited to IP, and may be a transmission packet having a flag indicating whether it can be discarded or not, such as ATM (TS) (transport stream) of MPEG2. .

The frame priority and stream priority described so far can be applied to a transmission medium or a data recording medium. Floppy disk as data recording medium,
It can be performed using an optical disk or the like.

Further, the recording medium is not limited to this, and it can be implemented in the same manner as long as it can record the program such as an IC card and a ROM cassette. Furthermore, an image / audio relay device such as a router or gateway that relays data may be targeted.

In addition, Stream Priority
(Priority between time-series data) and Frame Priority
By determining the time-series data to be retransmitted based on the information of the “ity” (priority within time-series data), preferential retransmission processing becomes possible. For example, when the receiving terminal is decoding based on the priority information, it is possible to prevent the retransmission of the stream or frame that is not the target of the processing.

Further, in addition to the priority which is the current processing target, the stream or frame having the priority to be retransmitted may be determined from the relationship between the number of retransmissions and the number of successful transmissions.

On the other hand, the terminal on the transmitting side also has
am Priority (time-series data priority) and F
frame Priority (Priority within time series data)
By determining the time-series data to be transmitted on the basis of the information, the preferential transmission processing becomes possible. For example, by determining the priority of streams and frames to be transmitted based on the average transfer rate and the number of retransmissions, it is possible to adaptively transmit video and audio even when the network is overloaded.

The above embodiment is not limited to two-dimensional image composition. An expression format in which a two-dimensional image and a three-dimensional image are combined may be used, or an image synthesizing method of synthesizing a plurality of images so that they are adjacent to each other such as a wide-field image (panoramic image) may be included. Further, the communication mode targeted by the present invention is not limited to the wired bidirectional CATV and B-ISDN. For example, video and audio are transmitted from the center side terminal to the home side terminal by radio waves (for example, VHF band, UHF band) or satellite broadcasting, and information is transmitted from the home side terminal to the center side terminal through an analog telephone line or N. It may be ISDN (video, audio and data do not necessarily have to be multiplexed). Also, I
A communication form using wireless such as rDA, PHS (Personal Handy Phone) or wireless LAN may be used.

Furthermore, even if the target terminal is a portable terminal such as a portable information terminal, the set top BO
It may be a desk-top type terminal such as X or a personal computer.

As described above, according to the present invention, it is possible to handle a plurality of video streams and a plurality of audio streams and to reflect the intention of the editor so that important scene cuts are mainly reproduced in synchronization with audio. Will be easier.

Embodiments of the present invention will be described below with reference to the drawings.

The embodiment described here mainly solves any of the problems (C1) to (C3) described above.

FIG. 41 shows the configuration of the transmitting apparatus according to the first embodiment. An image input terminal 2101 has an image size of, for example, 144 pixels vertically and 17 pixels horizontally.
It has 6 pixels. 2102 is a moving picture coding device,
Four components 1021, 1022, 1023, 102
4 (Recommendation H.26
1).

Reference numeral 1021 divides the input image into macroblocks (a square area of 16 pixels in the vertical direction and 16 pixels in the horizontal direction),
A switch 1022 for deciding whether to code this block by intra / inter is used to create a motion compensation image based on a locally decoded image that can be calculated from the previous coding result. And the difference between
A motion compensating unit that outputs the result in units of macroblocks. The motion compensation includes half-pel motion compensation having a long processing time and full-pel motion compensation having a short processing time. 1023
Is an orthogonal transforming means for performing DCT transformation on each macroblock, and 1024 is a variable length encoding means for performing entropy coding on the DCT transformation result and other coding information.

Reference numeral 2103 denotes a counting means, which counts the number of executions of the four constituent elements of the moving picture coding apparatus 2102,
The result is output to the conversion means for each input image. At this time,
From the motion compensating means 1022, the number of times of execution is counted for each of the two types of half pels and full pels.

Reference numeral 2104 is a converting means for outputting a data string as shown in FIG. Reference numeral 2105 denotes a transmission means, which is a variable length code from the moving picture coding device 2102,
The data string from the conversion means 2104 is multiplexed into a single data string and output to the data output terminal 2109.

With the above configuration, the receiving apparatus has essential processing (switch 1021, orthogonal transformation means 1023, variable length coding means 1024) and non-essential processing (motion compensation means 102).
It is possible to transmit the number of times of execution of 2).

Next, FIG. 48 is a flow chart of the transmission method according to the second embodiment.

Since the operation of this embodiment is similar to that of the first embodiment, the corresponding elements will be added. 8
An image is input at 01 (image input terminal 2101), and 8
At 02, the image is divided into macroblocks. After that, 80
By the conditional branch of 7, the processing from 803 to 806 is repeated until the processing for all macroblocks is completed. When each process is executed, the corresponding variable is incremented by 1 so that the number of processes 803 to 806 can be recorded in a specific variable.

First, at 803, it is determined whether the macroblock to be processed is to be coded intra / inter (switch 1021). In the case of inter, motion compensation is performed at 804 (motion compensation means 1022). Then 80
5, 806 performs DCT conversion and variable length coding (orthogonal transformation means 1023, variable length coding means 102).
4). When the processing for all the macroblocks is completed (when Yes in 807), in 808, the variable indicating the number of execution times corresponding to each processing is read and a data string as shown in FIG. 2 is generated, and this data is generated. The column and code are multiplexed and output. The above processing from 801 to 808 is repeatedly executed as long as the input image continues.

With the above configuration, the number of times each process is executed can be transmitted.

Next, FIG. 43 shows the configuration of the receiving apparatus according to the third embodiment.

In the figure, 307 is an input terminal for inputting the output of the transmitter of the first embodiment, and 301 is a variable length code and data based on the output of the transmitter of the first embodiment. It is a receiving means for extracting and outputting a column by demultiplexing, and at this time, the time required to receive one sheet of data is measured and this is also output.

Reference numeral 303 denotes a moving picture decoding device which receives a variable length code as an input, and is composed of five constituent elements. 303
Reference numeral 1 is a variable length decoding means for extracting the DCT coefficient and other encoded information from the variable length code, 3032 is an inverse orthogonal transformation means for performing an inverse DCT transformation process on the DCT coefficient, 30
Reference numeral 33 denotes a switch, which performs an operation of allocating the output to the upper and lower sides on the basis of the encoding information indicating whether the macroblock is encoded as intra or inter. 3034
Is a motion compensating means, which creates a motion compensating image using the previous decoded image and motion coding information, adds the output of the inverse orthogonal transforming means 3032 to this image, and outputs it. Thirty
Reference numeral 35 is an execution time measuring means, which measures the execution time from the input of the variable length code to the decoding device 303 to the completion of the decoding and output of the image, and outputs it. Thirty
2 is the number of executions of each element (variable length decoding means 3031, inverse orthogonal transformation means 3032, switcher 3033, motion compensation means 3034) from the data string from the reception means 301, and execution time from the execution time measurement means 3035. Is an estimation means for estimating the execution time of each element.

For example, if linear regression is used as the estimation method, the estimated execution time may be the objective variable y, and the number of executions of each element may be the explanatory variable x_i. In this case, the regression parameter a
_I can be regarded as the execution time of each element. Also,
In the case of linear regression, it is necessary to store a large amount of past data, which consumes a large amount of memory, but if this is disliked, estimation of internal state variables by a Kalman filter may be used. In this case, the observed value is the execution time, the execution time of each element is the internal state variable, and the observation matrix C
Can be considered as a case in which the number of executions of each element changes for each step. Reference numeral 304 denotes a number-of-times reducing means for changing the number of executions of each element so as to reduce the number of executions of full-pel motion compensation and increase the number of executions of half-pel motion compensation by a considerable number. The calculation method of this considerable number is as follows.

First, the execution times are estimated by receiving the execution times and estimated execution times of each element from the estimating means 302.
If this time exceeds the time required to receive the data from the receiving means 301, the number of full-pel motion compensation executions is increased and the number of half-pel motion compensation executions is reduced until the time is not exceeded. A decoded image output terminal 306 is provided.

The motion compensating means 3034 is in the case where it is instructed to perform half-pel motion compensation from the coded information. However, when the number of times of execution of half-pel motion compensation is exceeded, half-pel motion is compensated. Round and perform full-pel motion compensation as full-pel motion.

The first embodiment and the third embodiment described above
According to the embodiment of the present invention, the execution time of the decoding process is predicted from the estimated execution time of each element, and it seems that this exceeds the time (specified time) required to receive one sheet of data. For example, half-pel motion compensation, which has a long execution time, is replaced with full-pel motion compensation. As a result, the execution time can be prevented from exceeding the specified time, and the problem (C1) can be solved.

It should be noted that, in the IDCT calculation in the receiving device, by eliminating the use of high frequency components, the IDC
The processing time of T calculation can be reduced. That is, ID
Of the CT calculations, the calculation of the low frequency component may be regarded as an essential process and the calculation of the high frequency component may be regarded as a non-essential process, so that the number of calculations of the high frequency component in the IDCT calculation may be reduced.

Next, FIG. 49 is a flowchart of the receiving method according to the fourth embodiment.

Since the operation in this embodiment is similar to that in the third embodiment, the corresponding elements will be additionally described. At step 901, a variable a_ representing the execution time of each element
i is initialized (estimation means 302). At 902, multiplexed data is input and the time required for this is measured (reception means 301). At 903, this multiplexed data is separated into a variable length code and a data string and output (reception means 30.
1). At 904, each execution count is extracted from the data string (FIG. 2), and these are set to x_i. At 905, the actual number of executions is calculated from the execution time a_i of each element and each number of executions x_i (number reduction means 304). At 906, measurement of the execution time of the decryption process is started, at 907 a decryption process routine to be described later is started, and then at 908.
Then, the measurement of the execution time of the decoding process is finished (the moving image decoding device 303, the execution time measuring means 3035). 90
In 8, the execution time of each element is estimated from the execution time of the decoding processing in 908 and the actual number of executions of each element in 905, and a_i is updated (estimation means 302). The above processing is executed for each input multiplexed data.

In the decryption processing routine 907, 9
Variable length decoding is performed at 10 (variable length decoding means 303
1) and 911 perform inverse orthogonal transform (inverse orthogonal transform means 3
032) and 912, the processing branches at the intra / inter information extracted in the processing at 910 (switch 303
3). In the case of inter, motion compensation is performed at 913 (motion compensation means 3034). In this step 913, the number of executions of half-pel motion compensation is counted, and if it exceeds the actual number of executions obtained in 905, half-pel motion compensation is replaced with full-pel motion compensation and executed. After the above processing is completed for all macroblocks (step 914), this routine ends.

The second embodiment and the fourth described above
According to the embodiment of the present invention, the execution time of the decoding process is predicted from the estimated execution time of each element, and it seems that this exceeds the time (specified time) required to receive one sheet of data. For example, half-pel motion compensation, which has a long execution time, is replaced with full-pel motion compensation. As a result, the execution time can be prevented from exceeding the specified time, and the problem (C1) can be solved.

Next, FIG. 44 shows the structure of a receiving apparatus according to the fifth embodiment.

Most of the components of this embodiment are the same as those described in the second embodiment, and only two components are added and one component is modified. explain.

Reference numeral 402 is a modification in which the execution time of each element obtained as a result of the estimation by the estimating means 302 described in the second embodiment is output separately from the output to the number-of-times limiting means 304. Reference numeral 408 denotes a transmission means, which generates a data string as shown in FIG. 45 from the execution time of each element and outputs this. The maximum execution time is about 65 when expressed in 16 bits in units of microseconds.
It's enough because you can express the millisecond. 409
Is an output terminal for sending this data string to the transmitting means.

The receiving method corresponding to the fifth embodiment may have the step of generating a data string as shown in FIG. 45 added immediately after 808 in FIG.

Next, FIG. 46 shows the structure of the transmitting apparatus according to the sixth embodiment.

Most of the constituent elements of this embodiment are the same as those described in the first embodiment, and only the addition of two constituent elements will be described. Reference numeral 606 denotes an input terminal for receiving a data string output by the receiving device of the third embodiment, and 607 receives this data string,
The receiving means outputs the execution time of each element. 608
Is a determining means for determining the number of times of execution of each element, and the procedure is as follows. First, the processing in the switch 1021 is performed for all macroblocks in the image,
The number of times the switch 1021 is executed at this point is calculated. Further, the number of executions in the motion compensation means 1022, the orthogonal transformation means 1023, and the variable length coding means 1024 after this is
It can be uniquely determined by the processing results up to this point.
Therefore, using the number of times of execution and the execution time from the receiving unit 607, the execution time required for decoding on the receiving device side is predicted. This predictive decoding time is obtained as the sum of the product of the execution time of each element and the number of executions for each element. Then, the predictive decoding time is the code amount (for example, 16 kb) to be generated in the current image designated by the rate controller or the like.
ts) transmission time (for example, transmission speed is 64
If it is more than 250 msec for kbit / sec),
Increase the number of executions of full-pel motion compensation and reduce the number of executions of half-pel motion compensation so that the decoding time does not exceed the time required for transmission. (Because full-pel motion compensation has a shorter execution time, reduce the number of times. You can reduce the execution time with.

The moving picture coding device 2102 performs each processing based on the number of executions designated by the determining means 608. For example, if the motion compensating hand 1022 completes the half-pel motion compensation for the designated number of times of half-pel motion compensation execution, then only the full-pel motion compensation will be performed.

Also, the selection method may be devised so that the half-pel motion compensation is uniformly distributed in the image. For example, first, all macroblocks that require half-pel motion compensation are obtained, and this number (for example, 12) is divided by the number of half-pel motion compensation execution times (for example, 4) to obtain a quotient (3), and half-pel motion compensation is performed. It is also possible to apply the half-pel motion compensation only to those (0, 3, 6, 9) whose order from the beginning of the required macroblock is divisible by this quotient.

The fifth embodiment and sixth embodiment described above
According to the embodiment of the present invention, the estimated execution time of each element is transmitted to the transmission side, the transmission side predicts the execution time of the decoding process, and this is required to receive one sheet of data. Replace the half-pel motion compensation, which has a long execution time, with full-pel motion compensation so as not to exceed the expected time (specified time). This makes it possible to prevent half-pel motion compensation information in the sent encoded information from being discarded, prevent the execution time from exceeding the specified time, and solve the problem (C2).

In the non-essential processing, inter-macroblock coding is performed with normal motion compensation, 8 × 8 motion compensation,
It may be divided into three parts of overlap motion compensation.

Next, FIG. 50 is a flow chart of the transmission method according to the seventh embodiment.

Since the operation in this embodiment is similar to that in the sixth embodiment, the corresponding elements will be additionally described. 1
At 001, the initial value of the execution time of each process is set. 8
An image is input at 01 (input terminal 2101), and the image is divided into macroblocks at. At 1002, it is determined which of the macroblocks is to be coded, intra / inter (switch 1021). As a result,
Since the number of executions of each process from 1005 to 806 is known, in 1003, the actual number of executions is calculated from this number of executions and the execution time of each process (determination means 60).
8).

Thereafter, the conditional branch of 807 is performed until the processing for all macroblocks is completed until 1005
The processing from 1 to 806 is repeated.

When each process is executed so that the number of processes 1005 to 806 can be recorded in a specific variable, the corresponding variable is incremented by one. First, at 1005, the process branches based on the determination result at 1002 (switch 1021). In the case of inter, motion compensation is performed at 804 (motion compensation means 102).
2). Here, the number of times of half-pel motion compensation is counted, and if it exceeds the actual number of executions obtained in 1003, half-pel motion compensation is not executed but full-pel motion compensation is executed instead. After that, at 805 and 806,
DCT transform and variable length coding are performed (orthogonal transform means 10
23, variable length coding means 1024). When the processing for all the macroblocks is completed (when Yes in 807), in 808, the variable indicating the number of execution times corresponding to each processing is read and a data string as shown in FIG. 2 is generated, and this data is generated. The column and code are multiplexed and output. 10
In 04, the data string is received, and the execution time of each process is extracted and set.

The above processing from 801 to 1004 is
Repeatedly executed as long as the input image continues.

According to the paragraph starting with “Mata” at the end of the explanation part of the fifth embodiment and the seventh embodiment described above, the estimated execution time of each element is transmitted. To the sender side, the sender side predicts the execution time of the decoding process, and the half-pel with a long execution time does not exceed the time (specified time) that would be required to receive one sheet of data. Replace motion compensation with full-pel motion compensation. This makes it possible to prevent half-pel motion compensation information in the sent encoded information from being discarded, prevent the execution time from exceeding the specified time, and solve the problem (C2).

Next, FIG. 47 shows the structure of the transmitting apparatus according to the eighth embodiment.

Most of the constituent elements of this embodiment are the same as those described in the first embodiment, and only the addition of four constituent elements will be described.

Reference numeral 7010 denotes an execution time measuring means, which measures the execution time from when the image is input to the encoding device 2102 to when the encoding of the image and the output of the code are completed, and outputs this. Reference numeral 706 denotes each element (switch 1021, motion compensation means 10) from the data string from the counting means 2103.
22, orthogonal transformation means 1023, variable length decoding means 102
It is an estimating means that receives the number of executions of 4) and the execution time from the execution time measuring means 7010 and estimates the execution time of each element. The estimation method may be the same as that described in the estimation means 302 of the second exemplary embodiment. Reference numeral 707 denotes an input terminal for inputting a frame rate value from the user, 70
Reference numeral 8 is a determining means for obtaining the number of times of execution of each element, and the procedure is as follows.

First, the switch 1021 processes all macroblocks in the image, and the number of executions of the switch 1021 at this point is obtained. Further, the number of times of execution thereafter by the motion compensation means 1022, the orthogonal transformation means 1023, and the variable length coding means 1024 can be uniquely determined by the processing results up to this point. Next, the sum of the products of the number of executions and the estimated execution time of each element from the estimation means 706 is calculated for each element to calculate the predictive coding time. Then, if the predictive coding time is equal to or longer than the time that can be used to code one image obtained from the reciprocal of the frame rate from 707, the number of full-pel motion compensation executions is increased to determine the number of half-pel motion compensation executions. cut back.

This increase / decrease processing and the calculation of the predictive coding time are repeated until the predictive coding time becomes less than or equal to the usable time.
By repeating, the number of times of execution is determined.

The moving picture coding device 2102 performs each processing based on the number of executions designated by the determining means 608. For example, if the motion compensating hand 1022 completes the half-pel motion compensation for the designated number of times of half-pel motion compensation execution, then only the full-pel motion compensation will be performed.

Further, the selection method may be devised so that the half-pel motion compensation is uniformly distributed in the image. For example, first, all macroblocks that require half-pel motion compensation are obtained, and this number (for example, 12) is divided by the number of half-pel motion compensation execution times (for example, 4) to obtain a quotient (3), and half-pel motion compensation is performed. It is also possible to apply the half-pel motion compensation only to those (0, 3, 6, 9) whose order from the beginning of the required macroblock is divisible by this quotient.

According to the eighth embodiment described above, the execution time of each process is estimated, the execution time required for encoding is predicted in advance based on this estimated execution time, and the predicted encoding time is The problem (C3) can be solved by determining the number of executions so that the time is equal to or less than the time that can be used for encoding the image determined by the frame rate.

In order to detect the motion vector, the motion compensating means 1022 detects the vector having the smallest SAD (sum of absolute differences for each pixel) among the vectors in the range of 15 pixels on the left, right, upper and lower sides. There is a full search motion vector detection method, but there is also a 3step motion vector detection method (see H.261, an.
next. Described in). This is done by selecting nine points having an even arrangement relationship in the search range and selecting the point having the smallest SAD. Next, in the fitted range near this point,
Select 9 points and select the point with the smallest SAD. The 3step motion vector detection method is to execute such processing again.

These two methods are regarded as non-essential processing methods, the execution times are estimated, the execution times required for encoding are predicted based on the estimated execution times, and the predicted execution times are less than the user-specified time. As described above, the number of executions of the full search motion vector detection method may be appropriately reduced, and the number of executions of the 3step motion vector detection method may be increased instead.

Further, in addition to the 3step motion vector detection method, a motion vector detection method based on a fixed number of searches that further simplifies the process or a motion vector detection method that returns only a (0,0) motion vector as a result is used. good.

[0299] Next, FIG. 51 is a flow chart of the transmission method according to the ninth embodiment.

Since the operation of this embodiment is similar to that of the eighth embodiment, the corresponding elements will be added. For detailed operations in each flow, refer to the corresponding element description. Also, since it is almost the same as the second embodiment,
Only the differences will be described.

At 1101, the initial value of the execution time of each process is set in the variable a_i. The frame rate is input at 1102 (input terminal 707). 1103 is 110
2 frame rate, execution time of each process a_i, 100
The actual number of executions is determined from the number of executions of each process obtained from the intra / inter determination result in 2 (determination means 7
08). 1105 and 1106 are for measuring the execution time of the encoding process. 1104 estimates the execution time of each process from the execution time of 1106 and the actual number of executions of each process, and updates the variable a_i (estimation unit 70).
6).

According to the ninth embodiment described above, the execution time of each process is estimated, the execution time required for encoding is predicted in advance based on this estimated execution time, and the predicted encoding time is The problem (C3) can be solved by determining the number of executions so that the time is equal to or less than the time that can be used for encoding the image determined by the frame rate.

In the second embodiment, 808
At the time of generating the data string in, the 2-byte area may be added immediately after the start code shown in FIG. 2 and the binary representation of the code length may be added here.

Furthermore, in the fourth embodiment, 90
At the time of inputting the multiplexed data in 2, the code length is extracted from this 2-byte area, and the code transmission time obtained from this code length and the code transmission speed is used for the execution count calculation in 905. However, the number of executions of half-pel motion compensation is reduced so that the code transmission time is not exceeded.

In the first embodiment, 210
It is also possible to add a 2-byte area immediately after the start code shown in FIG. 2 at the time of generating the data string in 4, and add a binary representation of the code length here.

Furthermore, in the third embodiment, 30
At the time of inputting the multiplexed data in 1, the code length is extracted from this 2-byte area, and the code transmission time obtained from this code length and the code transmission speed is used for the execution count calculation in 304. However, the number of executions of half-pel motion compensation is reduced so that the code transmission time is not exceeded.

Also, in the fourth embodiment, 909
Immediately after that, the actual number of times half-pel motion compensation is performed is recorded and the maximum value thereof is calculated. If this maximum value is less than a sufficiently small value (for example, 2 or 3), a data string (data string consisting of a specific bit pattern) indicating that half-pel motion compensation is not used is generated and transmitted. You may. Furthermore, in the second embodiment,
Immediately after 808, it is confirmed whether or not this data string is received, and if a data string indicating that half-pel motion compensation is not used is received, the motion compensation process is always set to full-pel motion compensation in 808. Is also good.

Further, this idea can be applied in addition to motion compensation. For example, by not using high frequency components in DCT calculation, the processing time of DCT calculation can be reduced. That is, in the receiving method, when the ratio of the execution time of the IDCT calculation to the total execution time exceeds a certain value, a data string indicating that fact is transmitted to the transmission side. On the transmitting side, when this data string is received, D
In the CT calculation, only the low frequency component may be calculated and all the high frequency components may be set to zero.

Further, although the embodiment has been described here using the image, each of the above methods may be applied to a sound other than the image.

Also, in the third embodiment, 303
At 4, the actual number of executions of half-pel motion compensation is recorded and the maximum value thereof is calculated. If this maximum value is less than a sufficiently small value (for example, 2 or 3), a data string (data string consisting of a specific bit pattern) indicating that half-pel motion compensation is not used is generated and transmitted. You may. Furthermore, in the first embodiment,
When a data string indicating that half-pel motion compensation is not used is received, the motion compensation process at 1022 may always be full-pel motion compensation.

Further, this idea can be applied to other than motion compensation. For example, by not using high frequency components in DCT calculation, the processing time of DCT calculation can be reduced. That is, in the receiving method, when the ratio of the execution time of the IDCT calculation to the total execution time exceeds a certain value, a data string indicating that fact is transmitted to the transmission side.

On the transmitting side, when this data string is received, only low frequency components may be calculated in the DCT calculation, and all high frequency components may be set to zero.

Furthermore, although the embodiment has been described here using images, the above method may be applied to sounds other than images.

As is clear from the above description,
For example, according to the first embodiment and the third embodiment,
Predict the execution time of the decoding process from the estimated execution time of each element, and if this exceeds the time (specified time) required to receive one sheet of data, the half-pel with a long execution time Replace motion compensation with full-pel motion compensation. As a result, the execution time can be prevented from exceeding the specified time, and the problem (C1) can be solved.

Further, for example, according to the fifth and seventh embodiments, the estimated execution time of each element is transmitted to the transmission side, and the transmission side predicts the execution time of the decoding process. Then, half-pel motion compensation, which has a long execution time, is replaced with full-pel motion compensation so that this does not exceed the time (specified time) that would be required to receive one sheet of data. This makes it possible to prevent half-pel motion compensation information in the sent encoded information from being discarded, prevent the execution time from exceeding the specified time, and solve the problem (C2).

Further, for example, according to the ninth embodiment,
Estimate the execution time of each process, based on this estimated execution time,
Execution time required for encoding is predicted in advance, and the number of executions is determined so that the estimated encoding time is equal to or less than the time that can be used for encoding an image, which is determined by the frame rate. Can be resolved.

As described above, according to the present invention, a function (CGD: Comput) for gradually lowering the quality even if the calculation load increases.
(Ational Graceful Degradation) can be realized, and the benefits of implementation are enormous.

Further, a computer is made to execute all or some of the steps (or means) described in any one of the embodiments described above or the operation of each means. A magnetic recording medium that records the program,
A recording medium such as an optical recording medium may be created and the same operation as described above may be performed by a computer using the recording medium.

[0319]

As described above, according to the present invention,
For example, it is possible to change the information framework according to the situation by dynamically determining the framework of each data management information, transmission management information, and control information used by the transmitting terminal and the receiving terminal. It can be changed according to the transmission path.

[Brief description of drawings]

FIG. 1 is a schematic configuration diagram of an image / sound transceiver according to an embodiment of the present invention.

FIG. 2 is a diagram showing a reception management unit and a separation unit.

FIG. 3 is a diagram showing a method of transmitting and controlling images and sounds using a plurality of logical transmission paths.

FIG. 4 is a diagram showing a method for dynamically changing header information added to image or audio data to be transmitted.

5A to 5B are diagrams showing a method of adding AL information.

6A to 6D are diagrams showing an example of a method of adding AL information.

FIG. 7 is a diagram showing a method of dynamically multiplexing and separating a plurality of logical transmission lines to transmit information.

FIG. 8 is a diagram showing a transmission procedure of a broadcast program.

FIG. 9A is a diagram showing a method of transmitting an image or sound in consideration of a reading time of the program or data and a start-up time when the program or data is present in the receiving terminal, and FIG. 9B: The program or data is transmitted. Diagram showing the method of transmitting images and sound in consideration of the start-up time when reading programs and data when

FIG. 10A to FIG. 10B are diagrams showing a method of dealing with zapping.

FIG. 11 is a diagram showing a specific example of a protocol that is actually transmitted and received between terminals.

FIG. 12 is a diagram showing a specific example of a protocol that is actually transmitted and received between terminals.

FIG. 13 is a diagram showing a specific example of a protocol that is actually transmitted and received between terminals.

FIG. 14 is a diagram showing a specific example of a protocol that is actually transmitted and received between terminals.

FIG. 15 is a diagram showing a specific example of a protocol that is actually transmitted and received between terminals.

FIG. 16 is a diagram showing a specific example of a protocol that is actually transmitted and received between terminals.

FIG. 17 is a diagram showing a specific example of a protocol that is actually transmitted and received between terminals.

FIG. 18 is a diagram showing a specific example of a protocol that is actually transmitted and received between terminals.

FIG. 19 is a diagram showing a specific example of a protocol that is actually transmitted and received between terminals.

FIG. 20 is a diagram showing a specific example of a protocol that is actually transmitted and received between terminals.

FIG. 21 is a diagram showing a specific example of a protocol that is actually transmitted and received between terminals.

FIG. 22 is a diagram showing a specific example of a protocol that is actually transmitted and received between terminals.

FIG. 23 is a diagram showing a specific example of a protocol that is actually transmitted and received between terminals.

FIG. 24 is a diagram showing a specific example of a protocol that is actually transmitted and received between terminals.

FIG. 25 (a)-(b): CGD demo system configuration diagram of the present invention

FIG. 26 is a block diagram of the CGD demo system of the present invention.

FIG. 27 is a diagram showing a method of adding a priority when an encoder is overloaded.

FIG. 28 is a diagram describing a method of determining priority at a receiving terminal when overloaded.

FIG. 29 is a diagram showing a temporal change in priority.

FIG. 30 is a diagram showing stream priority and object priority.

FIG. 31 is a schematic configuration diagram of an image encoding / decoding device according to an embodiment of the present invention.

FIG. 32 is a schematic configuration diagram of a voice encoding / decoding device according to an embodiment of the present invention.

FIG. 33 (a)-(b): Diagrams showing a priority addition unit and a priority determination unit for managing the priority of processing at the time of overload.

FIG. 34 is a diagram showing a granularity for adding a priority.

FIG. 35 is a diagram showing a granularity for adding a priority.

FIG. 36 is a diagram showing a granularity for adding a priority.

FIG. 37 is a diagram showing a method of assigning priorities to multi-resolution image data.

FIG. 38 is a diagram showing a configuration method of a communication payload.

FIG. 39 is a diagram showing a method of associating data with a communication payload.

FIG. 40 is a diagram showing correspondence between object priority, stream priority, and communication packet priority.

FIG. 41 is a configuration diagram of a transmission device according to the first embodiment of the present invention.

FIG. 42 is an explanatory diagram of the first embodiment.

FIG. 43 is a configuration diagram of a reception device according to a third embodiment of the present invention.

FIG. 44 is a configuration diagram of a receiving device according to a fifth embodiment of the present invention.

FIG. 45 is an explanatory diagram of the fifth embodiment.

FIG. 46 is a configuration diagram of a transmission device according to a sixth embodiment of the present invention.

FIG. 47 is a configuration diagram of a transmission device according to an eighth embodiment of the present invention.

FIG. 48 is a flowchart of a transmission method according to the second embodiment of the present invention.

FIG. 49 is a flowchart of a receiving method according to the fourth embodiment of the present invention.

FIG. 50 is a flowchart of a transmission method according to the seventh embodiment of the present invention.

FIG. 51 is a flowchart of a transmission method according to the ninth embodiment of the present invention.

FIG. 52 is a configuration diagram showing an example of an image / sound transmission device of the present invention.

FIG. 53 is a block diagram showing an example of an image / sound receiving apparatus of the present invention.

FIG. 54 is a diagram for explaining priority adding means for adding priority to video and audio of the image / audio transmitting apparatus of the present invention.

FIG. 55 is a diagram for explaining a priority determining unit that interprets the priority added to the video and audio of the image / audio receiving apparatus of the present invention and determines whether or not decoding processing is possible.

[Explanation of symbols]

11 Reception management unit 12 Separation part 13 Transmitter 14 Image decompression section 15 Image expansion management unit 16 Image composition section 17 Output section 18 Terminal control unit 301 Receiving means 302 Estimating means 303 Video decoding device 304 Number of times reduction means 306 output terminal 307 input terminal 3031 Variable length decoding means 3032 inverse orthogonal transform means 3033 Switch 3034 Motion compensation means 3035 Execution time measuring means 4011 Transmission management unit 4012 image coding unit 4013 Reception management unit 4014 Image decoding unit 4015 Image synthesizer 4016 Output section 4101 Image coding apparatus 4102 image decoding apparatus

   ─────────────────────────────────────────────────── ─── Continued front page    (31) Priority claim number Japanese Patent Application No. 9-226027 (32) Priority date August 22, 1997 (August 22, 1997) (33) Priority claiming country Japan (JP) (31) Priority claim number Japanese Patent Application No. 9-226045 (32) Priority date August 22, 1997 (August 22, 1997) (33) Priority claiming country Japan (JP) (31) Priority claim number Japanese Patent Application No. 9-332101 (32) Priority date December 2, 1997 (December 2, 1997) (33) Priority claiming country Japan (JP) (72) Inventor Hiroshi Arakawa             1006 Kadoma, Kadoma-shi, Osaka Matsushita Electric             Sangyo Co., Ltd. F-term (reference) 5C059 MA00 MA05 MA14 MA23 MC38                       ME01 NN15 NN21 PP01 PP05                       PP06 PP07 PP12 RB01 RB09                       RC01 RC02 RC04 RE02 RF23                       SS01 SS06 TA07 TA43 TA61                       TA62 TA75 TA76 TC19 TC22                       TC36 TC37 TD07 TD11 TD14                       UA02                 5K034 AA17 CC02 DD01 EE10 MM22

Claims (6)

[Claims] 1. When (1) the actual transfer rate of image or audio information exceeds a target transfer rate, or (2) the elapsed time from the start of transfer processing and the information added to the encoded information. As a result of comparison with the time to be decoded or output, when it is determined that the writing of the encoded information to the transmission buffer is delayed, the priority added to the encoded information The data processing method is characterized in that the transmission of the information is thinned out by using the degree. 2. When (1) the actual transfer rate of image or audio information exceeds the target transfer rate, or (2) the elapsed time from the start of transfer processing and the information added to the encoded information. As a result of comparison with the time to be decoded or output, when it is determined that the writing of the encoded information to the transmission buffer is delayed, the priority added to the encoded information The data processing device is characterized in that the transmission of the information is thinned out by using the degree. 3. The divided time series data and the priority information thereof are paired and sequentially input, and (1) when the information of the divided time series data is damaged, the damaged data Re-transmission request processing is performed to request re-transmission, and (2) when the divided time-series data is lost continuously or at high frequency, the re-transmission request is performed only for data with high priority. A data processing method characterized by performing processing. 4. The divided time series data and the priority information thereof are paired and sequentially input, and (1) when the information of the divided time series data is damaged, the damaged data Re-transmission request processing is performed to request re-transmission, and (2) when the divided time-series data is lost continuously or at high frequency, the re-transmission request is performed only for data with high priority. A data processing device characterized by performing processing. 5. The divided time-series data and priority information thereof are paired and sequentially output, and the high-priority data is prioritized in accordance with the transmission amount of the divided time-series data. A data processing method characterized by transmitting. 6. The divided time-series data and priority information thereof are sequentially output as a pair, and the high-priority data is prioritized in accordance with the transmission amount of the divided time-series data. A data processing device characterized by transmitting.