JP2003008640A - Stream distribution system - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a stream distribution system in which a transmitting end and relay devices actively discard some parts of a multimedia stream so as to match a stream data transmission rate and a network bandwidth. SOLUTION: A multimedia stream transmitted from a sending device 10 is sent to a receiving device 16 through a network A11, a relay device A12, a network B13, a relay device B14, a network C15. Considering the arrival time of the multimedia stream to the receiving device 16 notified from the receiving device 16, the sending device 10, the relay devices A12 and B14 discard some parts of the multimedia stream in accordance with the bandwidths of the networks A11, B13 and C15. The receiving device 16 decodes the stream with no missing data.

Description

Detailed Description of the Invention

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a stream distribution system, and in particular, it distributes stream data such as audio and moving images that can be reproduced when a part of the data files are read without downloading the entire data file. Stream distribution system.

[0002]

2. Description of the Related Art Reproducing a data file such as voice data or moving image data in real time while downloading it on a network such as the Internet is called "streaming". In addition, data in such a data format that can be downloaded and reproduced at the same time is called stream data.

Audio data, moving image data and the like are generally downloaded to a storage medium such as a hard disk and then reproduced. Therefore, the user has to wait until the entire data file such as audio data or moving image data is downloaded. On the other hand, stream data such as audio and moving images can be downloaded and reproduced almost at the same time, so that the user does not have to wait for the reproduction until the entire data file is downloaded.

In the conventional stream distribution method, the receiving device side immediately decodes and reproduces the received stream data at the time of receiving the stream data such as the voice and the moving image transmitted from the transmitting device side. Without
The received fixed amount of stream data is accumulated in the buffer memory, and then the accumulated fixed amount of stream data is decoded and reproduced.

After starting the decoding on the receiving device side, the receiving device does not underflow the stream data due to insufficient processing capacity of the transmitting device or insufficient bandwidth of the network. In response, an instruction is issued to partially discard the stream data and reduce the stream data rate.

[0006]

However, in the conventional method, the transmitting device passively reduces the stream data rate after receiving the instruction from the receiving device. Therefore, the stream data is transmitted according to the bandwidth of the network. There is a large time lag when matching rates. Also,
The stream data once sent to the network continues to exist on the network until it is received by the transmission destination (reception device side), and there is a drawback that resources on the network may be wasted.

The present invention has been made in view of such a situation, and it is possible to reduce the time lag when the stream data rate corresponds to the network bandwidth and to effectively use the resources on the network. It is something to do.

[0008]

A stream distribution system according to claim 1 is a stream transmitting apparatus for transmitting a multimedia stream to a network, a stream receiving apparatus for receiving the multimedia stream, a stream transmitting apparatus and a stream receiving apparatus. A stream distribution system in which a stream relay device that relays stream transmission between devices is connected via a network, and a stream transmission device transmits a multimedia stream via the network, and then a reception device operates. A network delay time detecting means for detecting a network delay time until the multimedia stream is received, and the reception of the multimedia stream by the stream receiving apparatus may be performed before the multimedia stream is reproduced by the stream receiving apparatus. If the first determining means for determining whether or not the first receiving means determines that the reception of the multimedia stream by the stream receiving apparatus is not in time for the reproduction of the multimedia stream by the stream receiving apparatus, First control means for partially discarding the media stream, transmitting the remaining multimedia stream through the network, and controlling the data rate of the multimedia stream to correspond to the bandwidth of the network. The relay device includes a stream reproduction time detection means for detecting a reproduction time of the multimedia stream, and a second judgment for whether or not the reception of the multimedia stream by the stream reception device is in time for the reproduction of the multimedia stream by the stream reception device. Judgment means If the second determining means determines that the reception of the multimedia stream by the stream receiving apparatus is not in time for the reproduction of the multimedia stream by the stream receiving apparatus, the multimedia stream is partially discarded and the remaining multimedia stream is discarded. Second stream control means for transmitting the media stream through the network and controlling the data rate of the multimedia stream so as to correspond to the bandwidth of the network, wherein the stream receiving device receives the multimedia received through the network. Detecting means for detecting whether or not a part of the decoding unit of the multimedia stream is discarded by the stream transmitting device or the stream relay device from the stream, and each data forming the multimedia stream received via the network. Among the code units, a part of the decoding unit is provided with a decoding unit that decodes a unit that has not been detected as being discarded by the stream transmission device or the stream relay device. Further, the network delay time detecting means compares the time when the multimedia stream is transmitted to the stream receiving device via the network with the time when the multimedia stream reaches the stream receiving device via the network, The delay time is detected, and the first determination means determines the estimated arrival time of the multimedia stream transmitted from the stream transmission device to the stream reception device based on the reproduction time information of the multimedia stream itself, as the reproduction time information. When it is determined that the playback time indicated by is not earlier than the predetermined reference time, the first control means partially discards the multimedia stream and leaves the remaining multimedia stream via the network. To the stream receiver It may be adapted to controlled so. Further, the second determination means compares the reproduction time information of the multimedia stream itself with the internal time information of the stream relay device, and the internal time information of the stream relay device becomes the reproduction time indicated by the reproduction time information. On the other hand, if it is determined that it is not earlier than the predetermined reference time, the second
The control means may partially discard the multimedia stream and control the remaining multimedia stream to be transmitted to the stream receiving device via the network. Further, the stream receiving device includes an arrival time information transmitting means for transmitting to the stream transmitting device arrival time information indicating a time at which the multimedia stream transmitted from the stream transmitting device via the network, and the multimedia stream. Stream discarding detection means for detecting that the stream has been discarded, and the decoding unit including the discarded portion in the stream transmitter or stream relay device is discarded, and the data rate of the multimedia stream is set to the network bandwidth. It is possible to further include a third control means for controlling so as to correspond. The stream transmission device according to claim 5 is a stream transmission device that transmits a multimedia stream to a stream reception device via a network, and is connected to the network after transmitting the multimedia stream via the network. Network delay time detecting means for detecting a network delay time until the stream receiving device receives the multimedia stream, and reception of the multimedia stream by the stream receiving device,
The determining unit that determines whether or not the multimedia stream can be played back in time, and the determination unit that determines that the multimedia stream reception by the stream receiving apparatus is not in time for playing the multimedia stream Control means for partially discarding, transmitting the remaining multimedia stream through the network, and controlling so as to correspond to the bandwidth of the network, and the network delay time detecting means includes the multimedia stream through the network. The network delay time is detected by comparing the time when the packet of the predetermined stream transmission protocol for transmitting the packet is transmitted with the time when the packet reaches the stream receiving device, and the multimedia stream itself is detected by the determining means. Have Based on the raw time information, when it is determined that the estimated arrival time of the packet transmitted from the stream transmission device to the stream reception device is not earlier than a predetermined reference time with respect to the reproduction time indicated by the reproduction time information, The control means is characterized by partially discarding the multimedia stream, transmitting the remaining multimedia stream through the network, and controlling the bandwidth so as to correspond to the bandwidth of the network. The stream relay apparatus according to claim 6 is a stream relay apparatus that relays a multimedia stream transmitted from a stream transmitting apparatus via a network to a stream receiving apparatus, and based on a time when the multimedia stream is relayed, According to the stream relay time determining means for determining whether or not the relay of the multimedia stream to the stream receiving apparatus is in time for the reproduction of the multimedia stream by the stream receiving apparatus, and the determination result of the stream relay time determining means And a control means for partially discarding the stream and transmitting the remaining multimedia stream via the network.The stream relay time determination means determines the multimedia stream itself or the stream transmission program of the multimedia stream. The reproduction time information of the protocol is compared with the internal time information indicating the internal time of the stream relay device, and the internal time indicated by the internal time information is defined in advance with respect to the reproduction time indicated by the reproduction time information. When it is determined that the multimedia stream is not earlier than the reference time, the control means may partially discard the multimedia stream and transmit the remaining multimedia stream through the network to control the bandwidth so as to correspond to the network bandwidth. Can be The stream receiving device according to claim 7 is a stream receiving device that receives and plays back a multimedia stream transmitted from the stream transmitting device via a network and a stream relaying device, the stream transmitting device via the network. Arrival time information transmission means for transmitting arrival time information indicating the arrival time of the transmitted multimedia stream to a stream transmission device, and stream partial discard detection means for detecting that the multimedia stream has been partially discarded. And a third control means for discarding the decoding unit including the discarded portion in the stream transmission device or the stream relay device and controlling the data rate of the multimedia stream so as to correspond to the bandwidth of the network. Is characterized by. The stream delivery method according to claim 8 relays stream transmission between a stream transmission device that transmits a multimedia stream to a network, a stream reception device that receives the multimedia stream, and a stream transmission device and a stream reception device. A stream distribution method in a stream distribution system in which a stream relay apparatus is connected via a network, wherein a stream transmission apparatus transmits a multimedia stream via a network and a reception apparatus receives the multimedia stream. The network delay time detection step of detecting the network delay time until the start of the communication and the reception of the multimedia stream by the stream receiving apparatus are in time for the reproduction of the multimedia stream by the stream receiving apparatus. In the first determination step of determining whether or not it is determined in the first determination step that the reception of the multimedia stream by the stream receiving apparatus is not in time for the reproduction of the multimedia stream by the stream receiving apparatus, A first control step of partially discarding the media stream, transmitting the remaining multimedia stream through the network, and controlling the data rate of the multimedia stream to correspond to the bandwidth of the network. The relay device determines whether or not the stream reproduction time detection step of detecting the reproduction time of the multimedia stream and the reception of the multimedia stream by the stream reception device are in time for the reproduction of the multimedia stream by the stream reception device. In the second determining step and in the second determining step, when it is determined that the reception of the multimedia stream by the stream receiving apparatus is not in time for the reproduction of the multimedia stream by the stream receiving apparatus, the multimedia stream is partially analyzed. A second control step of discarding and transmitting the remaining multimedia stream through the network and controlling the data rate of the multimedia stream to correspond to the bandwidth of the network, the stream receiving device A detection step of detecting whether or not a part of the decoding unit of the multimedia stream has been discarded by the stream transmitting apparatus or the stream relay apparatus from the multimedia stream received via the multimedia stream received via the network. A decoding step in which a part of the decoding units of each of the decoding units forming the multimedia stream is not detected as being discarded by the stream transmitting apparatus or the stream relay apparatus. . The stream distribution program according to claim 9,
A stream transmission device that transmits a multimedia stream to a network, a stream reception device that receives the multimedia stream, and a stream relay device that relays stream transmission between the stream transmission device and the stream reception device are connected via a network. Which is a stream distribution program for controlling a stream distribution system, wherein a multimedia stream is transmitted to a stream transmission device via a network,
A network delay time detecting step of detecting a network delay time until the receiving device receives the multimedia stream, and whether or not the reception of the multimedia stream by the stream receiving device is in time for the reproduction of the multimedia stream by the stream receiving device. In the first determining step of determining, and in the first determining step, when it is determined that the reception of the multimedia stream by the stream receiving apparatus is not in time for the reproduction of the multimedia stream by the stream receiving apparatus, the multimedia stream is partially divided. And then send the rest of the multimedia stream over the network,
A first control step of controlling the data rate of the multimedia stream so as to correspond to the bandwidth of the network, and causing the stream relay device to detect a stream reproduction time of detecting the reproduction time of the multimedia stream; A second determination step of determining whether or not the reception of the multimedia stream by the receiving apparatus is in time for the reproduction of the multimedia stream by the stream receiving apparatus, and the receiving of the multimedia stream by the stream receiving apparatus in the second determination step. However, if it is judged that the stream receiving device cannot play the multimedia stream in time, the multimedia stream is partially discarded, and the remaining multimedia stream is transmitted through the network. The second control step of controlling the data rate of the stream so as to correspond to the bandwidth of the network is executed, and the stream receiving apparatus is configured to decode the multimedia stream from the multimedia stream received via the network. Part of the decoding unit that detects whether the stream transmission device or the stream relay device has discarded the part, and among the decoding units that make up the multimedia stream received via the network, part of the decoding unit is the stream. And a decoding step of decoding a signal which has not been detected to be discarded by the transmission device or the stream relay device. In the stream distribution system according to the present invention,
The stream transmission device detects a network delay time from the transmission of the multimedia stream via the network until the reception device receives the multimedia stream, and the reception of the multimedia stream by the stream reception device is performed by the stream reception device. Determine whether or not to play the multimedia stream by
When it is determined that the reception of the multimedia stream by the stream reception device is not in time for the reproduction of the multimedia stream by the stream reception device, the multimedia stream is partially discarded and the remaining multimedia streams are transmitted through the network. Then, the data rate of the multimedia stream is controlled so as to correspond to the bandwidth of the network, and the stream relay device detects the reproduction time of the multimedia stream,
It is determined whether the reception of the multimedia stream by the stream receiving device is in time for the reproduction of the multimedia stream by the stream receiving device, and the reception of the multimedia stream by the stream receiving device determines the reproduction of the multimedia stream by the stream receiving device. When it is determined that it is not in time, the multimedia stream is partially discarded, the remaining multimedia stream is transmitted through the network, and the data rate of the multimedia stream is controlled to correspond to the bandwidth of the network. The stream receiving apparatus determines whether a part of the decoding unit of the multimedia stream is discarded by the stream transmitting apparatus or the stream relay apparatus from the multimedia stream received via the network. Out of the decoding units that make up the multimedia stream received via the network and are not detected as being discarded by the stream transmission device or stream relay device. To do.

[0009]

BEST MODE FOR CARRYING OUT THE INVENTION An embodiment of the stream distribution system of the present invention will be described below. FIG. 1 is a diagram showing a system configuration example of an embodiment of a stream distribution system of the present invention.

As shown in FIG. 1, this embodiment comprises a transmitting device 10, a relay device A12, a relay device B14, and a receiving device 16. The transmitting device 10 is adapted to transmit stream data via the network A11. The relay device A12 is the network A1.
1 and the network B13 are connected. Further, the relay device B14 includes a network B13 and a network C15.
Is connected to. The receiving device 16 is a network C15.
The stream data is received via.

In response to a request from the receiving device 16, the transmitting device 10 has a network A11, a network B13,
It is a computer on the side that provides stream data such as audio and moving images through the network C15. As shown in FIG. 1, the transmitter 10 is connected to a network A11 such as the Internet.

FIG. 2 is a block diagram showing a configuration example of the transmission device 10. As shown in FIG.
PEG (Moving Picture Expert)
sGroup encoder 101, CPU (centr)
al processing unit) 102, RO
M (read only memory) 103, RA
M (random access memory) 10
4, network delay time detection unit 105, network interface 106, HDD (hard disk)
drive: hard disk drive) 107.

Here, the MPEG encoder 101 is a device for encoding a video signal from the outside into an MPEG2 transport stream, and the encoded transport stream is recorded in the HDD 107. C
The PU 102 is a processor that controls the entire transmission device 10. The ROM 103 is a read-only memory (recording medium) in which a control program for operating the CPU 102 is recorded. RAM 104 is a CPU
The data temporarily required for the 102 to perform various processes is stored. The network delay time detection unit 105
IP packet transmission time and reception time information (transmitting device 10
Is the time information indicating the time when the receiving device 16 received the IP packet transmitted by the. The receiving time information packet is transmitted from the receiving device 16 to the transmitting device 10), and the delay time of the network is detected. The network interface 106 receives an IP packet such as a stream transmission request, reception time information, and display start time information, and transmits a stream data IP packet and the like. HDD1
Reference numeral 07 denotes a storage medium that reads and writes data on a disk (disk) that rotates at high speed, and stores stream data to be transmitted to the receiving device 16 in response to a stream transmission request from the receiving device 16. There is.

Returning to FIG. 1, the receiving device 16 is a personal computer of a user who receives a distribution service of stream data such as voice and moving images on the network C15. The receiving device 16 is connected to the network C15, as shown in FIG. The receiving device 16
In addition to the personal computer, may be configured with a VCR or a television device.

FIG. 3 is a block diagram showing a configuration example of the receiving device 16. As shown in FIG.
PEG decoder 161, CPU 162, ROM 163,
The RAM 164, the stream partial discard detecting means 165, the network interface 166, the HDD 167, and the monitor 168 are provided.

Here, the MPEG decoder 161 is an HD
This is a device for decoding the MPEG transport stream stored in D167 and outputting it to the monitor 168.
The CPU 162 is a processor that controls the reception device 16 as a whole. The ROM 163 is a read-only memory (recording medium), and stores a control program for operating the CPU 162. RAM 164 is C
The PU 162 temporarily stores data necessary for performing various processes. Stream partial discard detection means 165
Detects that an IP packet including a part of each picture (decoding unit) of I, P, B that composes the transport stream of MPEG2 is discarded by the transmission device 10, the relay device A12, or the relay device B14. A means for invalidating the entire P picture or B picture including the discarded portion. The network interface 166 transmits IP packets such as a stream transmission request, reception time information, and display start time information, and receives stream data IP packets and the like. The HDD 167 is a storage medium that reads / writes data from / to a disk (disk) that rotates at high speed, and stores stream data transmitted from a transmission device.

Returning to FIG. 1, the relay device A12 is a device for flowing an IP packet to the network on the receiving device 16 side in the middle of the transmission path between the transmitting device 10 and the receiving device 16, and is connected to an unrelated network. Has the role of reducing the traffic of the entire network by not passing IP packets. The relay device A12 connects the network A11 and the network B13. Note that on the Internet, the relay device A12 is called a router.

FIG. 4 is a block diagram showing a configuration example of the relay device A12. As shown in FIG. 4, the relay device A12
Includes a CPU 122, a ROM 123, a RAM 124, a stream relay time determination unit 125, and a network interface 126. The CPU 122 is a processor that controls the entire relay device A12. ROM12
3 is a read-only memory (recording medium)
A control program for operating the U122 is recorded. The RAM 124 temporarily stores data necessary for the CPU 122 to perform various processes. The stream relay time determination unit 125 determines whether or not the IP packet arrives by the time required by the receiving device 16, and when it determines that the IP packet will be delayed by the time required by the receiving device 16, a predetermined time is determined. Discard the P picture or B picture based on the condition. Network interface 126
Relays a stream transmission request, reception time information, display start time information, stream data IP packet, and the like.

Returning to FIG. 1, the relay device B14 is a device for flowing an IP packet to the network on the receiving device 16 side in the middle of the transmission path between the transmitting device 10 and the receiving device 16, and is connected to an unrelated network. Has the role of reducing the traffic of the entire network by not passing IP packets. The relay device B14 connects the network B13 and the network C15. In addition, on the Internet, the relay device B14 is called a router.

FIG. 5 is a block diagram showing a configuration example of the relay device B14. As shown in FIG. 5, the relay device B14
Includes a CPU 142, a ROM 143, a RAM 144, a stream relay time determination unit 145, and a network interface 146. The CPU 142 is a processor that controls the entire relay device B14. ROM14
3 is a read-only memory (recording medium)
A control program for operating the U142 is recorded. The RAM 144 temporarily stores data necessary for the CPU 142 to perform various processes. The stream relay time determination unit 145 determines whether or not the IP packet arrives by the time required by the reception device 16, and when it determines that the IP packet will be delayed by the time required by the reception device 16, a predetermined time is determined. Discard the P picture or B picture based on the condition. Network interface 146
Relays a stream transmission request, reception time information, display start time information, stream data IP packet, and the like.

Although stream data can be applied to data such as audio and moving images, in the present embodiment, MPEG (Moti) as shown in FIG. 6 is used.
onPicture Experts Group) P
The stream data of moving images in the S (Program Stream) and TS (Transport Stream) formats will be described as an example.

As shown in FIG. 6, the MPEG stream data is divided for each frame (one still image which is a unit forming a moving image), and the frame includes an I picture frame, a B picture frame, And P picture frame. The I picture frame is an image without reference to the preceding and following frames, the B picture frame is a bidirectional predictive coding image to the preceding and following frames, and the P picture frame is a forward predictive coding of the frame. It is an image.

The GOP (Group) shown in FIG.
In order to enable random access of stream data, an intra-frame coded image by information closed only within a screen called an I picture is periodically inserted, and this intra-frame coded image is at least 1 It is the structure of a group of images. In FIG. 6, 1
A GOP is composed of 15 pictures and has a duration of 0.5 seconds (S
ec).

FIG. 7 shows an MPEG stream converted to UDP / I.
The structure when divided into P packets for transmission is shown for each picture (I picture, P picture, and B picture).
Are each divided into one or more UDP / IP packets. The UDP / IP packet has an IP header defined by a protocol, a UDP header, and a predetermined streaming header, and the streaming header is a picture No. indicating the temporal decoding order of pictures. , IP packet serial No. (IP packet No.), the number of IP packets forming a picture, discard level 1 time as a guideline for deleting B pictures, and discard level 2 time as a guideline for discarding B pictures and P pictures.

It should be noted that in FIG.
Transmitters 10 are connected to the network C15
Although one receiver 16 is connected, the number of receivers is not limited. There are no restrictions on the network topology (physical arrangement of network devices, etc.).

Next, the operation of this embodiment will be described. FIG. 8 is a flowchart showing the operation of transmitting apparatus 10 of the present exemplary embodiment. First, it is determined whether or not there is a stream transmission request from the reception device 16. That is, the reception confirmation of the stream transmission request storing the stream name information issued by the receiving device 16 is performed (step S2).
01), if the stream transmission request is not received (NO in step S201), the reception confirmation of the stream transmission request is performed again (step 201).

On the other hand, when the stream transmission request is received (YES in step S201), the procedure described below is performed.
Each picture of the MPEG stream having the structure shown in FIG. 6 is transmitted to the receiving device 16 as a UDP / IP packet having the structure shown in FIG.

First, the values of the network delay time 1 and the network delay time 2 are set to 0 (step S2).
02). The network delay time 1 represents the time until the transmitted IP packet arrives at the receiving device 16. When the receiving device 16 receives the picture IP packet including the picture data of the stream sent from the transmitting device 10, the receiving device 16 transmits the reception time information IP packet in which the internal time of the receiving device 16 that received the picture IP packet is described. Send to 10. Then, the transmitter 10
Is the transmission time of the transmitted IP packet and the reception device 16
The reception time information sent from the device is compared with the reception time described in the IP packet, and the time obtained by subtracting the reception time from the transmission time is set as the network delay time 1.

The network delay time 2 represents the elapsed time from the transmission of the picture IP packet transmitted next to the picture IP packet corresponding to the last received time information IP packet. This network delay 2
Is obtained by comparing the internal time of the transmitter 10 when the transmitter 10 transmits the picture IP packet with the current time. That is, the network delay time 2 is obtained by subtracting the internal time of the transmitter 10 when the transmitter 10 transmits the picture IP packet from the current time.

Next, it is confirmed that all the data in the stream have been transmitted (step S203). If there is untransmitted stream data (NO in step S203), the reception of new reception time information is confirmed (step S20).
4) If received (YES in step S204)
For, the network delay time 1 (= the latest IP packet reception time-the latest IP packet transmission time) is calculated and set (step S205).

After the latest reception time information IP packet is received, it is confirmed whether or not there is a picture IP packet which has not yet received the reception time information IP packet (step S206), and the latest reception time is received. Information IP
Reception time information IP after the packet is received
If there is a picture IP packet that has not yet received the packet, the network delay time 2 (= the elapsed time after the transmission of the IP packet for which the time information has not been received) is set (step S207).

The network delay time is the time until the IP packet transmitted from the transmission device 10 arrives at the reception device 16, and the larger value of the values of the network delay time 1 and the network delay time 2 is set. It is set as a network delay time (step S208).

Next, the reception of the display start time information IP packet is confirmed (whether it is received or not) (step S2).
09), if it has been received, as will be described later, the discard level 1 threshold and the discard level 2 threshold are set by performing a predetermined calculation using the display start time (step S).
210).

The discard level 1 threshold is an index (unit is time) for accommodating a narrow network bandwidth by thinning out stream data in the transmitter 10.
Is. I of the first picture of the multimedia stream
The transmission time of the first picture IP packet, which is the time when the P packet is transmitted, is compared with the display start time indicated by the display start time information IP packet, the time difference between them is calculated, and a value smaller than the time difference obtained by the calculation (for example, the time difference To 0.7) is set as the discard level 1 threshold.

The discard level 2 threshold is an index (unit is time) for thinning the stream data to correspond to a narrower network bandwidth in the transmitter 10. The head picture IP packet transmission time and the display start time indicated by the display start time information IP packet are compared, the time difference between them is calculated, and a value smaller than the time difference obtained by the calculation is calculated (for example, the time difference is multiplied by 0.5). Value) is set as the discard level 2 threshold.

Then, as will be described later, the time difference (margin time) between the predicted arrival time of the picture IP packet at the receiving device 16 and the display time of this picture IP packet (=
(Display time-expected arrival time) is less than the time indicated as the discard level 1 threshold, it is determined that the network bandwidth has become narrower, and in order to cope with this, the discard level is set to 1 and the B picture is set. And discard and send.

As will be described later, the time difference (margin time) (= display time-estimated arrival time) between the estimated arrival time of the picture IP packet at the receiving device 16 and the display time of this picture IP packet is discarded. When the time is less than the level 2 threshold value, it is determined that the bandwidth of the network has become narrower. In order to cope with this, the discard level is set to 2, and P and B pictures are discarded and transmitted.

Next, it is judged whether or not the margin time is smaller than the discard level 2 threshold value (step S211). The spare time is the time when the picture IP packet arrives at the receiving device 16 (the estimated arrival time of the receiving device), and the receiving device 16 monitors the picture included in the picture IP packet by the monitor 16
8 is the difference from the time displayed on the display 8 (display time of the receiving device) and indicates the time margin, which is calculated by the following equation. Extra time = Time at which the receiver is displayed-Estimated time at which the receiver arrives

The larger the margin time, the less likely the stream data underrun caused by the lack of stream data received on the receiving device 16 side occurs. The expected arrival time of the receiving device is calculated by adding the network delay time to the current time (current time + network delay time). The display time is PTS (Presentation Time St) at the display start time.
amp: reproduction time information) It is calculated by adding the offset time (display start time + PTS offset time).

The PTS is display time information included in the MPEG program stream or the MPEG transport stream, and the PTS offset time is the time from the beginning of the MPEG stream.

If it is determined in step S211 that the margin time is smaller than the discard level 2 threshold, the discard level is set to 2 (step S212). On the other hand, when it is determined in step S211 that the surplus time is greater than or equal to the discard level 2 threshold, it is determined whether the surplus time is less than the discard level 1 threshold (step 2
15) If it is determined that the margin time is smaller than the discard level 1 threshold, the discard level is set to 1 (step S2
16). On the other hand, when it is determined that the surplus time is greater than or equal to the discard level 1 threshold value, the process proceeds to step S213.

Then, according to the set discard level,
After discarding the picture and waiting until the current time reaches the transmission time of the picture IP packet to be transmitted next (step S213), or immediately if the transmission time has already passed, the next picture IP packet is transmitted immediately. (Step S
214), the procedure returns to step S203, and step S203
The subsequent processing is repeatedly executed.

The transmission time of the picture IP packet is expressed by the following equation. Picture IP packet transmission time = start picture IP packet transmission time + PTS offset time

The picture IP packet has a structure as shown in FIG. 7, and the discard level 1 time in the streaming header is obtained by subtracting the discard level 1 threshold value from the picture IP packet display time as shown in the following equation. It is calculated. Discard level 1 time = Picture IP packet display time-Discard level 1 threshold

The discard level 2 time is as shown in the following equation.
It is calculated by subtracting the discard level 1 threshold from the picture IP packet display time. Discard level 2 time = Picture IP packet display time-Discard level 2 threshold

FIG. 9 is a flow chart showing the operation of the relay device A12 and the relay device B14 of the present invention. Since the operations of the relay device A12 and the relay device B14 are basically the same, only the operation of the relay device A12 will be described here, and the description of the operation of the relay device B14 will be omitted.

First, the CPU 12 of the relay device A12
2, the reception confirmation of the picture IP packet (determination as to whether or not the picture IP is received) is performed (step S3).
01), if it is determined that the picture IP packet is not received, the process returns to step S301 and step S30.
The processes after 1 are repeatedly executed. On the other hand, picture I
When it is determined that the P packet is received, the discard level 1 time stored in the streaming header of the received picture IP packet is compared with the internal time of the relay device A12, and the discard level 1 time is set in the relay device A12. It is determined whether the internal time has passed (step S30).
2).

When it is determined in step S302 that the internal time of the relay device A12 has passed the discard level 1 time stored in the streaming header of the received picture IP packet, the received picture IP packet is a B picture. Is determined (step S303). As a result, when it is determined that the received picture IP packet is a B picture, this picture IP packet is discarded (step S304) and is not transferred to the reception device 16.

On the other hand, when it is determined in step S302 that the internal time of the relay device A12 has not passed the discard level 1 time stored in the streaming header of the received picture IP packet, or step S3
If the received picture IP packet is determined not to be a B picture in 03, or if the received picture IP packet is discarded in step S304, the process proceeds to step S305.

In step S305, the discard level 2 time stored in the received picture IP packet is compared with the internal time of the relay device A12, and the discard level 2 time stored in the streaming header of the picture IP packet is compared. If the internal time of the relay device A12 has passed, it is determined whether the received picture IP packet is a P picture IP packet (step S306). As a result, when it is determined that the received picture IP packet is a P picture IP packet,
This picture IP packet is discarded (S307) and is not transferred to the receiving device 16, and the process returns to step S301 to confirm the reception of the picture IP packet again.

On the other hand, in step S305, the discard level 2 stored in the received picture IP packet
If it is determined that the internal time of the relay device A12 has not passed, or if it is determined in step S306 that the received picture IP packet is not a P picture IP packet, the received picture IP packet is received. It transfers to 16 (step S308).

FIG. 10 is a flow chart showing the operation of the receiving apparatus 16 of this embodiment. First, the CPU 162 of the reception device 16 transmits a transmission request including stream name information of a desired stream to the transmission device 10 via the network interface 166 (step S4).
01), the picture IP packet transmitted from the transmission device 10 in the format as shown in FIG. 7 is received and supplied to the MPEG decoder 161. The MPEG decoder 161 is a C
The picture IP packet supplied from the PU 162 is decoded and supplied to the monitor 168. The monitor 168 is M
A moving image or a still image corresponding to the decoded picture IP packet supplied from the PEG decoder 161 is displayed on the screen.

Next, it is confirmed whether the stream has been received (step S402). When the stream has been received, this process is terminated. On the other hand, if the reception of the stream has not ended, the reception of the new picture IP packet is confirmed (step S403), and the new picture IP packet is received.
If no packet is received, the process returns to step S402 to confirm the reception end of the stream again.

When a new picture IP packet is received in step S403, it is determined whether or not the newly received picture IP packet is an invalid picture IP packet (step S404). Invalid I
The P packet is a picture No. included in the streaming header of the picture IP packet that has been discarded by the transmission device 10, the relay device A12, or the relay device B14 in the past. (FIG. 7) has the same picture IP packet No. This is a picture IP packet of (IP packet No.). This takes into account the possibility that a single picture may be split into multiple picture IP packets.

Whether or not the packet is an invalid IP packet in step S404 is determined as follows. That is, as will be described later, in step S408, the received picture number (picture N of the streaming header is
o. ) Is determined, and if the received picture numbers are discontinuous, the missing picture number, that is, the picture with the non-arrival picture number is set as an invalid picture in step S414, and the picture is dealt with as the invalid picture. The IP packet is determined to be an invalid IP packet. In step S404, the newly received picture IP packet is invalid I
If it is determined that the packet is a P packet, the newly received picture IP packet is discarded (step S41).
0).

On the other hand, if it is determined in step S404 that the newly received picture IP packet is not an invalid IP packet, packet reception time information indicating the time when the picture IP packet is received is transmitted to the transmission device 10 (step S405).

Next, it is judged whether or not the decoding of the stream has already started (step S406). As a result, when it is determined that the decoding of the stream has already started, the reception of this picture IP packet causes
The same picture number as this picture IP packet. It is determined whether or not the number of picture IP packets having "1" has been completed. That is, the same picture No. is set for the number of packets forming the picture (FIG. 7). With picture I
It is determined whether a P packet has been received (step S
407).

As a result, when the picture IP packets for the number of packets forming the picture are received and it is determined that all the picture IP packets forming the picture are complete, this picture is consecutive with the picture decoded immediately before. The picture No. of this picture. When,
Picture No. of the picture decoded immediately before. It is confirmed by comparing with (step S408), and if they are continuous, decoding of this picture is started (step S409).

On the other hand, if it is determined in step S406 that the decoding has not started yet, in order to cope with the fluctuation of the network delay time, for example, the HDD 1
It is determined whether the stream data stored in 67 is for 10 seconds (20 consecutive pictures) (step S411). As a result, when it is determined that the stream data is in the local environment for 10 seconds (stored in the HDD 167), the decoding of the stream data is started (step S412), and the display start time information is transmitted to the transmission device 10. It is transmitted (step S413).

If it is determined in step S408 that the aligned pictures are not continuous with the picture decoded immediately before, the number of the received picture No. The picture is designated as an invalid picture (step S414).

After discarding the IP packet in step S410, or when it is determined in step S411 that 10 seconds worth of data has not yet accumulated, or in step S407, the same picture number. If it is determined that all the picture IP packets having the same are not available, or after the display start time information is transmitted in step S413, or after the decoding of the picture is started in step S409, or in step S414,
Picture No. not received. After making the picture of the above into an invalid picture, it returns to step S402, and step S40
The processes after 2 are repeatedly executed.

As described above, the transmission device 10, the relay device A12, and the relay device B14 each actively
Since the picture IP packet can be appropriately discarded according to the network bandwidth, it is possible to reduce the time lag when the stream data rate corresponds to the network bandwidth and to waste the resources on the network unnecessarily. Can be suppressed.

A program for executing the above-described processing is a CD-ROM (compact disc).
read only memory), DVD (dig
It can be provided by recording it on various recording media such as an digital versatile disc), a floppy (registered trademark) disk, and a memory card. Then, the program controls the operation of each device, and the program-controlled device executes a predetermined process instructed by the program.

In the above embodiment, MPE is used.
The case of handling G stream data has been explained,
It is not limited to this.

Further, it goes without saying that the configurations and operations of the above-described embodiments are examples, and can be changed as appropriate without departing from the spirit of the present invention.

[0066]

As described above, according to the stream distribution system of the present invention, the stream transmitting device transmits the multimedia stream via the network and the receiving device receives the multimedia stream from the network. The delay time is detected, it is determined whether the reception of the multimedia stream by the stream receiving device is in time for the reproduction of the multimedia stream by the stream receiving device, and the reception of the multimedia stream by the stream receiving device is performed by the stream receiving device. If it is determined that the multimedia stream cannot be played in time, the multimedia stream is partially discarded, the remaining multimedia stream is transmitted through the network, and the data rate of the multimedia stream is changed to the network. The stream relay device detects the reproduction time of the multimedia stream and determines whether the reception of the multimedia stream by the stream reception device is in time for the reproduction of the multimedia stream by the stream reception device. If it is determined that the reception of the multimedia stream by the stream reception device is not in time for the reproduction of the multimedia stream by the stream reception device, the multimedia stream is partially discarded and the remaining multimedia streams are discarded. The data rate of the multimedia stream transmitted over the network is controlled so as to correspond to the bandwidth of the network, and the stream receiving device receives the multimedia stream from the multimedia stream received through the network. Some of the decode unit of the A stream, detects whether or not destroyed by the stream transmission apparatus, or stream relay device,
Among the decoding units that make up the multimedia stream received via the network, some of the decoding units are decoded if they have not been detected as being discarded by the stream transmission device or the stream relay device. Therefore, the stream transmission device or the stream relay device can actively thin out the multimedia stream to partially correspond to the bandwidth of the network, and suppress the time lag when the stream data is reproduced by the reception device. be able to. Further, it is possible to suppress wasteful consumption of resources on the network.

[Brief description of drawings]

FIG. 1 is a system configuration diagram showing a stream distribution system of the present invention.

FIG. 2 is a block diagram showing a configuration of a transmission device.

FIG. 3 is a block diagram showing a configuration of a receiving device.

FIG. 4 is a block diagram showing a configuration of a relay device A.

5 is a block diagram showing a configuration of a relay device B. FIG.

FIG. 6 is a diagram showing a picture structure of an MPEG stream.

FIG. 7 is a diagram showing a UDP / IP packet structure for transmitting an MPEG stream.

FIG. 8 is a flowchart showing the operation of the transmission device.

FIG. 9 is a flowchart showing the operation of relay devices A and B.

FIG. 10 is a flowchart showing the operation of the receiving device.

[Explanation of symbols]

10 transmitter 11 network A 12 relay device A 13 network B 14 relay device B 15 network C 16 receiver 102, 122, 142, 162 CPU 103, 123, 143, 163 ROM 104, 124, 144, 164 RAM 101 MPEG encoder 105 network delay time detection unit 106, 126, 146, 166 network interface 107, 167 HDD 125, 145 stream relay time determination unit 161 MPEG decoder 165 stream partial discard detection unit 168 monitor

Claims (9)

[Claims] 1. A stream transmitting apparatus for transmitting a multimedia stream to a network, a stream receiving apparatus for receiving the multimedia stream, and a stream relay for relaying stream transmission between the stream transmitting apparatus and the stream receiving apparatus. A stream distribution system in which a device is connected via the network, the stream transmitting device transmits the multimedia stream via the network, and then the receiving device receives the multimedia stream. Network delay time detecting means for detecting the network delay time up to, and the reception of the multimedia stream by the stream receiving device in time for the reproduction of the multimedia stream by the stream receiving device. And a first determining means for determining whether or not the reception of the multimedia stream by the stream receiving apparatus is in time for the reproduction of the multimedia stream by the stream receiving apparatus. If so, the multimedia stream is partially discarded, and the remaining multimedia stream is transmitted through the network,
A first controlling the data rate of the multimedia stream to correspond to the bandwidth of the network;
The stream relay device detects the playback time of the multimedia stream, and the stream receiving device receives the multimedia stream by the stream receiving device. Second determination means for determining whether or not the reproduction of the media stream is in time, and reception of the multimedia stream by the stream reception device by the second determination means determines whether the multimedia stream is received by the stream reception device. When it is determined that it is not in time for playback, the multimedia stream is partially discarded, and the remaining multimedia stream is transmitted via the network,
A second controlling the data rate of the multimedia stream to correspond to the bandwidth of the network;
The stream receiving apparatus discards a part of a decoding unit of the multimedia stream from the multimedia stream received via the network, by the stream transmitting apparatus or the stream relay apparatus. Detecting means for detecting whether or not the decoding is performed, and among the decoding units forming the multimedia stream received via the network, a part of the decoding unit is generated by the stream transmitting device or the stream relay device. A stream distribution system, comprising: a decoding unit that decodes information that has not been detected to be discarded. 2. The network delay time detecting means,
The network delay time is calculated by comparing the time when the multimedia stream is transmitted to the stream receiving device via the network with the time when the multimedia stream reaches the stream receiving device via the network. And the estimated time of arrival at the stream receiving device of the multimedia stream transmitted from the stream transmitting device, based on the reproduction time information of the multimedia stream itself, by the first determining means, When it is determined that the reproduction time indicated by the reproduction time information is not earlier than a predetermined reference time, the first control means partially discards the multimedia stream, and the remaining part of the multimedia stream is discarded. Multimedia streams over the network The stream delivery system according to claim 1, wherein the stream delivery system controls to transmit to the stream receiving apparatus. 3. The reproduction time information of the multimedia stream itself is compared with the internal time information of the stream relay device by the second determination means, and the internal time information of the stream relay device is reproduced. When it is determined that the playback time indicated by the time information is not earlier than a predetermined reference time, the second control means partially discards the multimedia stream and leaves the remaining multimedia. The stream distribution system according to claim 1 or 2, wherein the stream is controlled to be transmitted to the stream receiving device via the network. 4. The arrival time information transmission in which the stream reception device transmits arrival time information indicating the arrival time of the multimedia stream transmitted from the stream transmission device via the network to the stream transmission device. Means, stream partial discard detection means for detecting that the multimedia stream has been partially discarded, discarding the decoding unit including the discarded portion in the stream transmission device or the stream relay device, 4. The stream distribution system according to claim 1, further comprising third control means for controlling the data rate of the multimedia stream so as to correspond to the bandwidth of the network. 5. A stream transmitting apparatus for transmitting a multimedia stream to a stream receiving apparatus via a network, wherein the multimedia stream is transmitted via the network, and then the stream receiving apparatus connected to the network is received. Network delay time detection means for detecting a network delay time until the device receives the multimedia stream, and determining whether or not the reception of the multimedia stream by the stream receiving device is in time for the reproduction of the multimedia stream. And a determination unit that determines that the reception of the multimedia stream by the stream receiving apparatus cannot be performed in time for the reproduction of the multimedia stream. Abandoned
The remaining multimedia stream is transmitted via the network, and control means for controlling so as to correspond to the bandwidth of the network, the network delay time detecting means, the multimedia stream via the network. By detecting the network delay time by comparing the time when the packet of a predetermined stream transmission protocol for transmitting the packet is transmitted with the time when the packet reaches the stream receiving device, Based on the reproduction time information of the multimedia stream itself, the estimated arrival time of the packet transmitted from the stream transmission device to the stream reception device is a predetermined criterion with respect to the reproduction time indicated by the reproduction time information. Determined not to be earlier than the time In this case, the control means may partially discard the multimedia stream, transmit the remaining multimedia stream through the network, and control the bandwidth so as to correspond to the bandwidth of the network. Stream transmitter. 6. A stream relay apparatus for relaying a multimedia stream transmitted from a stream transmitting apparatus via a network to a stream receiving apparatus, wherein the multimedia stream of the multimedia stream is based on a time when the multimedia stream is relayed. Relay to the stream receiving device, in accordance with the determination result of the stream relay time determination means for determining whether or not the reproduction of the multimedia stream by the stream receiving device is in time, the determination result of the stream relay time determination means,
Control means for partially discarding the multimedia stream and transmitting the remaining multimedia stream via the network, the stream relay time determining means for the multimedia stream itself or the multimedia stream. The reproduction time information of the stream transmission protocol is compared with the internal time information indicating the internal time of the stream relay device, and the internal time indicated by the internal time information is compared with the reproduction time indicated by the reproduction time information. When it is determined that the time is not earlier than a predetermined reference time, the control means partially discards the multimedia stream, and transmits the remaining multimedia stream via the network, Corresponds to network bandwidth A stream relay device characterized by being controlled so that 7. A stream receiving device for receiving and playing a multimedia stream transmitted from a stream transmitting device via a network and a stream relay device, wherein the stream receiving device transmits from the stream transmitting device via the network. Arrival time information indicating the arrival time of the multimedia stream, arrival time information transmitting means for transmitting to the stream transmitting device, stream partial discard detecting means for detecting that the multimedia stream is partially discarded, Third control means for discarding a decoding unit including a portion discarded in the stream transmission device or the stream relay device, and controlling the data rate of the multimedia stream so as to correspond to the bandwidth of the network. Specially equipped with Stream receiver to collect. 8. A stream transmitting apparatus for transmitting a multimedia stream to a network, a stream receiving apparatus for receiving the multimedia stream, and a stream relay for relaying stream transmission between the stream transmitting apparatus and the stream receiving apparatus. A stream distribution method in a stream distribution system in which a device is connected via the network, wherein the stream transmission device transmits the multimedia stream via the network, and the reception device transmits the multimedia stream. A network delay time detecting step of detecting a network delay time until a stream is received; and the receiving of the multimedia stream by the stream receiving device, the receiving of the multimedia stream by the stream receiving device. A first determination step of determining whether or not it is in time for the reproduction of the audio stream, and in the first determination step, the reception of the multimedia stream by the stream receiving apparatus is performed by the stream receiving apparatus. , The multimedia stream is partially discarded, the remaining multimedia stream is transmitted through the network, and the data rate of the multimedia stream is set to the bandwidth of the network. And a first control step of controlling the stream reproduction time to detect the reproduction time of the multimedia stream, the stream relay apparatus detects the reproduction time of the multimedia stream, and the stream reception apparatus detects the reproduction time of the multimedia stream. A second determination step of determining whether or not reception is in time for the reproduction of the multimedia stream by the stream receiving apparatus; and in the second determination step, reception of the multimedia stream by the stream receiving apparatus, When it is determined that the reproduction of the multimedia stream by the stream receiving apparatus is not in time, the multimedia stream is partially discarded, and the remaining multimedia stream is transmitted via the network, and the multimedia stream is transmitted. And a second control step of controlling the data rate of the multimedia stream to correspond to the bandwidth of the network, the stream receiving apparatus is configured to extract the multimedia stream from the multimedia stream received via the network. A step of detecting whether or not a part of the decoding units of the video stream has been discarded by the stream transmission device or the stream relay device; and a decoding step of each decoding unit constituting the multimedia stream received via the network. Of these, a decoding step of decoding a part of the decoding unit that is not detected as being discarded by the stream transmission device or the stream relay device, is provided. 9. A stream transmitting apparatus for transmitting a multimedia stream to a network, a stream receiving apparatus for receiving the multimedia stream, and a stream relay for relaying stream transmission between the stream transmitting apparatus and the stream receiving apparatus. A device is a stream distribution program for controlling a stream distribution system connected via the network, wherein the multimedia transmitter transmits the multimedia stream to the stream transmission device via the network, A network delay time detecting step of detecting a network delay time until receiving the multimedia stream; and the receiving of the multimedia stream by the stream receiving apparatus, A first determining step of determining whether or not it is in time to play a multimedia stream; and in the first determining step, the reception of the multimedia stream by the stream receiving apparatus is performed by the stream receiving apparatus. , The multimedia stream is partially discarded, the remaining multimedia stream is transmitted through the network, and the data rate of the multimedia stream is set to the bandwidth of the network. And a stream reproduction time detecting step of detecting a reproduction time of the multimedia stream in the stream relay apparatus, the multimedia control apparatus according to the first aspect of the invention, A second determination step of determining whether or not the stream reception is in time for the reproduction of the multimedia stream by the stream reception device; and the reception of the multimedia stream by the stream reception device in the second determination step. However, if it is determined that the multimedia stream cannot be played back in time by the stream receiving device, the multimedia stream is partially discarded, and the remaining multimedia stream is transmitted via the network, A second control step of controlling the data rate of the media stream to correspond to the bandwidth of the network, and causing the stream receiving apparatus to select the multimedia stream from the multimedia stream received via the network. A detection step of detecting whether a part of the decoding unit of the media stream is discarded by the stream transmitting device or the stream relay device, and each decoding unit forming the multimedia stream received via the network. Among them, a stream distribution program, characterized in that a decoding step of decoding a part of the decoding unit that is not detected as being discarded by the stream transmission device or the stream relay device.