JPH11163934A - System and device for transmission, reception device, real-time dynamic picture, system and device for sound transmission, control method for the same and storage device - Google Patents

Abstract

PROBLEM TO BE SOLVED: To improve reliability at least regarding sound by transmitting encoded dynamic picture data and encoded sound data to be transferred in real time to a network as a separate stream respectively without time-division multiplexing them into one bit stream. SOLUTION: A transmission side device encodes sound and a dynamic picture at encoding circuits 1 and 2, and transmits sound data as they are to a reception side by way of a network. Also, dynamic picture data are made into plural packets and successively transmitted. Since a control circuit 6 of the reception side device knows a period of the sound data and the dynamic picture data to be received in accordance with communication condition instructed by the transmission side in advance, it issues a retransmission demand when it cannot receive the sound data even at that time (counted by timers 9 and 10) and does not issue the retransmission request in the case of the dynamic picture but notifies the transmission side of an Ack (acknowledgement) as a normal reception is performed.

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

[0001] 1. Field of the Invention [0002] The present invention relates to a system and apparatus suitable for real-time data transmission using a general-purpose computer network, a control method thereof, and a storage medium.

[0002]

2. Description of the Related Art Generally, real-time moving image and voice transmission using moving image encoding processing is performed between apparatuses having a block configuration shown in FIG. In the transmitting side device, after the audio and image data are respectively encoded by the encoding circuit, they are time-division multiplexed by the multiplexing circuit and transmitted to the transmitting circuit. Here, the multiplexing circuit and the transmission circuit time-division multiplex the encoded audio data and the encoded image data, for example, as shown in FIG. ((B) in the figure), and sends it out to the network.

On the receiving side, the segmented stream data received from the network is returned to the original time-division multiplexed stream data, and then separated into audio encoded data and video encoded data. Pass to
Decode into audio data and video data. If the decoded data is audio, predetermined processing is performed, and if it is video, processing for display is performed and output to the display device.

[0004] Further, for data transmission over a network generally used in computers, several procedures are determined to ensure the reliability of transmission. For example, T which is generally used in the computer world
In the CP / IP protocol, an acknowledgment for a transmission packet as shown in FIG. 14 is defined, and a procedure for retransmitting a packet "X" for which no acknowledgment is obtained is taken on the transmission side. Also, a technology such as a sliding window as shown in FIG. 15 has been introduced for network efficiency. This contributes to the effective use of the network bandwidth by a technique in which a plurality of packets may be transmitted over the network even before an acknowledgment is obtained.

[0005]

However, the above-mentioned real-time video and audio transmission apparatus has no problem in a highly reliable network environment, but has a problem such as a local area network (LAN) mainly used in computers. The network environment shared by unspecified majority has the following problems.

That is, when an error occurs on a network that secures these reliability and effectively uses the bandwidth, especially when an attempt is made to transmit moving images and voices in real time, a large amount of data is transmitted. A problem occurs that causes retransmission.

[0007] For example, Video as shown in the previous figure
Pack video and audio data in CD format,
When trying to transmit using a network, the GOP pattern is set to 15 frames of IBBP and the transmission rate is set to 1.5.
Mbps, it takes about 4 to transmit one GOP.
2 packs are needed, and about 84 frames are required for network frames. It is not known where the audio data is contained in these frames unless the received frame is returned to the pack and the packets in the pack are separated. In addition, the occurrence of retransmission results in further deterioration of the data transmission conditions on the transmission side and the network, and also causes a problem on the reception side such that audio and video are interrupted due to data interruption.

The present invention is directed to a transmission system, a transmission device, a reception device, and a storage medium that can increase the reliability of at least the second data when the first and second data are transferred in real time. It is intended to provide.

In another aspect of the present invention, encoded video data and encoded audio data transferred in real time are transmitted to a network as separate streams without time-division multiplexing into one bit stream. It is another object of the present invention to provide a real-time moving image and audio transmission system and apparatus, a control method thereof, and a storage medium that make it possible to enhance at least the reliability of audio.

[0010]

To solve this problem, for example, a transmission system according to the present invention has the following configuration. In other words, it is composed of a transmitting device that transmits the first data and the second data to be combined and reproduced, each being real-time data, in a packet format to another device via a network, and a receiving device that receives the same. System, wherein the transmitting device receives, from the receiving device, information indicating normal reception,
Transmitting means for transmitting a predetermined number of second packets of a plurality of first data to the receiving-side device via the network in response to a first predetermined number of packets of the second data; Retransmitting means for retransmitting the packet of the corresponding data when the request is made, wherein the receiving side apparatus receives the first packet of the second data after receiving the packet of the immediately preceding second data.
Receiving the second data packet following the second data within the allowable time, or receiving the first data packet within the second allowable time after receiving the immediately preceding first data packet. A first response means for returning information indicating that the packet has been normally received to the transmitting apparatus when a first data packet subsequent to the first data packet is received; And a second response unit that issues a retransmission request to the transmitting side device when the data packet of the transmission side is not received.

A real-time moving picture and audio transmission system according to another invention has the following configuration. In other words, the system includes a transmitting apparatus that transmits moving image data and audio data in a packet format in real time to another apparatus via a network and transmits a moving image and audio, and a receiving apparatus that receives the transmitting apparatus. As long as the transmission-side device receives information indicating normal reception from the reception-side device, for one audio data packet, a plurality of moving image data packets are transmitted via the network. A moving image transmitting means for transmitting to a receiving device; and a retransmitting device for retransmitting a packet of the data in response to a retransmission request from the receiving device. If the next audio data packet data is received within a first allowable time after receiving
When a packet of the next moving image data is received within a second allowable time after receiving a packet of the immediately preceding moving image data, a first response for returning information indicating normal reception to the transmitting side device Means for receiving the next packet of audio data even after the first permissible time has elapsed,
Second response means for issuing a retransmission request to the transmitting device;
If the next packet of moving image data is not received even after the second allowable time period or longer, it is assumed that the packet of the moving image data has been received, and information indicating normal reception is returned to the transmitting side. And third response means.

[0012]

Embodiments of the present invention will be described below in detail with reference to the accompanying drawings.

As will be apparent from the following description, in the embodiment, first, the moving picture and audio data transmitting side receives the time interval (GOP pattern), bit rate, and number of network frames of the data to be transmitted. Send to the side. On the receiving side, there are two timers, a timer for counting a time interval at which audio information will be received based on information such as a time interval sent from the transmitting side, and a timer for holding a time interval at which video information will be received. Then, a packet counter that holds the number of packets received by the network is initialized. Thereafter, the transmitting side sends out audio data at time intervals based on the information sent earlier, and sends out moving picture data. If the network is not congested, these data can be delivered to the receiving side without any problem. However, if the network is congested, some data may be lost on the network or may not arrive in time. . In this case as well, the retransmission request is transmitted to the transmitting side when the audio data is not received by the two timers and the packet counter set previously, and at the fixed time indicated by the timer when the moving image data is not received. Acknowledgment is forcibly transmitted, and as an error process, for example, the image data at the time of successful reception is continuously displayed.

FIG. 1 is a system block diagram of the embodiment, showing respective functional block diagrams of a transmitting apparatus and a receiving apparatus.

In the transmitting apparatus, for example, a moving image signal obtained from a camera and an audio signal obtained from a microphone are converted into coded data by respective coding circuits 1 and 2. The encoded audio and image data are passed to the transmission-side control circuit 3, and as shown in FIG. 2A, the audio data is only audio data and remains in a small chunk, while the moving image data is ), And sends it to the transmission circuit 4. The transmission circuit 4 further divides the data into several blocks if necessary, as shown in FIG. 2B, for example, in a form conforming to the standard of the network, and transmits the divided blocks to the network.

Thereafter, the transmission control circuit 3 performs the retransmission processing as shown in FIG. 3 when the arrival confirmation of the audio data is not performed based on the transmission state obtained from the transmission circuit 4 (when the retransmission request is received). Execute

The transmitted audio data and image data are received by the receiving circuit 5 of the receiving device, and are decoded by the audio decoding circuit 7 and the moving image decoding circuit 8 via the receiving control circuit 6 and finally. Is output.

Here, the receiving side control circuit 6 checks the timers 9 and 10 and the counter 11 which are set based on the transmission information (GOP pattern, transmission interval, bit rate, number of frames, etc.) received earlier. The reception interval (timed by the timer 9) of the audio data is the reception interval + α (>
0), a retransmission request is transmitted to the transmitting side (FIG. 3). On the other hand, as shown in FIG. 4, a reception interval in which moving image data is calculated from a transmission interval and a bit rate +
If not received within β (> 0), the control circuit 6 forcibly sends ACK to the transmitting side (returns ACK as normal reception) and sends an error signal to the video decoding circuit 8. hand over. When receiving the error signal, the moving picture decoding circuit 8 maintains the previous screen. The timer 9 is for checking the reception interval of audio data, and the timer 10 is for checking the reception interval of moving image data.

This is because, for example, in the case of a moving image, even if it is not received within the set time (reception interval + β), it is only for the instantaneous frame that constitutes a continuous moving image. It is. That is, even if the immediately preceding image is continuously displayed (even if it freezes), it is only a moment and does not significantly affect human vision.

On the other hand, with respect to voice, when a system in which both devices are provided with a camera and a microphone is constructed, there is a problem that a conversation is interrupted. The reason is that even if the output state immediately before is maintained as in the case of a video, the intention of the speaker is not transmitted to the other party unless audio data that could not be received is reproduced.

As for the operation on the transmitting side, as described above, it is only necessary to perform processing of transmitting data and retransmitting the corresponding data only when there is a retransmission request. 5 shows the state transition of the control circuit 6 on the side of FIG.
Will be described. In the following description, when the time set in the timers 9 and 10 has elapsed, respective interrupts are performed.

In FIG. 5, the receiving side control circuit controls the timer 9 and the timer 1 according to the transmission conditions transmitted from the transmitting side.
0, set the counter 11, and enter the "Idle" state. This state is maintained until audio data is received, and when other data is received, the received content is discarded (ignored). When receiving the audio data, the control circuit 6
Transitions to the "A Receive" state. In this state, the timer 9 and the timer 10 are cleared, the number of packets is set in a counter holding the number of received packets required for moving image transmission, and a flag (Error) indicating that no reception error has occurred in the moving image decoding circuit 8 is set.
mes). That is, at this time, the timers 9 and 10 start measuring the time. Also, an acknowledgment (Ack) indicating that the audio data has been received is transmitted to the transmitting side, and the state shifts to the “Video Wait” state.

In the "Video Wait" state, the flag (Error Mes) and the time of the moving image data sent from the transmitting side are checked. If the moving image data packet is received within the time set by the timer 10, The state transitions to the “Normal Proc” state. Here, after returning the received moving image data to the original pack state, the data is transferred to the moving image decoding circuit 8, the timer 10 is cleared, and the counter value is reduced by one. Then, a reply to the transmitting side that the moving image data has been normally received (Ack)
Is transmitted, the state returns to the original state Video Wait.

Thereafter, in the "Video Wait" state, when the value of the timer 10 is equal to or less than the set value and the value of the counter is other than 0, the state is maintained to prepare for the reception of the next moving image data. .

If the value of the counter 11 is other than 0 and the image data is not received even if the value of the timer 10 becomes equal to or more than the set value, "Video Wait"
State to the “VError” state (Timer
10 Int interrupt).

In the "VError" state, the timer 1
It clears 0 and decrements the counter value by one assuming that the corresponding moving picture packet has been received. Then, in order to instruct the moving picture decoding circuit 8 to maintain the display of the immediately preceding picture, the flag Error_mes is set, and the Ack is transmitted to the transmitting side as if it was normally received.
The state returns to the “Video Wait” state.

Thus, when the value of the counter 11 becomes 0, the state proceeds to the "Idle" state. The point to be noted here is that the timer 9 sets “A Rec
That is, the clocking is continued from the point of clearing in the "eive" state. If the value of the timer 9 exceeds the set time, the state of “Idle” is changed to “A Error”.
transition to the “r” state (Timer 9 Int interrupt occurs). In the “A Error” state, the timer 9 is cleared, a retransmission request for voice is issued to the transmitting side, and “I
dle "state.

FIG. 6 shows a specific configuration of the system. The illustrated system includes, for example, two information processing apparatuses 1000 and 2000 (for example, personal computers) connected to a network, and forms a so-called video conference system.

In the example described above, one device is described as a transmitting device and one device is described as a receiving device. However, each of the illustrated devices 1000 and 2000 functions as both a transmitting device and a receiving device. Since 100 to 111 and 200 to 211 are the same, the apparatus 1000 will be described here.

Reference numeral 100 denotes a CPU for controlling the entire apparatus, and 1
01 is an R that stores a boot program, BIOS, etc.
OM, 102 is a RAM that functions as a main memory. A hard disk 103 stores an OS and an application program for functioning as a video conference system. Reference numeral 104 denotes a network interface, for example, a card for connecting to Ethernet. 105 is a keyboard, 106 is a mouse, 10
7 is a display device. Reference numeral 108 denotes a capture unit that captures an image captured by the video camera 109 and generates encoded data. An audio input unit 110 encodes an audio signal from the microphone 111. An audio output unit 112 decodes the audio signal and outputs the decoded signal to the speaker 113. Timers 114a and 114b function as the timers 9 and 10 described above. The counter 11 is a register or a RAM in the CPU 100.
The variable secured in 102 is substituted.

In the above configuration, the devices 1000 and 200
When the power is turned on to 0 and a program for performing the audio / video transfer process and a program for displaying the received voice / video are activated, a television start system is constructed.

The transmission process of the apparatus 1000 will be described with reference to the flowcharts of FIGS.

First, in step S1, processing at the initial stage of connection, that is, transmission conditions of audio and video to the destination apparatus 2000 (transfer period of encoded audio data,
The transmission cycle and the number of packets of the encoded moving image data are transmitted.

Thereafter, the process proceeds to steps S2, S3, and S4, and controls the capture unit 110 and the audio input unit 110 to capture and input audio of each video frame constituting the moving image according to the conditions. Input the encoded data.

As described above, the encoded audio data is transferred as it is, and the encoded encoded data is packed in a predetermined size, and then the audio data packet, the moving image data packet, the moving image data packet. Packets (set number of packets), audio data, etc. are transferred.

During this time, A is transmitted from the device 2000 as the transfer destination.
Since the ck information or the retransmission request is received, the transfer is continued when the Ack is received.

When a retransmission request is received, an interrupt process shown in FIG. 8 is performed. Specifically, in step S5, the packet of the latest audio data among the transmitted audio data is retransmitted.

As is apparent from the above description, since only the voice data packet is received for the retransmission request, it is sufficient to retransmit the most recently transmitted voice data packet. That is, the apparatus 2000 may simply issue a retransmission request, and does not need information for specifying the data to be retransmitted.

On the other hand, the apparatus 1000 also receives audio data and moving image data packets from the apparatus 2000, and reproduces and outputs the packets.

The processing in this case will be described with reference to FIG.

First, in step S11, the device 20
The communication condition transmitted from 00 is received. In the next step S12, the timer 114 is set according to the received communication condition.
a, 114b determine the allowable time to be counted, and determine the initial value to be given to the counter (the register of the CPU 100 is used).

In step S13, first, an audio data packet is received and output to the audio output unit 112 for reproduction and output. Also, in step S14, the timer 11
4a and 114b are cleared to zero, and the counter is set to the previously determined initial value.

Next, the flow advances to step S15 to wait for reception of a moving image data packet, and when it has been received, a reproduction process is performed. Then, in step S16, the counter is decremented by "1", and the timer 114b is cleared to zero. Thereafter, in step S17, steps S15 and S16 are repeated until the value of the counter becomes "0". When the counter reaches "0", it is time to receive the next audio data packet, so the flow returns to step S13, and the above processing is repeated.

During the above processing, if the timer 114a measures the set time, that is, if it is not possible to receive the audio data packet when it is time to receive it, the timer 114a of FIG. Perform interrupt processing.

Here, in step S21, the timer 114
a is cleared to zero, and a retransmission request is sent to the device 2 in step S22.
000.

Also, when there is an interrupt from the timer 114b, that is, when the moving image data cannot be received even after the previously determined time has elapsed since the previous packet of the moving image data was received, The processing of FIG. 11 is performed.

First, in step S31, the timer 114b is cleared to zero, and in step S32, the counter is decremented by "1" so that the reception has been performed.

Next, in step S33, the video of the previous frame is displayed again, and in step S34, the Ack information is transmitted to the device 2000 as if it were normally received.

As described above, according to the embodiment, even in the real-time moving image and audio transmission in a network environment shared by an unspecified number of users, such as a computer network, the audio is not interrupted, and a good condition is maintained. Can be maintained.

Further, even when a transmission error occurs, the load on the network is small and a good network state can be supplied.

Although the present invention has been described with reference to an example including a plurality of devices (for example, a camera, a microphone, a speaker, a display device, and a host computer), it goes without saying that a device in which these devices are integrated may be used. Absent.

Further, the object of the present invention is that although a device on the transmitting side requires hardware for inputting video and audio and a device on the receiving side requires hardware for audio output, it is basically based on a personal computer. It can be built. Therefore, a storage medium storing the program code of software for realizing the functions of the above-described embodiments is supplied to a system or an apparatus, and the computer (or CPU or MPU) of the system or apparatus stores the program code stored in the storage medium. Is also achieved by reading and executing

In this case, the program code itself read from the storage medium realizes the functions of the above-described embodiment, and the storage medium storing the program code constitutes the present invention.

As a storage medium for supplying the program code, for example, a floppy disk, hard disk, optical disk, magneto-optical disk, CD-ROM, CD
-R, a magnetic tape, a nonvolatile memory card, a ROM, or the like can be used.

When the computer executes the readout program code, not only the functions of the above-described embodiment are realized, but also the OS (Operating System) running on the computer based on the instructions of the program code. ) May perform some or all of the actual processing, and the processing may realize the functions of the above-described embodiments.

Further, after the program code read from the storage medium is written into a memory provided in a function expansion board inserted into the computer or a function expansion unit connected to the computer, based on the instructions of the program code, It goes without saying that the CPU included in the function expansion board or the function expansion unit performs part or all of the actual processing, and the processing realizes the functions of the above-described embodiments.

As described above, according to the present embodiment,
The coded video data and coded audio data transmitted in real time are transmitted to the network as separate streams without time-division multiplexing into one bit stream, so that the By setting the hardware and the protocol so that the priority can be freely set, and suppressing the occurrence of retransmission, it is possible to suppress the occurrence of the above-described problem that both audio and video are interrupted.

Also, in a real-time moving picture and audio transmission in a network environment shared by an unspecified number of users such as a computer network, a relatively good state can be maintained without interruption of audio. In addition, even when a transmission error occurs, it is possible to supply a good network state with little burden on the network.

Further, in the embodiment, a plurality of moving images are transmitted for one audio packet. However, the present invention is not limited to this, and the number thereof can be variously modified. Also,
In the embodiment, a moving image is used as the first data, and audio data is used as the second data. However, the present invention is not limited thereto, and audio data may be used or a combination of moving images may be used.

[0060]

As described above, according to the present invention, it is possible to increase the reliability of at least the second data when the first and second data are transferred in real time.

According to another aspect of the present invention, the encoded moving image data and the encoded audio data transferred in real time are not time-division multiplexed into one bit stream, but are separated into a network as separate streams. , And at least the reliability of voice can be increased.

[0062]

[Brief description of the drawings]

FIG. 1 is a diagram illustrating an example of a real-time moving image and audio transmission device according to an embodiment.

FIG. 2 is a diagram illustrating an example of a transmission bit stream in the embodiment.

FIG. 3 is a diagram illustrating an example of an operation when an error occurs during transmission of audio data in the embodiment.

FIG. 4 is a diagram illustrating an example illustrating an operation when an error occurs in transmission of moving image data in the embodiment.

FIG. 5 is a state transition diagram illustrating an operation of a reception-side controller used in the embodiment.

FIG. 6 is a diagram showing a specific system configuration diagram in the embodiment.

FIG. 7 is a flowchart showing an operation processing content in the transmitting device.

FIG. 8 is a flowchart showing the operation processing content in the transmitting device.

FIG. 9 is a flowchart illustrating an operation processing procedure in the reception-side device.

FIG. 10 is a flowchart illustrating an operation processing procedure in the receiving device.

FIG. 11 is a flowchart illustrating an operation processing procedure in the receiving device.

FIG. 12 is a diagram illustrating a configuration example of a general real-time audio and video transmission system.

FIG. 13 is a diagram illustrating an example of a general transmission bit stream.

FIG. 14 is a diagram showing a sequence of acknowledgment and retransmission performed in the computer network.

FIG. 15 is a diagram showing a sliding window used in a computer network.

[Explanation of symbols]

 REFERENCE SIGNS LIST 1 audio encoding circuit 2 video encoding circuit 3 transmission control circuit 4 transmission circuit 5 reception circuit 6 reception control circuit 7 audio decoding circuit 8 video decoding circuit 9, 10 timer 11 counter 20 multiplexing circuit 21 separation circuit

Claims (16)

[Claims] 1. A transmitting apparatus for transmitting first data and second data to be combined and reproduced, each being real-time data, to another apparatus via a network in a packet format, and a receiving apparatus for receiving the same. In the system configured as described above, as long as the transmission-side device receives information indicating normal reception from the reception-side device, a plurality of packets for a first predetermined number of packets of the second data are Transmitting means for transmitting a predetermined number of second packets of the first data to the receiving side device via the network, and retransmission for retransmitting the packet of the data in response to a retransmission request from the receiving side Means, wherein the receiving-side device receives a second data packet following the second data within a first allowable time after receiving the immediately preceding second data packet. In the case, or when the first data packet subsequent to the first data is received within the second allowable time after the reception of the immediately preceding first data packet, the packet is normally received by the transmission side device. The first to return information indicating that
And a second responding unit that issues a retransmission request to the transmitting device when the next packet of the second data is not received even after the first allowable time has elapsed. Characteristic transmission system. 2. A transmission device for transmitting, in a packet format, first data and second data to be combined and reproduced, each being real-time data, to another device via a network, wherein: As long as the information indicating the normal reception is received, the predetermined number of second packets of the plurality of first data is received via the network for the first predetermined number of packets of the second data. A transmission device comprising: a transmission unit that transmits a packet to a side device; and a retransmission unit that retransmits a packet of data when there is a retransmission request from the reception side. 3. A receiving apparatus for receiving, via a network, first data and second data to be combined and reproduced, each being real-time data, in a packet format. When a packet of the second data following the second data is received within a first allowable time after reception, or within a second allowable time after receiving a packet of the immediately preceding first data. Receiving a packet of the first data following the first data, the first device returns information indicating normal reception to the transmitting device.
And a second responding unit that issues a retransmission request to the transmitting device when the next packet of the second data is not received even after the first allowable time has elapsed. Receiving device. 4. A transmission-side device that reads and executes a computer to transmit first data and second data to be combined and reproduced, each being real-time data, in a packet format to another device via a network. And a storage medium storing a program for causing the system to function as a system including a receiving device that receives the program, wherein the transmitting device is configured to receive, from the receiving device, information indicating normal reception. Image transmitting means for transmitting a predetermined number of second packets of a plurality of first data to the receiving side device via the network in response to a first predetermined number of packets of the second data; When there is a retransmission request from, a program code that functions as retransmission means for retransmitting the packet of the corresponding data, If a second data packet following the second data is received within a first allowable time after receiving the immediately preceding second data packet, or the immediately preceding first data packet is received. Then, when a packet of the first data following the first data is received within a second allowable time, the first device returns information indicating normal reception to the transmitting device.
And a program code functioning as second response means for issuing a retransmission request to the transmission side device when the next second data packet is not received even after the elapse of the first allowable time. The storage medium in which it was stored. 5. A transmitting apparatus for transmitting moving image data and audio data in packet format in real time to another apparatus via a network and transmitting a moving image and audio data, and a receiving apparatus for receiving the transmitting apparatus. In the system, the transmitting device transmits a plurality of moving image data packets to the network for one audio data packet as long as information indicating normal reception is received from the receiving device. Moving image transmitting means for transmitting to the receiving side device via the receiving side, and retransmitting means for retransmitting a packet of the corresponding data when a retransmission request is issued from the receiving side, wherein the receiving side device has When receiving the next audio data packet data within the first allowable time after receiving the data packet, or after receiving the immediately preceding moving image data packet A second response unit for returning information indicating that the packet has been normally received to the transmitting side device when a packet of the next moving image data is received within the second allowable time; Second responding means for issuing a retransmission request to the transmitting side device when the next packet of audio data is not received, and not receiving the next packet of moving image data even if the second allowable time period or more is reached A real-time moving image and audio transmission system, comprising: a third responding unit that receives a packet of the moving image data and returns information indicating normal reception to the transmitting side. . 6. The transmission-side apparatus, prior to transmitting audio and moving image data to the reception-side apparatus, sets the first allowable time, the second allowable time, and one audio data packet. The real-time moving image and audio transmission system according to claim 4, wherein the number of packets of the moving image data is notified to the receiving side device. 7. The transmitting apparatus according to claim 5, further comprising: a voice input unit for transmitting voice data to be transmitted to the receiving side device; and a moving image input unit for inputting a moving image. A real-time moving image and audio transmission system according to the section. 8. The real-time video and audio transmission system according to claim 7, wherein the audio input device includes a microphone, and the video input unit includes a video camera. 9. The apparatus according to claim 5, wherein the receiving side device includes a reproducing unit for reproducing and outputting based on the received audio data packet and the moving image data packet.
A real-time moving image and audio transmission system according to the section. 10. The reproduction device according to claim 9, wherein, when reproducing the moving image data, when the third response unit is activated, the reproduction of the immediately preceding image is maintained.
A real-time moving image and audio transmission system according to the section. 11. An audio and video reproducing apparatus for reproducing and outputting moving image data and audio data in a packet format received via a network, wherein the apparatus is within a first permissible time after receiving a packet of the immediately preceding audio data. When the next audio data packet data is received, or when the next moving image data packet is received within the second allowable time after receiving the immediately preceding moving image data packet, A first responding unit for returning information indicating that the packet has been received, a second responding unit for issuing a retransmission request to the transmission source device when the next packet of voice data is not received even after the first allowable time has elapsed. Response means for receiving the moving image data packet when the next moving image data packet is not received even after the second permissible time period has elapsed. And a third responding means for returning information indicating that the information has been normally received to the original device. 12. The audio and moving image reproducing apparatus according to claim 11, further comprising reproducing means for reproducing and outputting based on the received audio data packet and moving image data packet. 13. The reproduction apparatus according to claim 1, wherein the reproduction unit maintains reproduction of the immediately preceding image when reproducing the moving image data, when the third response unit is activated.
3. The audio and video reproduction device according to item 2. 14. A method for controlling an audio and video reproduction apparatus for reproducing and outputting moving image data and audio data in packet format received via a network, comprising: receiving a packet of the immediately preceding audio data; When the next audio data packet data is received within the allowable time, or when the next packet of the moving image data is received within the second allowable time after receiving the packet of the immediately preceding moving image data, the transmission is performed. A first response step of returning information indicating normal reception to the source apparatus, and a retransmission request to the transmission source apparatus when the next audio data packet is not received even after the first allowable time or more. A second response step of issuing a moving image data packet when the next moving image data packet is not received even after the second allowable time period or more. And a third response step of returning information indicating normal reception to the transmission source device. 15. A transmission-side device for transmitting moving image data and audio data to another device via a network in real time in a packet format by transmitting and receiving the moving image data and audio data in a computer and receiving the same. A storage medium storing a program that functions as a system configured by a receiving device, wherein the transmitting device receives, from the receiving device, information on audio data as long as information indicating normal reception is received. A moving image transmitting unit that transmits a plurality of moving image data packets to the receiving side device via the network for one packet, and when there is a retransmission request from the receiving side, the corresponding data packet is transmitted. A program code functioning as a retransmitting means for retransmitting, wherein the receiving side apparatus receives a packet of the immediately preceding audio data, When the next audio data packet data is received within the first allowable time, or when the next video data packet is received within the second allowable time after receiving the immediately preceding video data packet, First responding means for returning information indicating that the data has been normally received to the transmitting device; and when the next audio data packet is not received even after the first allowable time has elapsed, the transmitting device is transmitted to the transmitting device. A second responding unit for issuing a retransmission request; and when the next moving image data packet is not received even after the second permissible time period, the packet of the moving image data is received. A storage medium storing a program code for functioning as third response means for returning information indicating that the information has been normally received. 16. A storage medium storing a program for functioning as an audio and video reproduction apparatus for reproducing and outputting moving image data and audio data in packet format received via a network, and receiving a packet of the immediately preceding audio data. Then, when the next audio data packet data is received within the first allowable time, or when the next moving image data packet is received within the second allowable time after receiving the immediately preceding moving image data packet. First responding means for returning information indicating normal reception to the transmission source device if received, and transmitting the next audio data packet if the next audio data packet is not received even if the first allowable time or more is reached. Second responding means for issuing a retransmission request to the original device; and when the next moving image data packet is not received even after the second allowable time period or more, the moving image A storage medium storing a program functioning as a third response unit for returning information indicating that data packets have been normally received to the transmission source device, assuming that a data packet has been received.