JP2002058002A - Video telephone device - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a video telephone device wherein, even when packet data arrives continuously after a jitter has been generated in a network, a receiving buffer is hard to overflow, a delay in a display image generated by displaying the packet data arriving to be later than an ordinary timing is prevented and dynamic picture image data received after the receiving buffer has become empty due to the jitter can be displayed normally without being destroyed. SOLUTION: The video telephone device is provided with a jitter generation means which decides whether the jitter is generated in the network and a bit- rate control means which lowers an encoding bit rate to a prescribed rate when the jitter is generated in the network. Additionally, the device is provided with a dynamic picture image data output control means which does not output the dynamic picture image data created from the input packet data when the jitter is generated in the network.

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention
rotocol) Regarding a videophone device using a network, in particular, even if packet data continuously arrives after jitter has occurred in the network, the reception buffer is unlikely to overflow, thus delaying from normal timing. The present invention relates to a videophone device capable of preventing a delay of a display image caused by displaying arrived packet data, and capable of displaying normally without discarding moving image data received after a reception buffer becomes empty due to jitter.

[0002]

2. Description of the Related Art In recent years, videophone devices (systems) using an IP network have become widespread. But,
When jitter occurs in the IP network and the packet data stays in the network, the packet data accumulated in the network after the jitter is eliminated is input to the reception buffer of the receiving videophone device later than usual. In other words, jitter (fluctuation) in the network
Has occurred, packet data that should have arrived at a normal (normal) timing during the period in which the jitter has occurred arrives at a shorter interval than the normal reception timing after the jitter is eliminated. .

[0003] It has been pointed out that in the conventional videophone apparatus, the reception buffer overflows because the packets continuously arrive after the jitter is eliminated.

In addition, when the display interval time of moving image data of the receiving side videophone device is the same as the capturing time of moving image data of the transmitting side videophone device, the capacity of the reception buffer becomes empty due to the occurrence of jitter. If there is a period during which the moving image (display) data is not updated on the image display device at a normal timing, even if the jitter is eliminated and the packet data arrives and the display of the moving image data is resumed, this display is not displayed. The display delay for the period not performed, that is, the display delay of the moving image data due to the jitter delay cannot be eliminated. This will be described with reference to FIGS. 6 and 2A to 2C.

In the following description, the video telephone 6
At 00, it is assumed that the call connection is normally performed with another videophone device connected to the network, and communication of audio data and moving image data has started. The configuration of the other videophone device is the same as that of the videophone device 600.

[0006] Here, it is assumed that the flow control device 114 has acquired information on the encoding method and the decoding method of audio data and moving image data used by each terminal at the time of call connection. In addition, the interval at which the moving picture data of the transmitting videophone device 600 is fetched and the receiving videophone device 60
It is assumed that the interval at which moving image data of 0 is displayed is the same, and one frame of moving image data is transmitted as one packet data. For example, the moving image input device 103 on the transmitting side has a moving image capturing capability of 30 frames per second, packetizes one frame of data into one packet and transmits the data, and the moving image display device on the receiving side also displays a moving image of 30 frames per second. This is the case, for example, when it has an image display capability.

FIG. 6 is an internal configuration diagram of a conventional videophone device. According to FIG. 6, the videophone device 600 includes a voice input device 102, a video input device 103, an encoding device 106, a packetizing device 108, a transmission buffer 111, a communication control device 113, a reception buffer 112
, A streaming device 109, a decoding device 107,
It has an audio output device 104 and a moving image buffer 105, and is connected to the other videophone device via a communication line 116.

[0008] The transmission buffer 111 of the video telephone 101 on the transmitting side stores moving image data which is input from the moving image input device 102, is encoded by the device 106, and is packetized by the packetizing device 108. This packet data is transmitted to the communication line 116 via the communication control device 113.
, And input to the receiving-side videophone device 101. Communication control device 113 of receiving side videophone device 101
Of the packet data stored in the receiving buffer 112 via the streaming buffer 109, the data added at the time of the packetization by the streaming device 109 is deleted to become moving image data, and the moving image data is decoded by the decoding device 107. Is output to This moving image data is sent to the image display device, and the image is displayed.

FIG. 2A shows a video telephone 6 on the transmitting side.
12 is a diagram illustrating an example of a timing at which the transmission buffer 111 of 00 transmits packet data to the communication line 116 via the communication control device 113. FIG. As shown in FIG. 2A, the transmission buffer 111 sends out packet data at a certain timing. FIG. 2B shows packet data received by the reception buffer 112 of the videophone device 600 on the receiving side via the communication control device 113. In FIG. 2, the time axis is taken from the left side to the right side.

When the numbers given to the packets shown in FIGS. 2A and 2B are the same, it indicates that these packets are the same. That is, FIG.
The packet denoted by “1” in (a) is the same as the packet denoted by “1” in FIG. That is, FIG.
(A) and (b) show the video telephone device 60 on the transmitting side.
0 indicates that the packet “1” transmitted from the transmission buffer 111 to the communication line 116 via the communication control device 113 is stored in the reception buffer 112 via the communication control device 113 of the videophone device 600 on the receiving side. ing. At this time, since it takes time from transmission from the transmission buffer 111 to reception by the reception buffer 112, the packet data marked “1” in FIG.
It is described on the right side of the packet data marked “1” in FIG.

In the example shown in FIG. 2, it is assumed that jitter has occurred on the communication line 116 during the period indicated by "J1" in (b). That is, it is assumed that jitter has occurred on the IP network. Therefore, during the “J1” period, the receiving-side videophone device 600 does not transmit the packet data “2” and the subsequent packet data from the transmission buffer 111 of the videophone device 600 on the transmitting side. Is not input to the receiving buffer 112 of the.
When jitter occurs in this way, a period occurs in which the packet output from the transmission buffer 111 of the transmission-side videophone device does not reach the reception buffer 112 of the reception-side videophone device.

When the jitter is eliminated, FIG.
As shown in the period indicated by "J2" in (b), packets arrive continuously. That is, the packet data accumulated in the network reaches the reception buffer 112 continuously at a transmission interval by the transmission buffer 111, that is, at an interval shorter than a normal packet reception interval. FIG.
As shown in (b), after the arrival of the packet data "8", the delay in the arrival of the packet data due to the jitter is eliminated, and the packet data arrives at a normal timing.

FIG. 2C shows an example of the update status of the moving image data in the image buffer 105. As described above, the interval at which the image buffer 105 of the video telephone device 600 on the receiving side updates the display data (moving image data) displayed on the image display device is determined as described above.
Is the same as the interval at which moving image data is fetched from the moving image input device 103. FIG. 2C illustrates updating of moving image data obtained by decoding the packet data received by the reception buffer 112 illustrated in FIG. FIG. 2C also shows FIG.
As in (a) and (b), the time axis is taken from the left to the right, and the moving image data with “1” is the same data (moving image data) as the packet data with “1”. Shall be included. That is, the packet data “1” input to the reception buffer 112 becomes moving image data “1” that is streamed by the streaming device 1 and decoded by the decoding device 107 and is input to the moving image buffer 105. .

In the "J1" period in FIG. 2B, data is not updated in the image buffer 105 because jitter occurs in the network. That is, since there is no packet data after the packet data “2” in the reception buffer 112, the image buffer 105 cannot be updated. Accordingly, during this time, the moving image data before the occurrence of the jitter, that is, the moving image data of the packet number “1” is displayed on the image display device.

In the “J2” period of FIG. 2B, the reception buffer 112 receives packet data continuously. Therefore, the moving image data is sequentially updated in the image buffer 105. That is, the moving image data of the packet number “2” is displayed on the image display device at an abnormal timing after the jitter is eliminated.

As described above, the receiving-side videophone device 6
00 is an interval at which display data (moving image data) to be displayed on the image display device is updated.
Have the same interval for capturing moving image data from the moving image input device 103. Therefore, once jitter occurs in the “J1” period, the interval at which the display data displayed on the image display device is updated does not return to the normal timing. That is, when the reception buffer 212 is once emptied even if the image buffer 205 is updated by decoding the packet data that has arrived continuously, FIG.
A delay occurs in the display image as in the display data subsequent to the display data indicated by “2” shown in (c).

In other words, as described above, if the moving picture input capability of the video input device of the transmitting video telephone device 600 and the moving image display capability of the receiving video telephone device 600 are the same, the receiving video telephone device 600 Image display of 1
If a frame is delayed, all subsequent image displays will also be delayed. The delay of the display image is not canceled until the packet loss occurs in the network and the transmitted moving image data is lost or the moving image data is discarded so that the reception buffer 112 returns to a normal timing. Would. It has been pointed out that when jitter occurs in the network, display delay accumulates.

Further, the receiving-side videophone device 600 includes:
If the display interval time of the moving image data is shorter than the capturing time of the moving image data of the transmitting videophone device, once the capacity of the receiving buffer 112 becomes empty due to the occurrence of jitter,
It has been pointed out that it takes a long time from when the jitter is eliminated and the packet starts to arrive to when the moving image data is displayed at the normal (normal) timing. Hereinafter, this will be described with reference to FIGS.
This will be described with reference to (b) and (d). Note that FIG.
2D, the time axis is taken from left to right as in FIGS. 2A and 2B, and the moving image data with “1” as in FIG. 2C is “1”. And the same data as the packet data marked with.

FIG. 2D shows the receiving-side videophone device 60.
0 is the same as FIG. 2C except that the update interval at which display data is displayed on the image display device is shorter than the interval at which the transmitting videophone device 600 takes in moving image data from the moving image input device 103. Condition. For example, the moving image input device 103 on the transmitting side has a moving image capturing capability of 10 frames per second, and packetizes and transmits one frame of data in one packet, whereas the moving image display device on the receiving side has 30 frames. This is a case where a moving image display capability of a frame per second is provided, and display can be performed at an interval shorter than a packet interval to be received.

As shown in FIG. 2D, in the period corresponding to the period "J1" during which jitter occurs, the display image is not updated with the display data marked "1".
During the period corresponding to “J2”, the reception buffer 112
, The moving image is displayed at intervals shorter than the moving image capturing capability of the moving image input apparatus 103 on the transmission side. The moving image data marked “8” is stored in the image buffer 205 at normal timing. In this manner, a certain period of time (until the moving image data is displayed at normal timing, that is, until the update interval of the image buffer 105 becomes the same as the transmission interval of the video telephone device on the transmission side (see FIG. 2). If
It is pointed out that it takes a time until the moving image data “8” is displayed.

On the other hand, in Japanese Patent No. 2947302, encoded data of an input signal is hierarchized, and components having a large influence on image quality are transmitted to packets first, or the input signal is band-divided. There is disclosed a first related art in which encoding is performed in accordance with a band, and packets are transmitted first from encoded data of a low-frequency component.

Japanese Patent Application Laid-Open No. H11-177623 discloses that a receiver device monitors jitter, controls this jitter and the amount of buffering of received data, and makes an adjustment for reproducing reproduced data. Two prior arts are disclosed. More specifically, the second conventional technique has a problem that both the buffering amount for receiving a transmission packet is reduced as much as possible to speed up cueing and the buffering amount is increased to eliminate interruption of reproduction. In order to solve the problem, the buffering amount on the receiving side should be reduced for the output from the transmission jitter calculator that indicates a long-term change in the network state, such as cutting off silence parts in conversation or discarding discardable data. Slowly change the buffering amount on the receiving side for short-term changes,
When the short-term deterioration state is eliminated, the buffering amount is returned to the original amount, and the buffering amount is controlled to be set optimally with respect to the temporal change of the delay amount.

[0023]

However, according to the first prior art, when a jitter delay occurs in a high-speed packet network, data obtained from a packet received within a video frame synchronization held by a receiving station is transmitted. Even if the image is decoded using, there is no loss of data that has a large effect on image formation, and it is possible to decode an image without disturbance without losing the real-time performance, but it is possible to continuously decode after jitter occurs. It cannot be said that any countermeasure is taken against the overflow of the reception buffer due to the received packet, the accumulation of display delay, the time required for returning to the normal display timing, and the like.

According to the second prior art, since packet data of a normal size is sent, it takes a very long time to receive the packet data at a normal timing after the jitter is eliminated. I need it. Therefore, the display delay is accumulated, and it takes time to return to the normal display timing.
It does not indicate an effective countermeasure.

The present invention has been made in view of the above-described problems, and when a jitter occurs in a network, a transmitting-side videophone device lowers an encoding bit rate of moving image data, thereby enabling a receiving-side videophone device. It is an object of the present invention to provide a video telephone apparatus which increases the apparent capacity of a receiving buffer of the type described above and prevents overflow of the receiving buffer due to packets that continuously arrive after the occurrence of jitter.

Another object of the present invention is that the display interval time of the moving image data of the receiving video telephone device is the same as the capturing time of the moving image data of the transmitting video telephone device, and a buffer is generated on the network. In this case, the display of the moving image data is not updated until the receiving videophone device receives the packet at the normal timing, and the display is updated after receiving the packet at the normal timing. It is an object of the present invention to provide a videophone device that prevents accumulation.

Another object of the present invention is to provide a buffering device on a network in which the time required to capture moving image data in the transmitting videophone device is longer than the display interval time of video data in the receiving videophone device. In such a case, a videophone device is provided in which the transmitting videophone device reduces the encoding bit rate of moving image data, thereby shortening the time required for the receiving videophone device to start displaying packets at normal timing. That is.

Another object of the present invention is to resume the display by displaying the coded data as the frame data until the moving image data can be displayed at a normal timing after the occurrence of the jitter. To reduce the time it takes
It is another object of the present invention to provide a videophone device capable of displaying a high-quality image.

Another object of the present invention is to provide a method for displaying moving image data, which has arrived during the period in which the jitter is occurring, until the moving image data can be displayed at a normal timing after the occurrence of the jitter. An object of the present invention is to provide a videophone device capable of updating a display.

[0030]

In order to achieve the above object, according to the invention of the video telephone apparatus according to the first aspect,
It is characterized by having jitter determining means for determining whether jitter has occurred in the network, and bit rate control means for lowering the encoding bit rate to a predetermined rate when jitter has occurred in the network.

According to the second aspect of the present invention, in the videophone device of the first aspect, the jitter determining means generates jitter in the network when packet data is not input at a timing determined at the time of call connection. It is characterized by judging that there is.

According to a third aspect of the present invention, in the video telephone set of the second aspect, the jitter determining means determines whether the packet data is input at a timing from a sequence number sequentially incremented and added to the input packet data. It is characterized in that it is determined whether or not it is.

According to a fourth aspect of the present invention, in the videophone device of the second aspect, the jitter determining means determines whether or not packet data is input at a timing based on time stamp information added to the input moving image data. Is determined.

According to a fifth aspect of the present invention, in the video telephone of the second aspect, the jitter determining means determines whether or not packet data is input at a timing based on time stamp information added to the input audio data. Is determined.

According to a sixth aspect of the present invention, in the video telephone of the first aspect, the jitter judging means includes a step of judging a jitter in the network when the capacity of the packet data input to the receiving buffer is equal to or smaller than the threshold value. It is characterized in that it is determined that an error has occurred.

According to a seventh aspect of the present invention, in the video telephone according to any one of the first to sixth aspects, the bit rate control means reduces the encoding bit rate of the moving image data.

According to the eighth aspect of the present invention, in the video telephone according to any one of the first to seventh aspects, when jitter is generated in the network, the moving image data generated from the input packet data is transmitted. A moving image data output control unit that does not perform output.

According to the ninth aspect of the present invention, in the videophone device of the eighth aspect, the moving image data output control means sets the interval at which the moving image data is output to the image display device as the moving image of the transmitting videophone device. When the interval is shorter than the capture interval, moving image data is output even if jitter occurs in the network.

According to the tenth aspect, the eighth aspect is provided.
Wherein the moving image data output control means outputs the moving image data even if jitter occurs in the network when the moving image data is intra-frame encoded data. .

According to the eleventh aspect of the present invention, an eighth aspect is provided.
Or, in the videophone device of 10, when the network has a jitter, the video data output control means further comprises: a video decoding data buffer for storing video data created from the input packet data. If the latest video data is not created than the video data stored in the video code data buffer,
The moving image data stored in the moving image code data buffer is output.

According to the invention of the twelfth aspect, the jitter judging means for judging whether or not the jitter is generated in the network, and, when the jitter is generated in the network, the encoding is performed on the transmitting-side video telephone. Bit rate changing means for lowering the bit rate to a predetermined rate;
It is characterized by having.

According to the thirteenth aspect, according to the first aspect,
In the videophone device of (2), when the moving image data output control means outputs the moving image data to the image display device at an interval shorter than the moving image capturing interval of the transmitting videophone device, the jitter in the network is reduced. It is characterized in that moving image data is output even if it occurs.

According to the fourteenth aspect of the present invention, the first aspect
In the videophone device of the second aspect, the moving image data output control means outputs the moving image data even when jitter occurs in the network when the moving image data is data encoded in a frame. I have.

According to the fifteenth aspect, the first aspect is provided.
The video telephone apparatus according to 2 or 14, further comprising a moving picture decoding data buffer for storing moving picture data created from the input packet data when the network has a jitter; The means outputs the moving image data stored in the moving image code data buffer when the latest moving image data is not created than the moving image data stored in the moving image code data buffer.

[0045]

Embodiments of the present invention will be described below in detail with reference to the accompanying drawings.

Hereinafter, a videophone device according to the present invention will be described. In addition, a video telephone system having a plurality of video telephone devices according to the present invention has a plurality of the video telephone devices connected to an IP network. This videophone device determines whether or not jitter has occurred in the network, and if the jitter has occurred in the network, reduces the encoding bit rate to a predetermined rate. When jitter occurs in the network, the moving image data created from the input packet data is not output. If the interval at which the moving image data is output to the image display device is shorter than the moving image capturing interval of the transmitting videophone device, the moving image data is output even if jitter occurs in the network. You may.
If the moving image data is intra-frame encoded data, the moving image data may be output even if jitter occurs in the network. In addition, when the network is experiencing jitter, the network further includes a moving image decoding data buffer that stores moving image data created from the input packet data, and the moving image decoding data buffer stores a moving image code data buffer. If the latest moving image data has not been created, the moving image data stored in the moving image code data buffer may be output.
Details will be described below.

<Configuration of First Videophone Apparatus> FIG.
1 is a configuration diagram illustrating a configuration of a first videophone device according to the present invention. The videophone device 101 is a circuit device that is connected to a videophone device (not shown) connected to a network via a communication line 116 and communicates with the videophone device such as audio data and moving image data.

The voice input device 102 is for inputting voice, that is, converting voice into a voice signal (data) and inputting it to the device. For example, a microphone or the like can be used. The moving image input device 103 converts a moving image such as a moving image capture device to be taken into a videophone device into moving image data and inputs the moving image data to the device. For example, a CCD (chargecoup)
led device) camera and NTSC (national television)
system committee) A camera or the like can be used.

The encoding device 106 includes the speech input device 10
2 and connected to the moving image input device 103 to encode the input audio data and moving image data to produce encoded data, and to use the encoded data as a packetizing device 1
08. The packetizing device 108 converts the encoded data into a predetermined packet which is a data format for performing communication on the communication line 116 via the communication control device 113, and creates packet data.

The transmission buffer 111 is a packetizing device 1
08 is a buffer memory for storing the packet data input from 08. This packet data is transmitted to the communication control device 1
13, the communication control device 113, the communication line 1
The video data is transmitted to the other party's videophone device via the communication device 16. The communication control device 113 transmits the packet data stored in the transmission buffer 111 to the other videophone device via the communication line 116. Also, it receives packet data (speech data and moving image data are encoded and packetized) transmitted from the other party's videophone device via the communication line 116 and inputs the packet data to the reception buffer 112. The reception buffer 112 is a buffer memory for temporarily storing packet data received from the communication line 116 via the communication control device 113.

The streaming device 109 deletes packet header information and the like added at the time of packetization, and converts the packet data stored in the reception buffer 112 into audio data,
Then, the data is divided into video data and stream data. The decoding device 107 includes a streaming device 109
And decodes the data input from to create audio data and moving image data. The audio data is output to the audio output device 104, and the moving image data is output to the moving image buffer 105.

The sound output device 104 outputs sound, that is, converts a sound signal into sound and outputs the sound. For example, a speaker or headphones can be used. The moving image buffer 105 includes an image display device (LCD, not shown).
(Liquid crystal display), a buffer for storing moving image data reproduced by the display, and sends the moving image data to the image display device. In addition,
The image display device may be inside the videophone device 101 or outside.

The data control device 110 performs various controls such as a call connection with the other party's videophone device and control of the encoding device 106 and the decoding device 107 during stream communication. The flow control device 114 includes a communication control device 113
Network status information such as network jitter information, packet loss information, and round trip time information sent from a communication partner via
And coding of voice / video of the partner terminal at the start of communication
The encoding device 106 and the communication control device 113 are controlled based on the decoding capability and the like (encoding control, transmission control).

The data input / output device 115 receives an instruction from the user to the video telephone device 101 and notifies the data control device 110 of the instruction. Further, predetermined information such as information indicating an incoming call from the partner terminal, information of the own terminal, and information input from the data control device 110 is notified to the user. For example, it has input means such as numeric keys and key buttons, a communication destination number is input, and an LED (light emitti
It has a notifying means such as an ng diode) lamp or an LCD, and notifies the user of the state of the terminal itself and displays information input from the data control device 110.

<First Operation> Hereinafter, a case will be described where a jitter occurs in a network of a video telephone which outputs moving image data to an image display device at the same interval as the video capturing interval of the video telephone of the transmitting side according to the present invention. The operation (first operation) will be described with reference to FIG. 1 and FIG. In this description, it is assumed that the capturing interval of the moving image data of the transmitting videophone device 101 and the display interval of the moving image data of the receiving videophone device 101 are the same. That is, the conditions are the same as in the description of the related art shown in FIG. It is assumed that the packet data transmission intervals of the transmitting side and the receiving side videophone device 101 are the same.

The video telephone apparatus 101 includes the reception buffer 1
Based on the timing of the packet data input to 12, it is determined whether or not jitter has occurred in the network. If jitter has occurred in the network, the bit rate of video data encoding is reduced, and small encoded data is transmitted. Make it. Details will be described below.

FIG. 3 takes a time axis from left to right as in FIG. FIG. 3A shows the packet data input to the reception buffer 112 of the transmitting videophone device 101. FIG. 3B shows the transmitting-side videophone device 1.
01 shows packet data transmitted from the transmission buffer 111, and (c) shows packet data input to the reception buffer 112 of the video-phone apparatus 101 on the receiving side.
(D) shows moving image data developed from the packet data in the moving image buffer 105 of the video telephone device 101 on the receiving side. The numbers given to FIGS. 3B to 3D are as follows.
This indicates that they all have the same moving image data. That is, the packet data in FIG.
The packet data and moving image data of (b) to (d)
Even if the same numbers are given, they are different.

The video telephone device 101 acquires the timing of receiving packet data when a call is connected to the video telephone device 101 of the communication partner. This can be calculated, for example, from the coding / decoding method of moving image data determined after exchanging information on the coding / decoding method of audio data / moving image data used by each terminal. The reception interval of (packet) data obtained by this calculation is
The information may be registered in the data control device 110.

FIG. 3A shows the transmission side videophone device 10.
1 shows an example for explaining packet data input to the reception buffer 112 in FIG. As shown in FIG. 3A, after the packet data "1" is input, jitter occurs in the network, and the packet data "2" is not input at a predetermined (normal, normal) timing. After the jitter is eliminated, the packet data from packet data “2” to packet data “7” is stored in the reception buffer 1.
12 are successively input. The packet data after the packet data "8" is input at a predetermined (normal, normal) timing.

The reception buffer 112 always monitors the reception timing of the packet data in the reception buffer 112. The monitoring of the reception timing may be performed by monitoring whether packet data is received at the predetermined (packet) data reception interval as described above. That is, since the packet data is added with the sequentially incremented sequence number at the time of transmission, it is necessary to compare the reception timing of the packet data determined at the time of the call connection with the lost (unreached) packet data. Thus, lost packet data (packet data not input at normal timing) can be grasped. That is, the reception buffer 112 checks the coincidence between the number of packets that should have arrived at the time of determination calculated from the reception timing determined at the time of call connection and the sequence number added to the input packet data, and thereby determines the packet loss Monitor the occurrence. Note that it may be determined whether or not packet data is received at the above-described reception interval using time stamp information or the like added to the moving image data. Alternatively, it may be determined whether or not packet data is input at a normal timing by using time stamp information or the like added to an audio packet (audio data).

When the packet data does not reach the predetermined timing, the reception buffer 112 determines that jitter has occurred in the network, and notifies the flow control device 114 of this. The flow control device 114 that has received this notification sets the data control device 110 to lower the (encoding) bit rate in the encoding of the moving image data.
To the encoding device 116 via.

As shown in FIG. 3A, packet data “2” is not input to the receiving buffer 112 of the transmitting-side videophone device 101 at a predetermined timing. Therefore, the reception buffer 112 is connected to the network (communication line 1).
It is determined that jitter has occurred in step 16), and this is notified to the flow control device 114. The flow control device 114
As described above, the encoding device 106 is notified to reduce the encoding bit rate.

The transmitting-side videophone device 1 receiving the above notification
01 coding apparatus 106 lowers the (coding) bit rate. Accordingly, the encoding device 106 performs code amount control at the time of encoding, produces encoded data having a smaller data amount than normal encoded data, and sends the encoded data to the transmission buffer 111 via the packetizing device 108. Store. Therefore, after the bit rate is lowered, the transmission buffer 111
Is smaller than the packet data of the normal packet data (packet data "1" to "3" shown in FIG. 3B) (the packet data "Packet data" shown in FIG. 3B). 4 "). The bit rate may be a predetermined rate or an arbitrary rate depending on the situation of jitter.

The communication control unit 113 controls the transmission buffer 1 regardless of whether or not jitter has occurred in the network.
The packet data stored in 11 is transmitted to the communication line 116 at normal timing. That is, both the packet data before the bit rate is reduced (labeled “1” to “3”) and the packet data after the bit rate is reduced (labeled “4” and thereafter) are in order. Are transmitted to the communication line 116 at a predetermined timing.

FIG. 3C shows the receiving-side videophone device 10.
The reception timing of the packet data input to one reception buffer 112 is shown. As shown in FIG.
During the period indicated by "1", the packet data does not arrive because jitter has occurred as described above. Then, "J
In the period indicated by "2", the jitter is eliminated and the packet data accumulated in the network continuously arrives.
After the packet data described as “7”, the packet data arrives at a normal timing. Note that the packet data subsequent to the packet data marked with “4” are encoded at a reduced bit rate as described above, and are therefore shown small in FIG. 3C.

The reception buffer 112 of the video telephone 101 on the receiving side monitors the reception timing of the packet data in the same manner as the video telephone 101 on the transmission side to receive the packet data. It monitors whether the timing delay has been resolved and packets are received at the normal timing. As this monitoring method, any method may be used. As described above, the sequence number added to the packet data is compared with the reception timing of the packet data calculated based on the information determined at the time of the call connection. It may be done by doing. Also, it may be determined whether or not the reception timing is normal based on the time stamp information included in the moving image data, or the reception timing may be determined based on the time stamp information included in the audio packet. Is also good.

When the reception buffer 112 detects that jitter has occurred in the network (communication line 116), it notifies the flow control device 114 of this. The flow control device 114 receiving this notification instructs the decoding device 107 via the data control device not to output the moving image data to the moving image buffer 105. Upon receiving this instruction, the encoding device 107 does not output the produced moving image data to the moving image buffer 105. That is, as shown in the “J2” period of FIG. 3C, the packet data (“2” to
Even if the “6” packet data) has arrived at the reception buffer 112, no moving image data is input to the moving image buffer 105 while this instruction is issued. Therefore,
The moving image data during this period (“2” to “6”) is not displayed on the image display unit.

When the timing at which the packet data reaches the reception buffer 112 returns to normal, the reception buffer 112
Notifies the flow control device 114 of this. The flow control device 114 notifies the encoding device 107 via the data control device 110 to output the encoded moving image data to the moving image buffer 105. The encoding device 107 that has received the notification outputs the encoded moving image data to the moving image buffer 105. That is, the moving image buffer 105 receives moving image data (indicated as “7”) as shown in FIG. 3D and outputs this to the image display unit. As a method of determining whether the timing at which the packet data is input to the reception buffer 112 is a normal timing, for example, the determination may be made based on the sequence number added to the packet data as described above, The determination may be based on the time stamp information included in the moving image data, or the determination may be based on the time stamp information included in the audio data.

That is, after the jitter occurs in the network (communication line 116), while the arrival delay (of packet data) occurs due to the jitter, the image buffer 105 transmits the data before the occurrence of the arrival delay to the image display device. Keep displaying. In the case shown in FIG. 3D, while the arrival delay due to the jitter occurs, the data “1” before the arrival delay occurs.
Is continuously displayed. The display of the image display device is updated after the packet data starts to reach the reception buffer 112 at a normal timing. Therefore, in the case shown in FIG. 3D, after the moving image data indicated by “7” is decoded by the decoding device 107 and stored in the moving image buffer 105, the display on the image display device is updated. Thus, even when jitter occurs in the network, it is possible to prevent display delay of the image display device.

The communication side videophone device 101 returns the bit rate of the coding reduced by the coding device 106 to the original bit rate at a predetermined timing. For example, the predetermined timing may be a case where the number of packets that have continuously arrived at the reception buffer 112 is compared with the number of packets that have not arrived, and it is determined that they match. That is, the communication-side videophone device 1
After the timing of the packet data input to the No. 01 reception buffer 112 returns to normal, the encoding bit rate is returned to the original. Further, after the occurrence of jitter, it may be after packet data has arrived at the reception buffer.If the difference between the timing of packet data input to the reception buffer and the above-mentioned normal timing is smaller than a predetermined difference, May be returned.

Whether the timing at which the packet data is input is the predetermined timing may be determined based on the time stamp information as described above, or may be determined using a voice packet. Is also good.
As described above, the predetermined timing may be a timing irrelevant to the timing at which the packet data is stored in the receiving buffer 112 of the receiving videophone device 101.

According to the above-described embodiment, when jitter occurs in the network, the video-telephone device on the transmitting side lowers the encoding bit rate to prevent accumulation of packet delay in the video-telephone device on the receiving side. Becomes possible. Also, even when packet data arrives continuously after the jitter is eliminated, it is possible to delay the overflow of the receiving buffer of the receiving videophone device as compared with the case where the encoding bit rate has not been reduced. .
That is, by reducing the bit rate, the apparent capacity of the reception buffer, that is, the number of storable packets increases. Therefore, even if packet data continuously arrives after the jitter of the network is eliminated, it is possible to store a large amount of packet data.

As described above, the decoding device 107
Alternatively, only predetermined data may be output to the moving image buffer 105 from when the output of the decoded moving image data to the moving image buffer 105 is stopped until the decoding is restarted. As a result, the moving image (display) data is not updated at all until the jitter delay is eliminated. For example, assuming that the predetermined data is intra-frame encoded data, the difference between frames and the encoded data (Intra Frame / Intra Block) are not output to the moving image buffer 105 and the image display is performed. Nothing is displayed on the device. This prevents display data from being updated at all while preventing display delay.

The bit rate may be changed based on the capacity of the reception buffer 112. That is, when the capacity of the reception buffer 112 becomes equal to or less than the first threshold, the bit rate is reduced, and the capacity of the reception buffer 112 may be set to the second threshold (the same value as the first threshold). ), The bit rate may be returned to the original value. That is, when packet loss or jitter occurs, the amount of packet data input to the reception buffer 112 is reduced.
If the amount (capacity) of the data stored in the storage unit 2 becomes equal to or less than the threshold value, it may be determined that jitter has occurred in the network.

<Second Operation> Hereinafter, referring to FIG. 4, the interval at which the receiving-side videophone device 101 displays moving image (display) data on the image display device is determined by the moving image of the transmitting-side videophone device 101. The operation (second operation) of the videophone device in a case where the input device 103 is shorter than the interval for capturing moving image data will be described. FIGS. 4 (a) to 4 (d)
FIGS. 3A to 3D show a case where only the above conditions are different.
That is, the difference is that the interval at which the moving image (display) data is displayed on the image display device of the receiving videophone device 101 is shorter than the condition of FIG. 3. 4A to 4C are the same, and FIG. 3D and FIG.
Only (d) is different.

As described above, the transmitting-side videophone device 10
When the bit rate of the first encoding device 106 is reduced, the reception buffer 112 of the receiving videophone device 101
As shown in FIG. 4C, “J2” in which the jitter has been eliminated after the “J1” period in which the jitter has occurred in the network,
During the period, the packet data “2” and “3” coded at the normal bit rate arrive, and then the bit rate is reduced and the packet data after the coded packet data “4” arrives.

Here, the bit rate is set according to the first
The operation is performed in the same manner as described above. That is, it is assumed that the occurrence and elimination of jitter are confirmed by monitoring the reception timing of the reception buffer 112 of the transmission-side videophone device 101 or the amount of stored data, and the encoding bit rate is changed.

In this case, unlike the above-described first operation, the flow control device 114 outputs the moving image data decoded by the decoding device 107 to the moving image buffer 105. That is, as shown in FIG. 4D, the moving image data after the moving image data “2” is output to the moving image buffer 105 and displayed on the image display device, and the moving image indicated by “7” is displayed. From the image data, the image data is stored in the image buffer 105 at a normal timing and displayed on the image display device.

As described above, when jitter occurs in the network, the timing at which moving image (display) data is displayed on the image display device returns to normal earlier by reducing the bit rate. Further, as described above, since the encoding bit rate is reduced, it is possible to prevent overflow of the reception buffer.

Note that the videophone device 101 uses the first
Is determined based on the display capability of the image display device, the packet data reception interval of the reception buffer 112 calculated based on the coding method to be used determined at the time of the call connection, and the like. , May be performed by comparing. That is, if the result of this comparison satisfies the condition in the first operation described above, the first operation in which the moving image data is not output to the image buffer 105 is performed, and if the condition in the second operation is satisfied, The data may be controlled to be output to the image buffer 105.

<Configuration of Second Video Phone> A second video phone according to the present invention will be described below. FIG. 5 is a configuration diagram showing the internal configuration of the second videophone device according to the present invention. Videophone device 501 shown in FIG.
Is different from the videophone device 101 shown in FIG. 1 in that the videophone device 101 further includes a moving image decoded data buffer 517.

The moving image data output from the decoding device 507 is stored in the moving image buffer 505 in the same manner as the first videophone device before jitter occurs in the network. In the second videophone device, jitter occurs in the network and the reception buffer 51
When the capacity of the second apparatus becomes empty, or when the capacity of the second apparatus becomes equal to or smaller than a preset threshold value,
Moving image data produced in
17 is stored. This operation is performed until the reception buffer 512 receives packet data at normal (normal) timing. Whether or not packet data has been input at normal timing can be determined as described above.

The decoded moving image data buffer 517 can store moving image data for each display unit such as one frame. When the moving image data is stored for each display unit in this manner, the moving image decoded data buffer 5
17 outputs the moving image data to the moving image buffer 505 when the receiving buffer 112 does not store moving image data for the next display unit.

As described above, even when the jitter in the network is large and it takes a long time for the packet data to reach the reception buffer 512 at a normal timing, the display of the moving image data can be updated. Becomes

When the reception buffer 112 detects that a jitter has occurred in the network by the method described above, the reception buffer 112 lowers the encoding bit rate to the other party (transmission side) videophone device via the communication control device 113. , And the transmitting-side videophone device that has received the notification may lower the coding bit rate.

[0086]

As is apparent from the above description, according to the present invention, in a video telephone apparatus using an IP network, when a network jitter (fluctuation) in which the arrival time of a packet changes occurs, transmission is performed. The reception timing of the reception buffer that stores the packet data received by the local device is monitored, and when a packet is not received at the reception timing or when the capacity of the reception buffer falls below a predetermined threshold value, the trigger is used. Transmits coded packet data at a reduced bit rate for moving image data to be transmitted, preventing overflow of the reception buffer even if data arrives continuously after jitter has occurred on the receiving device. Becomes

Further, after the receiving side device receives the moving image data and jitters occur during the reception of the moving image data and the receiving buffer storing the moving image data becomes empty, the received moving image data can be displayed normally without being discarded. It is possible to restart.

Also, in addition to normal control parameters such as speed priority and image quality priority, jitter information is provided as a control parameter, and a moving picture coding apparatus for controlling the amount of code is thereby provided. As a result, the encoding bit rate on the transmitting side can be reduced by using this as a trigger, and as a result, more packets can be stored in the receiving buffer of the receiving videophone device, and overflow does not easily occur.

Also, the reception timing of the moving image data reception buffer is monitored, and if the packet data cannot be received at the reception timing, the bit rate is reduced and the moving image data is coded, whereby the packet data It performs a simple operation of checking the arrival timing,
If it is detected that the packet has not arrived at the normal timing, the transmitting videophone device can use this as a trigger to lower the encoding bit rate. Therefore, since the apparent capacity of the receiving videophone device becomes large, overflow does not easily occur even if packet data is continuously input after the jitter is eliminated.

Also, a threshold value is set in advance for the moving image data reception buffer, and when the buffer capacity becomes smaller than this value, the bit rate is reduced and the moving image data is decoded to store the moving image data in the reception buffer. It is possible to judge whether jitter has occurred in the network by a simple operation of checking the amount of packet data that has been transmitted, and the encoding bit rate is triggered by the fact that the amount of stored packet data has reached a predetermined value or less. It is possible to lower it. Therefore, the receiving-side videophone device can store more packet data, and is less likely to overflow even if packet data is continuously input after the jitter is eliminated.

Also, the reception timing of the audio packet reception buffer is monitored, and if the packet cannot be received at the reception timing, the bit rate is reduced and the moving picture data is encoded, so that the videophone on the reception side can be used. The device is unlikely to overflow even if packet data is continuously input after the jitter of the network is eliminated.

Further, a threshold value is set in advance for a reception buffer for receiving audio data, and when the buffer capacity becomes smaller than the set buffer capacity, the bit rate is reduced to encode the moving image data. The telephone device is unlikely to overflow even if packet data is continuously input after the jitter of the network is eliminated.

Also, when jitter occurs and packet reception is delayed, by transmitting moving image data encoded at a reduced bit rate, the apparent capacity of the receiving buffer of the receiving videophone device is increased, and jitter is reduced. It is possible to prevent overflow after the cancellation.

Further, when jitter occurs and packet reception is delayed, a request is made to the partner terminal to reduce the encoding bit rate of moving image data, so that a packet produced at a normal bit rate after the jitter is eliminated. This makes it possible to make it more difficult for the reception buffer to overflow as compared with the case where the input is made.

When jitter occurs, the bit rate is reduced to encode the moving image data. When the data arrives at the moving image data receiving buffer, the bit rate is restored to the original value and the moving image data is encoded. Is performed, the display delay of the moving image data that occurs when all the moving image data that continuously arrives after the jitter is eliminated is prevented.

Also, when data arrives at the audio data reception buffer while the moving image data is being encoded at a reduced bit rate when jitter occurs, the bit rate is restored to the original value. The encoding bit rate can be reduced only when jitter occurs in the network.

During the encoding with the bit rate lowered when the jitter occurs, if the influence of the jitter of the moving image data received after the jitter is eliminated and the reception starts at a normal timing, the bit rate is reduced. By returning to the original state, the encoding bit rate can be reduced only when jitter occurs on the network.

If the bit rate is lowered and the moving image data is being coded when the jitter occurs, the audio data received after the jitter has started to be received at normal timing without the influence of the jitter. By restoring the bit rate, the encoding bit rate can be reduced only when jitter occurs on the network.

After the jitter occurs and the moving image data receiving buffer becomes empty, even if the received moving image data is decoded, the decoded moving image data is not output to the image buffer for displaying.
It is possible to prevent display delay that occurs after the occurrence of jitter.

After jitter has occurred and the moving image data receiving buffer has been emptied, the received moving image data is decoded, and only the data obtained by decoding the data decoded by frame decoding is output to the image buffer and displayed. By doing so, it is possible to prevent a display delay that occurs after the occurrence of jitter, and to prevent the display of moving image data from being stopped at all for the purpose of preventing the display delay.

After the jitter occurs and the moving image data receiving buffer becomes empty, the received moving image data is decoded and stored in the buffer, the receiving state of the receiving buffer storing the received data is determined, and the received moving image data is stored in the buffer. By outputting the decoded data to the image buffer and displaying the decoded data, a display delay occurring after the occurrence of jitter is prevented, and the display of moving image data is prevented from being performed at all to prevent the display delay. It becomes possible.

[Brief description of the drawings]

FIG. 1 is a first internal configuration diagram showing the internal configuration of a videophone device according to the present invention.

FIG. 2 is a timing chart for explaining packet data transmission / reception and moving image data display timing of a videophone device according to a conventional technique.

FIG. 3 is a first timing chart for explaining packet data transmission / reception and moving image data display timing of the videophone device according to the present invention.

FIG. 4 is a second timing chart for explaining packet data transmission / reception and moving image data display timing of the videophone device according to the present invention.

FIG. 5 is a second internal configuration diagram showing the internal configuration of the videophone device according to the present invention.

FIG. 6 is an internal configuration diagram of a videophone device according to a conventional technique.

[Explanation of symbols]

 DESCRIPTION OF SYMBOLS 101 Video telephone apparatus 102 Audio input device 103 Video input device 104 Audio output device 105 Video buffer 106 Encoding device 107 Decoding device 108 Packetization device 109 Streaming device 110 Data control device 111 Transmission buffer 112 Reception buffer 113 Communication control Device 114 Flow control device 115 Data input / output device 116 Communication device 501 Videophone device 502 Audio input device 503 Video input device 504 Audio output device 505 Video buffer 506 Encoding device 507 Decoding device 508 Packetization device 509 Streaming device 510 data control device 511 transmission buffer 512 reception buffer 513 communication control device 514 flow control device 515 data input / output device 516 communication line 517 video decoding data Ffa 600 TV phone device

Continued on the front page F-term (reference) 5C059 KK34 KK35 PP04 RB02 RC04 SS07 SS08 SS30 TA60 TC20 TC21 TD12 UA02 UA05 UA38 5C061 BB03 CC03 5C064 AA01 AB04 AC01 AD06 AD14 BA07 BB05 BC16 BD02 BD05 BD08 CC13 CC11 KK SS07

Claims (15)

[Claims] 1. A television comprising: a jitter judging unit for judging whether or not a jitter is generated in a network; and a bit rate control unit for lowering an encoding bit rate when the jitter is generated in the network. Telephone equipment. 2. The videophone according to claim 1, wherein said jitter determining means determines that jitter has occurred in the network when packet data is not input at a timing determined at the time of call connection. apparatus. 3. The apparatus according to claim 2, wherein said jitter judging means judges whether or not the packet data is inputted at the timing from the sequence number sequentially incremented and added to the inputted packet data. Video phone equipment. 4. The television according to claim 2, wherein said jitter determining means determines whether or not packet data is input at said timing based on time stamp information added to the input moving image data. Telephone equipment. 5. The videophone according to claim 2, wherein said jitter determining means determines whether or not packet data is input at said timing based on time stamp information added to input audio data. apparatus. 6. The apparatus according to claim 1, wherein said jitter judging means judges that jitter has occurred in the network when the capacity of the packet data input to the receiving buffer is equal to or smaller than a threshold value. Video phone equipment. 7. The videophone apparatus according to claim 1, wherein said bit rate control means reduces a coding bit rate of moving image data. 8. A moving image data output control unit that does not output moving image data created from input packet data when jitter occurs in the network, further comprising: The videophone device according to any one of claims 1 to 7. 9. The moving image data output control means according to claim 1, wherein an interval of outputting moving image data to the image display device is shorter than a moving image capturing interval of the transmitting videophone device, and jitter occurs in the network. 9. The videophone device according to claim 8, wherein the videophone device outputs moving image data. 10. The moving image data output control means, when the moving image data is data encoded in a frame, outputs moving image data even when jitter occurs in a network. The videophone device according to claim 8. 11. When the network has a jitter, the video data output control means further comprises: a video decoding data buffer for storing video data created from the input packet data; If the latest moving image data is not created than the moving image data stored in the moving image code data buffer,
The videophone device according to claim 8, wherein the videophone device outputs the video data stored in the video code data buffer. 12. A jitter judging means for judging whether or not jitter has occurred in a network, and a bit rate change for causing a transmitting side video telephone apparatus to reduce an encoding bit rate to a predetermined rate when jitter has occurred in the network. A videophone device comprising: means. 13. The video data output control means, if the interval at which the video data is output to the image display device is shorter than the video capture interval of the video telephone device on the transmitting side, jitter occurs in the network. 13. The videophone device according to claim 12, wherein the videophone device outputs moving image data. 14. The moving image data output control means, when the moving image data is intra-frame coded data, outputs the moving image data even if jitter occurs in the network. The videophone device according to claim 12. 15. When the network has a jitter, the video data output control means further comprises: a video decoding data buffer for storing video data created from the input packet data; If the latest moving image data has not been created than the moving image data stored in the moving image code data buffer,
15. The videophone device according to claim 12, wherein the video phone device outputs the video data stored in the video code data buffer.