JP2007067826A - Audio and animation communication system and data communication terminal - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To control disturbance of synchronization timing of audio data and image data without deterioration of realtime property of audio packet in the bidirectional realtime communication of the audio data packet and image data packet. <P>SOLUTION: A TV adapter 102 obtains the transmission time information of audio packet of the communication partner having received with the IP phone 103 and the transmission time information of the audio packet added to the image packet, and compares these two transmission time information pieces. The TV adapter 102 reproduces the image when these information pieces are identical, and cancels the image when the audio packet transmission time added to the image packet is delayed. When the audio packet transmission time information added to the image packet is leading, the TV adapter 102 determines that the image is buffered. <P>COPYRIGHT: (C)2007,JPO&INPIT

Description

  The present invention relates to a data communication apparatus for transmitting and receiving audio data and video data via a network such as a LAN, the Internet, and an IP communication network, and a data communication system using the data communication apparatus.

  In recent years, with the spread of broadband, packetized audio data and video data are transmitted to the other party's device via a network such as the Internet and IP communication network, and the packetized voice data and video data are transmitted from the other device. Data communication apparatuses that perform real-time communication in both directions by receiving them have been put into practical use.

  In this type of data communication device, audio data and video data are each divided into a plurality of packets and transmitted sequentially over the network. However, if the data is transmitted through a route with no bandwidth guarantee, the data is lost. Due to occurrence of a shift in data transmission time or the like, audio data and video data cannot be transmitted and received in real time, and synchronization timing of audio and video reproduction may be disturbed.

  The real-time property and the synchronization timing shift in the transmission / reception of different packets are essentially in a trade-off relationship, and it is impossible to completely satisfy both.

  In particular, since audio data has a small packet size, the influence of transmission on a route for which bandwidth is not guaranteed is large, and real-time performance may be deteriorated due to packet loss and delay, and audio interruption may occur.

  In addition, in the two-way real-time communication of audio and video, the user places more importance on the audio quality and the real-time property than the video quality as convenience.

In order to solve these problems, when the receiving terminal recognizes that the packet loss or the fluctuation of the packet exceeds a certain value, it sends a data amount change request packet to the transmitting terminal, and changes the data amount, A technique is known in which a receiving terminal controls data received from a transmitting terminal during real-time communication so that real-time communication is possible without causing sound interruption or image disturbance (see Patent Document 1).
JP 2000-332829 A

  However, the following problems occur in the above conventional technique. In the technique described in Patent Document 1, the receiving terminal side requests the transmitting terminal to change the transmission data amount, and the transmitting terminal side changes the packet size of the voice packet to be transmitted, changes the encoding method, video There are cases where control is performed to reduce the amount of transmission data by changing the frame rate, image quality parameter, and the like.

  The above-mentioned control causes deterioration in sound quality due to changes in the encoding method, degradation in video quality due to changes in the frame rate and image quality parameters, etc., and gives users dissatisfaction with the performance of real-time communication of audio and video. The problem was not solved.

  The present invention solves such a conventional problem, and in two-way real-time communication between audio data and video data, the real-time property of the audio data is not impaired, and the synchronization timing of the audio data and the video data is disturbed. The purpose is to prevent the user from being dissatisfied with the communication performance due to the deterioration of the sound quality and the video quality due to the decrease in the data amount of the packet.

  In order to achieve the above object, the present invention provides a data communication system for bidirectionally transmitting and receiving packetized audio data and video data, as shown in claim 1, wherein the audio packet transmission time information is converted into an audio packet and The packet transmission unit that is added to the video packet and sequentially transmits the received audio packet and the audio packet transmission time information of the video packet are compared. If they are equal, the video data is reproduced, and the audio packet transmission time information of the video packet is When late, the video data is discarded, and when the audio packet transmission time information of the audio packet is late, an apparatus including a packet receiving unit that buffers the video packet is provided.

  According to the present invention, when performing bidirectional real-time communication between audio and video, the audio transmission time information is independently and sequentially transmitted in two packets, that is, an audio packet and a video packet. Compare transmission time information, if equal, play video data, discard video data if the audio packet transmission time information of the video packet is late, discard video data if the audio packet transmission time information of the audio packet is late Playback can be delayed, so that the real-time nature of the audio data is not impaired, the disturbance of the synchronization timing of the audio data and the video data is suppressed, and the sound quality and video quality due to the decrease in the amount of packet data are reduced. There is an effect that the user is not dissatisfied with the communication performance due to deterioration or the like.

  In the present invention, the data communication terminal device adds the transmission time information of the audio data to the two packets of the audio packet and the video packet and sequentially transmits the packet transmitting unit, the receiving unit that receives the two packets, and the two A comparison unit that compares each transmission time information of audio data included in the packet, and, as a result of comparison by the comparison unit, if the time information is substantially the same as the transmission time information, the received audio data and video data If the transmission time information added to the video packet is time information delayed from the transmission time information added to the audio packet, the received audio data is reproduced. The received video data is discarded, and the transmission time information added to the video packet is the audio packet. In the case of time information earlier than the transmission time information added to the network, it is determined that the received audio data and video data are not synchronized, and any processing for buffering the received video data is performed. An audio video communication system characterized by a configuration including a control unit.

  As a result, the audio transmission time information is independently transmitted in two packets of an audio packet and a video packet. On the receiving side, the arrival time of the video packet is compared by comparing the transmission time information of the two packets. It can be determined whether it is the same or early or late, simply and at high speed, and it plays back video data, and it can suppress the disturbance of the synchronization timing of audio data and video data without impairing the real time property of audio data, In addition, the user is not dissatisfied with the communication performance due to the deterioration of the sound quality and video quality due to the decrease in the amount of packet data.

  In addition, the data communication terminal device includes the packet transmission unit that sequentially transmits the transmission time information of the audio data to the two packets of the audio packet and the video packet, the reception unit that receives the two packets, and the two packets. When the time information is substantially the same as the transmission time information as a result of comparison by the comparison unit that compares the transmission time information of the audio data to be transmitted and the comparison unit, the received audio data and the video data are synchronized. When the received video data is reproduced after the received video is reproduced and the transmission time information added to the video packet is later than the transmission time information added to the audio packet, the received audio data and video data Discards the received video data because it is determined that it is not synchronized with the video, and sends the transmission time information added to the video packet to the audio packet. If the received time information is earlier than the transmitted time information, the control unit performs any process of buffering the received video data by determining that the received audio data and video data are not synchronized. It is good also as a structure provided.

  Even in this case, the audio transmission time information is independently transmitted in the two packets of the audio packet and the video packet, and the arrival time of the video packet is reached by comparing the transmission time information of the two packets on the receiving side. Whether it is the same as the time, or early or late, can be easily and quickly determined to play back the video data, and the real-time performance of the audio data can be reduced without disturbing the synchronization timing of the audio data and video data. In addition, the user will not be dissatisfied with the communication performance due to the deterioration of the sound quality and video quality due to the decrease in the amount of packet data.

  Further, the data communication terminal apparatus sequentially transmits and receives at least the audio data and the audio packet including the transmission time information of the audio data, and the video packet including the video data and the transmission time information of the audio data. A TV adapter is provided, and the telephone device includes transmission time information A of voice data included in a voice packet transmitted from the local telephone device to the remote telephone device, and voice included in a voice packet received from the remote telephone device. A voice communication control unit for notifying the own TV adapter of the data transmission time information B1 is included, and the TV adapter includes transmission time information among the transmission time information of the voice data transmitted from the own telephone device. A transmission unit that transmits a video packet including A to the partner TV adapter, and the partner TV adapter. As a result of the comparison by the receiving unit that receives the video packet including the transmission time information B2 of the audio data, the comparison unit that compares the transmission time information B1 and the transmission time information B2, the transmission time information When B1 is substantially the same time information as the transmission time information B2, the received video is determined to be synchronized with the received audio data, and the received video is reproduced, and the transmission time information B2 is the transmission time information. In the case of time information later than B1, it is determined that the received audio data is not synchronized and the received video data is discarded. When the transmission time information B2 is time information earlier than the transmission time information B1, And a control unit that performs any of the processes for determining that the received audio data is not synchronized with the received audio data and buffering the video data. Between the communication terminal device, to interactively transmit and receive audio data and video data via a communications network may be a voice video communication system according to claim.

  Thus, the user can use the two terminals at different installation locations by performing video shooting and playback processing with the TV adapter on the local network and voice input and playback processing with the telephone device.

  Furthermore, the data communication terminal device sequentially transmits / receives at least audio data and a voice packet including transmission time information of the audio data sequentially, and a video packet including video data and transmission time information of the audio data. A TV adapter is provided, and the telephone device includes transmission time information A of voice data included in a voice packet transmitted from the local telephone device to the remote telephone device, and voice included in a voice packet received from the remote telephone device. A voice communication control unit for notifying the own TV adapter of the data transmission time information B1 is included, and the TV adapter includes transmission time information among the transmission time information of the voice data transmitted from the own telephone device. A transmission unit that transmits a video packet including A to the partner TV adapter, and the partner TV adapter. As a result of the comparison by the receiving unit that receives the video packet including the transmission time information B2 of the audio data, the comparison unit that compares the transmission time information B1 and the transmission time information B2, the transmission time information When B1 is substantially the same time information as the transmission time information B2, the received video is determined to be synchronized with the received audio data, and the received video is reproduced, and the transmission time information B2 is the transmission time information. In the case of time information later than B1, it is determined that the received audio data is not synchronized, and the received video data is discarded. When the transmission time information B2 is time information earlier than the transmission time information B1, It may be configured to include a control unit that performs any of the processes for determining that the received audio data is not synchronized with the received audio data and buffering the video data.

  As a result, the user can use the two terminals at different installation locations by performing video shooting and playback processing with the TV adapter on the local network and inputting and playing back sound with the telephone device.

  Further, the data communication terminal apparatus sequentially transmits and receives at least the audio data and the audio packet including the transmission time information of the audio data, and the video packet including the video data and the transmission time information of the audio data. A TV adapter is provided, and the telephone device includes transmission time information A of voice data included in a voice packet transmitted from the local telephone device to the remote telephone device, and voice included in a voice packet received from the remote telephone device. A voice communication control unit for notifying the own TV adapter of the data transmission time information B1 is included, and the TV adapter includes transmission time information among the transmission time information of the voice data transmitted from the own telephone device. A transmission unit that transmits a video packet including A to the partner TV adapter, and the partner TV adapter. As a result of the comparison by the receiving unit that receives the video packet including the transmission time information B2 of the audio data, the comparison unit that compares the transmission time information B1 and the transmission time information B2, the transmission time information When B1 is substantially the same time information as the transmission time information B2, the received video is determined to be synchronized with the received audio data, and the received video is reproduced, and the transmission time information B2 is the transmission time information. In the case of time information later than B1, it is determined that the received audio data is not synchronized, and the received video data is discarded. When the transmission time information B2 is time information earlier than the transmission time information B1, A predetermined number of audio packets, and a control unit that performs any of the processes by determining that the received audio data is not synchronized with the received audio data and buffering the video data. Each signal, to notify the sending time information of the audio data to the TV adapter from the telephone device may be a voice video communication system according to claim.

  Thereby, the processing load of this function is reduced and a CPU with a low processing speed can be used.

  Furthermore, the data communication terminal device sequentially transmits / receives at least audio data and a voice packet including transmission time information of the audio data sequentially, and a video packet including video data and transmission time information of the audio data. A TV adapter is provided, and the telephone device includes transmission time information A of voice data included in a voice packet transmitted from the local telephone device to the remote telephone device, and voice included in a voice packet received from the remote telephone device. A voice communication control unit for notifying the own TV adapter of the data transmission time information B1 is included, and the TV adapter includes transmission time information among the transmission time information of the voice data transmitted from the own telephone device. A transmission unit that transmits a video packet including A to the partner TV adapter, and the partner TV adapter. As a result of the comparison by the receiving unit that receives the video packet including the transmission time information B2 of the audio data, the comparison unit that compares the transmission time information B1 and the transmission time information B2, the transmission time information When B1 is substantially the same time information as the transmission time information B2, the received video is determined to be synchronized with the received audio data, and the received video is reproduced, and the transmission time information B2 is the transmission time information. In the case of time information later than B1, it is determined that the received audio data is not synchronized, and the received video data is discarded. When the transmission time information B2 is time information earlier than the transmission time information B1, A predetermined number of audio packets, and a control unit that performs any of the processes by determining that the received audio data is not synchronized with the received audio data and buffering the video data. Each signal may be configured to notify the transmission time information of the audio data to the TV adapter from the telephone device.

  This also makes it possible to use a CPU with a low processing speed that reduces the processing load of this function.

  Hereinafter, an audio video communication system according to an embodiment of the present invention will be described with reference to the drawings.

  FIG. 1 is a configuration diagram of an audio video communication system 100 according to an embodiment of the present invention. The audio video communication system 100 is configured by connecting a local audio video communication device 112 and a counterpart audio video communication device 113 via the Internet 106.

  In this embodiment, the audio / video communication devices 112 and 113 are implemented as IP videophones. Not limited to the configuration of the embodiment, a system may be configured by connecting three or more audio / video communication apparatuses.

  The own-side audio-video communication device 112 connects the own-side TV adapter 102 and the own-side IP phone 103 to a local network such as a LAN (or wireless LAN) under the router 104, and the own-side TV adapter 102 transmits and receives video. The own IP phone 103 transmits and receives voice. The TV 101 is a television that is connected to the TV adapter 102 on its own side and displays a moving image. A modem 105 is provided above the router 104 and is connected to the audio / video communication device 113 on the other side via the Internet 106.

  In the other party audio video communication device 113, the other party TV adapter 110 and the other party IP phone 109 are connected to a local network such as a LAN (or wireless LAN) under the router 108, and the other party TV adapter 110 transmits and receives video. Then, the other party's IP phone 109 transmits and receives voice. The TV 111 is a television that is connected to the partner TV adapter 110 and displays a moving image. A modem 107 is provided above the router 108 and is connected to the audio / video communication device 112 on the own side via the Internet 106.

  FIG. 2 is a block diagram showing the configuration of the own TV adapter 102 used in the audio / video communication system 100 of FIG. The counterpart TV adapter 110 has the same configuration.

  In FIG. 2, the own TV adapter 102 is mainly composed of a network interface 201, a camera 202, a video communication control unit 203, a video codec 204, a display control unit 205, a display output unit 206, a system control unit 207, and a remote control unit 208. It is configured.

  The network interface 201 is an interface for connecting to the router 104 and the own IP phone 103 via a local network such as a LAN (or wireless LAN). When transmitting video data, video data captured by the camera 202 is compressed and encoded by the video codec 204.

  The compressed and encoded video data is processed into a video packet by the video communication control unit 203, and the video packet 400 (details will be described later with reference to FIG. 4) is sent from the network interface 201 to the partner audio video communication device 113 via the Internet 106. Sent.

  When receiving video data, the video packet 400 is received by the network interface 201 via the Internet 106 from the counterpart TV adapter 110 of the counterpart audio video communication device 113.

  The received video packet 400 is subjected to packet analysis / decomposition / buffering processing in the video communication control unit 203, decompressed and decoded by the video codec 204, and combined for display in a frame by the display control unit 205. The other party's video is reproduced and displayed on the TV 101.

  In addition, the remote controller control unit 208 operates the own TV adapter 102 using the remote controller, and the system control unit 207 controls the entire own TV adapter 102. It is also possible to display the video on its own side from the display output unit 206 with the camera 202.

  FIG. 3 is a block diagram showing the configuration of the own IP phone 103 used in the audio / video communication system 100. The other party IP phone 109 has the same configuration.

  In FIG. 3, the own IP phone 103 is mainly configured by a network interface 301, a voice communication control unit 302, a voice control unit 303, a display control unit 304, an LCD 305, a system control unit 306, and a receiver 307.

  The network interface 301 is an interface connected to the router 104 and the own TV adapter 102 via a local network such as a LAN (or wireless LAN). When the telephone voice is transmitted, the voice data input from the receiver 307 is compressed and encoded by the voice control unit 303.

  The audio data that has been compression-encoded is processed into an audio packet by the audio communication control unit 302, and an audio packet 1100 (details will be described later with reference to FIG. 11) is sent from the network interface 301 to the counterpart audio-video communication device 113 via the Internet 106. Sent out.

  At the time of telephone voice reception, the voice packet 1100 is received by the network interface 301 via the Internet 106 from the partner IP telephone 109 of the partner voice video communication device 113.

  The received voice packet 1100 is subjected to packet analysis / decomposition / buffering processing by the voice communication control unit 302, decompressed and decoded by the voice control unit 303, and voice is output from the receiver 307. The system control unit 306 controls the own IP phone 103 as a whole.

  The display control unit 204 performs display processing such as setting and status of the IP phone, and displays on the LCD 305.

  FIG. 11 is a configuration diagram of a voice packet 1100 transmitted and received between the own IP phone 103 and the other party IP phone 109.

  The voice packet 1100 has an RTP (real time protocol) packet configuration conforming to the RFC 1889 standard of IETF used in real time communication. Specifically, it comprises an RTP header 1101, a time stamp 1102, an SSRC identifier 1103, a CSRC identifier 1104, a payload header (voice) 1105, and payload data (voice) 1106.

  The voice packet 1100 is generated at the time of transmission by the voice communication control unit 302 of the own IP phone 103 shown in FIG. 3, and is analyzed and decomposed at the time of reception. In the IP phone 109, the voice packet is processed in the same manner.

  The time stamp 1102 carries the transmission time information of the voice packet 1100. It should be noted that the time stamp 1102 that is the transmission time information of the audio packet is the audio time stamp 1102A that is transmitted from the own audio / video communication device 112 to the other audio / video communication device 113, and from the other audio / video communication device 113. What is transmitted to the own audio / video communication device 112 is defined as an audio time stamp 1102B, and is used in the following description.

  FIG. 4 is a configuration diagram of a video packet 400 transmitted and received between the own TV adapter 102 and the other TV adapter 110.

  The video packet 400 has a configuration in which a header of an RTP (real time protocol) packet used in real time communication is extended. Specifically, RTP header 401, time stamp 402, SSRC identifier 403, CSRC identifier 404, payload header (video) 405, payload data (video) 406 and voice transmission time information header extension 407 (details will be described later in FIG. 5) Is added and configured.

  In the time stamp 402, transmission time information of the video packet 400 is mounted. The voice transmission time information header extension 407 is extension information that does not conform to the standard of the RTP packet, and includes transmission time information of the voice packet 1100.

  FIG. 5 is a block diagram of the voice transmission time information header extension 407 of FIG.

  The voice transmission time information header extension 407 is a data type that is an identifier of the voice transmission time information header extension and a data length 501 indicating the number of bytes of the voice transmission time information header extension 407 and transmission time information of the voice packet 1100. It consists of an audio time stamp 502.

  That is, the transmission time information is set to a value equal to the transmission time transmission information mounted on the time stamp 1102 of the voice packet 1100 in FIG.

  The video packet 400 is generated at the time of transmission by the video communication control unit 203 of the local TV adapter 102 and the counterpart TV adapter 103 shown in FIG. 2, and is analyzed and decomposed at the time of reception.

  For convenience of explanation, the audio time stamp 502, which is transmission time information of the audio packet 1100 included in the audio packet transmission time information header extension 407 of the video packet 400, is transmitted from the own audio video communication device 112 to the other audio video communication device. It is determined that the audio time stamp 502A is transmitted to 113, and 502B is the audio time stamp 502A transmitted from the partner audio / video communication device 113 to the own audio / video communication device 112, and is used in the following description.

  FIG. 6 is a configuration diagram of the own time stamp packet 600 notified from the own IP phone 103 to the own TV adapter 102.

  The own time stamp packet 600 is transmitted from the own IP phone 103 to the other party IP phone 109, the data type 601 indicating the data type that is the identifier of the own time stamp packet 600 and the number of bytes of the time stamp packet. Voice time stamp 602 of the voice packet (transmission) that is the transmission time information of the voice packet 1100 (the setting value is equal to 1102A described in FIG. 11), and the other party's IP telephone 103 received from the other party's IP telephone 109 It is composed of a voice time stamp 603 (set value is equal to 1102B described in FIG. 11) of the voice packet (reception) which is transmission time information of the voice packet 1100 sent from the side IP phone 109 to its own IP phone 103. ing.

  The self-time stamp packet 600 is generated and transmitted by the voice communication control unit 302 of the own-side IP phone 103, received by the video communication control unit 203 of the own-side TV adapter 102, analyzed and decomposed, and later-described video data. Used during reproduction (details will be described later with reference to FIG. 7).

  FIG. 10 is a configuration diagram of the counterpart time stamp packet 1000 notified from the counterpart IP phone 109 to the counterpart TV adapter 110.

  The counterpart time stamp packet 1000 is transmitted from the counterpart IP phone 109 to the own IP phone 103 from the data type 1001 indicating the data type that is the identifier of the counterpart time stamp packet 1000 and the number of bytes of the timestamp packet. Voice time stamp 1002 of the voice packet (transmission) which is the transmission time information of the voice packet 1100 (the setting value is equal to 1102B described in FIG. 11), and the voice packet transmitted from the partner IP phone 109 to the own IP phone 103 It is composed of a voice packet 1003 which is transmission time information 1100 (the set value is equal to 1102A described in FIG. 11).

  The counterpart time stamp packet 1000 is generated and transmitted by the counterpart IP phone 109, received by the counterpart TV adapter 110, analyzed and disassembled, and video data reproduction described later, as with the own time stamp packet 600. Sometimes used (details will be described later in FIG. 7).

  6 and FIG. 10 have the same format. The packet 600 in FIG. 6 is used by the own audio / video communication device 112, and the packet 1000 in FIG. What is different is.

  Next, in the configuration of the system 100 of FIG. 1, the real-time property of the audio data is not impaired between the own IP telephones 103 and 109 and the own TV adapters 102 and 110, and the synchronization timing of the audio data and the video data is not affected. With reference to the sequence diagram shown in FIG. 7, an explanation will be given of the operation that suppresses the disturbance and does not give the user unsatisfactory communication performance due to the deterioration of the sound quality and the video quality due to the decrease in the data amount of the packet.

  FIG. 7 is a sequence diagram for explaining the operation of the audio video communication system. This sequence includes the video communication control unit 203 and the audio communication control unit 302 included in the local data communication device 112 and the video control communication unit and the audio communication control unit similar to those illustrated in FIGS. Between.

  When the network connection is completed and the power is turned on (step S701), the own IP phone 103 continuously sends a pairing request message to the own TV adapter 102. When the network connection is completed and the power is turned on, the own-side TV adapter 102 responds to a pairing request message from the own-side IP phone 103 and performs pairing between the own-side IP phone 103 and the own-side TV adapter 102. Is established (step S701).

  The other party IP phone 109 and the other party TV adapter 110 establish pairing by the same operation (step S702).

  When the handset is off-hooked from its own IP phone 103 and dialed to the other party's IP phone 109, when the INVITE message is sent by the standard call control protocol SIP call control communication, the other party's IP phone 109 rings and the other party's IP phone 109 Responds with a 200 OK message to its own IP phone 103 to establish voice communication capable of two-way conversation (step S703).

  Video communication between the own TV adapter 102 and the other TV adapter 110 is also established by setting the SDP (Session Description Protocol) video packet transmission / reception included in the SIP message of the SIP call control communication (step S704).

  When the voice communication is established, two-way voice packet communication is started between the IP phone 103 on the own side and the IP phone 109 on the other side. Voice packet 1100 including voice time stamp 1102A (see the description of FIG. 11) is transmitted from own IP phone 103 to counterpart IP phone 109 (step S705).

  Similarly, voice packet 1100 including voice time stamp 1102B (see the description of FIG. 11) is transmitted from partner IP phone 109 to own IP phone 103 (step S706).

  Next, the voice time stamp 602 (set value is equal to 1102A) sent from the own IP phone 103 to the own TV adapter 102 to the other party IP phone 109 and the voice time stamp 603 (setting) received from the other party IP phone 109 The own-side time stamp packet 600 with the value equal to 1102B is notified to the own-side TV adapter 102 (step S707).

  Similarly, the voice time stamp 1002 (set value is equal to 1102B) transmitted from the own IP phone 103 to the other party TV adapter 110 from the other party IP phone 109 and the voice time stamp 1003 received from the own IP phone 103 ( The partner side time stamp packet 1000 including the set value is equal to 1102A is notified to the partner side TV adapter 110 (step S708).

  When video communication is established, bi-directional communication of video packets between the own TV adapter 102 and the other TV adapter 110 starts. The own TV adapter 102 transmits the video packet 400 including the audio time stamp 502A (refer to the description of FIG. 5 for the symbol A) to the other TV adapter 110 (step S709).

  Similarly, the partner TV adapter 110 transmits the video packet 400 including the audio time stamp 502B (refer to the description of FIG. 5 for the sign B) to the own TV adapter 102 (step S710).

  With the above operation, the own TV adapter 102 has two audio time stamps 502B included in the video packet 400 and the audio time stamp 603 included in the own time stamp packet 600 (the setting value is equal to 1102B). Get audio time stamp.

  With the two audio time stamps, it is possible to identify a determination of the relative speed of arrival time of two packets, that is, an audio packet and a video packet.

The own-side TV adapter 102 performs lip sync reproduction (reproduction of video in which the movement of the lip that produces voice and voice is synchronized) by the following comparison determination based on the two audio time stamp values.
(1) When the audio time stamp 502B is equal to the audio time stamp 1102B, it is determined that there is no synchronization deviation between the video packet and the audio packet, and the video reproduction process is started.
(2) When the audio time stamp 502B is larger than the audio time stamp 1102B, it is determined that the video packet is behind the audio packet, and the video packet is discarded and the video is not reproduced.
(3) When the audio time stamp 502B is smaller than the audio time stamp 1102B, it is determined that the audio packet is behind the video packet, the video packet is buffered, and the video is audio after the next audio packet arrives. And the timing is reproduced (step S711).

  Note that the purpose of the present invention is to suppress the disturbance of the reproduction timing of audio data and video data without impairing the real-time property of the audio, and therefore the audio in the determination processes (1), (2), and (3) described above. The comparison of the time stamps does not need to be strict, and for example, the determination comparison may be performed with a tolerance of deviation within a predetermined range so that it is “equal” when “substantially equal” in the comparison.

  The counterpart TV adapter 110 also performs lip sync playback similar to that of the own TV adapter 102 based on the audio time stamp 1102A and the audio time stamp 502A (step S712).

  Next, the operation of own IP phone 103 will be described with reference to the flowchart shown in FIG. FIG. 8 is a flowchart for explaining the operation of the own IP phone 103. The flowchart is mainly implemented by the voice communication control unit of FIG. The other party's IP phone 109 operates in the same flow.

  When the network connection of the own IP phone 103 is completed and the power is turned on (step S801), a pairing request message is continuously transmitted to the own TV adapter 102.

  When the own TV adapter 102 responds to the pairing request, pairing is established between the own IP phone 103 and the own TV adapter 102 (step S802).

  When the handset 307 is off-hooked from the own-side IP phone 103 and dialed to the other-side IP phone 109, an INVITE message is transmitted by communication of the SIP call control message, and a two-way conversation can be performed by receiving a response of the 200 OK message. Voice communication is established (step S803).

  When the voice communication is established, the local IP phone 103 starts two-way communication of voice packets with the counterpart IP phone 109 (step S804).

  At this time, the own IP phone 103 transmits a voice packet including the voice time stamp 1102A to the counterpart IP phone 109 (step S805).

  In addition, own IP phone 103 receives the voice packet including voice time stamp 1102B from partner IP phone 109 (step S806).

  Next, the own IP phone 103 sequentially reproduces the received voice in real time (step S807).

  Next, the voice time stamp 602 (set value is equal to 1102A) transmitted from the partner IP phone 109 to the own TV adapter 102 from the own IP phone 103 and the voice time stamp 603 (setting) set from the partner IP phone 109 The own-side time stamp packet 600 including the value equal to 1102B is notified to the own-side TV adapter 102 (step S808).

  The voice packets are sequentially transmitted and received, and the voice communication ends when the call ends (step S809).

  As described above, the IP phone 103 on the own side performs standard-compliant call control operations except for step S808, and the processing load on the voice communication control unit 302 does not increase, thereby improving the real-time property. There is no loss.

  Next, the operation of the own TV adapter 102 will be described with reference to the flowchart shown in FIG. FIG. 9 is a flowchart for explaining the operation of the own TV adapter 102. The flowchart is mainly executed by the video communication control unit 203 of FIG. The counterpart TV adapter 110 operates according to the same flow.

  When the network connection is completed and the power is turned on (step S901), the own-side TV adapter 102 responds to a pairing request message from the own-side IP phone 103, and the own-side IP phone 103 and the own-side TV adapter 102 And pairing are established (step S902).

  As described with reference to FIG. 7, when the local IP phone 103 establishes voice communication with the counterpart IP phone 109 through call control, the local TV adapter 102 establishes video communication with the counterpart TV adapter 110 ( In step S903), bidirectional communication of the video packet is started (step S904).

  The own TV adapter 102 transmits the voice time stamp 602 (set value is equal to 1102A) transmitted from the own IP phone 103 to the other party IP phone 109 and the voice time stamp 603 (set value is the same as the set value is 1102A). The self-time stamp packet 600 loaded with “equal to 1102B” is received (step S905).

  The own-side TV adapter 102 transmits the video packet 400 including the audio time stamp 502A (refer to the description of FIG. 5 for the code A) to the other-side TV adapter 110 (S906).

  In addition, the own TV adapter 102 receives the video packet 400 including the audio time stamp 502B (refer to the description of FIG. 5 for the code B) from the other TV adapter 110 (step S907).

  As a result of the above operation, the own-side TV adapter 102 obtains two audio time stamps of the audio time stamp 502B included in the video packet 400 and the audio time stamp 603 included in the own-side time stamp packet 600 (the setting value is equal to 1102B). obtain.

  Based on the two audio time stamps, it is possible to determine a relative speed of arrival time of two packets of the audio packet and the video packet.

  The own TV adapter 102 compares whether the audio time stamp 502B is equal to the audio time stamp 1102B (step S908).

  When the audio time stamp 502B is equal to the audio time stamp 1102B as a result of the determination in step S908, the own TV adapter 102 determines that there is no synchronization between the video packet and the audio packet, and starts the video reproduction process (step S909).

  If they are not equal in step S908, the TV adapter 102 on the local side compares whether the audio time stamp 502B is greater than the audio time stamp 1102B (step S910).

  When the audio time stamp 502B is larger than the audio time stamp 1102B, the own-side TV adapter 102 determines that the video packet is behind the audio packet, discards the video packet, and does not reproduce the video (step S911).

  When the audio time stamp 502B is smaller than the audio time stamp 1102B in the determination result in step S910, the own-side TV adapter 102 determines that the audio packet is behind the video packet, buffers the video packet, and After the audio packet arrives, the video is reproduced in synchronization with the audio (step S912).

  The video packets are sequentially transmitted and received, and the video communication ends (step S913).

  As described in the description of the sequence diagram, the comparison of steps S908 and S910 may be performed with a predetermined tolerance.

  As described in the above embodiments, in the realization of an IP videophone that performs a two-way call in real time on the Internet without bandwidth guarantee, the IP phone that transmits and receives voices does not significantly increase the load of packet processing in this embodiment. And real-time performance is not impaired.

  Further, the TV adapter for transmitting and receiving video may receive the same time as the arrival time of the audio packet by comparing the audio transmission time information received independently in the two packets of the audio packet and the video packet, or Since it is determined whether it is early or late, and video data is reproduced and controlled, it is possible to suppress synchronization deviation with audio easily and at high speed.

  Further, since the amount of data of packets to be transmitted / received is not reduced, it is possible to reproduce audio and video without giving the user unsatisfactory communication performance due to deterioration of sound quality and video quality.

  Although the present invention has been described in detail based on examples, these examples are merely examples, and the present invention is not limited to the examples. In this embodiment, the IP phone and the TV adapter are separated, but the same effect can be obtained by integrating the IP phone and the TV adapter.

  Further, in this embodiment, the video reproduction comparison determination is performed based on the voice packet transmission time information every time a packet is received. However, every time a packet is received according to the memory and CPU processing capability of the IP phone and the TV adapter. Alternatively, the same effect can be obtained by making a video reproduction comparison determination based on the voice packet transmission time information at predetermined intervals.

  Further, in this embodiment, the IP telephone is integrated with the telephone, but the same effect can be obtained even if the IP telephone is separated into the IP telephone adapter and the telephone.

  Further, in this embodiment, the TV adapter is separated from the TV, but the same effect can be obtained even if the TV adapter and the TV are integrated.

  In this embodiment, the Internet 106, the modems 105 and 107, and the routers 104 and 108 are provided. However, any one of the Internet 106, the modems 105 and 107, and the routers 104 and 108 is omitted as necessary, and the own IP phone is provided. The same effect can be obtained by providing 103 and the other party's IP telephone 109, the own TV adapter 102, the other party's TV adapter 110, the television TV 101 and the TV 111 on a network such as a LAN in the same IP address space.

  The audio-video communication device of the present invention can synchronize audio data and video data without losing the real-time property of audio data when performing bidirectional real-time communication of audio and video via a network such as a LAN, the Internet, or an IP communication network. It is possible to suppress timing fluctuations and does not reduce the amount of packet data to be transmitted / received, so audio and video can be played without giving users dissatisfaction with communication performance due to degradation of sound quality and video quality. Therefore, the present invention can be used as a two-way real-time data communication device for audio / video and a system using the same.

Configuration diagram of an audio-video communication system according to an embodiment of the present invention The block diagram which shows the structure of the own side TV adapter used with the audio video communication system of a present Example. The block diagram which shows the structure of the own side IP telephone used with the audio video communication system of a present Example. The block diagram of the video packet transmitted / received between the own side TV adapter and the other party TV adapter of a present Example Configuration diagram of header extension for voice transmission time information in this embodiment Configuration diagram of own time stamp packet notified to own TV adapter from own IP phone of audio / video communication system of this embodiment Sequence diagram for explaining the operation of the audio-video communication system of the present embodiment Flowchart for explaining the operation of the own IP phone of this embodiment Flowchart for explaining the operation of the own TV adapter of this embodiment Configuration diagram of the other party time stamp packet notified from the other party IP phone of this embodiment to the other party TV adapter Configuration diagram of voice packet transmitted / received between own IP phone and partner IP phone of this embodiment

Explanation of symbols

DESCRIPTION OF SYMBOLS 100 Audio video communication system 102 Own side TV adapter 103 Own side IP telephone 109 Other party IP telephone 110 Other party TV adapter 112 Own side audio video communication apparatus 113 Other party audio video communication apparatus 203 Video communication control part 302 Audio communication control part 400 Video packet 407 Audio transmission time information header extension 600 Local time stamp packet 1000 Remote side time stamp packet 1100 Audio packet

Claims (6)

In an audio / video communication system for transmitting and receiving audio data and video data bidirectionally between data communication terminal devices connected via a communication network,
The data communication terminal apparatus includes a packet transmitting unit that sequentially transmits audio data transmission time information added to two packets of an audio packet and a video packet, a receiving unit that receives the two packets, and the two packets. A comparison unit that compares each transmission time information of the audio data;
As a result of comparison by the comparison unit,
If the time information is substantially the same as the transmission time information, the received audio data and the video data are determined to be synchronized, and the received video is reproduced.
When the transmission time information added to the video packet is time information delayed from the transmission time information added to the audio packet, it is determined that the received audio data and the video data are not synchronized, and the received video Destroy the data,
If the transmission time information added to the video packet is earlier than the transmission time information added to the audio packet, the received video data is determined to be out of synchronization with the received audio data. A control unit for performing any processing for buffering,
An audio-video communication system characterized in that audio data and video data are interactively transmitted and received through a communication network between data communication terminal devices having these functions. A data communication terminal device for bidirectionally transmitting / receiving packetized audio data and video data to / from a counterpart device via a communication network,
A packet transmission unit that sequentially transmits the transmission time information of audio data to two packets of an audio packet and a video packet, and a reception unit that receives the two packets;
A comparison unit for comparing transmission time information of audio data included in the two packets;
As a result of comparison by the comparison unit,
If the transmission time information is substantially the same time information, it is determined that the received audio data and video data are synchronized, and the received video is reproduced,
When the transmission time information added to the video packet is time information delayed from the transmission time information added to the audio packet, it is determined that the received audio data and the video data are not synchronized, and the received video Destroy the data,
If the transmission time information added to the video packet is earlier than the transmission time information added to the audio packet, the received video data is determined to be out of synchronization with the received audio data. A data communication terminal device comprising: a control unit that performs any processing for buffering the data. In an audio / video communication system for transmitting and receiving audio data and video data bidirectionally between data communication terminal devices connected via a communication network,
The data communication terminal device includes: a telephone device that sequentially transmits / receives at least audio data and an audio packet including transmission time information of the audio data; and a TV that sequentially transmits / receives video packet including image data and transmission time information of the audio data. With an adapter,
The telephone device includes transmission time information A of voice data included in a voice packet transmitted from the local telephone device to the counterpart telephone device, and transmission time information of voice data included in a voice packet received from the counterpart telephone device. A voice communication control unit for notifying B1 to the TV adapter on its own side,
In the TV adapter, a transmission unit that transmits a video packet including transmission time information A to the other TV adapter among transmission time information of the audio data transmitted from the own telephone device;
A receiving unit for receiving a video packet including transmission time information B2 of audio data created in the same manner as described above by the other-side TV adapter;
A comparison unit for comparing the transmission time information B1 and the transmission time information B2,
As a result of comparison by the comparison unit,
When the transmission time information B1 is substantially the same time information as the transmission time information B2, the received video is determined to be synchronized with the received audio data, and the received video is reproduced.
If the transmission time information B2 is time information delayed from the transmission time information B1, it is determined that the received audio data is not synchronized, and the received video data is discarded.
If the transmission time information B2 is earlier than the transmission time information B1, it is determined that the transmission time information B2 is not synchronized with the received audio data, and a control unit that performs any processing for buffering video data is provided. Prepared,
An audio-video communication system characterized in that audio data and video data are interactively transmitted and received through a communication network between data communication terminal devices having these functions. A data communication terminal device for bidirectionally transmitting / receiving packetized audio data and video data to / from a counterpart device via a communication network,
At least, a telephone device that sequentially transmits and receives audio data and audio packets including transmission time information of the audio data, and a TV adapter that sequentially transmits and receives video data and video packets including the transmission time information of the audio data,
The telephone device includes transmission time information A of voice data included in a voice packet transmitted from the local telephone device to the counterpart telephone device, and transmission time information of voice data included in a voice packet received from the counterpart telephone device. A voice communication control unit for notifying B1 to the TV adapter on its own side,
In the TV adapter, a transmission unit that transmits a video packet including transmission time information A to the other TV adapter among transmission time information of the audio data transmitted from the own telephone device;
A receiving unit for receiving a video packet including transmission time information B2 of audio data created in the same manner as described above by the other-side TV adapter;
A comparison unit for comparing the transmission time information B1 and the transmission time information B2,
As a result of comparison by the comparison unit,
When the transmission time information B1 is substantially the same time information as the transmission time information B2, the received video is determined to be synchronized with the received audio data, and the received video is reproduced.
If the transmission time information B2 is time information delayed from the transmission time information B1, it is determined that the received audio data is not synchronized, and the received video data is discarded.
If the transmission time information B2 is earlier than the transmission time information B1, it is determined that the transmission time information B2 is not synchronized with the received audio data, and a control unit that performs any processing for buffering video data is provided. A data communication terminal device characterized by comprising. 4. The audio-video communication system according to claim 3, further comprising notifying the transmission time information of the audio data from the telephone device to the TV adapter every time a predetermined number of audio packets are received. 5. The data communication terminal device according to claim 4, further comprising notifying the transmission time information of the audio data from the telephone device to the TV adapter every time a predetermined number of audio packets are received.

Images