JP2005045740A - Device, method and system for voice communication - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide a device, a method and a system of voice communication in which the real time voice information amount is reduced in high-quality communication having a BGM and a sound effect function via the Internet. <P>SOLUTION: A VoIP system has a real time transmission mode for transmitting real time voice data in which the voice data of a user or a real time voice data by synthesizing the voice data with the BGM etc. between VoIP clients by an RTP packet, and a batch transfer mode for transferring only the BGM in advance of the real time voice data by the RTP packet. Also, an RTCP packet can notify the communication party of the reception buffer size of the RTP packet. The VoIp client checks the gathered voice level of a user, and when listening to no voice and judging that the user does not speak, the VoIP client sets the batch transfer mode and transmits only the BGM which should be synthesized with the voice and transmitted in advance by the reception buffer size. <P>COPYRIGHT: (C)2005,JPO&NCIPI

Description

  The present invention relates to a call device, a call method, and a call system that perform a so-called Internet telephone using, for example, VoIP (Voice over Internet Protocol), and is particularly suitable for exchanging real-time acoustic data composed of high-quality voice and sound. TECHNICAL FIELD The present invention relates to a simple call device, a call method, and a call system.

  VoIP is a technology that enables voice communication via an IP network by encapsulating voice in IP packets. In order to make a call by VoIP, it is necessary to exchange a series of information such as acquisition of information on the other party to be called, calling of the other party to be called, and response. For these purposes, SIP (Session Initiation Protocol) etc. A call control protocol is used.

  There are an increasing number of systems that transmit real-time data such as real-time audio data such as real-time moving image data and audio data using a communication network such as the Internet including such VoIP. In public line networks such as the Internet, since a plurality of users supply network bandwidth, a congestion control method, that is, a congestion avoidance method and a sedation method at the time of occurrence of congestion is a major issue. Therefore, with a change in the number of communication modes in which real-time property is important, congestion control techniques in real-time data communication have become important (see Patent Document 1 below).

  For example, in Patent Document 2 below, a server including a speech encoding device that gives a relatively high bit amount to voiced sound having an important meaning in a speech section and reduces the number of bits in the order of unvoiced sound and background noise is a client terminal. A technique for suppressing the total transmission bit amount of voice to be transmitted is disclosed.

  By the way, conventionally, in real-time voice communication using VoIP, the main information of human voice is included in the frequency band near 4 to 5 KHz, so the number of channels is 1 and the sampling frequency is 8 KHz. Although it is usual to set the frequency to ˜16 KHz, it is difficult to transmit background noise and atmosphere in such a narrow band.

  If the number of channels is set to 2 or more and the sampling frequency is raised to 44.1 KHz, which is the same level as CDDA (Compact Disk Digital Audio), background noise and atmosphere can be transmitted. In addition, existing music content can be used for voice communication. It is also possible to add a diverted high-quality BGM function or the like.

  Here, when the number of channels is set to 2 or more, the sampling frequency is raised to 44.1 KHz, which is the same level as CDDA, and uncompressed 16-bit Linear PCM (Phase Code Modulation) data is transmitted, The transmission bit rate becomes 1411.2 Kbps or more, and it is difficult to use it in an actual Internet environment. Therefore, it is necessary to use a high-efficiency encoding method at present, but in real-time voice communication that exchanges high-quality real-time acoustic data such as a BGM function added due to recent improvements in CPU performance. It has become possible to adopt a high-efficiency encoding method.

  There are several high-efficiency encoding methods. First, frequency decomposition is performed, the signal is divided into a plurality of subbands, and each block is encoded using the psychoacoustic characteristics. Most of them adaptively change and entropy code a predetermined frequency component with a minimum number of bits. With this high-efficiency encoding, the bit rate becomes variable according to the frequency component to be encoded.

Japanese Patent Application Laid-Open No. 2001-3200430 JP 2001-5474 A

  However, even if an attempt is made to suppress the transmission rate by the method described in Patent Document 2 described above, in high-quality real-time voice communication, there is a BGM addition function with high-quality sound in addition to conversation. Regardless of whether it is present or not, there is a problem that a certain amount of data is transmitted and received, and the network bandwidth is occupied.

  The present invention has been proposed in view of such conventional circumstances, and has reduced the amount of information of real-time audio during high-quality voice communication having BGM and sound effect functions via the Internet. It is an object to provide a call device, a call method, and a call system.

  In order to achieve the above-described object, a call device according to the present invention includes a voice conversion unit that converts collected sound into an electric signal in a call device that transmits and receives at least voice data via the Internet, and the electric device. Synthesizing means for synthesizing additional data with audio data converted into a signal, data packet generating means for generating the data packet storing the audio data and / or additional data to be synthesized with the audio data, and at least the data packet Control packet generation means for generating a control packet storing management information for managing transmission and reception; transmission means for transmitting the data packet and the control packet to one or more other call devices via the Internet; Receiving means for receiving data packets and control packets from other call devices; Control means for controlling transmission and reception of packets and control packets, wherein the control means transmits in real time a first data packet storing the voice data or synthesized data obtained by synthesizing additional data with the voice data. Switching control between the first mode and the second mode in which the second data packet storing only the additional data is transmitted collectively.

  In the present invention, the first data packet storing only the audio data is transmitted in real time (real-time transfer), and only the additional data such as background music and / or sound effects is synthesized with the audio. And a second mode for batch transmission (batch transfer) of the second data packet storing the data, and by switching control of this, only the additional data can be postponed as necessary.

  Further, the control means can control to switch to the second mode in which the collective transmission is performed when the collected sound is lower than a predetermined volume level, and uses timing when the user is not speaking. Thus, the additional data can be postponed.

  Further, the control means can control to perform batch transmission in the second mode before performing real-time transmission in the first mode with the other call device, and immediately before the user starts a call. The additional data can be postponed using the timing.

  Furthermore, the control means transmits the first data packet storing only the audio data in the first mode for a predetermined time after collectively transmitting the second data packet in the second mode. In the predetermined period, that is, the period in which the additional data is postponed, only the voice data is used, and the transmission rate can be reduced as compared with the data packet including the voice data and the additional data.

  In addition, after the batch transmission of the second data packet in the second mode, the control means stops transmission of the data packet for a predetermined time when the collected sound is lower than a predetermined volume level. By controlling the additional data in advance, it is not necessary to transmit data when the user is not speaking.

  Further, the reception means has a reception buffer for buffering received data packets, and the control means is a buffer information indicating the capacity of the reception buffer and a buffer indicating the amount of data buffered in the reception buffer. Occupancy information is stored in the control packet, and the second data is based on the buffer information and the buffer occupancy information of the other call device stored in the control packet from the other call device received by the receiving means. The packet transmission rate can be controlled, and the amount of additional data to be postponed can be made variable in accordance with the buffer size and the occupation rate of other communication devices.

  The call method according to the present invention is a call method in which at least audio data is transmitted / received via the Internet, an audio conversion step of converting collected sound into an electric signal, and additional data in the audio data converted into the electric signal. A data packet generating step for generating a data packet storing the voice data and / or additional data to be combined with the voice data, and a control storing management information for managing at least transmission / reception of the data packet A control packet generating step for generating a packet; a transmission step for transmitting the data packet and the control packet to one or more other communication devices via the Internet; a data packet from the one or more other communication devices; A receiving step for receiving the control packet; and the data packet and the control packet A first mode for transmitting in real time a first data packet storing the voice data or synthesized data obtained by synthesizing the voice data with additional data; Switching control is performed between the second mode in which the second data packet storing only the additional data is transmitted collectively.

  The call system according to the present invention is a call system that transmits and receives at least audio data via the Internet, wherein each of the call devices is converted into an electric signal, voice converting means for converting the collected voice into an electric signal, and Managing at least transmission / reception of the data packet, combining means for synthesizing additional data with the voice data, data packet generating means for generating the data packet storing the voice data and / or additional data to be synthesized with the voice data Control packet generation means for generating a control packet storing management information; transmission means for transmitting the data packet and the control packet to one or more other call devices via the Internet; and the one or more other call calls Receiving means for receiving data packets and control packets from the device; and the data Control means for controlling transmission / reception of packets and control packets, and the reception means of one of the at least two communication devices has a reception buffer for buffering received data packets, and the control means Buffer information indicating the capacity of the buffer is stored in the control packet, and the control means of the other communication device executes the first data packet storing the voice data or synthesized data obtained by combining the voice data with additional data. The one communication device that controls switching between a first mode for time transmission and a second mode for batch transmission of a second data packet storing only the additional data. The transmission rate of the second data packet is controlled based on the buffer information of the one communication device stored in the control packet from Characterized in that it.

  In the present invention, one call device notifies the size of the reception buffer to the other call device by means of a control packet, and the other call device uses a second buffer based on the size of the reception buffer of the one call device. In the mode, it is possible to control the transmission rate for collectively transmitting only the additional data.

  According to the communication device of the present invention, in the communication device that transmits and receives at least audio data via the Internet, the audio conversion means that converts the collected sound into an electric signal, and the audio data converted into the electric signal. Combining means for combining additional data, data packet generating means for generating a data packet, control packet generating means for generating a control packet storing at least management information for managing transmission / reception of the data packet, the data packet, and the data packet Transmission means for transmitting a control packet to one or more other communication devices via the Internet, reception means for receiving a data packet and control packet from the one or more other communication devices, and the data packet and control packet Control means for controlling transmission / reception of the audio data. A first mode for transmitting in real time a first data packet that stores data or synthesized data obtained by synthesizing the voice data and a second data packet that stores a second data packet that stores only the additional data. Since the mode is controlled to be switched, if the second data packet consisting only of additional data such as background music and / or sound effects to be synthesized with the voice is transmitted in a batch at a desired timing before the voice, While the additional data that has been postponed is played back on the receiving side, only the audio data can be transmitted in real time, so the network bandwidth that is always occupied is narrowed when performing real-time communication with high sound quality and large amount of information. be able to.

  According to the call system of the present invention, one call device notifies the other call device of the capacity of the reception buffer, and the other call device that receives this notifies the capacity of the reception buffer of the one call device serving as the reception side. It is possible to determine the transmission rate of additional data to be postponed by batch transmission, and it is efficient when exchanging high-quality, high-quality real-time acoustic data consisting of voice and additional data via a network. The network bandwidth can be narrowed.

  Hereinafter, specific embodiments to which the present invention is applied will be described in detail with reference to the drawings. In this embodiment, the present invention is applied to a VoIP client as a call device in which two or more call devices make a call by VoIP over the Internet and a VoIP system as a call system including the same. The VoIP client in this embodiment can reduce the amount of information of real-time voice and intentionally narrow the bandwidth.

  First, an outline of a VoIP system that performs network communication using VoIP in the present embodiment will be described. FIG. 1 is a schematic diagram showing an example of a VoIP system in the present embodiment. The VoIP system according to the present embodiment realizes high-quality voice communication that can share not only a call but also various sound effects and BGM, for example, between two or more channels and between users. Note that the VoIP system in the present embodiment is performed between two call devices (hereinafter referred to as VoIP clients). However, the number of VoIP clients constituting the VoIP system is not limited to two. There may be two or more participants who can participate in network communication.

  As shown in FIG. 1, the VoIP system 100 includes a VoIP client 111 such as a PC (Personal Computer), for example, and a VoIP client 121 connected thereto via the Internet 130.

  In the VoIP system 100, the user 110 of the VoIP client 111 and the user 120 of the VoIP client 121 are VoIP applications (soft phones), which will be described later, installed in their VoIP clients 111 and 121, for example. A headset composed of a microphone and headphones or a handset composed of a microphone and a receiver is used to communicate with a communication partner via the Internet 130.

  The Internet 130 is a network environment that is spread all over the world by connecting a plurality of communication lines such as general public lines and information communication networks. Currently, broadband transmission is enabled by the widespread use of broadband and high-speed communication lines. A network is configured with a communication line of 500 kbps or higher using an optical fiber, an asymmetric digital subscriber line, radio, or the like.

  Connected to the Internet 130 are a VoIP server 131 for controlling VoIP communication, a web server 134 for managing data such as sound source data 132 and download user information 133, and the like. Each VoIP client 111, 121 stores sound source data 113, 123 downloaded from the web server 134 by the web browser 112, 122 or the like included in each VoIP client 111, 121, or purchased or edited by itself. Yes.

  The sound source data 132 stored in the database of the web server 134 and the sound source data 113 and 123 possessed by the user by downloading, for example, are music used for BGM (Back Ground Music) or the like, or sound of waves -Various sound effects such as clapping sound, thunder, bell sound, etc. These sound source data can be used in network communication between users.

  Next, the VoIP client will be described. FIG. 2 is a block diagram showing the functions of the VoIP client constituting the VoIP system. As shown in FIG. 2, the VoIP client 20 for performing such Internet communication receives the data from the calling party and the transmission means 21 for transmitting data to the participants participating in the communication, that is, the calling party. Receiving means 41.

  The transmission means 21 is connected to a microphone, and is compressed into a microphone capture (MIC capture) 22 that catches the external (user) voice and, for example, MP3 (MPEG (Moving Picture Experts Group) 1 audio Layer 3) or MPEG4 Sound effect file storage section 23 storing sound source files of various sound effects (SE), BGM file storage section 24 storing sound source files of various compressed BGM, sound effect files and BGM files Decoders 25 and 26 that respectively read and decode the sound, gain adjustment units 27 to 29 that adjust the volume by controlling the gain of the sound, sound effect, and BGM captured by the microphone, and the synthesis unit 30 that synthesizes these three sounds An encoder 31 for compressing and encoding the synthesized sound, and a data packet for the encoded data. A packetizing unit 32a as a data packet generating means to be stored in an RTP (Real-Time Transport Protocol) packet, and control information to be described later, and an RTP packet and an RTCP (Real-Time Control Protocol) packet A control unit 34 that controls transmission / reception, an RTCP packetizing unit 32b as a control packet generation unit that stores the control information and management information that controls RTP packets in the RTCP packet, and transmission that transmits RTP packets, RTCP packets, and the like Part 33. The RTP packet sent from the sending unit 33 is sent via the Internet 130 to another VoIP client to be communicated.

  On the other hand, the receiving means 41 for receiving data sent from the transmitting means 21 of the communication partner via the Internet 130 includes a receiving section 42 for receiving RTP packets and RTCP packets, and an RTP depacketizing section for depacketizing the received RTP packets. (Depacketize) 43a, an RTCP depacketizing unit (depacketize) 43b for depacketizing an RTCP packet, a dejitter unit (de-jitter) 44 for correcting the arrival time of the RTP packet, and a portion where an error has occurred in the sent RTP packet Alternatively, a packet compensator 45 that compensates for missing parts such as a lost part during transmission, a decoder 46 that decodes and decompresses data from the packet compensator 45, and an incoming call that is compressed to MP3, MPEG4, etc. Ring tone file (Ring Tone) that stores sound source files File) storage unit 47, decoder 48 that reads and decodes the ringtone file, gain adjustment units 49 and 50 that adjust the gains of decoders 46 and 48, and synthesis unit 30 in transmission means 21 described above. PCM data for transmission, that is, a gain adjusting unit 52 that adjusts the gain of the transmitter's own sound source, and a combining unit that combines the gain-adjusted transmission data, data transmitted from the communication partner, and ringtone data 53, an output unit 54 that outputs the data synthesized by the synthesis unit 53 to the headphones (HP), a gain adjustment unit 51 that adjusts the gain of the ringtone separately from the ringtone output to the synthesis unit 53, And a speaker (SP) 55 for outputting a gain-adjusted ring tone to the outside.

  Next, a data transmission / reception method of the VoIP client will be described. First, on the transmission side, the microphone capture 22 catches the user's voice input by the microphone and sends it to the gain adjustment unit 27. The gain adjusting unit 27 multiplies the audio data by a gain coefficient k1 in accordance with a user instruction or automatically, and adjusts the gain to a desired magnitude.

  The sound effect file storage unit 23 and the BGM file storage unit 24 store, for example, a hard disk drive (HDD), a ROM (read only memory), or a magneto-optical device in which a sound source file compressed in advance by a compression technique such as MP3 or MPEG4 is stored. It consists of a disk, and the decoders 25 and 26 read out these compressed encoded data, and convert the read compressed encoded data into PCM data. Furthermore, the decoders 25 and 26 send the converted PCM data to the gain adjustment units 28 and 29, respectively, and the gain adjustment units 28 and 29 respectively send the received data according to a user instruction or automatically. Multiply the gain coefficients k2 and k3 to adjust the gain to a desired size.

  The sound, BGM, and sound effect of the real-time acoustic data whose gain is adjusted by the gain adjusting units 27 to 29 are supplied to the synthesizing unit 30, and the synthesizing unit 30 adds the outputs of the gain adjusting units 27 to 29 while performing saturation processing. The addition result is output to the encoder 31. The encoder 31 encodes, for example, MPEG4 in order to transmit the addition result through the network. The encoded data is supplied to an RTP packetizing unit 32 that packetizes the data according to a real-time transport protocol (RTP).

  The RTP packetization unit 32a packetizes the encoded data into RTP packets, and the packetized data is transmitted from the transmission unit 33 to the VoIP client of the communication partner via the Internet 130.

  Further, the control unit 34 transmits in real time an RTP packet as a first data packet in which only audio data or synthesized data obtained by synthesizing BGM and sound effects into audio data is stored as encoded data to be transmitted in the RTP packet. A first mode (real-time transmission mode) for performing (real-time transfer) and a second mode (collective transmission mode) for collectively transferring RTP packets as second data packets storing, for example, only BGM to be synthesized with voice data And switching control.

  Specifically, when it is determined that the sound captured by the microphone capture 22 is less than a predetermined volume level, that is, at a timing when the user is not speaking, only BGM is transferred at once. Alternatively, after the call control, only BGM is batch transferred before exchanging real-time acoustic data. Thus, when only BGM is transferred at once, it is not necessary to transmit BGM for a predetermined period, and only audio data is transmitted as real-time transmission.

  In this way, when only BGM is transferred at once, the sound from the microphone capture 22 is muted, and the output of the decoder 25 for decoding the sound effect is also muted, so that the output of the decoder 31 is set to only BGM. it can. Alternatively, for example, the RTP packetizing unit 32a may read out a necessary amount of the compressed sound source file stored in the BGM file storage unit 24, store it in the RTP packet, and convert it into an RTP packet.

  In addition, the control unit 34 manages various management information stored in a report block of an RTCP packet, which will be described later, and generates control information stored in an extended portion of the RTCP packet. The control information includes a fluctuation absorbing buffer provided to cancel the jitter of the packet received by the RTCP receiving unit 42, a capacity of the PCM buffer for storing uncompressed PCM data before being reproduced by the headphones 54, and the like. Is to be notified. Further, in the extended portion of the RTCP packet transmitted / received by the VoIP client in the present embodiment, the compression rate of the real-time acoustic data stored in the RTP packet transmitted by the other VoIP client serving as the communication partner is designated. Information such as the encoding band and the number of sub-band divisions, and information indicating a retransmission request for a lost packet in which an error occurs or is lost during transmission can also be described.

  The RTCP packetizing unit 32b packetizes the control information and management information into RTCP packets, and the packetized data is also transmitted from the transmitting unit 33 to the VoIP client of the communication partner via the Internet 130.

  In a VoIP call, before transmitting an RTP packet, the control unit 34 performs call signaling to another VoIP client to be communicated via the transmission unit 33 using a call control protocol such as SIP. Then, after the RTP / RTCP session is established between other VoIP clients, the transmission unit 33 can transmit the RTP packet and the RTCP packet. Details of the RTP / RTCP packet will be described later.

  On the other hand, on the receiving side, the receiving unit 42 receives RTP packets, RTCP packets, and the like sent via the Internet 130. Then, the RTCP depacketizing unit 43b decomposes the received RTCP packet, and the control unit 34 in the batch transfer mode in which only BGM is batch transferred, among the control information stored in the extended portion of the RTCP packet. The transmission rate of BGM data to be collectively transferred is controlled based on the sizes of the fluctuation absorbing buffer and PCM buffer of the other party.

  In addition, after the RTP depacketization unit 43 converts the packet to a reverse packet, the dejitter unit 44 corrects the data in which the arrival times are unequal due to, for example, a delay in reception due to a network condition or the like, and equalizes the data. This correction is performed based on the RTP time stamp and sequential number that are depacketized and decomposed from IP and UDP (User Datagram Protocol). Thereafter, the packet compensator 45 ensures the packet that is lost or cannot be received in the transmission / reception of the network, specifically, the same packet as the previous or subsequent packet is used instead of the lost packet. The packet loss is compensated by requesting retransmission of the lost packet or receiving the lost packet again. The compressed data such as MPEG4 obtained in this way is decoded by the decoder 46, and the gain adjustment unit 49 multiplies the gain coefficient k5 by the user or automatically adjusts the gain.

  The decoder 48 generates a ring tone sound source file including a ring tone for calling or a music used for calling when a call is signaled from another VoIP client, that is, when a call is received. Read from the ringtone file storage unit 47. The ring tone data from the file storage unit 47 is selected in advance according to the user's request, and is decoded by the decoder 48 while being read into a RAM (not shown) according to the timing of the incoming call. The sound file of the ringtone is also compressed to MP3, MPEG4, etc., like the sound file of BGM, etc., and is stored in the file storage unit 47 for a plurality of files as ringtone data for each file. Then, the decoder 48 decodes the read sound source file, and gain adjustment is performed by the gain adjusting units 50 and 51 multiplying the decoded sound source data by a user instruction or automatically by gain coefficients k6 and k7, respectively.

  The synthesizer 53 adds the received data received from the call partner and the transmission data output from the gain adjuster 52, which is transmitted from the ring tone and the communication partner and gain-adjusted by the gain adjuster 49. . The gain adjusting unit 52 multiplies the transmission data by a gain coefficient k4 that is a loopback volume level set by the user in order to share the transmission data to be transmitted with the other party.

  The data synthesized by the synthesis unit 53 is output via the headphones 54 and transmitted to the user. In addition, the ring tone data read from the ring tone file storage unit 47 is configured to be output from the speaker 55 separately from the headphones 54.

  Here, a buffer for storing batch-transmitted BGM data is prepared before or after the decoder 46, or after the gain adjusting unit 49, and the real time composed of voice or voice and sound effect sent after the batch transmission. BGM data is read from the buffer so as to be sequentially synthesized and output to the acoustic data. Further, although the above-described PCM buffer is provided in the subsequent stage of the decoder 46 or the subsequent stage of the gain adjusting unit 49, the BGM data collectively transferred thereto may be buffered.

  In such a VoIP client, the control unit 34 switches between a real-time transmission mode for transferring real-time acoustic data and a batch transmission mode for batch-transferring only BGM as necessary, for example, by the user. If only the BGM is transferred at a time when the BGM is not transferred, the real-time acoustic data of only the sound may be transmitted and then transmitted as the real-time acoustic data while the batch-transferred BGM is reproduced. The amount of data can be reduced, and the network bandwidth occupied by the VoIP client can be reduced.

  Furthermore, by widening the dynamic range as compared with a general telephone, it is possible to transmit and receive a synthesized sound in which a high-quality sound such as stereo and sound composed of BGM and sound effects are synthesized. Even if is added, conversation (voice) is not masked. In addition, since the sound read from the sound source file such as the sound effect and BGM and the sound spoken by the communication person, that is, the microphone sound can be adjusted separately, the sound effect and the volume of the BGM can be set to desired levels. The user can adjust the user's feelings, for example, to the communication partner.

  In addition, by outputting the ringtone separately from the headphone 54 and the speaker 55, for example, the user temporarily leaves the VoIP client 20 during VoIP communication using the headphone 54, or even after the call ends. Even when 54 is left set, the ringtone can be output from the speaker 55 to the outside.

  Next, software used for the VoIP client will be described. FIG. 3 is a diagram illustrating a software module including a general VoIP client application for a PC. The VoIP client performs the function shown in FIG. 2 by executing the software module 1 corresponding to the protocol of each layer based on the open system interconnection (OSI) architecture shown in FIG. Achieve.

  In FIG. 3, each layer will be described from the lower layer to the upper layer. First, the functions as the physical layer 2 include a universal serial bus (USB) camera driver 2a, a USB audio driver 2b, and various drivers 2c. This is a layer that matches the physical conditions of the transmission conditions of video data from the camera driver 2a and audio data from the audio driver 2b.

  Next, the function as the data link layer includes an operating system (OS) 3. This is for executing data transmission without error between adjacent nodes.

  The network layer function includes an Internet Protocol (IP) control unit 4. The network layer selects a communication path used for data transmission / reception and performs communication control such as flow control and quality control. IP, which is a connectionless packet transmission protocol that does not pursue reliability, leaves the reliability assurance function, flow control function, and error recovery function to the upper layers (transport layer and application layer).

  The function as the transport layer includes a transport control protocol (TCP) / user datagram protocol (UDP) control unit 5.

  In the transport layer, end-to-end transmission is performed using an IP address. Regardless of the type of network, flow control and sequence control are performed according to the required quality class. TCP has a reliability guarantee function, attaches a sequence number to each byte of transmitted data, and retransmits the data if no acknowledgment (Acknowledgment: ACK) is sent from the receiving side. UDP provides a function for transmitting datagrams between applications. When streaming an audio / video using an IP network, generally, a transport protocol such as TCP that retransmits when an error occurs cannot be used. TCP is a protocol for one-to-one communication, and information cannot be transmitted to a plurality of partners. Therefore, UDP is used for such applications. In other words, when performing highly reliable communication such as requiring retransmission control, TCP packets are generated as TCP communication, and communication with high real-time properties such as transmission of audio and video data and SIP described later is performed. When performing, UDP communication is generated as UDP communication.

  UDP is designed to allow application processes to transmit data to other application processes on a remote machine with minimal overhead. Therefore, the information entered in the UDP header is only the source port number, destination port number, data length, and checksum, and there is no field for entering the number indicating the order of packets in TCP. When the order of the packets is changed due to transmission through the network, processing for returning the order to the correct state cannot be performed. Also, there is no field for inputting time information such as a time stamp at the time of transmission in TCP or UDP.

  The functions as the session layer 6 include a session initiation protocol (SIP) control unit 11, an RTP / RTCP control unit 12, a codec (codec) 13, speech sound and BGM or sound effect synthesis processing software. There are a holding tone control unit 14, a BGM synthesis unit 15, and a ringing tone control unit 16 which are configured. The session layer 6 controls information transmission. Manage conversation modes between applications and control conversation units.

  The SIP control unit 11 performs call control using a signaling protocol of the application layer 6 (standardized by RFC3261) for establishing, changing, and terminating a multimedia session on the IP network.

  Of the RTP / RTCP control unit 12, RTP is a protocol for transmitting transmission data in which BGM and sound effects are combined with audio data and compressed and encoded. Time information and order information are added to the transmission data. And has a function of transmitting and receiving voice data through the network. RTCP is a control protocol for controlling RTP, and has functions of performing RTP flow control, clock synchronization, data reproduction time recognition, information source recognition, and the like.

  The codec 13 has a function of compressing and decoding the transmission sound (transmission data) described later with high efficiency.

  The function as the presentation layer includes VoIP call control 7. The presentation layer manages the expression format of information transmitted and received by the application, and performs data conversion and encryption. The VoIP call control unit 7 controls the entire VoIP call function.

  As a function of the uppermost application layer, there is a graphical user interface (GUI) 8 for visually operating a computer. The application layer is a layer for managing external specifications of communication functions used in the user program and exchanging information based on the external specifications. The GUI 8 provides an interface for user operation and handles user input information. .

  FIG. 4 is a schematic diagram illustrating an example of a GUI. As shown in FIG. 4, the GUI 8 includes an application control unit 61 that performs termination processing of the VoIP client application 1, an information display unit 62, a dial unit 63, a headset volume unit 64, a speaker volume unit 65, and a sound. An effect selection display section 66, a sound effect control section 67, a BGM selection display section 68, and a BGM control section 69 are provided.

  The information display unit 62 displays a dial number when the user dials the number of the other party to be communicated, a partner state such as whether or not the communication partner is busy, and the like. The dial unit 63 includes a numeric keypad for dialing a VoIP counterpart. The headset volume 64 adjusts the volume output from the headset, and the speaker volume unit 65 adjusts the volume output from the speaker. The sound effect selection unit 67 displays a sound effect sound source data file that can be used by the user. For example, a gunshot sound, lightning sound, applause sound, cheering sound, etc. can be selected. Play and stop sound, and adjust volume. Thereby, a user can express feelings to the other party with various sound effects.

  The BGM selection display unit 68 displays a BGM sound source data file such as a sound of a Tahiti wave that can be used by the user. Further, the BGM control unit 69 further plays and stops the BGM and the volume of the BGM. By performing the adjustment, the user's mood and the atmosphere of the place can be communicated to the communication partner by the volume selected and adjusted by the user himself, like the sound effect.

  Next, the hardware configuration of the VoIP client that executes the software module 1 will be described. FIG. 5 is a block diagram showing a hardware configuration of the VoIP client. As shown in FIG. 5, in the VoIP client 20, a CPU (Central Processing Unit) 221 includes various programs constituting the software module stored in a ROM (Read Only Memory) 222 or a RAM (Random) from the storage unit 228. Access Memory) 223 executes various processes in accordance with various programs constituting the software module 1 described above. In addition, a timer (not shown) performs time counting operation and supplies time information to the CPU 221. The RAM 223 also appropriately stores data necessary for the CPU 221 to execute various processes.

  The CPU 221, ROM 222, and RAM 223 are connected to each other via a bus 224. An input / output interface 225 is also connected to the bus 224.

  The input / output interface 225 includes an input unit 226 including a keyboard and a mouse, a display including a CRT and an LCD, an output unit 227 including a headphone and a speaker, a storage unit 228 including a hard disk, a modem, a terminal A communication unit 229 composed of an adapter or the like is connected.

  The communication unit 229 performs communication processing via the Internet (not shown). Data provided from the CPU 221 is transmitted. The communication unit 229 outputs data received from the communication partner to the CPU 221, RAM 223, and storage unit 228. The storage unit 228 communicates with the CPU 221 to store and delete information. The communication unit 229 also performs analog signal or digital signal communication processing with other clients.

  A drive 230 is connected to the input / output interface 225 as necessary, and a magnetic disk 241, an optical disk 242, a magneto-optical disk 243, a semiconductor memory 244, or the like is appropriately mounted, and a computer program read from these is loaded. It is installed in the storage unit 228 as necessary.

  Next, a method of executing a VoIP call by executing various programs constituting the software module 1 shown in FIG. 2 by the VoIP client having such a hardware configuration will be described.

  FIG. 6 is a diagram for explaining the RTP / RTCP packet transmission / reception function controlled by the VoIP call control unit 7 in the general PC software module shown in FIG. As shown in FIG. 6, the control of the VoIP call control unit 7 can be divided into threads 1 to 5. The thread 1 is a process until the sender transmits an RTP packet, that is, a process of transmitting a data packet containing data including voice uttered by the sender, and the threads 2 and 3 transmit the transmitted RTP packet. Processing until reception and reproduction, that is, processing until the sender hears data including the voice of the other party with whom communication is made.

  The thread 4 is automatically generated by a thread library attached to the operating system (OS) 3. The thread library performs calculation resource allocation, that is, scheduling, on the main processor (CPU 221 shown in FIG. 5) of the thread 1, thread 2, and thread 3 according to priority.

  The thread 5 is a main thread of the GUI 8 that is an application. The thread 5, the thread 2, and the thread 3 are generated and destroyed, and the thread 1, the thread 2, and the thread 5 are generated according to a programmed algorithm or a user operation. The thread 3 is controlled.

  In the thread 1 which is the transmission processing of the RTP / RTCP packet, the user's voice is captured from the microphone and PCM data is received (Capture). If necessary, the BGM synthesizing unit 5 etc. The sound effects and BGM and sound are synthesized (Effect or Mixing), and the codec 13 compresses and encodes the data (encode). Then, the RTP / RTCP control unit 12 converts the compression-encoded data into an RTP packet (packet) and transmits the packet (send).

  In addition to the RTP packet transmission process, an RTCP packet transmission process that stores control information for controlling the codec compression encoding method in the communication target communication device is performed.

  On the other hand, in the RTP packet receiving process, a fluctuation absorbing buffer provided before and after the decoding process provided on the receiving side of the VoIP client, that is, a fluctuation absorbing buffer (Forward Jitter buffer) BF1 for storing the encoded byte stream PCM, and a decoding A PCM buffer (Backward Jitter buffer) BF2 for storing data that has been converted into an uncompressed linear PCM is used.

  In the RTP / RTCP packet reception function, in the thread 2, the RTP / RTCP control unit 12 receives the RTP packet (Receive), and detects a lost packet such as a data loss occurring on the network and a data error of the transmission packet. Compensation processing is performed (parse), and the data is stored (push & pop) in the fluctuation absorbing buffer BF1 provided before the decoding processing. This is decoded and expanded by the codec 13 (decode), and stored as uncompressed data in a fluctuation absorbing buffer (PCM buffer) BF2 provided after the decoding process (push).

  In the thread 3, the USB camera driver 2a, the USB audio driver 2b, and the various drivers 2c read unpopulated data from the fluctuation absorbing buffer BF2 (pop) and reproduce the read data (sound device).

  Further, the effect effect and the synthesis (Effect or Mixing) of BGM and sound performed in the thread 1 may be performed after the data is read from the fluctuation absorbing buffer BF2 in the thread 3 and before the reproduction. Alternatively, a PCM buffer for storing uncompressed data may not be prepared, and data decoded by reading data from the fluctuation absorbing buffer may be reproduced without being stored in the fluctuation absorbing buffer BF2.

  Here, the VoIP system according to the present embodiment realizes high-quality real-time communication, that is, for example, has a high-quality BGM addition function in addition to conversation, and whether or not the user utters voice. Regardless of this, a certain amount of data is transmitted and received to prevent the network bandwidth from being occupied.

  For this reason, the VoIP call control means 7 shown in FIG. 3 checks the amount of voice information of the user captured from the microphone. For example, when the conversation is interrupted, the voice is less than a predetermined volume level and the amount of voice information is small. In some cases, the transmission of audio data is temporarily stopped, and the RTP / RCTP control unit 12 is controlled so that only BGM data is prefetched and transmitted in batch. That is, as described above, normally, a real-time transmission mode in which real-time acoustic data obtained by synthesizing voice, BGM, and sound effects is transmitted in an RTP packet is set to a real-time transmission mode, and it is determined that the user is not speaking. At this timing, the mode is switched to the batch transmission mode in which only BGM is batch-transmitted.

  Here, as will be described later, the RTCP packet transmitted from the RTP / RCTP control unit 12 can be transmitted by describing the size of its own fluctuation absorbing buffer BF1, BF2 by extension, and each VoIP client It is possible to recognize the fluctuation absorbing buffer size of another VoIP client as a communication partner. Therefore, when it can be determined that the voice information of the user is small and the user is not speaking, the VoIP client can collectively transmit only the BGM data corresponding to the fluctuation absorbing buffer size of the other VoIP clients.

  In this way, by deferring additional data such as BGM scheduled to be sent together with the voice, it is not necessary to send the additional data in real time, and the information to be sent in real time is only the voice data or the voice and sound effects. Real-time acoustic data is synthesized, and the network bandwidth can be narrowed.

  Next, the RTP / RTCP packet in the VoIP system in the present embodiment will be described more specifically.

  The RTP / RTCP packet is transmitted / received after the RTP / RTCP session is established by the call control of the SIP control unit 11 shown in FIG. FIGS. 7A and 7B are diagrams showing the structure of RTP packets and the format of the header. FIGS. 8A to 8C are the structures of RTCP APP packets that can be expanded by the application and their headers. It is a figure which shows the example of a format of (RTCP APP Application-defined Header) and a transmission report.

  RTP is a transport protocol for transmitting / receiving voice and moving images in real time in an IP network such as the Internet, and is recommended by RFC1889. RTP is located in the transport layer and is generally used with RTCP over UDP.

  As shown in FIG. 7A, the RTP packet includes an IP header, a UDP header, an RTP header, and RTP data. In the RTP header, as shown in FIG. 7B, from the top, a version information storage unit (V: version, for example, V = 2) F1, a padding storage unit (P: padding) F2, and an extension bit storage unit (X: extension) F3, CSRC (contributing source) count storage unit (CC) F4, marker information (M: marker) storage unit F4, marker bit storage unit (M: maker) F5, payload type information storage unit (PT : Payload type) F6, sequence number information storage unit (sequence number) F7, time stamp storage unit (time stamp) F8, SSRC identifier storage unit (synchronization source identifier) F9, and CSRC identifier storage unit F10 are provided to store CSRC identifiers. Real-time acoustic data is added behind the part F10.

  The version information storage unit F1 stores information indicating the RTP version. For example, when indicating RTP2, the version information indicating that is stored.

  The count storage unit F4 indicates the number of CSRCs (contributing transmission source identifiers) indicated in the header.

  The payload type information storage unit F6 stores information indicating the type of real-time acoustic data, for example, information indicating video or audio.

  The sequence number information storage unit F7 counts up every time an RTP packet is transmitted / received in an RTP session, and stores a sequence number for recognizing the order of the RTP packets to be transmitted / received.

  The time stamp storage unit F8 stores time stamp information related to the date and time when real-time acoustic data is created and updated.

  Information for identifying the source on the data transmission side in the RTP session is stored in the SSRC identifier storage unit F9 and the CSRC identifier storage unit F10. The SSRC identifier is a synchronous transmission source identifier, is an identifier assigned so that a plurality of streams to be handled in combination by the same user share the same value, and the CSRC identifier is a contributing transmission source identifier, indicating a stream source . This is used when a plurality of streams are mixed and provided as one stream data.

  Here, the real-time voice is SSRC, the sound source data (additional data) of the mixing processing part is regarded as one CSRC, the CSRC count storage unit (CC) F4 is incremented by one, and the CSRC is given.

  When transmitting real-time acoustic data according to RTP, the RTP / RTCP control unit 8 shown in FIG. 3 stores various types of information in the storage units and recognizes the various types of information stored in the storage units. Process to extract acoustic data.

  While RTP is a protocol that transmits / receives audio / video data itself, RTCP periodically evaluates line quality such as packet loss, delay jitter, and round trip, and matches the bandwidth. A protocol for transmitting / receiving information to realize real-time communication.

  By using this RTCP, it is possible to perform dynamic processing such as changing the transmission rate by estimating the network state or the like based on information fed back from the other party. In addition, since information indicating who is currently sending data and who is receiving it is also sent at the same time in the RTCP packet, it is possible to know the current participant information.

  As shown in FIG. 8A, the RTCP packet includes an IP header, a UDP header, an RTCP header, and RTCP data. In the header of the RTCP APP packet, which is an expandable RTCP packet, as shown in FIG. 8B, the version information storage unit (V: version, for example, V = 2) F11, the padding storage unit ( P: padding) F12, subtype storage unit F13, packet type (PT) storage unit F14, report length storage unit (length) F16, SSRC / CSRC identifier storage unit F17, ASCII (American Standard Code for Information), A Name storage unit F18 described in US standard code for information exchange) is provided, followed by a data storage unit (Application-Dependent Data) F19 in which application-specific data is stored.

  The type of RTCP packet is recorded in the packet type PT storage unit F14, and in this embodiment, this packet type PT = APP (Application: application specific information) = 204 (packet type value) is described. APP indicates that it is a packet for notifying application-specific control information outside the RTCP standard.

  As described above, the RTP / RTCP control unit 8 shown in FIG. 3 uses the RTCP packet as a RTCP APP packet, and as described above, the fluctuation for receiving the RTP packet used to absorb the jitter when the RTP packet is received as the retransmission request for the missing packet. The size of the absorption buffer can be described in the data storage unit F19 and transmitted. At the same time, according to an instruction from the VoIP call control unit 3, it is possible to describe and transmit information on frequency components to be compressed in transmission data to be transmitted by the communication partner.

  Here, the RTCP packet includes an RTCP SR (Sender Report,) packet of a type sent from the sender of the RTP data and an RTCP packet RTCP RR (Receiver Report,) of a type sent from the receiver of the RTP data. is there.

  The SR packet is transmitted from a terminal that is transmitting a stream to another terminal. Transmission information (sender info) that is information regarding a stream transmitted by the own device and reception of the stream for each received stream Including a report block (reception report block) for reporting the state (packet discard rate, jitter, etc.) to the transmitting device, and the RR packet is for notifying information on the stream received from the other communication device, Similarly, each received stream includes a report block for reporting the reception status of the stream to the transmitting apparatus.

  As shown in FIG. 8C, this report block includes a synchronization source (SSRC) identifier of a packet sender, an RTP loss rate, the number of lost RTP packets, a reception sequence number, and jitter of an arrival time interval. The average value, the transmission time of the last received SR (LSR: Last SR timestamp), and the time (DLSR: Delay since Last SR) from when the SR was last received until this RR is sent are entered.

  Therefore, the transmission side manages the number of transmission RTP packets and the number of transmission RTP bytes when transmitting RTP data, and the reception side receives the number of received RTP packets when receiving RTP data. Management information such as the number of lost RTP packets and jitter of arrival time is managed.

  FIG. 9 is a diagram showing messages exchanged when exchanging RTCP APP messages in the present embodiment. In the RTCP packet including the SR block and the RTCP packet including the RR block exchanged between UA1 and UA2 which are both RTP Sender and Receiver, as shown in FIG. 9, the encoded bandwidth (Encode Bandwidth), number of subband division blocks (sub band Numbers), fluctuation absorption buffer size before and after decoding processing unit (convertible to time) (Forward and Backward Jitter Buffer size), data size queued in the absorption buffer ( (Convertible to time) (Buffer queued size), and RTP packet retransmission request information (Re-Request: sequence number) can be described. The sequence number is used for the RTP packet retransmission request.

  Here, as the application-specific information RTCP APP packet, in addition to the information on the buffer sizes of the fluctuation absorbing buffers BF1 and BF2 before and after the decoding process, the data amount occupying the fluctuation absorbing buffers BF1 and BF2, the VoIP client on the receiving side Describes the encoding bandwidth of the transmission data of the VoIP client on the transmitting side, the number of subband division blocks, etc., and by specifying the compression encoding method of the communication partner, the transmission rate of the stream received by itself And the time from the transmission of the RTCP APP packet including the SR (Sender Report) block to the reception of the RTCP APP packet including the RR (Receiver Report) block. For RTT (Round Trip Time) In response, it is also possible to request retransmission of the missing packet.

  Next, a method for reducing the information amount of real-time data using such RTCP APP packet will be described in more detail. FIG. 10 is a diagram showing a processing sequence when only BGM data is batch-transmitted when audio is actually interrupted.

  As shown in FIG. 10, when transmitting / receiving high-quality sound data in which voice is synthesized with BGM between VoIP clients UA1 and UA2 constituting the VoIP system, first, as described above, the RTCP APP packet is used. Then, the buffer sizes of the fluctuation buffers BF1 and BF2 before and after the decoding processing unit are exchanged (D1). The buffer sizes of these fluctuation absorbing buffers BF1 and BF2 can be converted into transmission time.

  Then, as a real-time transmission mode, voice and acoustic data are transmitted and received by RTP packets (D2). At this time, the real-time voice is set to SSRC = 1234, the CC field F4 described above is incremented, and the BGM audio is set to, for example, CSRC = 5678 to transmit / receive data.

  Thereafter, in one VoIP client UA1, for example, when it is determined that the conversation is interrupted due to, for example, the user leaving the communication apparatus for some reason, and the voice collected by the microphone is interrupted, if there is room in the network bandwidth, The mode is switched to the transmission mode, and BGM data is transmitted in batch (batch) by the buffer size (for example, n seconds) of the fluctuation absorbing buffer BF2 after the decoding process on the other side (D3). At this time, the CSRC (= 5678) used when mixing and transmitting real-time voice and GGM is used for SSRC. That is, it transmits as SSRC = 5677.

  As a result, the VoIP client UA1 on the transmission side can transmit only voice for a time corresponding to n seconds of the BGM data transmitted in a batch.

  If no voice is detected within n seconds, the VoIP client UA1 need not transmit anything. When the VoIP client UA1 transmits the voice again, the VoIP client UA1 again mixes with the BGM, increments the sequence number by n size, and compresses the sound as an RTP packet to resume transmission with the VoIP client UA2 that is the other party. do it.

  In the present embodiment, when the voice information of the person captured from the microphone is small, for example, when the conversation is interrupted, for example, the transmission of the voice data is temporarily stopped, only the data for BGM is pre-read, By transmitting, the network bandwidth can be utilized more efficiently when the user is not speaking.

  In the above-described embodiment, when the conversation is interrupted, other acoustic data to be mixed is collectively transmitted. However, it is also possible to pre-read and transmit the BGM data at the time of call control. It is.

  For example, the user of the call device normally recognizes the target of the call before the start of the call, and therefore selects the BGM to be mixed first, and then actually makes a call, as shown in FIG. The SIP control unit 11 performs call control. Then, after the RTP / RTCP media session is probable and the buffer size of the fluctuation absorbing buffer BF2 after the other party decoding process is exchanged with each other by the RTCP APP packet, before the RTP packet of the voice data is transmitted / received, the batch transmission mode As a result, only the BGM is sent in a batch, so that the VoIP client on the sending side needs to send only the voice for the time of BGM data sent in batches, reducing the amount of information transmitted in real time. The same effects as described above can be obtained.

  Further, after the call control as described above, additional data such as BGM is collectively transmitted before transmission / reception of the voice data, and in response to a user request such as changing the BGM, the voice is transmitted as described above. Of course, the additional data may be transmitted at a time when the transmission is interrupted, that is, the real time transmission mode and the collective transmission mode may be switched and transmitted as necessary.

It is a schematic diagram which shows an example of the VoIP system in embodiment of this invention. It is a block diagram which shows the principal part which consists of the VoIP client 111 and the internet 130 among the VoIP systems in embodiment of this invention. It is a figure which shows the VoIP client application at the time of using general PC as a VoIP client which the VoIP system in embodiment of this invention has. It is a schematic diagram which shows an example of GUI in the said VoIP client application. It is a block diagram which shows the hardware constitutions of the VoIP client (VoIP client) in the VoIP system in embodiment of this invention. It is a functional block diagram which shows the process which transmits / receives an RTP / RTCP packet among the VoIP client applications shown in FIG. (A) And (b) is a figure which shows the structure of a RTP packet, and the format of a header, respectively. (A) thru | or (c) is a figure which shows the format example of the structure of the RTCP APP packet expandable by an application, its header (RTCP APP Application-defined Header), and a transmission report. It is a figure which shows the message exchanged at the time of the exchange of RTCP APP message in the VoIP system in this Embodiment. It is a figure explaining the telephone call method in this Embodiment, Comprising: It is a figure which shows the process sequence at the time of carrying out batch transmission of only BGM data, when the audio | voice has interrupted.

Explanation of symbols

1 VoIP client application, 21 transmission means, 22 microphone capture, 23 sound effect file reading unit, 24 BGM file reading unit, 25, 26, 31 decoder, 27, 28, 29 gain adjustment unit, 30 synthesis unit, 31 encoder, 32 Packetizing unit, 33 transmitting unit, 41 receiving means, 42 receiving unit, 43 depacketizing unit, 44 dejittering unit, 45 packet compensating unit, 46 decoder, 47 reading unit, 48 decoding unit, 49, 50, 51, 52 gain adjusting unit , 53 synthesis unit, 54 output unit, 55 speaker, 100 VoIP communication system, 111, 121 VoIP client, 110, 120 user, 112, 122 web browser, 134 web server, 113, 123 sound source data, 130 interface Tsu door

Claims (17)

In a communication device that transmits and receives at least audio data via the Internet,
Audio conversion means for converting the collected sound into an electrical signal;
Synthesizing means for synthesizing additional data with the audio data converted into the electrical signal;
Data packet generating means for generating a data packet storing the audio data and / or additional data to be synthesized with the audio data;
Control packet generating means for generating a control packet storing management information for managing at least transmission and reception of the data packet;
Transmitting means for transmitting the data packet and the control packet to the one or more other communication devices via the Internet;
Receiving means for receiving data packets and control packets from the one or more other communication devices;
Control means for controlling transmission and reception of the data packet and the control packet,
The control means includes a first mode for transmitting in real time a first data packet storing the voice data or synthesized data obtained by synthesizing additional data with the voice data, and second data storing only the additional data. A communication apparatus that controls switching between a second mode in which packets are collectively transmitted. The call device according to claim 1, wherein the control means performs switching control to the second mode in which the collective transmission is performed when the collected sound is lower than a predetermined volume level. The call device according to claim 1, wherein the control means controls to perform batch transmission in the second mode before performing real-time transmission in the first mode with the other call device. The control means controls to transmit the first data packet storing only the audio data in the first mode for a predetermined time after batch transmission of the second data packet in the second mode. The communication device according to claim 1, wherein: The control means controls the transmission of the data packet to be stopped for a predetermined time when the collected sound is less than a predetermined volume level after the second data packet is collectively transmitted in the second mode. The communication device according to claim 1, wherein: The call device according to claim 1, wherein the additional data is background music and / or sound effects. The receiving means has a receiving buffer for buffering received data packets,
The control means stores buffer information indicating the capacity of the reception buffer in the control packet, and stores the buffer information of the other communication apparatus stored in the control packet from the other communication apparatus received by the reception means. The communication device according to claim 1, wherein a transmission rate of the second data packet is controlled based on the transmission rate. The control means stores buffer occupancy information indicating the amount of data buffered in the reception buffer in the control packet, and the other means stored in the control packet received by the receiving means from the other call device. The call device according to claim 7, wherein a transmission rate of the second data packet is controlled based on buffer information and buffer occupancy information of the call device. In a calling method that transmits and receives at least audio data via the Internet,
An audio conversion process for converting the collected sound into an electrical signal;
A synthesis step of synthesizing additional data with the audio data converted into the electrical signal;
A data packet generating step for generating a data packet storing the audio data and / or additional data to be synthesized with the audio data;
A control packet generation step of generating a control packet storing management information for managing at least transmission and reception of the data packet;
A transmission step of transmitting the data packet and the control packet to one or more other communication devices via the Internet;
Receiving a data packet and a control packet from the one or more other communication devices;
A control process for controlling transmission and reception of the data packet and the control packet,
In the control step, a first mode for transmitting in real time a first data packet storing the voice data or synthesized data obtained by synthesizing additional data with the voice data, and second data storing only the additional data. A call method characterized by switching control of a second mode in which packets are collectively transmitted. The call method according to claim 9, wherein, in the control step, when the collected voice is less than a predetermined volume level, the control is switched to the second mode in which the batch transmission is performed. The call method according to claim 9, wherein, in the control step, control is performed so as to perform batch transmission in the second mode before performing real-time transmission in the first mode with the other call device. In the control step, after the second data packet is collectively transmitted in the second mode, the first data packet storing only the audio data is transmitted in the first mode for a predetermined time. The call method according to claim 9, wherein: In the control step, after the second data packet is collectively transmitted in the second mode, if the collected sound is less than a predetermined volume level, control is performed to stop transmission of the data packet for a predetermined time. The call method according to claim 9, wherein: The call method according to claim 9, wherein the additional data is background music and / or sound effects. In the control step, buffer information indicating a capacity of a reception buffer for buffering the received data packet is stored in the control packet, and the control information from the other call device received in the reception step is stored in the control packet. The call method according to claim 9, wherein a transmission rate of the second data packet is controlled based on buffer information of another call device. In the control step, buffer occupancy information indicating the amount of data buffered in the reception buffer is stored in the control packet, and the reception unit stores the other information stored in the control packet received from the other call device. The call method according to claim 15, wherein a transmission rate of the second data packet is controlled based on buffer information and buffer occupancy information of the call device. In a call system that transmits and receives at least audio data via the Internet,
Each of the above communication devices
Audio conversion means for converting the collected sound into an electrical signal;
Synthesizing means for synthesizing additional data with the audio data converted into the electrical signal;
Data packet generating means for generating a data packet storing the audio data and / or additional data to be synthesized with the audio data;
Control packet generating means for generating a control packet storing management information for managing at least transmission and reception of the data packet;
Transmitting means for transmitting the data packet and the control packet to the one or more other communication devices via the Internet;
Receiving means for receiving data packets and control packets from the one or more other communication devices;
Control means for controlling transmission and reception of the data packet and the control packet,
The reception means of one of the at least two communication devices has a reception buffer for buffering the received data packet, and the control means stores buffer information indicating the capacity of the reception buffer in the control packet,
The control means of the other call device stores a first mode for transmitting in real time a first data packet storing the voice data or synthesized data obtained by synthesizing the voice data with additional data, and stores only the additional data. A second mode of batch transmission of the second data packet, and a buffer of the one communication device stored in the control packet from the one communication device received by the receiving means. A call system, wherein the transmission rate of the second data packet is controlled based on the information.

Images