KR100911771B1 - A apparatus of packet loss concealment with realtime voice communication on internet and method thereof - Google Patents

Abstract

The present invention relates to a terminal apparatus and a method for concealing packet loss of voice communication, and particularly, to a terminal for real-time communication with an audio signal through the Internet, a variable bit rate in which the total bit rate of an audio signal and surplus information input in real time does not increase. A transmitter that encodes the data, records a sound quality evaluation result, converts the result into a real-time transmission format, and transmits the packet to the Internet; And a receiving end receiving a real time audio packet through the Internet, checking a sound quality evaluation result and feeding it back to the transmitting end, sequentially recording a packet decoded at a variable bit rate, removing jitter, feeding back a sound quality evaluation result, and outputting a reproduced voice signal. Characterized in that the configuration, the excess information is transmitted without increasing the transmission bit rate to recover the loss caused without affecting the bandwidth or data transmission rate of the occupied channel, and efficiently concealed to improve the quality of the voice signal It works.

Audio, Encoding, Decoding, Format, Packet, RTP, Real Time, Buffer, Quality, Internet

Description

A APPARATUS OF PACKET LOSS CONCEALMENT WITH REALTIME VOICE COMMUNICATION ON INTERNET AND METHOD THEREOF}

The present invention compensates for the data loss of the voice signal real-time communication device using the Internet. In particular, the redundant information for concealing the loss of packet data caused by the voice signal is transmitted without increasing the overall transmission bit rate and measured at the receiving end. The present invention relates to a terminal apparatus and a method for concealing packet loss of voice communication in which a transmission side determines a variable bit rate and a data type according to a sound quality evaluation result.

As social and intellectual activities of human beings become more active, cases of communicating to designated parties or communicating for communication often occur, and the amount of information to be delivered increases.

In addition, the communication time is not fixed regularly, and it is common to immediately connect and communicate with a desired counterpart when necessary.

Communication means the exchange of information with a far-distance or long-distance counterpart that cannot exchange information directly with the other party by using sounds, symbols, etc. There may be cases where you do not know where it is located, and communication is generally expensive.

The real-time communication method is a method of directly connecting with the other party and delivering the doctor and information, and the non-real-time communication method is to transmit the information to be delivered and to confirm and reply to the other party at a leisurely time.

As described above, regardless of where the other party is located on the earth and communicates in real time (NON-REALTIME) rather than in real time (REALTIME), the amount of information to be transmitted may be very large and the communication cost is relatively low One way is the Internet.

The Internet transmits and receives a packet through multiple paths, and takes a predetermined time to process the transmitted and received packet data, and the delay and jitter are issued by the characteristics of the network. It has been used for data communication.

RELIABILITY, VOICE QUALITY, QUALITY OF SERVICE, and STANDARDIZATION must be guaranteed in order to provide INTERNET TELEPHONY communication service that sends and receives voice level signals in real time using the Internet. It also includes factors such as CLARITY, DELAY, and ECHO, and if there is a problem with any one of them, it affects the overall voice quality.

However, the real-time communication technology using voice signals including audio signals continues to be developed and developed due to low communication costs in the case of using the Internet and to connect and communicate anywhere in the world, and to develop broadband communication system equipment.

Due to the development of the Internet communication technology, the real-time audio media communication service is more preferred.

When the Internet has congestion or a bottleneck of traffic (TRAFFIC) in the network connection, that is, when a lot of traffic occurs, a packet-level transmission delay or a loss due to a transmission error occurs. In the real-time communication service using the PSA, it is difficult to adequately secure quality or quality of service (QoS).

Therefore, for real-time applications, techniques that guarantee minimal performance to reduce packet loss, such as bandwidth limitation (BANDWIDTH LIMITATION), packet loss recovery (LOSS RECOVERY SCHEME), QUEUING, and packet classification (PACKET) CLASSIFICATION, FORWARD ERROR CORRECTION (FEC), etc.

Generally, about 5% or more loss occurs in streaming of packet data transmitted through the Internet, and when a silent section is generated due to the loss, the sound quality is degraded so that normal real-time communication is difficult. Many techniques are being developed to compensate for data loss.

In the prior art, a method of interpolating by approximating a silent section generated by lost packet data to the original audio signal waveform WAVEFORM is used.

In addition, in the prior art, the interpolation is performed using a noise or a method such as repeated playback in a section where a loss occurs.

In general, one packet due to the audio communication signal corresponds to about 20 to 30 milliseconds (MILLISECOND) time, so even in the event of loss of only one packet, the intermittent interval or silence interval becomes relatively large to interpolate. it's difficult.

In order to further develop such a prior art, an interpolation technique in a compression domain for an MPEG (MOVING PICTURE EXPERTS GROUP) audio codec is presented, "ERROR CONCEALMENT FOR COMPRESSED DIGITAL AUDIO" P. LAUBER AND R. SPERSCHNEIDER, IN AES 111TH CONVENTION, NY, PP. 1-11, SEPT. 2001 and "ERROR MITIGATION IN MPEG-AUDIO PACKET COMMUNICATION SYSTEMS" S. QUACKENBUSH AND P. DRIESSEN, IN AES 115TH CONVENTION, NY, PP. 1-11, OCT. 2003.

In addition, a method of using PARAMETRIC AUDIO MODELING is presented, such as "PACKET LOSS CONCEALMENT BASED ON SINUSOIDAL EXTRAPOLATION" J. LINDBLOM AND P. HEDELIN, IN PROC. ICASSP, PP.173-176, MAY 2002.

As described above, the algorithm for recovering the loss of packet data is called a packet loss concealment (PLC) algorithm.

The PLC algorithm is divided into two methods, a transmitter-based and a receiver-based method. “ADAPTIVE RECOVERY TECHNIQUES FOR REAL-TIME AUDIO STREAMS” WT LIAO, JC CHEN, MS CHEN, IN PROC. IEEE INFOCOM, ANCHORAGE, AK, VOL. PP. 815-823, APR. 2001.

The transmitter-based PLC algorithm uses redundancy information or side information transmitted together with the previous packet in place of the lost packet when the packet received at the receiver is lost. (FORWARD ERROR CORRECTION: FEC), INTERLEAVING, RE-TRANSMISSION, etc.

In addition, the receiver-based PLC algorithm restores the lost packet data by using valid packet data at the receiving end, that is, packet data received before the packet in which the loss occurred and packet data received thereafter. INSERTION, INTERPOLATION, and MODEL-BASED RECOVERY.

However, although the transmitter-based PLC algorithm of the related art has a good result for restoring lost packet data, the transmission bit rate is increased by additional transmission of the corresponding information, and the encoding of the previous frame or packet is performed. There is a problem that a processing delay occurs, such as a need to be processed again.

In addition, according to the prior art, the receiver-based PLC algorithm has an advantage of recovering the packet lost independently and independently at the receiver even without additional information provided by the transmitter, but is difficult to recover itself when the packet loss rate is high. Therefore, there is a problem in that the quality (QoS) of the voice-level communication signal drops sharply.

Problems such as "PACKET LOSS CONCEALMENT FOR AUDIO STREAMING BASED ON THE GAPES AND MAPES ALGORITHMS" H. OFIR AND D. MALAH, IN IEEE 24TH CONVENTION OF ELECTRICAL AND ELECTRONICS ENGINEERS, ISRAEL, PP. 280-284, NOV. 2006 and “G.729 Frame Loss Concealment Algorithm Robust to Continuous Frame Loss” Cho, Choong-Sang, Young-Han Lee, Hong-Kook Kim, and Korean Society of Speech Sciences Conference 1, pp. 307-310, 2007.5.

In addition, in the prior art as described above, when a loss occurs in a highly correlated voice signal, a certain amount of recovery can be performed by predicting data in a section in which the loss occurs, but in the case of a wideband audio stream such as music, a packet unit There is a problem that it is difficult to predict the correlation between signals, and thus it is difficult to restore itself without additional information.

The present invention has been made in view of the above-mentioned problems of the prior art, and in particular, when a loss occurs in a real-time voice level signal transmitted over the Internet, a voice signal packet of a loss interval is transmitted by transmitting surplus information without increasing the overall transmission bit rate. It is an object of the present invention to provide a terminal apparatus and a method for concealing packet loss of voice communication by recovering or concealing the data.

Another object of the present invention is to use a result of receiving a voice signal packet at a receiving side and evaluating sound quality, to determine a variable bit rate at the transmitting side, and to determine and transmit surplus information according to the type of additional voice data. It is an object of the present invention to provide a terminal apparatus and method for concealing a loss.

The present invention devised to achieve the above object, in the terminal for real-time communication to the audio signal via the Internet, the real-time input of the audio signal and the surplus information is encoded at a variable bit rate does not increase the overall bit rate and sound quality evaluation A transmitter that records the result, converts it into a real-time transmission format, and transmits the packet to the Internet; And a receiving end receiving a real time audio packet through the Internet, checking a sound quality evaluation result and feeding it back to the transmitting end, sequentially recording a packet decoded at a variable bit rate, removing jitter, feeding back a sound quality evaluation result, and outputting a reproduced voice signal. This presents the included configuration.

Preferably, the transmitting end is a sound capture unit for sampling and input the audio signal is applied in real time; An audio encoding unit for encoding an input audio signal into a digital signal and encoding the surplus information of the previous packet in a state where the total bit rate is not increased; A real time format unit which records the sound quality measurement result fed back to the audio signal encoded by the audio encoding unit and converts the result into a real time transmission format; A real-time packet unit for converting and transmitting the audio signal formatted by the real-time format unit into a packet for real-time transmission to the Internet; Characterized in that comprises a.

The receiver may include a real time receiver configured to receive a real time packet transmitted from an opposite terminal through the Internet; A real time analyzer for analyzing a packet received by a real time receiver to check a sound quality evaluation result and feeding back to the transmitter; An audio decoder for analyzing a variable bit rate from a signal applied by a real time analyzer and decoding a received packet; A buffer unit for recording the decoded packets in order and removing jitter; A quality measuring unit which sequentially receives the packets stored in the buffer unit and measures a sound quality evaluation result at each predetermined number period and feeds back to the transmitting end; An audio renderer for outputting an audio signal applied from the quality measurer; Characterized in that comprises a.

The audio encoding unit may be configured to encode an input audio signal and surplus information at a variable bit rate without increasing the overall bit rate.

In addition, the variable bit rate is characterized in that consisting of a configuration that is confirmed by analyzing the sound quality evaluation results confirmed and feedback from the receiving end.

In addition, the variable bit rate is characterized by consisting of a configuration for analyzing and confirming the sound quality evaluation result received in the packet from the other terminal.

The real-time format unit may be configured to record a measurement sound quality evaluation result fed back by a receiver in a prediction region MOS of a payload format.

The audio decoder may be configured to analyze a signal of the real-time analyzer to identify a variable bit rate and to decode the identified variable bit rate.

In addition, the signal received by the receiving unit is characterized in that the configuration comprises at least one or more of the excess information of the current packet and the previous packet.

In addition, the surplus information is characterized in that the configuration further comprises a surplus information for the previous packet of the previous packet.

In addition, the variable bit rate is characterized by a configuration made by the MPEG-2 (AAC).

In addition, the jitter is characterized by a configuration consisting of variations occurring in the Internet.

In addition, the buffer unit is configured to remove the jitter since the decoded packet signal is written to the buffer.

The quality measuring unit may be configured to measure a sound quality evaluation result after receiving an audio packet signal of a predetermined number period set from a buffer unit.

In addition, the sound quality evaluation is characterized in that the objective result is evaluated by using the NON-INTRUSIVE method of the ITU-T standard P563 (P.563).

In addition, the quality measuring unit is characterized in that the result of the sound quality evaluation expressed in the range of 1 to 5.

In addition, the result value of the sound quality evaluation is characterized in that consisting of a configuration that is fed back to the real-time format portion of the transmitter is recorded.

In addition, the real-time analysis unit is characterized in that consisting of a configuration for feeding back the confirmed sound quality evaluation results to the transmitting end.

The audio renderer may be configured to reproduce and output an audio signal of a packet unit applied from a buffer unit in real time through a sound card.

In order to achieve the above object, the present invention provides a method for concealing packet loss of a voice communication, the method comprising: inputting an audio signal by a transmitting end of a voice communication terminal and capturing it in real time; Encoding the captured audio signal together with the surplus information by the variable bit rate analyzed by the confirmed sound quality evaluation result; Converting the encoded audio signal and the measured sound quality evaluation result into a real-time transmission format; And converting an audio signal of a real time transmission format into a packet and transmitting it to the Internet. Present a configuration that includes a.

Preferably, the encoding process does not encode the surplus information based on the confirmed sound quality evaluation result.

The encoding process may be performed by encoding one surplus information or two surplus information together in a state in which the total bit rate does not increase according to the confirmed sound quality evaluation result.

In addition, the one surplus information is characterized in that the information on the previous packet of the packet to be currently encoded.

In addition, the two surplus information is characterized in that the information on the previous packet and the previous packet of the current packet to be encoded.

The measured sound quality evaluation result may be a result of accumulating the audio packets received from the counterpart by a predetermined predetermined number unit.

In order to achieve the above object, the present invention provides a method for concealing packet loss of voice communication, wherein the receiving end of the voice communication terminal receives and analyzes an audio packet in real time and feeds back the confirmed sound quality evaluation result to the transmitting end. Process of doing; Decoding the received packet at a variable bit rate analyzed from the sound quality and the result and storing the received packet in a buffer by a sequence number; And converting the sound quality evaluation measured from the packet of the predetermined number unit stored in the buffer into a real time audio signal while feeding back to the transmitter. This presents the included configuration.

Preferably, the confirmed sound quality evaluation result is characterized in that to confirm the recorded sound quality evaluation results by analyzing the received packet.

In addition, the variable bit rate is characterized by confirming by analyzing from the confirmed speech evaluation results.

In addition, the sound quality evaluation is characterized in that for every cycle in which packets are accumulated and recorded in a predetermined number unit set in the buffer.

In addition, the output of the audio signal is characterized by receiving and converting and outputting audio packets which are sequentially recorded in the buffer regardless of the measurement of the sound quality evaluation.

In order to achieve the above object, the present invention provides a method for concealing packet loss of voice communication, comprising: analyzing a packet received from a counterpart by a receiving end of a voice communication terminal and confirming a sound quality evaluation result; Varying the transmission bit rate according to the analyzed sound quality evaluation result and coding the captured audio signal and the redundant information; Accumulating the received packets in a predetermined predetermined unit of time and measuring an objective sound quality evaluation; And feeding back the measured sound quality evaluation result to the transmitting end to record and transmit in the designated area of the format; Present a configuration that includes.

Preferably, the step of coding the surplus information is any one selected from coding the surplus information or coding the surplus information of the previous packet and the previous packet according to the analysis result of the sound quality evaluation It is characterized by.

The present invention having the above-described configuration transmits the surplus information for recovering the loss of packet data transmitted through the Internet without increasing the overall transmission bit rate, thereby reducing the loss generated without affecting the bandwidth or data transmission rate of the occupied channel. There is an industrial benefit of rapid recovery.

In addition, according to the present invention, the variable bit rate and data type are determined and transmitted at the transmitting side by using the result of sound quality evaluation at the receiving side connected to the Internet. There is a convenient effect to increase the quality of.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS Hereinafter, preferred embodiments of a terminal apparatus and method for concealing packet loss of voice communication according to the present invention having the above configuration will be described in detail with reference to the accompanying drawings.

Example

1 is an overall functional configuration diagram according to an example of the Internet system to which the present invention is applied, and FIG. 2 is a terminal for concealing packet loss of voice communication according to an example of the present invention. Fig. 3 is a detailed field configuration diagram of a general audio packet for explaining the present invention, and Fig. 4 is a detailed field configuration diagram of an audio packet transmitted according to an example of the present invention. FIG. 6 is a diagram illustrating a real-time sound quality evaluation method according to an exemplary embodiment of the present invention, and FIG. 6 is a diagram illustrating a method of differentially performing surplus information encoding according to a sound quality evaluation result that a transmitter receives and is fed back according to an example of the present invention. FIG. 7 illustrates an example of the present invention, which measures buffering and sound quality evaluation results of packets transmitted and received between devices. FIG. 8 is a diagram illustrating a result detected by an experiment of the present invention, and FIG. 9 is a flowchart of a method of operating a transmitting end by a terminal device concealing packet loss of voice communication according to an example of the present invention. FIG. 10 is a flowchart illustrating a method of operating a receiving end by a terminal device for concealing packet loss of voice communication according to an example of the present invention.

In describing an example of the present invention, drawings and descriptions of well-known technical contents that are not directly related to the present invention are omitted, and thus the present invention is clearly communicated without obscuring the gist of the present invention.

Referring to FIG. 1, the Internet system to which the present invention is applied includes a terminal device 100 that inputs and receives voice signals, converts them into digital data, converts them into packet units, uses UDP / IP and RTP protocols, etc. 100 ') and the Internet 200 are included.

The terminal device 100, 100 ′ is configured to include a transmitting end 110 and a receiving end 120.

The transmitter 110 samples the input audio signal, encodes the received audio signal with the variable bit rate analyzed from the confirmed sound quality evaluation result, converts it into a real time format including the measured quality evaluation result, and processes the packet. The signal is transmitted to the Internet 200.

The receiver 120 is connected to the Internet 200 to receive a packet, extracts and confirms a sound quality evaluation result transmitted from the other party's transmitter, and feeds back the confirmed sound quality evaluation result to the transmitter 110, Analyze the results of the sound quality evaluation to identify and decode the encoded variable bit rate of the received packets, remove the jitter by recording them in the buffer in sequence number, and measure the sound quality evaluation results from the accumulated packets at a predetermined number of periods. The decoded packets are converted into audio signals in order and output.

The INTERNET 200 selects a counterpart designated by a plurality of computers or hubs distributed around the world, designated as communicating with each other by exchanging packet data with each other, such as a net or a network. Send and receive information.

Protocols for transmitting a packet through the Internet 200 include TCP (TRANSMISSION CONTROL PROTOCOL) and UDP (USER DATAGRAM PROTOCOL), and a technique for distinguishing and recognizing the other party connected to the Internet 200 includes: There is an IP (INTERNET PROTOCOL) protocol.

The IP distinguishes a sender and a receiver in the Internet 200 formed of a network, and is IP address information using numbers and English letters.

The TCP divides the digital data information into a packet or a datagram (hereinafter referred to as a packet) of a predetermined unit and assigns a serial number to each packet so that the data can be transmitted and received in a reliable state. If the packet information divided by each is received by changing the order by the given serial number, it is assembled according to the order. Check whether the other party's receiver designated by IP received the transmitted packet data without error, and check the status of the error. In this case, the reliability of the transmitted data is ensured by using a method of transmitting the packet of the serial number again.

The UDP is a big difference in not assigning a sequence number to each divided packet, and is widely used when the amount or size of data to be transmitted is very small.

Although UDP is mainly used for media transmission by real time communication due to the advantage of rapidly transmitting packet data, there is no process of retransmitting a packet in which loss occurs. Therefore, a packet loss conceals packet loss when a loss occurs. : PLC) technology.

The PLC technology is a technology that does not allow the receiver to feel packet loss because the previous packet is played instead of the lost packet.

In the present invention, RTP (REALTIME TRANSPORT PROTOCOL) protocol is basically used to specify the order of packets while using the UDP protocol.

Referring to FIG. 2, a terminal device for concealing packet loss of voice communication according to an example of the present invention will be divided into a transmitting end 110 and a receiving end 120.

According to the present invention, the terminal apparatus concealing packet loss of voice communication uses a redundant information coding scheme, which is one of the forward error correction (FEC) schemes.

The transmitter 110 includes a sound capture unit 111, an audio encoder 112, a real time format unit 113, and a real time packet unit 114, and the receiver 120 includes a real time receiver 121. , A real time analyzer 122, an audio decoder 123, a buffer 124, a quality measurer 125, and an audio renderer 126.

The sound capture unit 111 of the transmitter 110 samples the input audio signal with a predetermined frequency and inputs it as a digital signal.

The audio encoding unit 112 encodes an audio signal input by the sound capture unit 111, and is provided with feedback from the real-time analyzer 122 of the receiver 120 to provide the confirmed sound quality. The evaluation result is analyzed to determine a variable bit rate and to encode an input audio signal according to the determined variable bit rate.

The real-time analyzer 122 of the receiving end 120 analyzes the audio packet transmitted from the transmitting end 110 of the counterpart terminal device connected through the Internet 200, thereby extracting and confirming the recorded sound quality evaluation result.

That is, the audio encoding unit 112 determines the bit rate or variable bit rate to be encoded of the audio signal to be transmitted using the sound quality evaluation result value informed by the other party.

For example, the variable bit rate is applied by a codec that supports a variable bit rate based on the RTP PAYLOAD FORMAT of MPEG-2 AAC defined in the IETF RFC (REQUEST FOR COMMENTS) DRAFT.

Therefore, the transmitting end transmits surplus information about the previous packet to conceal packet loss, that is, by adjusting the transmission bit rates of the surplus information of the current packet and the previous packet, respectively, so that the overall bit rate is maintained without increasing.

For example, when the sound quality evaluation result value is greater than or equal to a predetermined criterion, the input current audio signal is encoded and added to a frame of a packet to be transmitted, and when the sound quality evaluation result value is a value within a set range, the overall bit rate does not increase. In this case, the bit rate is varied so that the current audio signal and the audio signal of the previous packet are encoded at the same bit rate.

In addition, even when the result of the sound quality evaluation is less than or equal to a predetermined criterion, the bit rate is varied while the overall bit rate does not increase, and half of the bit rate is allocated to the packet N for the current audio signal. The same bit rate is allocated to the audio signal by 1) and the previous packet N-2, and encoded.

Such an example will be described in detail again in the accompanying description of FIG. 6.

The real-time format unit 113 checks the sound quality evaluation result value transmitted from the counterpart terminal device while the audio encoding unit 112 does not increase the overall bit rate, and encodes the current audio packet or transfers the current audio packet to the previous one. Encode an audio packet or convert a signal that encodes a current audio packet and a previous and previous audio packet into a real-time format (RTP PAYLOAD FORMAT).

In addition, at the same time as the above conversion, the quality measurement unit 125 of the receiver 120 receives the previous stream STREAM of the audio packet transmitted from the counterpart terminal device and estimates the sound quality evaluation result value of the converted real-time format. Record in the PREDICTION FIELD or MOS (MEAN OPINION SCORE) area.

The conversion by the real-time format unit 113 into a format suitable for real-time transmission is possible because it provides information necessary for continuity of packets and restoration of packet loss, and transmission of multiple audio packets and information on the same.

Since the RTP payload format formatted by the real time formatter 113 analyzes the prediction field received from the counterpart, the encoding state of the surplus information and the type (type) of the surplus information, that is, includes one or two previous packets. You will be asked to confirm the type.

The real time packet unit 114 finally converts a signal of a packet provided by the real time format unit 113 into an RTP packet and transmits it to the Internet 200.

At this time, the transmitter 110 applies the UDP / IP protocol, but applies the RTP protocol again for the order of the transmitted packets.

The real-time receiver 121 of the receiver 120 receives an RTP packet received from the Internet 200. In this case, as opposed to the transmitter 110, the RTP protocol is first applied and then the UDP protocol is applied.

The real-time analyzer 122 analyzes a signal provided by the real-time receiver 121 to extract and confirm a sound quality evaluation result value, and checks the checked sound quality evaluation result value in the audio encoding unit 112 of the transmitter 110. Feedback to.

The confirmed sound quality evaluation result value is transmitted from the other party's transmitter, and is applied to a variable bit rate by analyzing the excess information encoded by the other party's transmitter and the type of the redundant information, and the value applied to a codec supported by MPEG-2 AAC. do.

The audio decoder 123 receives the sound quality evaluation result value and the received packet received from the real-time analyzer 122, checks a variable bit rate encoded by the other party's transmitter, and checks the variable bit rate. DECODING the applied packet.

The buffer unit 124 records the audio packet decoded by the audio decoder unit 123 in the buffer in the order designated by the sequence number (SEQUENCE NO.) According to the RTP protocol. Since the data is written to the buffer unit, the delay variation of jitter generated in the Internet 200 is eliminated.

The jitter has been removed by the buffer unit 124 is provided to the quality measuring unit 125 and a predetermined number of audio packets are repeatedly accumulated, and the audio render unit 126 has a constant and regular time. Are provided at intervals.

When the audio packet is accumulated by a predetermined number, the quality measuring unit 125 measures the sound quality evaluation by the received audio packet, and transmits the measured sound quality evaluation result value to the real time format unit 113 of the transmitter 110. Feedback should be recorded to.

The quality measuring unit objectively evaluates sound quality using a non-intrusive method such as P. 563 (P.563), which is an INTERNATIONAL TELECOMMUNICATION UNION-TELECOMMUNICATION STANDARDIZATION SECTOR (ITU-T) standard.

The result of the evaluation is expressed as a MEAN OPINION SCORE value between 1 and 5.

The MOS value means 0 to 1 BAD, 2 to POOR, 3 to FAIR, 4 to GOOD, and 5 to EXCELLENT. In general, if the value is 3.6, voice communication is possible.

The audio renderer 126 reproduces the audio packet regularly applied and outputs it in a continuous state constantly.

As described above, since the audio signal is transmitted in real time in both directions, the communicating device is also referred to as an audio conferencing system (ACS).

Referring to FIG. 3, the detailed field configuration of a general audio packet will be described for the purposes of the present invention, and is for MPEG-2 and MPEG-4 AAC audio defined in IETF RFC (REQUEST FOR COMMENTS) DRAFT. This is the original format for exchanging sound quality evaluation methods based on PAYLOAD FORMAT.

The format may be one or more packets by being transmitted through one packet or multipath, and the packet includes a plurality of frames, but the following description is brief. In the case where it is not necessary to distinguish the frame to make it easy and easy to understand, the frame will be described and described in a format.

The format of FIG. 3 includes first, information for restoring when a frame is lost in the process of being transmitted through the Internet, second, information on whether the frame is divided into several RTP packets, and third, grouping and interleaving the frame. Fourth, since it provides the information on whether the INTERLEAVING) and the information data of the actual audio frame is included, it is a configuration that provides the ability to conceal the packet loss and synchronization.

3, the PVS region 301 indicates how many 32-bit PRD VECTOR regions 305, 306, and 307 are present.

The PRD VECTOR regions 305, 306, and 307 combine PQ (PREDICTABILITY QUANTIFIER) information for each frame by bit shift. If the value is “0”, the lost information cannot be predicted using another AAC frame. If the value is “1”, the state can be predicted from the previous frame or both frames before and after. If the value is “2”, the state is unpredictable from the previous frame, the next frame, or both frames before and after. Is displayed.

The M area 302 indicates how many frames are contiguous in the current packet, and indicates whether the frames are grouped or interleaved.

The F area 303 indicates whether the frame included in the current packet is divided, and indicates whether the frame is divided into several RTP packets.

The TYPE area 308 indicates the type of data. If the value is '0', the encoded original data is encoded. If the value is '1', the encoded original data is encoded original data but is redundant information data for packet recovery. A value indicates the encoded data compressed at a high compression rate to reduce the bit rate.

The SEQ area 309 indicates the sequence number of the frame, and the LENGTH area 310 indicates the length of the frame in bytes.

With reference to FIG. 4, the detailed field structure of the audio packet transmitted by the example of this invention is demonstrated.

4 is a modified RTP PAYLOAD FORMAT modified for the present invention, and the sound quality evaluation result periodically measured from the receiving end is recorded in the MOS region 410 in the PRD VECTOR region, which is a region for recording prediction information of the frame.

If the value recorded in the PVS region of the RTP PAYLOAD FORMAT is equal to or greater than "1", the value recorded in the MOS region 410 is referred to. If the value recorded in the PVS region is "0", the MOS region 410 is determined. Does not reference a value.

The value recorded in the MOS region 410 is a value between 1 and 5, and it may be confirmed whether a sound quality evaluation result is recorded.

That is, if the value recorded in the MOS region 410 is 1 or more, it is checked whether the PRD VECTOR exists and the sound quality evaluation result value is fed back.

When the transmitter transmits the surplus information for the previous frame, the audio data regions 430 and 440 which are the next order of the main frame 420 are grouped and transmitted.

Referring to Figure 5 by way of example of the present invention will be described the real-time sound quality evaluation method at the receiving end.

5 is for determining a variable bit rate for encoding the surplus information at the transmitter, to ensure a sound quality evaluation result used as information for analyzing the variable bit rate.

The receiver 120 of the terminal device according to the present invention analyzes the received RTP packet (510), analyzes the RTP PAYLOAD FORMAT (520), and decodes the compressed and transmitted audio data (S530).

In this case, if packet loss occurs, the data to be decoded is NULL, and the decoding and playback process are skipped. In the next step, the sound quality measurement is performed by generating RAW audio data generated by the arrival of N packets including the lost packet. Whenever possible, measure the sound quality.

The downsampled audio signal of the wideband audio signal is measured 540 for the sound quality evaluation measurement represented by the MOS, and the down sampled audio data is input to the actual sound quality evaluation block 550.

The sound quality evaluation method is based on the ITU-T standard P. 563, and uses a non-intrusive evaluation method that can independently perform sound quality evaluation without a reference sound source.

The configuration finally receives N RTP packets and outputs one MOS value.

Referring to FIG. 6, a method of differentially performing surplus information encoding according to a sound quality evaluation result fed back and received by a transmitting end will be described.

Referring to FIG. 6 in detail, for example, when the measured sound quality evaluation result value (MOS m) of the Nth transmitted packet is greater than or equal to the sound quality evaluation first reference value (MOS 1), the bit rate allocated to the channel The state of using (BITRATE f) altogether is shown above.

Shown in the middle is the bit rate assigned to the channel if the measured sound quality evaluation result value MOS m is less than the sound quality first reference value MOS 1 and greater than or equal to the sound quality second reference value MOS 2. It divides the bit rate (BITRATE f) to transmit the N-th packet to the bit rate (BITRATE p) and the bit rate (BITRATE r) to transmit the N-1th transmitted packet used as surplus information.

The bit rate “BITRATE f” allocated to the Nth transmitted packet is changed to “BITRATE p”, and a bit rate of “BITRATE r” is allocated for the N−1th transmitted packet to be used as surplus information. That is, BITRATE r = BITRATE f-BITRATE p.

In other words, when a loss occurs in the N-1 th packet, the transmission rate assigned to the channel for transmitting the packet is divided into two, the 1/2 transmission rate is allocated to the N th packet, and the packet is transmitted to the N-1 packet. Since the transmission rate is assigned at the rate of / 2, the transmission loss is concealed.

Shown below is BITRATE, which is one half of the bit rate (BITRATE f) assigned to the channel when the quality evaluation result value (MOS m) of the measured Nth packet is smaller than the quality evaluation second reference value (MOS 2). p is allocated for the Nth packet.

The bit rate (BITRATE r) allocated for the redundant information concealing the lost packet is divided into a first bit rate (BITRATE r1) and a second bit rate (BITRATE r2), and the first bit rate (BITRATE r1) is N-1. The second bit rate (BITRATE r2) is allocated for the N-2 th packet.

That is, BITRATE r = BITRATE r1 + BITRATE r2, and BITRATE f = BITRATE p + BITRATE r.

As described above, without increasing the overall bit rate or transmission rate allocated to the channel, the bit rate of the current N-th packet and the N-1 th packet used as surplus information for concealing a lost packet are varied at the same time. send. In the case where a large loss occurs, the respective bit rates of the N th, N-1 th, and N-2 th packets are changed and transmitted simultaneously, but the N th packet is more than the bit rate allocated to the N th th and N-2 th packets. The state of allocating a large number of bit rates to keep the quality of the audio signal optimal is shown.

Referring to FIG. 7, a state of measuring and feeding back buffering and sound quality evaluation results of packets transmitted and received between devices according to an example of the present invention will be described.

The transmitting side Tx encodes the audio packet in real time and transmits it to the receiving side Rx by applying the RTP protocol. The receiving side Rx decodes the received packets and sequentially writes them in the buffer in the order specified by the RTP protocol.

When the number of packets recorded in the buffer is set and becomes the designated number N, the receiving side Rx measures the MOS which is a sound quality evaluation result value, and feeds back the measured sound quality evaluation result value MOS to the transmitting side Tx. FEEDBACK).

The sender transmits packets of the next order according to the feedback quality evaluation result value (MOS) by applying the RTP protocol, but, if necessary, may transmit the previously transmitted packets together with the variable bit rate by applying the redundant information.

A state in which the above process is repeated every predetermined number N periods is shown.

Experiment example

The present invention as described above is experimented using the multi-party voice collaboration software RAT (ROBUST AUDIO TOOL) and MPEG-2 AAC open source.

Since the MPEG-2 AAC encoding and decoding API (APPLICATION PROGRAM INTERFACE) library is implemented in accordance with the RAT structure and added to the RAT program, audio communication is possible and the present invention is applied to a terminal device that conceals packet loss according to the present invention.

The source of audio input to the transmitter is 32 seconds of female voice, SQAM (SOUND QUALITY ASSESSMENT MATERIAL) stereo (STEREO) signal sampled at 32 KHz, and the encoding bit rate of MPEG-2 AAC is set to 146 Kbit / s.

The down-sampling factor for sound quality evaluation is set to 1/4 to secure the received voice for 8 KHz mono sound quality evaluation.

The measurement period N of the receiving end of the sound quality evaluation result is set to 8 seconds corresponding to 250 packets, and the sound quality evaluation reference value (MOS) for voice communication is set to 3.6 level, and the variable bit rate of the current packet and the previous packet for encoding the surplus information Are set to 1/2 each.

The sound source is prepared in a lossless state and a lossy state, respectively, for the experiment, and the pattern of the lossy sound source is produced with 7% loss using the GILBERT-ELLIOT model defined in ITU-T standard G.191.

As a result, the lost sound source is measured using a conventional transmitting terminal device and a receiving terminal device, and the measurement result is again measured using the transmitting terminal device and the receiving terminal device according to the configuration of the present invention. 8 is shown.

Referring to FIG. 8 again, the experiment was performed four times. In the conventional terminal device to which the present invention is not applied, the measured sound quality evaluation result (MOS) is 2.7 on average when the sound source lost 7% is transmitted. In this terminal device, the average was 3.6.

That is, when the present invention is applied, the sound quality of about 24% compared with the prior art is improved.

Referring to FIG. 9, a transmission terminal operating method by a terminal apparatus concealing packet loss of voice communication according to an example of the present invention will be described. S100).

The input audio signal as described above analyzes the sound quality evaluation result value confirmed to be fed back from the receiving end, thereby checking the set variable bit rate and encoding the audio signal at the determined variable bit rate in real time (S110).

By analyzing the sound quality evaluation result value, only the current packet is encoded or the previous packet is included and encoded together at a variable bit rate by MPEG-2 AAC.

The transmitting end formats the encoded audio packet into the RTP PAYLOAD FORMAT, receives the sound quality evaluation result measured from the receiving end, and includes the feedback in the format (S120). The configuration of the formatted packet is shown in detail in FIG.

The packet formatted as described above is packetized using the RTP protocol and transmitted to the other party (S130).

Referring to FIG. 10, a method of operating a receiver by a terminal device for concealing packet loss of voice communication according to an example of the present invention will be described. In real time, an audio packet is received through an RTP protocol through the Internet (S200).

The received audio packet is analyzed and extracted in real time to extract and confirm the recorded sound quality evaluation result value, and the confirmed sound quality evaluation result value is fed back to the transmitter (S210).

The variable bit rate of the received audio packet is checked by analyzing the value of the confirmed sound quality evaluation result, and the received audio packet is decoded using the determined variable bit rate (S220). The decoding of the audio packet is processed in the audio decoder section.

The decoded audio packet is sequentially recorded in the buffer according to the corresponding sequence number, thereby removing jitter generated while being transmitted through the Internet (S230).

After receiving the audio packet recorded in the buffer, the sound quality evaluation result of the accumulated packets is measured every predetermined number period, and the measured sound quality evaluation result value is fed back to the transmitter (S240). Irrespective of the accumulation of a predetermined number, the audio signal is regularly converted into an audio signal and reproduced and output in real time (S250).

Although the present invention has been described as an example, it is not necessarily limited to such an example, and various modifications can be made without departing from the spirit of the present invention. Therefore, the examples disclosed in the present invention are not intended to limit the technical idea of the present invention but to describe the present invention, and the scope of the technical idea of the present invention is not limited by these examples. The scope of protection of the present invention should be interpreted by the following claims, and all technical ideas within the scope equivalent thereto should be construed as being included in the scope of the present invention.

1 is an overall functional configuration diagram according to an example of the Internet system to which the present invention is applied;

2 is a diagram illustrating a terminal device function concealing packet loss of voice communication according to an example of the present invention;

3 is a view illustrating a detailed field configuration of a general audio packet for explaining the present invention;

4 is a detailed field configuration diagram of an audio packet transmitted by an example of the present invention;

5 is a diagram illustrating a real-time sound quality evaluation method according to an example of the present invention.

FIG. 6 is a diagram illustrating a method of differentially performing surplus information encoding according to a sound quality evaluation result of a transmitter receiving and being fed back according to an example of the present invention;

7 is a diagram illustrating a state of measuring and feeding back buffering and sound quality evaluation results of packets transmitted and received between devices according to an example of the present invention;

8 is a diagram showing results detected by one experiment of the present invention;

9 is a flowchart illustrating a method for operating a transmitting end by a terminal apparatus concealing packet loss of voice communication according to an example of the present invention;

10 is a flowchart illustrating a method of operating a receiving end by a terminal device for concealing packet loss of voice communication according to an example of the present invention.

          ** Explanation of symbols on the main parts of the drawing **

100, 100 ': terminal 110: transmitter

111: sound capture section 112: audio encoding section

113: real time format unit 114: real time packet unit

120: receiver 121: real-time receiver

122: real-time analysis unit 123: audio decoder unit

124: buffer section 125: quality measurement section

126: audio render unit 200: the Internet

Claims (32)

delete In a terminal device for real-time communication with audio signals through the Internet, A transmitter for encoding real-time input audio signals and surplus information at a variable bit rate at which the overall bit rate does not increase, recording a sound quality evaluation result, converting the result into a real-time transmission format, and transmitting the packet to the Internet; And A receiving end receiving a real-time audio packet through the Internet, checking a sound quality evaluation result and feeding it back to the transmitting end, sequentially recording a packet decoded at a variable bit rate, removing jitter, feeding back a sound quality evaluation result, and outputting a reproduced voice signal; The transmitter includes a sound capture unit that samples and inputs an audio signal applied in real time, and encodes the input audio signal into a digital signal and includes surplus information of a previous packet in a state where the total bit rate is not increased. An audio encoding unit for encoding, a real-time format unit for recording a sound quality measurement result fed back to the receiver by converting the audio signal encoded by the audio encoding unit into a real-time transmission format, and an audio signal formatted by the real-time format unit on the Internet in real time And a real time packet unit for converting and transmitting the packet into a transmission packet. The method of claim 2, wherein the receiving end, A real time receiving unit which receives a real time packet transmitted from the opposite terminal device through the Internet; A real-time analyzer for analyzing a packet received by the real-time receiver to check a sound quality evaluation result and feeding back to the transmitter; An audio decoder for analyzing a variable bit rate from a signal applied by the real time analyzer and decoding a received packet; A buffer unit which records the decoded packets in order and removes jitter; A quality measuring unit which sequentially receives the packets stored in the buffer unit and measures a sound quality evaluation result at each predetermined number period, and feeds back a feedback signal to the transmitting end; An audio renderer which outputs an audio signal applied from the quality measurer; Terminal for concealing packet loss of voice communication, characterized in that it comprises a. The method of claim 2, wherein the audio encoding unit, And a terminal for concealing packet loss of voice communication, characterized in that the audio signal and the surplus information are encoded at a variable bit rate without increasing the overall bit rate. The method of claim 4, wherein the variable bit rate, A terminal device for concealing packet loss of voice communication, characterized in that consisting of a configuration that is confirmed by analyzing the sound quality evaluation results confirmed and feedback from the receiving end. The method of claim 4 or 5, wherein the variable bit rate, A terminal device for concealing packet loss of voice communication, characterized in that configured to analyze and confirm the sound quality evaluation result received in the packet from the other terminal device. The method of claim 2, wherein the real-time format unit, And a packet recording apparatus for concealing packet loss of voice communication, characterized in that the receiver is configured to record the measured sound quality evaluation result fed back to the prediction region (MOS) in a payload format. The method of claim 3, wherein the audio decoder, And analyzing the signal of the real-time analyzer to confirm the variable bit rate and to decode the identified variable bit rate. The signal of claim 3, wherein the signal received by the receiver is A terminal apparatus for concealing packet loss of voice communication, characterized in that at least one of the surplus information of the current packet and the previous packet is included. The method of claim 9, wherein the surplus information, Terminal equipment for concealing packet loss of the voice communication, characterized in that the configuration further comprises a surplus information for the previous packet of the previous packet. delete The method of claim 3, wherein the jitter, And a terminal device for concealing packet loss of voice communication, characterized by a configuration consisting of mutations occurring in the Internet. The method of claim 3, wherein the buffer unit, And writing the decoded packet signal into a buffer to remove the jitter. The method of claim 3, wherein the quality measuring unit, And a terminal for concealing packet loss of voice communication, characterized in that the sound quality evaluation result is measured after receiving an audio packet signal of a predetermined number period set from the buffer unit. The method of claim 14, wherein the sound quality evaluation, Terminal equipment for concealing packet loss of voice communication, comprising objective results evaluated using the NON-INTRUSIVE method of the ITU-T standard P563 (P.563) . The method of claim 3 or claim 14, wherein the quality measuring unit, A terminal device for concealing packet loss of voice communication, characterized by expressing the result of the sound quality evaluation in the range of 1 to 5. The method of claim 16, wherein the result value of the sound quality evaluation, A terminal apparatus for concealing packet loss of voice communication, characterized in that the configuration is fed back to the real-time format portion of the transmitter. The method of claim 3, wherein the real-time analysis unit, And a terminal for concealing packet loss of voice communication, characterized in that said feedback is fed back to said transmitting end. The audio rendering unit of claim 3, wherein the audio renderer comprises: And a packet unit for concealing packet loss of voice communication, characterized in that the audio signal of a packet unit applied from the buffer unit is reproduced and output in real time through a sound card. delete In the method of concealing packet loss of voice communication, Inputting and real time capturing an audio signal by a transmitting end of a voice communication terminal device; Encoding the captured audio signal with surplus information by a variable bit rate analyzed as a result of the confirmed sound quality evaluation; Converting the encoded audio signal and the measured sound quality evaluation result into a real-time transmission format; And Converting the audio signal of the real time transmission format into a packet and transmitting it to the Internet; And the encoding process does not encode the surplus information according to the confirmed sound quality evaluation result. The method of claim 21, wherein the encoding process, And encoding one surplus information or two surplus information together without increasing the overall bit rate according to the confirmed sound quality evaluation result. The method of claim 22, wherein the one surplus information, A method for concealing packet loss in voice communication, characterized in that it is information about a previous packet of a packet to be currently encoded. The method of claim 22, wherein the two surplus information, A method for concealing packet loss in voice communication, characterized in that it is information about a previous packet and a previous packet of the current packet to be encoded. The method of claim 21, wherein the measured sound quality evaluation results, A method for concealing packet loss in voice communication, characterized in that the audio quality received from the other party is a result of accumulating the audio packets by a predetermined predetermined number unit. delete In the method of concealing packet loss of voice communication, Receiving and analyzing an audio packet in real time by a receiving end of the voice communication terminal device and feeding back the confirmed sound quality evaluation result to the transmitting end; Decoding the received packet at a variable bit rate analyzed from the sound quality and the result and storing the received packet in a buffer by a sequence number; And Feeding back a sound quality evaluation measured from a packet of a predetermined number unit stored in the buffer to a transmitting end and simultaneously converting the sound quality evaluation into a real time audio signal; And confirming the recorded sound quality evaluation result by analyzing the received packet. The method of claim 27, wherein the variable bit rate, And confirming by analyzing from the confirmed voice evaluation result. The method of claim 27, wherein the sound quality evaluation, A method for concealing packet loss in voice communication, characterized in that the measurement is performed for each period in which packets are accumulated and recorded in predetermined number units set in the buffer. The method of claim 27, wherein the output of the audio signal, And receiving and converting the audio packets sequentially recorded in the buffer regardless of the measurement of the sound quality evaluation. In the method of concealing packet loss of voice communication, Analyzing, by the receiving end of the voice communication terminal device, a packet received from the other party and confirming a sound quality evaluation result; Varying a transmission bit rate according to the analyzed sound quality evaluation result and coding a captured audio signal and redundant information; Accumulating the received packets on a predetermined predetermined unit of time and measuring an objective sound quality evaluation; And Feeding back the measured sound quality evaluation result to a transmitting end to record and transmit in a designated area of a format; Method of concealing packet loss of voice communication comprising a. 32. The method of claim 31, wherein coding the surplus information comprises: The packet loss of the voice communication, characterized in that it is any one selected from the coding of the surplus information, the coding of the surplus information of the previous packet or the coding of the surplus information of the previous packet and the previous packet according to the analyzed sound quality evaluation result. How to conceal.

Images