JP2003218932A - Error concealment apparatus and method - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide a most appropriate error concealment method capable of being dynamically selected during run time at a reception side. <P>SOLUTION: This invention provides the method and the apparatus for enabling dynamic selection and application of various error concealment techniques. A plurality of algorithms for concealing an error are prepared and one of the algorithms is used for concealing the error. A selection signal determines which of the algorithms is to be selected. The selection signal is created according to various parameters indicating computer processing capability and audio signal characteristics. <P>COPYRIGHT: (C)2003,JPO

Description

Detailed Description of the Invention

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention selects a received signal error concealment method and an error concealment method applied to the field of real-time streaming in a packet network in which lost packets are a major cause of failure on the receiving side. Regarding the device.

[0002]

2. Description of the Related Art Heretofore, much research has been extensively conducted on error concealment technology for audio data. Most of these methods can be classified into a method of reconstructing a packet lost on the receiving side, a method of reconstructing a packet lost on the transmitting side, or a method of reconstructing a packet lost on the transmitting / receiving side. They are shown in Non-Patent Documents 1, 2, 3, and 4, for example.

[Non-Patent Document 1] "IEEE" issued in March 1986.
E Transaction on Acoustic
s, Speech and Signal Proces
36 ", No. 3, Vol. 36, No.
O. disclosed in 3). J. Wasem, D.M. J. G
oodman, C.I. A. Dvorak, and H.M. G. P
"The Effect of Wavef"
orm Substitution on the Qua
light of PCM Packet Communic
applications (effect of waveform substitution on quality of PCM packet communication) ”

[Non-Patent Document 2] IEEE N, September / October 1998
Colin Perki disclosed in the network
ns, Orion Hodson, and Vicky Ha
"A Survey of Packe" by rdman
t Loss Recovery Technologies
for Streaming Audio (Survey of packet loss recovery technology for streaming audio) "

[Non-Patent Document 3] R. A Valenzuela and C.I.
N. "A new Voice Pa by Animalu
ccket Reconstruction Techni
que (new voice packet reconstruction technology) "

[Non-Patent Document 4] V. Hardman, A .; Sasse, M .; Ha
ndley, and A. "Relia by Watson
bleAudio for Use over Inter
net (reliable audio used over the Internet) "

[Non-Patent Document 5] 3GPP TS 26094 V4.
0.0 (2001-3)

[Non-Patent Document 6] Petre Pollak, Pave
l Sovka, and Ceps by Jam Uhril
tral Speech / Pause Detector
s

[Non-Patent Document 7] Werner Verhelst and March R disclosed in IEEE of 1993.
"An Overlap Tech" by olands
nice based on Waveform Sim
ilarity (WSOLA) for High Qu
Ability Time-Scale Modicat
ion of Speech (overlap technology based on waveform similarity for high quality timescale correction of speech) "

[0003]

Reconstructing a missing packet on the receiving side is easier than reconstructing the missing packet on the transmitting side, but the frequency of packet loss is low, and Effective only when the packet size is small. The method of reconstructing the packet lost at the other transmission side is redundancy, that is, additional information is added to the data packet, which requires extra bandwidth or end-to-end delay. Becomes longer (see Non-Patent Documents 1 and 2).

In the case of the method of reconstructing a lost packet on the transmitting side, it is required to add a lot of additional information. As shown in Non-Patent Documents 4 and 7, the additional information is added by adding a sequence number to each packet, adding a retransmission request when a packet is lost, or a packet to be retransmitted. Is complicated because it involves the addition of a coding request at a low bit rate.

The method of reconstructing a lost packet on the transmitting and receiving side has the advantage of both the reconstructing of the receiving side base and the reconstructing of the transmitting side base.
The error concealment technique is static and cannot be changed at runtime.

It is an object of the invention to allow the receiving side to dynamically select the most suitable error concealment method at runtime.

[0007]

The gaps that are missing must be filled with the most appropriate alternatives. In a voice packet network, the data to be filled may be in the form of silence or reconstructed data based on estimation and environmental factors. Since different error concealment techniques are effective for different loss patterns, it is better to evaluate that loss pattern and other environmental conditions by adding a "dynamic selector" prior to performing error concealment. Bring an ending.

The present invention maximizes the utility of known error concealment methods by selecting the most appropriate method at the receiver, based on the loss error pattern and the current conditions of network and resource levels. .

The present invention also provides selection criteria that determine which error concealment technique to use. The existing IP protocol on the Internet is a connectionless and best effort protocol that offers no quality of service guarantee. For real-time streaming applications, the best error concealment method can be used to deal with burst loss, but when congestion occurs, or when the receiver resources are suddenly depleted, the present invention provides lower A switch to the calculation method of can be achieved during runtime.

The present invention provides an apparatus for concealing errors using various methods during packet decomposition. In the case of burst loss, various techniques may be used over consecutive loss segments.

The present invention also provides a method of utilizing the characteristics of loss patterns to determine which technique should be used. In voice over IP applications, when a voice signal is detected before its loss, a signal reconstructed based on the information of the neighboring signals is used. If a non-voice signal is detected before the loss, a low power noise segment or silence is used.

To solve the above problems, the invented device uses the current resource level information and lost packet information. The device of the invention comprises a speech detector which detects segments of the good signal which are either speech signals or non-speech signals. Such a voice detector is, for example, one described in Non-Patent Documents 5 and 6.

Error concealment for speech signals is selected from the set of available error concealment algorithms. The selection criteria obtains the current activity status at the receiver and the current available resource level. If the resources available are low, error concealment, which uses very little CPU resources, is used. Different levels of thresholds are set for different algorithms, depending on their complexity.

The non-speech signal has a separate set of algorithms with threshold levels associated with each algorithm.

The present invention includes a selector module that selects the most appropriate technique for concealing errors in a packet network. The selector utilizes the results of other modules such as the "voice detection" module. The voice detection module looks at the voice segment and the non-voice segment of the template data. The template data is the data packet from the previous packet. Different sets of error concealment methods are used to handle speech and non-speech segments. Also, the selector considers the existing environmental conditions and decides whether to use a simpler algorithm when resources are scarce. In order for the error concealment method to work efficiently, analysis is performed in two ways. It is a short-term analysis and a long-term analysis. The short term analysis is based on the error loss based on the audio packets buffered in the selector module. Long-term analysis is done based on statistics collected by the receiver or a trusted network node, along with knowledge of packet loss to send to the receiver.

The first selection device according to the present invention is most suitable for concealing the loss of the digitized audio signal distributed via the packet switching network so that the digital audio can be decoded in real time at the receiving side. A device for selecting a proper error concealment method. This equipment includes a packet audio buffer used to perform packet buffering, a loss pattern detector used to determine the pattern of lost packets received during a particular time frame, and a digitized audio signal. And an error concealment selector that selects a final error concealment algorithm based on the results produced by the packet audio buffer, the loss pattern detector, and the audio signal analyzer. Become.

Preferably, in the selection device, the loss pattern detector has a packet size representing a length (number of bytes, number of bits, etc.) of a digital signal encapsulated by a data packet, a fixed length, or a variable size. The number of consecutive packet losses used to encapsulate a long digital signal in a unit of time and the packet spacing symbolizing the digital signal measured in the time between two packet losses, or substantially this Measure the corresponding signal (number of bits, etc.). Further, the loss pattern detector uses the packet size, the number of consecutive packet losses, and the packet interval or a format corresponding to the signal corresponding to the packet size to detect error packets in an audio or voice stream. Report an outbreak.

A first method according to the present invention is a method for determining characteristics of a lost packet used in the packet audio buffer and the loss pattern detector. This method consists of buffering the previous good packet, which encapsulates a fixed-length or variable-length digitized signal in unit time, and a fixed-length or variable-length digitized signal in unit time. Buffering the next good packet, encapsulated in, and determining the number of consecutive lost packets encapsulating a fixed-length or variable-length digitized signal per unit time; Encapsulating a variable-length digitized signal in a unit of time, determining the pattern of the previous good packet, and encapsulating a fixed-length or variable-length digitized signal in a unit of time. And determining the pattern of subsequent good packets.

The second method according to the present invention is to generate a statistic in a unit time or other specific time (variable time, etc.) to quantify the digital signal loss calculated or measured during any period. It is a pattern description method that provides a description of an error loss pattern by giving it.
The above statistics show that the interval between losses is the time between two digital signal losses, or the distance value expressed in units of packets, and the burst loss is at a unit time or other specific time (variable time, etc.). Between the erroneous digital signal or the maximum distance between the lost digital signals (hereinafter referred to as the "error or lost digital signal") and the error or lost digital signal when defined as the continuous loss of the digital signal From the first example of the last batch consisting of consecutive units of error-free digital signals, to the end of a specific period for error pattern description, to the last of the error or loss digital signals. The interval and the time from the beginning of a particular period for error pattern description to the beginning of the first example of an erroneous digital signal, A first spacing between the over or loss digital signals, and a maximum burst error or loss digital signals, and the average burst error or loss digital signal,
The last burst period of an error or loss digital signal, measured from the last batch of consecutive units of erroneous digital signals to the end of the specified period for error pattern description, and the specified for error pattern description And a first burst period of the error or lost digital signal measured from the beginning of the period to the beginning of the first example of the error-free digital signal.

A third method according to the present invention is a method of selecting an error concealment algorithm. The method comprises the steps of determining the remaining currently available computational resources for performing error concealment, and comparing the current resource with various error concealment algorithms based on error concealment computation time and resource consumption. And determining the characteristics of the digital signal encapsulated in the previous good packet, determining the error pattern of the digital signal received during a predetermined time frame, and said maximum interval, average. Interval, last interval, first interval, maximum burst error or loss digital signal, average burst error or loss digital signal, last burst period, and input collected from the first burst period,
Selecting an error concealment algorithm.

A fourth method according to the present invention is for the receiver to predetermine an error concealment algorithm before receiving a data packet encapsulating a digitized audio signal. The method includes receiving a feedback report about a packet loss rate from an active receiver of a data packet encapsulating a digitized signal, and reporting the occurrence of an error packet based on the pattern description method described above. And, based on the method of measuring, generating packet loss statistics and transmitting the statistics to a receiver that has received or is about to receive a digital signal stream delivered in consecutive data packets. .

In a fifth method according to the present invention, the selector is
It is a method that makes it possible to dynamically select a different error concealment method for burst loss. This method comprises switching the error concealment method from the first method to the second method when the average sample amplitude is smaller than the threshold T, and reducing the amplitude of each successive error concealed error packet. , Audio, or voice, gradually reduce the amplitude of the packet being played,
Due to the muting switching step, which eventually results in a mute effect, and a smoother transition between good and concealed packets, and vice versa, there are multiple boundaries at both the beginning and end of burst loss. A merging step for performing packet merging in the segment.

Preferably, said switching step comprises
Initializing a table containing a list of thresholds, mapping each threshold to an error concealment algorithm, calculating a threshold T based on the error loss pattern and environmental conditions; Comparing the established threshold value T with the values in the list of initialized threshold values, and selecting the algorithm closest to the calculated T value.

Preferably, the muting switching step comprises setting a silence threshold ST, multiplying each sampled signal in a packet by a constant C having a value less than 1, and said sampled signal. On the other hand, it comprises a step of calculating the average power P, a step of comparing the average power P with the silence threshold ST, and a step of switching to the muting mode when the average power P is smaller than the threshold ST.

Preferably, said merging step comprises determining a template window W1 consisting of n sample signals, and n samples of the error concealed signal at the boundary of the burst in said template window W1. It consists of multiplying with a sample, normalizing the resulting signal of the multiplication, and using said normalized signal instead of a segment of the error concealed packet at the boundary. .

A sixth method according to the present invention is a method of merging a boundary at the start and end of burst loss when n samples of the error concealed signal at the boundary are silent. The method comprises the steps of determining a template window W2 consisting of n sample signals, normalizing the n signals in the template window W2 to n signals in a boundary segment of a good packet, and an error at the boundary. Substituting the normalized sample for the segment of the concealed packet.

A packet receiver according to the present invention receives a data packet from a network, detects a loss of the received data packet, and outputs the detection result, and the packet receiving means. Based on the detection result output from the means, the selection device for performing the error concealment process by selecting the most appropriate error concealment method for the lost data packet, and the error concealed digital output from the selection device. And a conversion means for converting the signal into an analog signal and outputting the analog signal.

[0028] Preferably, in the above packet receiver, the packet receiving means comprises an error indicator field for displaying error information and a sequence number field for indicating a consecutive sequence number of each packet received. Add a header to the received data packet and output.

Preferably, the selecting means is a packet comprising the packet header and a payload consisting of one packet size data following the packet header, wherein the error indicator field is
A packet that indicates “no loss” and has the correct sequence number field is determined to be a good packet.

Preferably, the selecting means is a packet comprising the packet header, and the packet head is not followed by payload data, and the error indicator field indicates "loss" information, A packet whose sequence number field is in the correct order is determined to be a lost packet.

A seventh method according to the present invention is a method of determining the number of consecutive burst losses. The method comprises the steps of obtaining the header of each packet and examining the error indicator field, recording the sequence number x when the start of burst loss is encountered, and recording the sequence number when the end of burst loss is encountered. It consists of recording the number y and calculating the number of consecutive packet burst losses as (y−x) +1 packets.

Preferably, the method of detecting the onset of burst loss comprises the steps of detecting that the current packet is a lost packet and detecting that the previous packet is a good packet.

Preferably, the method for detecting the end of burst loss comprises the steps of detecting that the current packet is a lost packet and detecting that the following packet is a good packet.

An eighth method according to the present invention is based on the second method according to the present invention, which is an audio packet receiver or other reliable device capable of collecting network statistics and generating an error pattern format. Using error statistics generated in network nodes,
Select an audio error concealment algorithm without storing long-term audio data for analysis.

[0035]

BEST MODE FOR CARRYING OUT THE INVENTION Referring to the accompanying drawings,
An embodiment of the present invention will be described. The error concealment device shown in FIG. 1 includes a missing packet detector 1, a packet audio buffer 2, a loss pattern detector 3, and a multiplexer 4. The multiplexer 4 also has a function as a digital audio payload merger. Further, it has a signal analyzer 7 including a voice detector 5 and an energy calculator 6. It has an error concealment selector 8 and an algorithm holder 9. These elements 1 to 9 are controlled by the computer 10.

Next, the operation of the error concealment device will be described. The audio signal (payload) shown in FIG. 3A is carried in the packet 1. The voice signal is a voice signal for telephone communication, for example. The voice signal is signal-processed by pulse code modulation (PCM) and divided into a plurality of packets for transmission. In the example shown in FIG. 3A, an example in which the packet is divided into 14 packets for transmission is shown. In addition, in this example, an example in which packets 003 to 010 of 14 packets are lost is shown. Consider a case where a voice signal in which packets are partially lost is input to the lost packet detector 1 in this way.

The missing packet detector 1 detects the sequence number of a packet and determines whether or not the sequence number is a serial number, thereby determining whether or not there is a missing packet. For a valid packet that is not missing, a flag of 0 is added to the head of the packet and output. If a packet loss is found, a compensation packet is created, a sequence number is given to the compensation packet, and a flag 1 indicating that it is a compensation packet is set at the beginning and output.

The header of the packet output from the missing packet detector 1 is composed of an "error indicator" field and a "sequence number" field, as shown in FIGS. 2 (a) and 2 (b). A valid packet and a packet that has not been lost are composed of a header and a payload. Its payload is, as shown in FIG.
It consists of a PCM signal 23. FIG. 2 (b) shows a supplementary packet consisting of only a header without payload. The error indicator flag shows 0 if there is no error and 1 if it is a missing packet. The sequence number field indicates a serial sequence number including a missing packet. FIG. 3B shows an output signal from the missing packet detector 1, that is, a signal to which a compensation packet is added.

The packet audio buffer 2 receives the signal of FIG. 3B from the missing packet detector 1, temporarily holds the signal, delays it, and distributes the packet based on the flag. This delay corresponds, for example, to the size of the PCM signal of the packet arriving at the receiver. The valid packet to which the flag of 0 is added is directly output from the packet audio buffer 2 and output to the multiplexer 4, and also to the algorithm holder 9. The flag may be omitted in the output of the packet audio buffer 2.

As for the output from the packet audio buffer 2 to the loss pattern detector 3, only the header portion (including at least a flag and a sequence number) of each packet is output. Therefore, the loss pattern detector 3 receives the flag and the sequence number corresponding thereto. Figure 3
In the case of the signal of (b), the loss pattern detector 3 receives the flag pattern signal of 00111111110000, which is the flag pattern, and also receives the sequence number (001 to 014 in the case shown) for each flag. Further, only the PCM data part of the packet with the header removed is sent from the packet audio buffer 2 to the signal analyzer 7. The header may remain attached.

The signal analyzer 7 receives the flag pattern signal from the loss pattern detector 3 and takes in the PCM data contained in the valid packet to which the flag 0 belongs when the flag changes from 0 to 1 in the flag pattern. Figure 3
In the example of (b), the signal PCM2 is captured. The PCM signal taken into the signal analyzer 7 is analyzed as a template signal. The template signal may be a short signal generated by the PCM signal of one valid packet, or the P signal of a plurality of valid packets.
It may be a long one made of a CM signal. In the voice detector 5, the received template signal (signal PCM
2) determines whether it is a voice signal or a non-voice signal. That is, it is determined whether the voice signal is a telephone conversation signal or another signal. This determination can also be performed by the method shown in Non-Patent Document 6, for example. Therefore, the voice detector 5 outputs a signal indicating a voice signal or a non-voice signal. The energy calculator 6
The energy of the PCM signal, that is, the amplitude of the signal obtained by demodulating the PCM signal is generated, and a signal having a value indicating the average amplitude W thereof is output as energy information.

The algorithm holder 9 holds programs of various algorithms for producing a substitute PCM signal used for the compensation packet. 4 in FIG.
Two algorithms are shown by way of example. Figure 4 (a)
The algorithm 1 shown in 1 is an algorithm that uses the PCM signal of the valid packet immediately before the dropped packet and creates the PCM signal of the dropped packet. That is, all or part of the audio signal demodulated from the PCM signal of the valid packet is specified as the window W0, and the specified window covers the part of the window W0 and the audio signal part of the next missing packet. In this way, the expanded audio signal is modulated again into the PCM signal, and a supplement packet is created. This algorithm is called pitch waveform substitution.

Algorithm 2 shown in FIG. 4B is an algorithm for making a supplementary packet by directly copying the PCM signal of the valid packet immediately before the missing packet. This algorithm is called packet repetition. Figure 4
Algorithm 3 shown in (c) is the PC of the missing packet.
This is an algorithm for substituting the M signal with a PCM signal corresponding to a white noise signal to form a compensation packet. This algorithm is called noise substitution. Algorithm 4 shown in FIG. 4 (d) is an algorithm that substitutes the PCM signal of the missing packet with a PCM signal corresponding to a silent signal to create a compensation packet. This algorithm is called silence substitution.

As described above, a plurality of algorithms are prepared in the algorithm holder 9, and each algorithm is
An algorithm for preparing a PCM signal of a missing packet and a complementary packet is prepared. Which of the plurality of algorithms prepared in this way is selected is specified by the selection signal from the error concealment selector 8. Next, the error concealment selector 8 will be described.

The error concealment selector 8 receives the flag pattern signal from the loss pattern detector 3, the voice / non-voice signal and the energy information from the signal analyzer 7, and the information from the computer 10 indicating the degree of use of the computer. Receive some environmental information. As shown in Table 3, the CPU usage rate of the computer is indicated by the parameter t1. When the CPU usage rate is 70% or more, the parameter is set to 0, when the usage rate is 50% to 70%, the parameter is set to 0.15, and when the usage rate is 30% to 50%, the parameter is set to 0. 3. If the usage rate is 30% or less, the parameter is set to 0.5. Further, as shown in Table 4, the parameter t2 is set to 0 when the usage rate of the computer memory is 70% or more, and is set to 0.15 when the usage rate is 50% to 70%. Is 30% to 50%
Is set to 0.3, and when the usage rate is 30% or less, it is set to 0.4. The parameters t1 and t2 set in this way are added to set a margin value T.

The error concealment selector 8 judges from the signal from the voice detector 5 whether the PCM signal of the compensation packet to be created from now should be a voice signal or a non-voice signal. That is, if the valid packet immediately before the dropped packet is a voice signal, P as a voice signal is used.
A CM signal is created, and if it is a non-voice signal, a PCM signal as a non-voice signal is created.

Table 1 shows a selection for determining whether the pitch waveform should be replaced or the packet should be repeated when the voice signal is determined. That is, if the margin value T is 0.4 or more, a signal for selecting the pitch waveform substitution which is the algorithm 1 is output. When the margin value T is 0 to 0.4, the packet is repeated,
That is, a signal for selecting Algorithm 2 is output.

Table 2 shows the selection for determining whether to substitute the noise or the silence when the non-voice signal is determined. When the average power W is 0.3 or more, a selection signal for selecting Algorithm 3 which is a substitute for noise is output, and the average power W is 0 to 0.
If it is 0.3, a selection signal for selecting silence substitution, which is Algorithm 4, is output.

The algorithm selection signal selected as described above is output to the algorithm holder 9 to select an algorithm. A PCM signal of the supplement packet is created based on the selected algorithm, a header including a sequence number is further added, and the PCM signal is output to the multiplexer 4. Therefore, from the algorithm holder 9, FIG.
The created supplement packet of (d) and the valid packet of FIG. 3 (c) are sent from the packet audio buffer 2 to the multiplexer 4. The multiplexer 4 arranges the valid packet having no error and the generated supplementary packet in the order of the sequence number, and outputs them to the digital analog conversion unit. Note that when the number of selected algorithms is small, for example, when there are two, the voice detector 5
The selection may be performed only with the signal from, or the selection may be performed only with the signal from the energy calculator 6, or the selection may be performed only with the margin value T.

FIG. 5 is a flow chart for explaining an error concealment method selection process. First, the header is obtained from the packet in the packet audio buffer 2 (step S1). Then, it is checked whether or not a new packet exists in the packet audio buffer 2 (step S
2). If there is no new packet in the buffer 2, the selection process ends. If not, the flag of the error indicator field is checked (step S3). If the flag of the error indicator field is equal to 1, the current packet is recognized as a compensation packet (Fig. 2 (b)). If the value is 0, it is recognized as a valid packet (FIG. 2 (a)).

(Case 1: When receiving a compensation packet)
Error concealment is performed. The first information about the current resource usage is checked. The current CPU usage and memory usage are calculated (step S5). Then, the number of consecutive supplementary packets is counted (step S
6). This is done in the loss pattern detector 3. The method for obtaining this number of supplementary packets is described in the flowchart of FIG. First, the start of the compensation packet is detected (step S21). This is done by inspecting the error indicator field of the packet
This is realized by detecting that the previous packet is a valid packet and that the current packet is a supplementary packet. When the start of the supplement packet is detected, the sequence number x of the supplement packet is recorded (step S22). Then, the end of the compensation packet is detected (step S23). This is accomplished by inspecting the error indicator field to detect that the current packet is a supplemental packet and then the next packet to be a valid packet. When the end of the compensation packet is detected, the sequence number y of the compensation packet is recorded (step S2).
4). Then, the number of consecutive compensation packets is (y−
x) +1 pieces are calculated (step S25). Figure 5
See again. In step S6, consecutive compensation packets are processed. Each successive fill packet is processed separately. That is, the error concealment technique used is
It may be different between packets. In step S7, it is checked whether all the compensation packets have been processed. In step S8 it is checked whether all "n" packets have been processed. First, in the consecutive supplementary packets, the first n supplementary packets are processed (step S9). “N” is calculated based on the number of consecutive supplementary packets. Details of processing the first "n" packets are described in the flowchart of FIG. Further, after "n" packets have been processed, the remaining (here, p) packets are processed (step S10). Details of the processing of the p packets will be described in the flowchart of FIG.

(Case 2: When a valid packet is received)
When a valid packet is received, the data segment 23 shown in FIG. 2A is buffered and sent to the multiplexer 4 (step S4). In addition, this valid packet is used by the signal analyzer 7 as a template signal when a supplementary packet is detected next.

The processing of the first n supplementary packets is shown in FIG.
Are described in. First, the valid packets immediately before the n supplementary packets are sent to the voice detector 5 and stored as a template signal (step S31). Further, the voice detector 5 detects whether the PCM signal of the valid packet is a voice signal or a non-voice signal (step S3).
2). If the valid packet is a voice signal, the CPU
The margin value T is calculated according to the current environmental conditions such as use and memory availability. This margin value T is compared with a threshold value T1 of a list of predetermined error concealment techniques (Table 1) for processing a voice signal (step S35). It is determined to which range in Table 1 the calculated margin value T belongs, and an error concealment method is selected (step S36).
It is executed based on the selected algorithm. In step S32, if the valid packet is a non-voice signal, the average power W is calculated for all signals (step S33). The error concealment method is selected from the non-voice list (Table 2) based on the average power value (step S34). That is, the average power W is compared with the threshold value T2. If the average power W is small (0 in Table 2)
If it is considered negligible (if the value is 0.3), silence substitution is used. If the average power W is the threshold T
If it exceeds 2 (0.3 in Table 2), a noise substitute with an average power similar to white noise is generated,
That is an alternative to the compensation packet.

If the supplement packet lasts for a long time, the first n
The p packets (pluralities) that are replaced after the replacement of p packets gradually decrease in amplitude and eventually go to the mute effect.

The case where there are four error concealment methods that can be selected by the dynamic selector will be described. The four error concealment methods are two from the voice list and two from the non-voice list.
One is selected. The audio list consists of the following alternatives:

[Table 1] The non-speech list consists of the following alternatives:

[Table 2]

When the compensation packets are continuously received (inspected in step S6 of FIG. 5), for example, as shown in FIG.
As shown in, when eight consecutive compensation packets are received, n is

[Equation 1] Is set to.

The first n packets, ie the three supplementary packets, are processed in the manner shown in FIG. The valid packet before the three supplementary packets is held in the signal analyzer 7 as a template signal. This template signal is first sent to the voice detector 5 in order to check whether it is a voice signal or a non-voice signal (S3).
1). In this example, the template signal is a voice signal, and the voice detector 5 returns a voice flag (S3).
2). Then, the current resource usage is checked. Margin value T
Is calculated based on two reference values. They are C
The PU usage rate CPC and the memory usage rate MEC. If the CPU usage exceeds 70%, the margin value t1
Is 0. If the memory usage exceeds 70% of the total memory allocated, the margin value t2 is zero. The margin value is calculated as follows.

[Table 3]

[Table 4]

The calculation of the margin value T is expressed by the following equation (2).

[Equation 2]

If the current CPU utilization is 40% and the memory utilization is 50%, the margin value T is equal to 0.45. Since the previous valid packet is a voice signal, the allowance value T is compared with T1 (S35). If it is a non-voice signal, the average power W is compared with T2. In this case, the margin value T may be compared with T2 instead of the average power W. The selection criterion is that T is T1 or T2
, The error concealment method with the closest value is selected. In this case, since the margin value T is 0.45, the pitch waveform alternative is selected (S36). Therefore, the waveforms of the first three compensation packets are hidden using the pitch waveform substitution method.

Subsequent alternatives are based on the algorithm shown in FIG. The value of "X-n" shown in step S41 is (8-3) = 5 in this example. That is, X represents the number of consecutive supplementary packets, and n represents the number of supplementary packets that are positioned at the head and that repeat the same processing collectively. Table 5 is a table showing the relationship between the error concealment method selected previously and the error concealment method to be selected next. If the error concealment method of the previous compensation packet is the pitch waveform substitution, the error concealment method of the next compensation packet is C times (C <1) packet repetition. If the error concealment method for the previous compensation packet is packet repetition, the error concealment method for the next compensation packet is C times (C <1) packet repetition. If the error concealment method of the pre-compensation packet is noise substitution,
The error concealment method for the next compensation packet is also noise substitution. If the error concealment method of the previous compensation packet is silent substitution, the error concealment method of the next compensation packet is also silent substitution. In the above example, since the previous compensation packet was a pitch waveform substitute, packet repetition is performed next, and packet repetition is performed next. Its amplitude is gradually reduced and packet repetition is used continuously. Then, error concealment is repeated 5 times (step S
43). The packet replacement is executed for each packet until all the remaining five packets are hidden (step S44). After the fifth packet is concealed, the packet processing ends.

[Table 5]

In the case of successive substitutions as described above, the power of the substitutions is gradually reduced and when the average power falls below the silence threshold ST, all subsequent substitutions are replaced by the silence substitutions. FIG. 10 is a flowchart showing the muting switching process. In the muting switching process, for example, a step of setting a silence threshold ST and a step of multiplying each sample signal in the packet by a constant C having a value smaller than 1 (S51).
Then, for the sample signal obtained in step S51,
When the average power P of the packet or a part of the packet is calculated (S52), the average power P is compared with the silence threshold ST (S53), and when the average power P is smaller than the threshold ST, And a step (S54) of switching to the muting mode.

Merging may be performed at the boundary between the valid packet and the supplement packet to ensure a smooth transition. If the average power difference at the boundary is less than the predetermined value Y, no merging is required. If not, merging is needed. This is done in the multiplexer 4.

The merging process will be described below with reference to FIGS. 11 and 12. FIG. 11 is a flowchart illustrating a merging / merging process in the case where the error-concealed packet is not the silent substitute. The predetermined template window W consists of n samples of the signal. The end of consecutive valid packets is captured by the window We. The leading end of consecutive supplementary packets following this is captured in the window Ws (step S6).
1). Therefore, the windows We and Ws specify the boundary between the valid packet and the supplement packet. The average power in the window We is calculated. The power in the window Ws is normalized by this average power (step S6).
2). The sample signal of the window Ws is replaced so that the power change in the windows We and Ws changes smoothly (step S63).

FIG. 12 is a flow chart for explaining the merging / merging processing when the error concealed packet is a substitute for silence. Template window W2
Is defined only for the use of silence substitution. The template window W2 is normalized to the value of the average power of the boundary segment of the good packet (step S71). The normalized sample is then directly replaced for the boundary segment of the error concealed packet (step S72).

The selection process can be executed based on the information generated from the network performance statistics. The statistics are distributed over the network in a format similar to the Pattern_Loss structure by network nodes capable of generating such information. Alternatively, the information is generated based on packet audio loss patterns received by the packet audio buffer over time. This selection process is based on a long-term analysis, which is sometimes defined as the time interval of an error loss pattern that exceeds the time interval of the audio buffered in the packet audio buffer. Regarding error concealment over a long period of time, in order to perform error concealment on audio data received in the packet audio buffer 2 of FIG. 1, Pattern_L
Data elements in the oss structure can be used. That is, in the above, the short-term analysis (analysis that selects only the concealment method by looking at only the continuous packets that enter the packet audio buffer 2) has been described. Below, long-term analysis (packet audio buffer 2
The analysis of selecting the concealment method using the result of continuing checking the incoming packet for a predetermined long period (buffer frame period shown in FIG. 13) will be described. In the long-term analysis, the value of information, which will be described later, is updated every time the buffer frame period passes. The pattern loss detector 3 performs short-term analysis or long-term analysis.

In an audio broadcasting environment, these Patterns are used because error concealment is performed in the same way.
Statistics in the n_Loss structure type allow error concealment to be performed in an integrated manner. Although the receivers may start receiving the same audio broadcast at different times, the algorithms selected and executed may be the same. The pattern loss detector 3 may be switched off if the receiver has no computing power. The choice of error concealment by long-term analysis is limited to the limited loss pattern observed in the packet audio buffer (ie the flag pattern described above) and the statistical average of the loss patterns generated by the pattern loss detector 3 of FIG. (Information from a later description). The algorithm may be selected using both the long-term analysis result and the short-term analysis result, or the algorithm may be selected using either the long-term analysis result or the short-term analysis result. Good.

The error statistics generated by the pattern loss detector 3 are in the format as shown in the following structure measured during the buffer frame period. The temporal loss pattern, and the buffer frame period used in determining its duration, is defined as a unit of time or a block of digital audio signal composed of many digitized audio data packets. Note that it is more effective to set the buffer frame period longer than the packet period held in the packet audio buffer.

Pattern_Loss {Delay; / ^{* unit of} seconds or packet payload size (byte / bit) ^* / (delay 52 representing the size of the PCM signal of one packet on the time axis) Mean Loss Interval; / ^* Average time interval between packet losses ^* / (average of L_I) Max Loss Interval; / ^* Maximum time interval between two packet losses ^* / (L_I (max) 5
4) First Loss Interval; / ^* Time interval from the beginning of the buffer frame until the first packet loss is measured ^* / (L_I (p) 53) Last Loss Interval; / ^* The first group of the last group of error-free packets Time interval measured from the first packet to the end of the buffer frame ^*
/ (TLLI) Mean Burst Interval; / ^* Average time interval between two error / lossless packets ^* /
(Average of B_L) Max Burst Interval; / ^* Maximum time interval between two error / lossless packets ^* /
(B_L (max)) First Burst Interval; / ^* first group of consecutive errored / lost packets measured from the beginning of the buffer frame to the end of the last packet of the group of erroneous packets ^* / (FB
L) Last Burst Interval; / ^{* The} time interval measured from the first packet of the last group of erroneous packets to the end of the buffer frame.
^* / (LBI)}

FIG. 13 shows the above-mentioned Pattern_Los.
The method used to determine and describe the error pattern is described by generating statistics, as shown in the s structure. An example 51 of a buffer frame consisting of audio data packets of arbitrary length for which an error pattern is determined is given. The packet length in this embodiment has a fixed size and is determined by the parameter delay 52. Mean Loss In
tval) is the average of all loss intervals measured in the buffer frame. In this example, Mean Loss Interval = {L_I (m
ax) + L_I + L_I (p)} / total interval of error-free packets;

Average Burst Interval (Mean Burst)
Interval) is the average of all burst intervals measured in the buffer frame. In this example, Mean Burst Interval = {B_L
(Max) + B_L + B_L (n)} / total interval of erroneous packets;

The variable L_I (p) 53 is a Pattern
_ Loss structure First Loss Interval
Gives the value of the l element. The variable L_I (max) 54 is P
Max Loss Int of altern_Loss structure
Gives the value of the interval element. The variable L_I55 is one of the data values used to measure the loss interval within the buffer frame. The Last Burst Interval element of the Pattern_Loss structure is equal to the variable B_L (n) 56. The variable B_L57 is one of the burst error values measured in that buffer frame. The variable B_L (max) 58 is the Pattern_Los of the buffer frame shown as an example in FIG.
It is equivalent to the Max Burst Interval element of the s structure.

Table 6 below shows Pattern_Loss.
Much about how the error loss statistics obtained in the structure can be used by the error concealment selector to make a better decision on the choice of the error concealment algorithm that produces an optimal error concealed error-free audio stream. The following is an example. Error concealment is applied to audio packets stored in buffer 2 of FIG. The buffer 2 can in some cases only store audio signals of short duration. The specific time intervals of the audio signal and the lost time intervals shown here do not have a numerical value, but instead,
Use high and low display for relative comparison. For a detailed implementation, it is necessary to know the packet size and sampling rate of the audio signal and the type of error concealment algorithm available.

[Table 6]

The above-mentioned Pattern_Loss structure is an example of statistics used when describing an error loss pattern, and is not limited to this. Further, the buffer frame period, the interval between packet losses, and the like can be expressed not in time but in the number of bits.

FIG. 14 is a block diagram of a voice packet receiver having an error concealment selector module according to the present invention. The receiver 70 in FIG. 14 includes a voice packet reception unit 71, a voice packet decoding unit 72, an error concealment selector module unit 73, and a voice output unit 74. The voice packet receiver 71 receives a voice packet from the network and detects whether or not the received voice packet is missing. When the loss of the voice packet is detected, the detected information is output to the voice packet decoding unit 72. The voice packet decoding unit 72 includes the voice packet receiving unit 7
When the voice packet output from 1 is encoded voice, decoding is performed and PCM voice is generated. Also,
PC required by the error concealment selector module unit 73
Generate M packets. The error concealment selector module unit 73 corresponds to the block diagram shown in FIG. Further, the voice packet decoding unit 72 receives the packet loss information detected by the voice packet receiving unit 71 and outputs the information to the error concealment selector module unit 73. Error concealment selector module unit 7
3 determines whether the packet input from the voice packet decoding unit 72 is a good packet or a lost packet, and performs processing according to the result. The error concealment selector module unit 73 includes a packet header added by the voice packet receiving unit 71 and a payload following the packet header, and an error indicator field indicates "no loss" information, and a sequence of a sequence number field. A packet with the correct number sequence is determined as a good packet. Also, the error concealment selector module unit 73
In the packet header added by the voice packet receiving unit 71, a packet is determined as a lost packet if the error indicator field indicates “loss” information and the sequence number in the sequence number field is correct. The voice output unit 74 is processed by the error concealment selector module unit 73, converts the output digital voice into analog voice, and outputs the analog voice to a speaker or the like. The voice packet decoding unit 72 can be omitted when the voice packet output from the voice packet receiving unit 71 is unencoded PCM voice.

[0075]

The present invention allows different combinations of error concealment methods to be used in some cases. The most suitable option can be selected dynamically at runtime. When resources are low, a low complexity error concealment algorithm can be used, and if sufficient resources are detected, a more effective but more complex algorithm can be selected. The choice is from a set of predetermined algorithms with different properties.
Error concealment for a voice segment can be selected from a list of methods and, for non-voice segments, another method list. In the case of burst loss, where many consecutive packets are dropped, this selector allows various methods of concealing the error to handle the burst loss. The present invention also provides a complex audio concealment by an audio receiver with a small audio buffer size and low latency by using error statistics that measure the error pattern of packetized audio streams. To enable that. Based on the collected error statistics, the error concealment selector can make a better decision on the type of error concealment used for a particular audio lost frame, resulting in better error concealed audio output.

[Brief description of drawings]

FIG. 1 is a block diagram of a dynamic error concealment selector.

FIG. 2 (a) is a diagram showing fields (head and data) in a valid packet, and (b) is a field in a lost packet (no data, only header).
FIG.

FIG. 3 is a diagram showing signals at main points of the apparatus shown in FIG.

FIG. 4 is an explanatory diagram showing various algorithms.

FIG. 5 is a flowchart of error concealment selection processing.

FIG. 6 is a flow chart illustrating a method of obtaining the number of burst losses.

FIG. 7 is a flowchart of a selection process for processing the first n consecutive packets.

FIG. 8 is a flowchart of a selection process for processing the subsequent p consecutive packets.

FIG. 9 shows a part of a stream with 8 consecutive dropped packets.

FIG. 10 is a diagram showing a flowchart of a muting switching process.

FIG. 11 is a flowchart illustrating a merging process in the case where an error-concealed packet is not a silent substitute.

FIG. 12 is a flowchart illustrating a merging process in the case where an error-concealed packet is a substitute for silence.

FIG. 13 shows that the error pattern detected in the audio packet buffer is transformed into mathematical statistics used for error concealment.

FIG. 14 is a block diagram of a voice packet receiver including an error concealment selector module according to the present invention.

[Explanation of symbols]

2 packet audio buffer 3 Loss pattern detector 4 multiplexer 5 Voice detector 6 energy calculator 7 Signal analyzer 8 Error concealment selector 9 Algorithm holder

   ─────────────────────────────────────────────────── ─── Continued front page    (72) Inventor Baek Tan             Singapore 534415 Singapore, Thailand             Sen Avenue, Block 1022, 04-             No. 3530, Thai Sen Industrial             Estate, Panasonic Singapore             Research Institute Co., Ltd. (72) Inventor Masaki Sato             1006 Kadoma, Kadoma-shi, Osaka Matsushita Electric             Sangyo Co., Ltd. F-term (reference) 5K030 GA12 HA08 HB01 HB16 JT01                       LA06 LC02 MB13

Claims (16)

[Claims] 1. An error concealment device for detecting a packet loss of an audio signal composed of a plurality of packets and for concealing the lost packet, wherein the error concealment device generates a compensation packet, which is lost in a received valid packet sequence. Missing packet detection means (1) for detecting existing packets, extraction means (2, 3) having a buffer for extracting valid packets existing before the lost packets, and retained in the extracted valid packets Analyzing means (7) for analyzing a voice signal being generated, an algorithm holding means (9) for holding a plurality of algorithms and generating a compensation packet using any one of the plurality of algorithms, and the analyzing means. The selection means (8) for determining which of a plurality of algorithms is to be selected using the analysis result of the An error concealment device comprising: a multiplexer (4) for inserting a compensation packet at the position of the dropped packet. 2. A CPU (10) for further controlling operation.
And a memory (10) cooperating with the CPU, and a margin value measuring means (10) for measuring a margin value indicating a margin degree of the CPU and the memory. 2. The error concealment device according to claim 1, wherein which of the algorithms is selected is determined. 3. The error according to claim 1, wherein the analysis means (7) has a voice detector (5) for determining whether the voice signal contains voice or is in a silent state. Concealment device. 4. The error concealment device according to claim 1, characterized in that said analysis means (7) comprises an energy calculator (6) for calculating the energy of the speech signal. 5. The missing packet detecting means (1) adds a flag for identifying a valid packet or a missing packet to all the packets to be received. Error concealment device. 6. The error concealment device according to claim 5, wherein the extraction means extracts a flag and generates a flag pattern in which the flags are arranged. 7. The error concealment apparatus according to claim 6, wherein the selection means analyzes a flag pattern and determines which of a plurality of algorithms to select based on the analysis result of the flag pattern. . 8. The first algorithm, wherein the plurality of algorithms modify a voice signal of a valid packet immediately before a lost packet to produce a voice signal of the lost packet, and a valid packet immediately before the lost packet. The second algorithm that copies the voice signal of the above as it is and produces the voice signal of the missing packet, the third algorithm that produces the voice signal of the lost packet using white noise, and the third algorithm that uses the silent signal The error concealment device according to claim 1, further comprising a fourth algorithm for generating a voice signal of a packet. 9. An error concealment device for generating a compensation packet for detecting a packet loss of an audio signal composed of a plurality of packets and concealing the dropped packet, wherein the error concealment device receives a missing packet in a valid packet sequence. A valid packet existing before the missing packet, a step of analyzing a voice signal held in the extracted valid packet, a plurality of algorithms are held, A step of generating a supplementary packet using any one of a plurality of algorithms, a step of determining which of a plurality of algorithms is selected using the analysis result, And a step of inserting a compensation packet at a position. 10. A CPU (1) for further controlling operation.
0) and a margin value indicating a margin degree of the memory (10) that cooperates with the CPU, and which of the plurality of algorithms is selected is also determined by using the margin value. The error concealment method according to claim 9. 11. The error concealing method according to claim 9, wherein the analyzing step includes a step of determining whether the audio signal contains a voice or is in a silent state. 12. The error concealment method according to claim 9, wherein the analyzing step includes a step of calculating energy of a voice signal. 13. The error concealment method according to claim 9, further comprising the step of adding a flag for identifying whether each packet is a valid packet or a missing packet to all packets to be received. 14. The error concealment method according to claim 13, further comprising the step of extracting a flag and generating a flag pattern in which the flags are arranged. 15. The error concealment method according to claim 14, further comprising the step of analyzing a flag pattern and determining which of a plurality of algorithms is to be selected based on the analysis result of the flag pattern. 16. The first algorithm corrects a voice signal of a valid packet immediately before a lost packet to generate a voice signal of the lost packet, and a valid packet immediately before the lost packet. The second algorithm that copies the voice signal of the above as it is and produces the voice signal of the missing packet, the third algorithm that produces the voice signal of the lost packet using white noise, and the third algorithm that uses the silent signal 10. The error concealment method according to claim 9, further comprising a fourth algorithm for generating a voice signal of a packet.