KR101495879B1 - A apparatus for producing spatial audio in real-time, and a system for playing spatial audio with the apparatus in real-time - Google Patents

Abstract

The present invention relates to an apparatus for receiving music respectively performed by players on different spaces through an IP network in real time at a receiving end, mixing the music into one of the natural music in a multi-channel method, and generating the mixed music into a stereophonic sound, and a stereophonic sound reproduction system including the same. The apparatus and the stereophonic sound reproduction system restore music or audio packets with a minimum delay by applying a signal processing method, such as adaptive reproduction scheduling and packet loss hiding and merging, to an audio stream corresponding to the music received through the IP network, mix audio streams respectively transmitted from a plurality of players on independent spaces in a multi-channel sound signal processing method such that the audio streams can be one of the natural music, and create the mixed audio streams into a stereophonic sound through an arrangement of a plurality of speakers. The stereophonic sound generation apparatus includes a stereophonic sound generator.

Description

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a stereophonic real-time generation apparatus and a stereophonic real-

The present invention relates to a device for real-time generation of stereophonic sound based on an IP network, and a system for real-time reproducing the stereophonic sound. More particularly, the present invention relates to a method and apparatus for mixing music played by each performer in different independent spaces in a multi-channel manner, And a system for reproducing the stereophonic sound.

Stereophonic refers to sound with spatial information added to the listener's perception of direction, space, and distance when a listener in a space where no sound source is heard hears the sound.

Standardization of stereophony is underway in MPEG and ITU-R. MPEG-4 ALS (Audio Lossless Coding), MPEG-4 SLS (Scalable Lossless Coding), MPEG-4 IMAF (Interactive Music Application) Format, and MPEG-D Spatial Audio Object Coding (SAOC). In recent years, stereo sound technology combining MPEG Surround and MPEG SAOC has been intensively researched and developed. In addition, sound localization technology, sound field control technology, crosstalk control technology, binaural and ambsonic recording technology are being studied, and research on stereophonic output using virtual speaker is also being carried out.

However, the above-described techniques are performed by inputting a plurality of players or objects in the same space to produce a stereophonic sound. The generated stereophonic sound is transmitted together with the audio signal processed with the metadata, The technique is a technique to combine data to generate a stereophonic sound suited to the receiving end environment. The technique is different from a method in which each player transmits music played in an independent space through an IP network at a receiving end to generate stereophonic sound.

On the other hand, when audio data is transmitted through an IP-based packet network, since each packet is transmitted through different paths rather than through the same path, the time at which each packet arrives at the receiving end is not constant. That is, packets may be lost due to a network load, packets may not be received within a predetermined time, or packets may be received in a changed order, resulting in degraded sound quality.

Korean Patent Laid-Open Publication No. 2007-0033860 proposes a stereo sound generating apparatus for reproducing a multi-channel sound input signal with a two-channel speaker. However, this apparatus can not solve the above-described problem because it can not mix sound signals generated in different spaces into a channel signal.

The present invention has been conceived to solve the problems described above, and it is an object of the present invention to provide an apparatus and a method for generating stereophony in real time by mixing audio information obtained in different independent spaces on the basis of network jitter of each channel and generation time of each audio information And a stereoscopic real-time reproduction system having the same.

However, the objects of the present invention are not limited to those mentioned above, and other objects not mentioned can be clearly understood by those skilled in the art from the following description.

According to an aspect of the present invention, there is provided a method for transmitting audio information on a specific channel, the method comprising the steps of: receiving audio information on a specific channel; A jitter estimator for estimating a network jitter; A signal processor for selectively processing the currently received audio information based on the network jitter; And extracts audio information generated at the same time based on the generation time of each audio information among signal processed audio information received from different channels and mixes the extracted audio information to generate stereo sound in real time And a stereophonic sound generating unit.

Preferably, the stereo sound generating unit extracts audio information generated based on performance by a different player or musical instrument in different spaces with the audio information generated at the same time.

Preferably, the jitter estimating unit may calculate an average value of jits obtained so far, a variance value calculated from difference values between each jitter obtained up to now and jitter obtained before each jitter is acquired, and a predetermined weight To estimate the network jitter.

Preferably, the jitter estimation section uses a difference value between the weight used previously as the weight or the currently obtained jitter and the average value of jitters obtained up to now, and the calculated value of the previously used variance value.

Preferably, the jitter estimation unit may include: a delay time calculation unit for calculating a packet delay time of the specific channel based on a reception time of the currently received audio information and a reception time of the previously received audio information; And estimating the network jitter by comparing the packet delay time with a reference time. If the packet delay time is greater than or equal to the reference time, it is estimated that a spike has occurred in the specific channel. And estimates that the specific channel is in a normal operation if it is less than the predetermined time.

Preferably, the jitter estimation unit changes the packet delay time to the reference time if the packet delay time is equal to or greater than the reference time, and changes the packet delay time obtained using the audio information received after the specific channel to the reference time, And a reference time adjusting unit for restoring the reference time to an original state when the time is less than the predetermined time.

Preferably, the stereophonic sound generating apparatus further comprises: an audio information dividing unit dividing the currently received audio information in units of intervals according to the presence or absence of the audio signal using the time-base energy of the currently received audio information; A division information storage unit for storing the audio information divided in units of intervals and sequentially outputting audio information divided in units of intervals according to a sequence order; And a signal processing decision unit for deciding whether to process the audio information outputted in order by using one of a compression / decompression method, a loss concealment method and a merging method.

Preferably, the signal processing determination unit determines whether the audio information output at the current time is present or not based on whether the audio information output at the previous time is processed according to the loss concealment method The loss concealment method, and the merge method to determine whether to perform signal processing using the compression / decompression method, the loss concealment method, and the merge method.

Preferably, the signal processing unit outputs audio information output at the current time using the compression / decompression method if the audio information output at the previous time is not signal-processed according to the loss concealment method A compression / non-compression processing unit for processing the audio information; A merging processor for signaling the audio information output at the current time using the merging method if the audio information output at the current time is present and the audio information output at the previous time is signal processed according to the loss concealment method; And a loss concealment processing unit for processing the audio information output at the current time using the loss concealment method if there is no audio information output at the current time.

Preferably, the compression / non-compression processing unit is configured to compress the audio information output at the current time using one of a compression method and an uncompression method depending on whether the audio signal is present or not and whether spikes are generated in the specific channel. .

Preferably, the compression / non-compression processing unit calculates a difference value between the jitter obtained up to now and the jitter obtained before the jitter is acquired when the method of any one of the compression method and the non-compression method is selected Or the ratio between the network jitter and the audio information divided on the basis of the interval.

Preferably, the loss concealment processing unit compares the number of consecutive signal processes performed in accordance with the loss concealment method with a reference frequency, and determines whether or not the current And processes the audio information output in turn.

Preferably, the loss concealment processing unit further uses a linear decreasing scale function for eliminating a buzz sound when signaling the audio information output at the current time.

Preferably, the merging processor extracts sub-audio information from the audio information output at the current time, superimposes each sub-audio information on the audio information output at the previous time, .

Preferably, the stereophony generator determines whether the extracted audio information is received from a larger number of channels than the reference number, and if the extracted audio information is received from a larger number of channels than the reference number The extracted audio information is mixed using the loss concealment method to generate the stereo sound.

Preferably, the stereo sound generating device stores audio information received on the specific channel, and generates audio information received on the specific channel and information on the specific channel A reception information storage unit for storing the combined information; A decoding unit decoding audio information selected based on the network jitter among the stored audio information; And a processing information backup unit for backing up the signal-processed audio information.

Preferably, the stereophonic sound generating apparatus may further include a step of adjusting the first gain value of the audio information divided in units of intervals to equalize the volume when the signal-processed audio information is divided in units of intervals according to whether an audio signal exists or not A first gain adjustment unit; And a second gain value of the audio information divided by the interval including the echo component and the delay component is calculated, and the first gain value and the second gain value are calculated for each audio information segmented by the interval And a second gain reflector for reflecting the second gain.

Preferably, the first gain adjuster comprises: a group generator for generating audio information groups using information other than audio information including silence among the audio information divided in units of intervals; A gain calculating unit for calculating a peak value for a volume for each audio information group and calculating a first gain value using the peak value and the reference value; And comparing the first gain values of the two audio information groups located before and after the audio information group in accordance with the time order, and if there is a difference value between the two first gain values, And a gain comparator for adjusting a first gain value of an audio information group located later on the basis of the first gain value.

Preferably, the stereo sound generating unit generates mixing control coefficients using the energy correlation between the two audio information groups, and mixes the audio information using the mixing control coefficients.

Preferably, the second gain reflector calculates the second gain using an all pass filter and a feedback comb filter.

According to another aspect of the present invention, there is provided a jitter estimation method for estimating network jitter based on a reception time of audio information currently received on a specific channel and a reception time of audio information previously received on the specific channel, Estimation; A signal processor for selectively processing the currently received audio information based on the network jitter; And extracts audio information generated at the same time based on the generation time of each audio information among signal processed audio information received from different channels and mixes the extracted audio information to generate stereo sound in real time A stereo sound generating device including a stereo sound generating section; An audio information generating unit for recording performances of different performers or musical instruments in different spaces at the same time to generate respective pieces of audio information; An encoding unit encoding the generated audio information; An audio information transmitting unit for dividing the encoded audio information into packets and transmitting the divided audio information to the jitter estimating unit and notifying the generation time of the encoded audio information to the stereo sound generating unit; And a stereophonic sound reproducing unit for reproducing the stereophonic sound in real time.

The present invention can obtain the following effects by mixing and reproducing audio information obtained in different independent spaces on the basis of network jitter of each channel and generation time of each audio information to generate and reproduce stereophonic sound in real time.

First, we can minimize the network jitter estimation error through the optimum weight of jitter dispersion, and it can minimize buffering delay and packet loss by playout scheduling using it.

Second, in the process of mixing the performances of the performers transmitted through the IP network into one, the equalization of the volume and the diffusion effect of the audio frame for the audio signals of the N channels are generated, and the audio signals of the N channels are output to the M output signals It is possible to generate a stereo sound with improved sound quality by mixing through the adjusted timing corresponding to the time of the performance, saturation elimination that can be caused by mixing, and harmonization of the respective playing sounds.

FIG. 1 is a conceptual diagram schematically showing a real-time stereo sound generation system of IP network based multi-party independent performance according to an embodiment of the present invention.
2 is a conceptual diagram specifically showing an internal configuration of a packet control unit constituting the system shown in FIG.
3 is a conceptual diagram showing an internal configuration of a mixing and stereo sound generating unit constituting the system shown in FIG.
4 is a block diagram schematically illustrating a stereo sound reproducing system according to a preferred embodiment of the present invention.
FIG. 5 is a block diagram schematically showing an internal configuration of a stereophonic sound generating apparatus constituting the stereophonic sound reproducing system of FIG.
FIG. 6 is a block diagram schematically illustrating the internal structure of a jitter estimating unit included in the stereo sound generating apparatus of FIG. 5;
7 is a block diagram schematically illustrating an internal configuration that can be added to the stereophonic sound generating apparatus of FIG.
8 is a block diagram schematically showing an internal configuration of a signal processing unit constituting the stereophonic sound generating apparatus of FIG.
9 is a block diagram schematically showing an internal configuration that can be added to the stereophonic sound generating apparatus of FIG.

Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the accompanying drawings. In the drawings, the same reference numerals are used to designate the same or similar components throughout the drawings. In the following description of the present invention, a detailed description of known functions and configurations incorporated herein will be omitted when it may make the subject matter of the present invention rather unclear. In addition, the preferred embodiments of the present invention will be described below, but it is needless to say that the technical idea of the present invention is not limited thereto and can be variously modified by those skilled in the art.

In the music playing performed by multiple performers, when the performers of the respective musical instruments do not play in the same space but play in different independent spaces and transmit the music to be played through the IP (Internet Protocol) network in real time, The present invention relates to a method and apparatus for mixing musical performances transmitted from respective musicians so that musical performances become natural musical performances, and generating and reproducing mixed musics as stereophonic sounds.

The present invention applies a signal processing scheme such as adaptive reproduction scheduling, packet loss concealment, and merging to recover an audio stream of music performance transmitted through an IP network, thereby restoring music or audio packets with a minimum delay, A method of mixing an audio stream transmitted by a plurality of players through a multi-channel sound signal processing method so as to become a natural music performance, and a method of generating a mixed audio stream through a plurality of speaker arrays in a stereo sound .

The present invention mixes music played in independent spaces into one natural music as if played by a player in the same space, and generates stereoscopic sound and provides it to a user. The acoustic parameters used for stereophonic generation are stored as metadata so that the user can listen to the stereophonic sound in a desired manner.

FIG. 1 is a conceptual diagram schematically showing a real-time stereo sound generation system of IP network based multi-party independent performance according to an embodiment of the present invention.

In the present invention, instead of playing music in the same space by multiple players, each performer plays in real time in an independent space and transmits through the IP network, and generates stereo sound in real time on the receiving end based on the transmitted performance So various problems arise.

When music or audio data is transmitted through an IP-based packet network, since each packet is transmitted through different paths instead of the same path, the time at which each packet arrives at the receiving end is not constant. That is, a packet may be lost due to a network load, a packet may not be received within a predetermined time, or the order of transmission may be changed. In addition, the buffering delay is increased in a dynamically changing network situation or an abnormal network situation such as a spike, so that the sound quality deteriorates severely. This degradation in voice quality is also occurring in VoIP-based voice calls and music streaming services, and a method for solving the problem of increasing buffering delay and degrading sound quality is required.

In the case of a musical performance involving multiple musicians, when each musical performer plays in a separate space, the musical notes of each musical performer transmitted through the IP network are converted into one Listen to mixed performances and play. However, due to differences in the environment of each participant in different spaces, the audio volume of the musical performance transmitted by each participant is different, so that the audio volume of a player becomes large and the audio volume of a player sounds small Since the IP network delay transmitted from each player is different from each other and signals of a plurality of channels are mixed, inconsistencies in timing for synthesizing audio signals, unbalance in volume, and saturation of volume occur. As a result, there is a problem that the harmony between the respective playing sounds is not achieved and the sound quality is seriously degraded. There is a need for a method of harmonizing each channel by matching the mixing timing of the N-channel signals and equalizing the volume.

In order to produce stereo sound, it is necessary to mix the player's audio signal input through the N channel into the M channel, and to transmit the output signal in accordance with the listening environment possessed by the user. In order to generate stereophonic sound, it is necessary to adjust the uniformity of the volume of the audio signal, to adjust the gain reflecting the echo and delay, and to apply the mixing method applying the difference and correlation of the energy ratio between each channel and the channel. Also, there is a need for a method capable of determining whether a multi-channel sound source received in real time is composed of stereo sound.

Studies on audio streaming, stereo sound generation and reproduction have been conducted independently and independently. Unlike the case where the multiple participants transmit sound sources played in the same space to generate stereophonic sound, each player receives music played in an independent space, generates one music performance in real time, A stereophonic effect is provided to the user, and therefore, a method for solving various problems caused by generating audio in real time by combining audio streaming and stereophonic sound is required.

In order to estimate the jitter that causes deterioration in sound quality, a method of estimating jitter using a timestamp and sequence information of a test packet by transmitting a test packet on a network path, A method of estimating the network jitter by obtaining the average and variance of the jitter using the information, multiplying the fixed jitter dispersion weight by the variance, and adding it to the average of the jitter.

In order to prevent the degradation of the call quality due to the delay and jitter occurring in the network, the playout method measures network traffic for a predetermined time interval and compares the measured network traffic with a threshold value, thereby detecting one of the extension / compression / normal output And the buffering delay can be reduced by the signal processing of the jitter buffer. If the network delay suddenly increases and the jitter buffer is expected to be exhausted or the network delay suddenly decreases, the underflow and the overflow phenomenon A method of performing extension and compression can be used.

Also, in the packet loss concealment method of concealing a packet that is lost on the network due to packet loss, the voice parameter for the packet lost in the normally decoded packet is predicted, and the lost voice packet is decoded through the predictive parameter It is possible to use a method of restoring the pitch of the previous voice frame one by one using the frame information immediately before the loss is generated in order to restore the used or lost voice frame.

However, the above methods increase buffering delay and deteriorate sound quality in dynamically changing network conditions or abnormal network conditions such as spikes. In addition, the sound quality may be degraded in the signal processing process of compression, extension, loss concealment, merging, etc., and sound quality may be degraded. There is a need for a method for solving the problems of increased buffering delay and sound quality degradation.

In order to solve the above problems, an IP network-based real-time stereo sound generation system 100 for real-time stereo sound generation based on IP network according to an aspect of the present invention is characterized in that in a music performance in which a plurality of performers participate, The proposed method is based on multi - channel mixing of music through IP network to create one musical performance and reproduce the generated musical performance as stereoscopic sound.

First, in order to eliminate the phenomenon that sound quality deterioration occurs due to increase of buffering delay at a receiving end due to an abnormal network situation such as an IP network situation or a spike in which a music performance generated in an independent space changes dynamically, The jitter estimation error of the next network state can be reduced by rapidly detecting the transition from the steady state to the spike state of the network and the transition point from the spike state of the network to the steady state by using the information of the dynamic jitter distribution and the dynamic weighting function of the jitter dispersion, Maintain balance between delay and sound quality enhancement.

The jitter estimation method with high jitter estimation accuracy reduces the buffering delay and improves the sound quality by compressing and normally outputting the packets of the receiving end without expanding the packets of the receiving end. Compression is performed only in the silence period excluding the section where the audio signal exists, thereby maintaining a balance between the buffering delay and the sound quality improvement.

In case of packet loss, packet loss concealment is applied to each packet loss concealment method by dividing the lost packet into short and long intervals, thereby eliminating buzz sounds and improving sound quality.

And improves the sound quality by performing merging with the previous restoration packet without distorting the normally received packet at the time of merging the previously recovered previous packet with the currently received restoration.

The adaptive playback scheduling method and the packet loss concealment method are closely related to the mixing unit for generating stereophonic music, so that the stereophonic sound is generated in real time by reducing the delay as much as possible. When the packet control unit 130 receives a notification that an audio signal of a channel corresponding to 2/3 of the N-channel audio signals has arrived in the mixing unit for mixing the N-channel audio signals, The packet loss concealment in the short interval is used for normal output without performing the compression and merging process.

Second, there are problems such as inconsistency of mixing timings, unbalance of volume, saturation of volume, unbalanced harmony of each playing sound, deterioration of sound quality, and the like which occur in the process of mixing each player's performance sound transmitted through IP network A consecutive audio frame in which an audio signal is not a silence section is defined as a section and a gain value of each section obtained by a ratio of a volume reference value and a maximum peak value in one section is applied to an audio section, Thereby eliminating the loudness nonuniformity due to the independent environmental difference of each of

In order to prevent distortion of the audio signal caused by a sudden change in the audio gain value, the gain value is applied linearly to the gain value obtained through the difference between the previous gain value and the current gain value, Volume equalization is performed.

By independently calculating the audio gain values independently calculated for each channel, applying the pre-processing forward filter and the feedback filter to the spreading gain values that are generated by adding the generated echoes and the delay to the audio sections that are equalized in volume, Creating a feeling of stereophonic emanating to the headphone.

The mixing timing is adjusted so that there is no error in the parallax between each of the N channels by applying the transmitting end generation time, the network transmission delay time, the signal processing time, etc. of the audio frame of the audio section input from each of the N channels, , And mixes the musical performance audio stream transmitted from each of the N channels through the correlation ratio into M audio channels.

When mixing the input audio signal between channels, the uniform gain mixing coefficient for preventing saturation is calculated based on the volume limit threshold and the mixing coefficient at the ratio of the volume limit to the voice signal size, and applied to the audio section, prevent.

The mixed audio signal is converted into an analog signal by an amplifier and amplified. The amplified signal is transmitted to the speaker of the stereo environment section.

Third, for stereophonic production, a new virtual speaker is generated through the M-channel pre-post-left-right signals mixed and output, and used as a speaker output signal, and an audio signal output from the placed speakers Receive through a multi-channel input audio device and perform a listening test in L sections to evaluate whether the stereo sound is properly generated. If it is determined that the stereo sound is being generated properly, the currently applied parameter is applied to the stereo sound rendering model. On the other hand, when it is determined that the stereo sound is not generated properly, it is applied to the newly input audio section by changing the diffusion gain, the diffusion parameter, the mixing gain, and the mixing parameter.

As described above, the present invention can achieve the following effects.

First, network jitter estimation error is minimized through optimal weight of jitter dispersion, and playout scheduling using it minimizes buffering delay and packet loss.

Second, when the loss concealment is performed, the consecutive loss interval is detected and classified into the short-term and long-term loss concealment, and the erasure scaling function is applied to the concealed voice frame to prevent the buzz sound due to concealment of the long- Thereby preventing sound quality degradation. After the loss concealment, the merging method using the subframes of the normally received audio frames in the merging process prevents the deformation of the previously received audio frames, thereby improving the sound quality.

Third, in the process of mixing the performances of the performers transmitted through the IP network into one, the equalization of the volume and the spreading effect of the audio frame for the audio signals of the N channels are performed, and the audio signals of the N channels are output to the M output signals To produce a stereo sound with enhanced sound quality by mixing with adjusted timing corresponding to the time of the performance, saturation elimination that can be caused by mixing, and harmonization of the respective playing sounds.

Fourth, in order to measure the effect of the stereo sound production, the audio sound outputted from the speaker arranged is recorded to feedback the stereo sound generation diagram to the mixing engine through the result of the visual simulation of the listening test and the output sound wave, .

The following description will be made with reference to the drawings.

1, a stereophonic sound generating system 100 according to the present invention includes a sound recording and encoding unit 110, a packet generating and transmitting unit 120, a packet control unit 130 and a mixing and stereo sound generation unit 140 through gain control.

When the operation is started, the player plays in real time in each independent space, and the audio sound to be played is recorded by the music sound recording and encoding unit 110 and encoded through the audio encoding method, ), And is transmitted to the packet control unit 130 through the IP network. The packet generating and transmitting unit 120 transmits the time at which each packet is generated at each transmitting end to the stereo sound generating unit 140 of the receiving end independently of being transmitted including information on the time at which each packet is generated in the packet . In the present invention, since the system is configured to apply various audio codecs, there is no limitation on the audio codec.

The audio sound played in each of the independent spaces corresponds to one of the N channels, which is the input of the receiver, that is, the stereo sound generator 140. An audio packet is transmitted from the plurality of channels to the stereo sound generating unit 140, and packet control and audio gain value control are independently performed on the audio packet input to each channel.

The input audio packet of each channel is transmitted to the packet control unit 130. The packet control unit 130 estimates the jitter on the IP network path and simultaneously converts the packet into an audio signal through audio decoding. By using the estimated jitter, packet loss concealment, and adaptive playback method, the problem of jitter and delay is overcome with minimum buffering delay and the audio signal of packet unit is restored. The configuration and operation of the packet control unit 130 will be described in more detail with reference to FIG.

The restored audio signal of the packet unit is transmitted to the mixing and stereo sound generation unit 140 through the gain control. In the mixing unit through the individual control, an N-channel input audio signal is received, an appropriate direct gain and delay are applied to the corresponding channel to perform a diffusion effect of the audio output signal, and an N-channel input Mix audio signals. The output M channel is mapped to the speaker to be placed to generate stereo sound. The configuration and operation of the mixing and stereo sound generation unit 140 through gain control will be described in more detail with reference to FIG.

FIG. 2 is a conceptual diagram specifically showing an internal configuration of the packet control unit 130 constituting the system shown in FIG. The packet control unit 130 is configured to recover and reproduce a packet through an adaptive playout scheduling and packet loss concealment scheme in an IP network environment.

2, the packet control unit 130 includes an audio jitter buffer 210, a network jitter estimation unit 205, an audio decoder 215, a silence interval detection unit 220, an audio frame storage unit 225, A signal processing determination unit 230, an audio frame compression and normal output unit 235, a loss concealment unit 245, a merging unit 240, an audio frame backup unit 250, and a receiving end output device 255 .

When the operation is started, the audio packet generated at the transmitting terminal arrives at the receiving end, that is, the stereo sound generating section, through the communication module. The audio packets arriving at the receiving end are stored in the audio jitter buffer 210. The arrived audio packet includes a time stamp (Timestamp), a sequence number (Sequence Number), and encoded audio information in the case of the RTP header.

The received audio packet is stored in the jitter buffer 210, and at the same time, the network jitter estimator 205 estimates the network jitter of the audio packet to be received next through the audio packet information. The jitter estimation is classified into a steady state in which the jitter dispersion does not vary greatly, a jitter state in which the jitter changes within a certain range in the average value, and a spike state in which the jitter is varied by a large change in the average value. The steady state is set as the initial stage, and the network delay change is measured based on the arrival time of the packet currently arriving at the receiving end and the previous packet according to the time, and a spike occurs when the measured network delay change exceeds the spike detection threshold On the other hand, if it is smaller than the spike detection threshold value, it is determined that the normal state is maintained.

If it is determined that a spike has been detected from the steady state, the network is determined as the end point of the network steady state using the largest sequence of the packets received in the steady state before the spike detection, and the average of the jitter obtained in the steady state before the spike And variance. The average and variance of the stored jitter is used for spike detection when restored to a steady state after a spike has occurred.

In order to detect a change from a spike state to a steady state, a sequence number of a packet arriving at the time of spike detection is defined as a spike end point, and a packet number entered before that is defined as a spike start point. If all the packets between the spike start point and the end point are received, it is assumed that all the packets that have not been received due to spikes are received, and that the spikes are finished and are restored to a normal state.

If a packet arrives after the spike, the spike endpoint is defined as the sequence number of the packet received after the detection of the spike again, so that the spike section is judged to be still in progress and the spike section is increased. If all the packets reaching the extended spike are received, it is judged that the spike is over and the packet is restored to the normal state.

In order to prevent an error when a packet is lost in a spike due to a failure to receive all the packets of the spike interval, a spike interval detection threshold is defined using a change in delay at the time of spike detection, Expected maximum spike duration. If the spike maximum duration is not restored to a steady state, then the spike maximum duration is considered normal.

When the spike state is changed to the steady state, the mean and variance values of the steady state jitter before the spike temporarily stored at the time of detecting the spike are restored to estimate the jitter in the steady state without affecting the jitter calculated in the spike state.

In the steady-state jitter estimation, the next network jitter is estimated by summing the current jitter dispersion multiplied by the optimal weight of the jitter dispersion, which is a value obtained by predicting the jitter variation to the present jitter average.

The optimal weight of the jitter dispersion, which is a key factor for minimizing the jitter error, is a weight that minimizes the difference between the previously estimated jitter and the actual presently generated jitter, and the optimum weight value of the previous jitter dispersion is equal to the previously calculated jitter dispersion It is judged that the jitter is not changed and the optimum weight value of the jitter variance is set to the previous maximum weight value. If there is a change in the weight value, the ratio of the jitter variance to the previous jitter variance with respect to the value obtained by subtracting the previous jitter average value from the current jitter is applied as the optimum weight value of the jitter variance.

In the spike state, since the packets are received due to the abnormal network state and the packets are not received, the optimal weights of the average, dispersion, and jitter dispersion of the jitter in the spike interval are restored Allows you to adapt effectively to network conditions.

In order to deliver the received packet to the output device, the audio decoder 215 requests an audio packet from the jitter buffer 210. The jitter buffer 210 receiving the audio packet from the audio decoder 215 refers to the network jitter estimated by the network jitter estimator 205 and the audio packet information stored in the jitter buffer 210 and transmits the audio packet to the audio decoder 215 And delivers the audio packet to the audio decoder 215.

The audio decoder 215 receiving the audio packet decodes the audio packet, generates an audio frame storing the audio signal, and transmits the audio frame to the silence interval detecting unit 220.

The silence interval detector 220 measures the time axis energy of an audio frame generated through audio decoding to distinguish between an audio signal existence interval and a silence interval. Thereafter, the audio frame is delivered to the audio frame storage unit 225.

The audio frame storage unit 225 transmits an audio frame having a sequence number to be transmitted from the stored audio frame to the current output device to the audio signal processing determination unit 230 to compress and normalize the received audio frame, Or concatenation of the signals.

If there is an audio frame in the audio frame storage unit 225 and no loss concealment has occurred through the loss concealment unit 245, the audio frame is delivered to the audio frame compression and normal output unit 235.

However, if there is no audio frame to be output to the audio frame storage unit 225, the loss concealment unit 245 generates a loss concealment signal to output a signal.

However, if there is an audio frame in the audio frame storage unit 225 and the loss concealment is performed through the loss concealment unit 245, the loss concealment unit 245 extracts the lost concealed audio frame and the discontinuity point And transfers the audio frame to the merging unit 240 for elimination.

The audio frame compression and normal output unit 235 is divided into two cases where the IP network is switched from the spike state to the normal state and the normal state is maintained.

In the case of switching from the spike state to the normal state, in order to prevent an increase in the buffering delay due to the sudden incoming audio packet after the spike, the continuous silence interval stored in the jitter buffer 210 and the audio frame storage unit 225 And compresses it.

If it is determined that the decoded audio frame is a silent period and the current state is not a transition point from the spike state to the steady state, the current estimated jitter and jitter dispersion are compared with the overflow threshold value, Whether or not to make a normal output. That is, if the dispersion of the currently estimated jitter and jitter is smaller than the overflow threshold value, it is predicted that the current network situation is very smooth and the jitter fluctuation is small, and the normal output is performed. On the other hand, if the variance of the currently estimated jitter and jitter is larger than the overflow threshold value, the ratio of the estimated jitter to the length of the voice frame existing at the receiving end is compared to determine the compression and normal output for unstable network conditions.

If the ratio of the estimated jitter to the length of the speech frame present at the receiving end is smaller than the normal output threshold value, compression is performed. On the other hand, if it is larger than the threshold value, it performs normal output.

However, if it is determined that the decoded audio frame is a silent period and is a transition point from a spike state to a normal state, normal output is performed to prevent erroneous compression.

If the audio frame to be output at present is an audio signal existence period other than the silence period, the normal output is performed without performing compression. This is to prevent distortion of the audio signal, which degrades the communication quality through compression in the audio presence interval.

In the loss concealment section 245, the short loss loss concealment is performed if the number of continuous loss concealment times is smaller than the sound quality loss threshold value, and otherwise, the long loss loss concealment is performed.

When the short-interval loss concealment is performed, it is confirmed whether or not an audio frame exists in the audio frame storage unit 225. [ If an audio frame does not exist in the audio frame storage unit 225, a replacement signal of the lost voice frame is generated using the audio frame normally received before the loss in the audio frame backup unit 250.

If there is no audio frame in the audio frame storage unit 225 and there is an audio packet to be output to the jitter buffer 302, an audio packet is requested to the audio decoder 215 and a packet existing in the jitter buffer 210 And stores the decoded audio frame into the audio frame storage unit 225, and generates a concealed signal using the normally received audio frame before the loss and the normally received audio frame after the loss.

When a packet loss occurs in a long interval, a lost packet interval is repeated using only a normal received audio frame before a loss as in the short interval, so that a buzzing sound is generated which rapidly degrades the sound quality due to repeated identical or similar signals. In order to eliminate such a buzzing sound, in the present invention, if a previous audio frame used for packet concealment is silent, it is used as it is. If it is discriminated that an audio signal exists, a previous audio frame is applied to conceal the loss, The reduced scale function is applied directly to the restored audio frame to remove the buzzing sound.

The silence and audio signal reduction scale functions used in short and long interval packet loss concealment are buzz sound cancellation functions that are generated by measuring the listening effect for 20 listeners.

The merging unit 240 acquires the normally received audio frame at the audio frame storage unit 225 at the same time as acquiring the lost hidden audio frame. Subtracting the subframes from the normally received audio frame to prevent distortion occurring during merging to generate an alternate signal for superposition summing. Searches for a pseudo interval in the lost secreted audio signal using the generated substitute signal, and performs superposition-sum of the lost secreted audio frame and the substitute signal using the searched pseudo interval. The final merged signal is generated by connecting the normally received audio frame to the audio frame from which the discontinuity point is removed by the overlapping sum of the loss concealed voice frame and the replacement signal.

The adaptive playback scheduling method and the packet loss concealment method are closely related to the mixing section for generating stereophonic music together with the jitter estimation, so that the overall delay can be minimized and a stereophonic sound can be generated in real time. That is, when the packet control unit 130 receives a notification that an audio signal of a channel corresponding to 2/3 of the N-channel audio signals has already arrived in the mixing unit for mixing N-channel audio signals, the packet control unit 130 does not perform the compression and merging processes, but uses only the packet loss concealment in the short interval to output normally, thereby reducing the delay to a minimum and effectively performing the mixing.

The audio frame compression and normal output unit 235 according to the network jitter, the loss concealment unit 245 according to the packet loss, or the merging unit 240 are subjected to the signal processing process, And transmitted to the sound generating unit 140. At the same time, the audio frame is stored in the backup audio frame backup unit 250.

The characteristics of the audio frame compression and normal output unit 235 will be further described as follows.

- Problems that occur

In the case of live audio streaming using mobile VoIP, when the network conditions are congested and the network condition improves suddenly, there is an overflow that a lot of packets are received at the receiving end and a lot of packets are piled up. Congestion occurs and an underflow phenomenon occurs in which audio frames to be output to the receiving end are exhausted.

- Problems of existing technology

There is a large error in jitter estimation in a network situation or a spike situation that varies dynamically in a playout schedule scheme using a conventional jitter estimation scheme. Especially, in the steady state, the jitter estimation error is large in the period of change of the spike and the period of change from the spike period to the steady state, so that the buffering delay adjustment for the playout scheduling is not properly performed. In other words, since the playout scheduling method that performs compression using only an inaccurate jitter estimation and a length of an audio frame existing in a receiving end does not immediately reflect dynamic network network fluctuation situation, the buffering delay increases and packet loss Lt; / RTI >

The process is divided into the following two.

First, when returning from spikes to normal

First, the audio packet suddenly comes in and the buffering delay increases. Thereafter, the audio signal is compressed in the non-audio section of the jitter buffer 210 and the output audio frame storage unit 225. At this time, if the non-audio interval is not present in the output audio frame storage unit 225 and is present in the jitter buffer 210, decoding is requested to fill the output audio frame, and then audio is judged again. Then it compresses to Biodi.

Second, in the audio section,

If the current output is non-audio, use jitter information to determine whether the network is a turning point from spike to normal. Then, if the jitter and jitter variance are compared with the overflow threshold value in the non-audio interval, it is determined that the network is unstable due to the unstable jitter and the compression is performed. Otherwise, the normal output is performed.

If the ratio of the estimated jitter and the length of the frame stored in the output audio frame is smaller than the normal output threshold value, it is determined that there is an audio frame that can be normally output to the receiver until the next packet arrives,

Next, the characteristics of the loss concealment unit 245 and the merging unit 240 will be described in detail.

- cause

It conceals loss when packet loss occurs due to spike occurrence. If a normal frame is input after concealment, the frame is merged.

- existing problems

In the case of using the conventional scheme in the continuously lost interval, the lost secret packet is repeatedly generated and the buzz sound is generated, which degrades the sound quality.

- Process

The loss concealment is largely classified into short - term packet loss concealment and long - term packet loss concealment.

1. Short-term loss and long-term loss

If the number of consecutive loss concealment is smaller than the threshold, it is divided into short-term loss and long-term loss.

2. Short-term loss concealment

And checks whether an audio frame after loss is present in the output audio frame storage unit.

1) If there is no audio frame after the loss in the output audio frame storage unit and the normal packet does not exist in the jitter buffer 210 after loss, the audio frame usable for restoring the lost audio frame is an audio frame backup There is only a normally received audio frame before the loss of the part 250. [

And conceals the lost packet using the normally received audio frame before loss.

When the loss packet is concealed, it is determined whether the normally received audio frame is voiced or unvoiced.

If it is unvoiced, apply the unvoiced clearing scaling function. If the voiced sound is a voiced sound, keep the scale value until a certain interval. Then, after the designated interval, apply the voiced sound reduction scale function to conceal the lost frame.

2) If there is no audio frame after loss in the output audio frame storage unit 225 and there is an audio packet to be output to the jitter buffer 210, To be decoded.

The decoder decodes the audio packet of the jitter buffer 210 upon request and stores the decoded audio packet in the output audio frame storage unit.

Loss after the loss of the output audio frame storage unit and the audio frame before the loss of the audio frame storage unit 225 are used.

3) If there is an audio frame in the output audio frame storage unit, the loss of the normally received audio frame and the output audio frame after the loss of the audio frame backup unit 250, do.

3 is a conceptual diagram showing an internal configuration of a mixing and stereo sound generating unit 140 constituting the system shown in FIG. In this embodiment, the mixing and stereo sound generator 140 uses gain control.

3, the mixing and stereo sound generation unit 140 includes an audio volume gain adjustment unit 305, an audio diffusion adjustment unit 310, an audio mixing unit 325, a metadata generation unit 330, An amplifier 335, a stereo sound environment unit 345, an input device 340, and a stereo sound environment measurement unit 315. [

When the operation is started, audio gain adjustment is performed independently for audio frames input to each N channel.

The audio volume gain adjuster 305 aims to equalize the volume in preference in consideration of the diversity of the volume of the music signal played in each independent space. Information corresponding to a section in which an audio signal exists rather than a silence section is received through the silence section detection section 304 of the packet control section 130, and F consecutive audio frames are defined as sections, and each of the samples The peak value having the highest value among the peak values is searched. The peak value of the previous audio frame is compared with the peak value of the current audio frame. The gain value of each section is calculated by the ratio of the volume reference value and the maximum peak value through the maximum peak value in one section. If there is a difference between the gain value obtained from one section and the gain value of the subsequent section, the continuity of the audio signal is maintained by applying the gain value of the next section through smoothing between the sections. The adjusted gain value is applied to the restored packet unit of the voice frame signal to remove the loudness nonuniformity according to the environment difference. When volume equalization is applied, the volume reference value is applied, so that the volume saturation does not occur. However, if the volume saturation of the audio signal occurs in the event of an event, the audio gain value is reduced in proportion to the degree of saturation. In order to prevent distortion of the audio signal caused by a sudden change in the audio gain value, the change of the gain value is calculated through the difference between the previous gain value and the current gain value. If the gain value changes, the gain value is linearly changed by the difference of the gain value, so that the gain value is applied to the audio signal of the unit of measure. If there is no change in the gain value, the current gain value is directly applied to the audio signal.

The energy ratio and the degree of correlation between the N channels input from each player before gain adjustment are calculated through energy calculated in units of sections of audio frames input to each channel. The energy ratio and the correlation ratio between the obtained N channels are applied to the mixing of the audio mixing unit 325. The audio mixing unit 325 will be described in detail. The audio gain value used in the audio gain adjustment unit 305 may be used as a volume control element or metadata for a system engineer or a user to adjust a music section transmitted and played in each independent space.

The audio diffusion adjustment unit 310 defines the audio gain value independently calculated in each channel as a direct gain value, removes high frequency energy by applying a preprocessing filter to the audio frame received from the volume gain adjustment unit 305, Generates a spreading gain value to which echoes and delays are added through a filter (all pass filter) and a feedback comb filter. The feedback comb filter creates a variance of the signal and generates a delay dependent on the echo frequency. By applying the spreading gain value, various echoes and delay components of the audio signal are generated, thereby generating a diffusion effect of the audio signal. The spread gain value and the direct gain value are applied to the audio section and output. Diffusion gain is an intentional audio processing module that is combined with direct gain to give the speaker or headphone a sense of stereo sound emission. The diffusion gain value used in the audio diffusion adjustment unit 310 can be used as a volume adjustment element or metadata that is adjusted by a system engineer or a user.

The audio mixing unit 325 applies the packet generation time received from the packet dividing and transmitting unit 120, the transmitting end generation time of the audio section of the audio section inputted in each of N channels, the network transmission delay time, and the signal processing time So that there is no error in the parallax between each of the N channels, and the music performance audio stream transmitted from each of the N channels is mixed into M audio channels. If a packet corresponding to two thirds of the N channels has already arrived at the audio mixing unit 325, only a simple short-term packet loss concealment is applied to the packet control unit 130 of the channel that has not arrived yet, So that the delay can be delayed and the N channel can be mixed with the audio signal in the multi-channel manner in the mixing unit.

The audio mixing unit 325 mixes the audio signals of the N audio input channels by generating mixing control coefficients for forming a set of M output channels so that the stereo audio environment unit 345 can reproduce the performance. The mixing control coefficients are generated through the ratio of the energy between the N channels obtained by the volume gain adjuster 305, and the correlation ratio.

Since the situation of volume saturation may occur when mixing the audio signals inputted between the channels, a uniform gain mixing coefficient for preventing saturation of the volume is applied in order to prevent volume saturation. That is, when the mixing coefficient obtained from the k-th section is applied to the size of the i-th audio signal, if it is smaller than the volume limitation threshold value, it is determined that the mixing coefficient can be applied as it is. Here, the mixing coefficient obtained in the k-th section is a mixing coefficient applied to the audio signal inputted until the mixing coefficient is calculated in the (k + 1) -th section. On the contrary, when the mixing coefficient is applied to the amplitude of the i-th audio signal, it is determined that the volume saturation occurs when the volume limitation threshold is exceeded. The uniform gain mixing coefficient for preventing saturation Thereby preventing volume saturation.

i번째 오디오 신호에 적용될 믹싱 계수의 차이가 발생하면, i

번째 오디오 신호의 전체 샘플 단위로 믹싱 계수값을 선형적으로 변화시키면서 적용한다.In order to prevent the distortion of the audio signal caused by the variation of the mixing coefficient, the mixing coefficient applied to the (i-1)

When a difference in the mixing coefficient to be applied to the i-th audio signal occurs, i

And the mixing coefficient value is linearly changed in units of a whole sample of the first audio signal.

The mixing control coefficient used in the audio mixing unit 325 may be used as a volume control element or metadata that is adjusted by a system engineer or a user.

The mixed audio signal is converted into an analog signal by the amplifier 335 and amplified. The amplified signal is transmitted to the speaker of the stereo environment unit 345.

The stereo sound environment section 345 is composed of the number of speakers, arrangement type, position, and the like. The number of M channels output from the audio mixing unit 325 and the actual number of speakers do not necessarily match each other. A new virtual speaker can be generated through the pre-post-left-right signal of the output M channel and used as a speaker output signal. The number, arrangement, orientation information, height information, and level information of speakers used in the stereophonic environment unit 345 can be used as metadata.

The stereo environment environment measuring unit 315 receives the audio signals output from the speakers of the stereo sound environment unit 345 through the multi-channel input audio device. Expert listening tests are performed on the transmitted audio signal in units of L sections to evaluate whether the stereo sound is generated properly. If it is determined that the stereo sound is being generated properly, the currently applied parameter is applied to the stereo sound rendering model. On the other hand, if it is determined that the stereo sound is not properly generated, the volume gain adjusting unit 305, the diffusion gain adjusting unit 310, and the audio mixing unit 325 feedback the volume gain, spreading gain and spreading parameter, And changes the mixing parameter to apply to the newly input audio section.

The stereophonic rendering model consists of the ontology map and the sound field visual simulation result output from the speaker to compute the correlation between the stereophonic parameters satisfying the stereophonic listening test and the stereophonic sound model Check. The ontology map shows the optimized correlation between the generated stereo sound parameters and enables to distinguish the stereo sound generation level through the correlation.

The metadata generating unit 330 stores parameters generated by the mixing and stereo sound generating unit 140 using gain control as metadata. The stored metadata may be applied for a stereo reproduction system.

The features of the stereo sound generating system 100 described with reference to FIGS. 1 to 3 are summarized as follows.

The stereophonic sound generation system 100 can reduce the buffering delay only through the signal processing at the receiving end in the case of generating the stereophonic sound at the receiving end by transmitting the played sound played by the player in each independent space through the IP network, A jitter estimation method using an optimum weight function of jitter dispersion is used.

The stereophonic generation system 100 utilizes a playout scheduling scheme using jitter estimation to reduce jitter estimation errors.

The stereo sound generation system 100 may include a structure in which a lossy speech frame reconstruction method can be applied regardless of a type of a decoder by separating a decoder and a loss concealment unit; The packet loss concealment and merging method based on the detection result of the silent interval of the received packet is used.

The stereo sound generation system 100 detects a spike state in which the network jitter is rapidly changed using the header information of the packet received by the reception apparatus and the optimum jitter dispersion weight corresponding thereto, Lt; / RTI >

The stereo sound generation system 100 divides a network state into a spike state and a normal state in a rapidly changing network environment, and calculates a jitter distribution using a mean and variance of current network jitter according to the state of each network, Minimize estimation error.

The stereophony generation system 100 stores information of a packet normally received at the time of spike detection to detect a change from a steady state to a spike state of the network, Lt; / RTI >

The stereophonic generation system 100 temporarily stores the mean and variance of the jitter of the previous steady state during spike detection and uses the average and variance of the temporarily stored jitter when switched from the spike state to the steady state to change the jitter The jitter of the steady state is estimated without affecting the jitter.

The stereo sound generation system 100 calculates an error between the estimated jitter and the actual jitter in the steady state and the spike state of each network to obtain an optimum jitter dispersion weight.

The stereophonic sound generation system 100 effectively performs playout scheduling by using the jitter estimated by the jitter estimation method, the length of the audio frame and the audio packet existing at the receiving end.

The stereophonic sound generation system 100 judges only the compression and normal output excluding the extension in performing the playout scheduling.

The stereophonic sound generation system 100 distinguishes between a case where a spike is switched to a normal state and a state where a normal state is maintained during compression, and compression is performed accordingly.

The stereophonic sound generation system 100 is divided into a section where an audio signal exists and a silence section in which compression is performed, and compression is performed only in a section in which silence exists without performing compression in an interval in which an audio signal exists, thereby preventing sound quality degradation .

The stereophonic generation system 100 performs a normal output to prevent an error at a transition point from a spike to a steady state transition.

The stereophonic sound generation system 100 finds consecutive silence periods of the audio frame buffer and the jitter buffer in a state where the spike to the steady state is changed during compression, and compresses the consecutive silence periods to reduce the buffering delay.

Stereophonic sound generation system 100 recognizes consecutive packet losses when concealing packet loss and performs concealment in such a manner as to discriminate short-term and long-term loss concealances in order to eliminate buzz sounds.

The stereo sound generation system 100 restores the lost audio frame using only the previous audio frame in the case where there is no audio frame after the lost audio frame when the consecutive packet loss is a short interval, If there is an audio frame after the frame, the lost audio frame is recovered using the audio frame before loss and the audio frame after loss.

If the audio frame does not exist after loss in the audio frame buffer in the process of confirming the existence of the audio frame after loss, and the time of the timer for mixing does not have a problem, the stereo sound generation system 100 detects the audio frame after loss in the jitter buffer And decoded to be used for loss recovery.

The stereophonic sound generation system 100 distinguishes silence and audio signal existence intervals and applies an erasure scaling function to the audio signal existence interval in order to remove buzz sounds when the long interval loss is concealed.

The stereophonic sound generation system 100 extracts a subframe from a normally received audio frame to prevent degradation in sound quality during merging, and prevents a deformation of a normally arriving packet by performing a natural merge with a lost hidden audio frame using the extracted subframe.

Since the adaptive reproduction scheduling method and the packet loss concealment method are closely related to the mixing unit for generating stereophonic music, a 2/3 channel signal is received in the mixing unit so that the delay can be minimized and a stereophonic sound can be generated in real time The packet control unit does not perform the compression and merging process, but uses only the packet loss concealment in the short interval and outputs normally, thereby reducing the delay as much as possible.

The stereophonic sound generation system 100 performs a process of mixing the N channels including the performance sound of each player transmitted through the IP network into the M output signals in the process of matching the timing, uniformity of the volume, saturation of the volume, Maintain balanced balance of playing notes.

The stereo sound generation system 100 defines F audio frames in which audio signals exist, not silence periods, as sections, and calculates a gain of each section as a ratio of a volume reference value to a maximum peak value through a maximum peak value found in units of sections Calculate the value.

The stereo sound generation system 100 maintains the continuity of the audio signal by applying the gain value of the next section through the smoothing between the respective sections when the difference between the gain value obtained in one section and the gain value of the succeeding section is generated .

The stereo sound generation system 100 reduces the audio gain value at a rate according to the degree of volume saturation to prevent saturation of the volume if the volume saturation of the audio signal applied with the gain value in the unit of section occurs.

Stereophonic sound generating system 100 calculates the packet generation time from the packet segmentation and transmission unit input from each player before the gain adjustment through the energy calculated in units of sections of audio frames input to each channel, , And the correlation is calculated and applied to the mixing.

The stereo sound generation system 100 defines an audio gain value independently calculated for each channel as a direct gain value, and adds a pre-processing forward filter and a feedback filter to which echo and delay are added through an audio frame received from the volume gain adjusting unit Thereby generating a gain value.

Stereophonic generation system 100 combines diffusion gain and direct gain to give a speaker or headphone a sense of stereophonic sound emission.

The stereophonic sound generation system 100 applies timing such as a transmission terminal generation time, a network transmission delay time, a signal processing time, etc. of an audio frame of an audio section input in each of N channels so that there is no error in the parallax between the N channels And mixes music performance audio streams transmitted from each of the N channels into M audio channels.

The stereophonic sound generation system 100 generates a mixing control coefficient for generating a set of M output channels so that a performance can be reproduced in a stereo sound environment unit through an energy ratio and a correlation ratio between N channels obtained in the volume gain adjustment unit, Mix the audio signal of the channel.

The stereophonic sound generation system 100 applies a uniform gain mixing coefficient for preventing saturation of sound to prevent loudness saturation at the time of mixing audio signals inputted between respective channels.

The stereo sound generation system 100 prevents the saturation of volume by calculating and applying a uniform gain mixing coefficient for preventing volume saturation at a ratio of the volume limit and the voice signal size.

The stereo sound generation system 100 linearly changes the mixing coefficient value to prevent distortion of the audio signal caused by variation of the mixing coefficient.

The stereo sound generation system 100 combines audio mixing and a stereo environment to generate an initial stereo sound, and records the generated stereo sound to a multi-channel input device to generate an improved stereo sound.

The stereophonic sound generation system 100 generates a new virtual speaker through a pre-post-left-right signal of the M channel output from the audio mixing unit and uses it as a speaker output signal.

The stereo sound generation system 100 receives an audio signal output from the speakers of the stereo environment unit as multi-channel input audio, and performs a listening test of an expert in units of L to evaluate whether a stereo sound is properly generated.

The stereophonic generation system 100 applies the applied stereophonic parameters to the rendering model if the stereophony is properly generated. On the other hand, if it is determined that the stereo sound is not properly generated, a request for changing the stereo sound parameter is transmitted to the packet control unit, the volume gain adjusting unit, the diffusion adjusting unit, and the mixing coefficient adjusting unit to generate an improved stereo sound by applying new parameters .

The stereo sound generation system 100 configures an ontology map between the stereo sound parameters satisfying the stereo sound listening test in order to generate a stereo sound rendering model, and generates a sound field visual simulation result output from the ontology map and the speaker Make sure that the stereo model is applied properly.

Next, a preferred embodiment of the present invention, which can be inferred from the embodiments described in Figs. 1 to 3, will be described.

4 is a block diagram schematically illustrating a stereo sound reproducing system according to a preferred embodiment of the present invention.

4, the stereophonic sound reproducing system 400 includes an audio information generating unit 410, an encoding unit 420, an audio information transmitting unit 430, a stereo sound generating apparatus 500 and a stereo sound reproducing unit 440, .

The audio information generating unit 410 records performances of different performers or musical instruments in different spaces at the same time, and generates audio information for each of the performers.

The encoding unit 420 encodes the audio information generated by the audio information generating unit 410. In the present embodiment, the music sound recording and encoding unit 110 of FIG. 1 may be implemented by the audio information generating unit 410 and the encoding unit 420 of FIG.

The audio information transmitting unit 430 divides the audio information encoded by the encoding unit 420 into packet units and transmits the divided audio information to the jitter estimating unit of the stereo sound generating apparatus 430. The audio information transmitting unit 430 generates a stereo sound generation time And notifies the stereo sound generating unit of the apparatus 430. [ In this embodiment, the packet generating and transmitting unit 120 of FIG. 1 may be implemented by the audio information transmitting unit 430 of FIG.

The stereo sound reproducing unit 440 performs a function of real-time reproducing the stereo sound generated by the stereo sound generating device 430. [

Hereinafter, the stereo sound generating apparatus 500 will be described in more detail.

FIG. 5 is a block diagram schematically showing an internal configuration of a stereophonic sound generating apparatus constituting the stereophonic sound reproducing system of FIG.

5, the stereophonic sound generating apparatus 500 includes a jitter estimating unit 510, a signal processing unit 520, a stereo sound generating unit 530, a power source unit 540, and a main control unit 550.

The power supply unit 540 performs a function of supplying power to each configuration of the stereophonic sound generating apparatus 500. The main control unit 550 performs a function of controlling the overall operation of each configuration of the stereophonic sound generating apparatus 500.

The jitter estimator 510 estimates network jitter based on the reception time of the audio information currently received on the specific channel and the reception time of the audio information previously received on the specific channel when the audio information is received on the specific channel . In this embodiment, the network jitter estimation unit 205 of FIG. 2 may be implemented by the jitter estimation unit 510 of FIG.

The jitter estimation unit 510 uses the average value of the jitters obtained so far, the variance value calculated from the difference values between the jitter obtained before the respective jits obtained until now and the jitter obtained up to now, and the predetermined weight The network jitter can be estimated.

The jitter estimating unit 510 may use a difference value between the weight previously used as the weight or the currently obtained jitter and the average value of the jitters previously obtained and the calculated value of the previously used variance value.

The jitter estimator 510 may include a delay time calculator 511, a delay time comparator 512 and a reference time adjuster 513 as shown in FIG. FIG. 6 is a block diagram schematically illustrating the internal structure of a jitter estimating unit included in the stereo sound generating apparatus of FIG. 5;

The delay time calculating unit 511 calculates the packet delay time of a specific channel based on the reception time of the currently received audio information and the reception time of the previously received audio information.

The delay time comparator 512 estimates network jitter by comparing the packet delay time with a reference time, estimates that a spike occurs in a specific channel when the packet delay time is longer than a reference time, , It performs a function of estimating that a specific channel is operating normally.

The reference time adjustment unit 513 changes the packet delay time to the reference time if the packet delay time is longer than the reference time and restores the reference time when the packet delay time obtained using the audio information received later on the specific channel is less than the reference time .

Referring back to Fig.

The signal processor 520 selectively performs signal processing on the currently received audio information based on the network jitter estimated by the jitter estimator 510. In the present embodiment, the audio signal processing determination unit 230, the audio frame compression and normal output unit 235, the merge unit 240, the loss concealment unit 245, and the like of FIG. 2 correspond to the signal processing unit 520 of FIG. 5 Can be implemented.

The signal processing unit 520 may include a compression / non-compression processing unit 521, a merge processing unit 522, and a loss concealment processing unit 523 as shown in FIG. 8 is a block diagram schematically showing an internal configuration of a signal processing unit constituting the stereophonic sound generating apparatus of FIG.

The compression / non-compression processing unit 521 has audio information output at the current time, and if the audio information output at the previous time is not signal-processed according to the loss concealment method, the audio / As shown in FIG. In this embodiment, the audio frame compression and normal output unit 235 of FIG. 2 may be implemented by the compression / decompression processing unit 521 of FIG.

The compression / non-compression processing unit 521 can process the audio information output at the current time using either a compression method or an uncompression method depending on whether an audio signal is present and whether spikes are generated in a specific channel have.

The compression / non-compression processing unit 521 performs a compression / decompression process on the basis of the difference values between the jitter obtained before the jitter obtained and the jitter obtained before the jitter obtained, , Or a ratio between network jitter and audio information segmented on a segmental basis.

The merging processor 522 performs a function of processing the audio information output at the current time using the merging method if the audio information output at the current time is signaled and the audio information output at the previous time is signal processed according to the loss concealment method do. In the present embodiment, the merging unit 240 of FIG. 2 may be implemented by the merge processing unit 522 of FIG.

The merging processor 522 extracts the sub audio information from the audio information output at the current time and superimposes the sub audio information on the audio information output at the previous time so as to process the audio information output at the current time.

The loss concealment processing unit 523 performs a function of signal processing the audio information output at the current time using the loss concealment method if there is no audio information to be outputted at the current time. In this embodiment, the loss concealment unit 245 of FIG. 2 may be implemented by the loss concealment processing unit 523 of FIG.

The loss concealment processing unit 523 compares the number of consecutive signal processes performed in accordance with the loss concealment method with the reference number and outputs the result at the current time using either the short-term loss concealment method or the long-term loss concealment method And can process audio information.

The loss concealment processing unit 523 may further use a linear decreasing scale function for eliminating the buzz sound when processing the audio information output at the current time.

Referring back to Fig.

The stereo sound generating unit 530 extracts audio information generated at the same time based on the generation time of each audio information among signal processed audio information received from different channels, mixes the extracted audio information And performs a function of generating stereo sound in real time. In this embodiment, the audio mixing unit 325 of FIG. 3 may be implemented by the stereo sound generation unit 530 of FIG.

The stereophonic sound generator 530 may extract audio information generated based on performance by different performers or musical instruments in different spaces using the audio information generated at the same time.

The stereo sound generator 530 determines whether the extracted audio information is received from a larger number of channels than the reference number. If it is determined that the extracted audio information is received from a larger number of channels than the reference number, The stereo information can be generated by mixing the extracted audio information using the concealment method.

The stereo sound generating unit 530 may generate a mixing control coefficient using the energy correlation between the two audio information groups, and may mix the audio information using the mixing control coefficient.

FIGS. 7 and 9 are block diagrams schematically illustrating an internal configuration that can be added to the stereophonic sound generating apparatus of FIG.

7, the stereo sound generating apparatus 500 includes an audio information dividing unit 610, a divided information storing unit 620, a signal processing determining unit 630, a received information storing unit 640, a decoding unit 650 , A processing information backup unit 660, and the like.

The audio information dividing unit 610 divides the currently received audio information in units of intervals according to the presence or absence of the audio signal using the time axis energy of the currently received audio information. In this embodiment, the silence interval detection unit 220 of FIG. 2 may be implemented by the audio information division unit 610 of FIG.

The segmentation information storage unit 620 stores audio information segmented on a segment basis, and sequentially outputs audio information segmented on a segment basis in a sequence order. In this embodiment, the audio frame storage unit 225 of FIG. 2 may be implemented as the division information storage unit 620 of FIG.

The signal processing determination unit 630 performs a function of determining whether to process the audio information, which is sequentially output, by using one of a compression / decompression method, a loss concealment method, and a merge method. In this embodiment, the audio signal processing determination unit 230 of FIG. 2 may be implemented by the signal processing determination unit 630 of FIG.

The signal processing determination unit 630 determines whether the audio information output at the current time is compressed or not based on whether the audio information output at the current time is present and whether the audio information output at the previous time is processed according to the loss concealment method It is possible to decide whether to perform signal processing using any one of a compression method, a loss concealment method, and a merge method.

The reception information storage unit 640 stores audio information received on a specific channel, and combines and stores information on a specific channel and the generation time of audio information received on a specific channel in audio information received on a specific channel . In the present embodiment, the audio jitter buffer 210 of FIG. 2 may be implemented as the reception information storage unit 640 of FIG.

The decoding unit 650 decodes the selected audio information based on the network jitter among the audio information stored in the reception information storage unit 640. In this embodiment, the audio decoder 215 of FIG. 2 may be implemented by the decoding unit 650 of FIG.

The processing information backup unit 660 performs a function of backing up the processed audio information. In this embodiment, the audio frame backup unit 250 of FIG. 2 may be implemented by the processing information backup unit 660 of FIG.

The stereophonic sound generator 500 may further include a first gain adjuster 670 and a second gain reflector 680.

The first gain adjustment unit 670 adjusts the first gain value of the audio information divided in units of intervals to equalize the volume when the signal-processed audio information is divided in units of intervals according to the presence or absence of the audio signal. In this embodiment, the audio volume gain adjustment unit 305 of FIG. 3 may be implemented by the first gain adjustment unit 670 of FIG.

The first gain adjustment unit 670 may include a group generation unit 671, a gain calculation unit 672, and a gain comparison unit 673.

The group generation unit 671 generates audio information groups using information other than audio information including silence among the audio information divided in units of intervals.

The gain calculating unit 672 calculates a peak value for the volume for each audio information group, and calculates a first gain value using the peak value and the reference value.

The gain comparator 673 compares the first gain values of the two audio information groups located before and after the audio information groups according to the time order, and if there is a difference value between the two first gain values, And adjusts the first gain value of the audio information group located later based on the first gain value of the group.

The second gain reflector 680 calculates the second gain value of the audio information that includes the echo component and the delay component and is divided in units of intervals, and calculates a first gain value and a second gain value for each audio information And reflects the second gain value. In this embodiment, the audio diffusion adjustment unit 310 of FIG. 3 may be implemented by the second gain reflection unit 680 of FIG.

The second gain reflection unit 680 may calculate the second gain value using an all pass filter and a feedback comb filter.

It is to be understood that the present invention is not limited to these embodiments, and all elements constituting the embodiment of the present invention described above are described as being combined or operated in one operation. That is, within the scope of the present invention, all of the components may be selectively coupled to one or more of them. In addition, although all of the components may be implemented as one independent hardware, some or all of the components may be selectively combined to perform a part or all of the functions in one or a plurality of hardware. As shown in FIG. In addition, such a computer program may be stored in a computer readable medium such as a USB memory, a CD disk, a flash memory, etc., and read and executed by a computer to implement an embodiment of the present invention. As the recording medium of the computer program, a magnetic recording medium, an optical recording medium, a carrier wave medium, and the like can be included.

Furthermore, all terms including technical or scientific terms have the same meaning as commonly understood by one of ordinary skill in the art to which this invention belongs, unless otherwise defined in the Detailed Description. Commonly used terms, such as predefined terms, should be interpreted to be consistent with the contextual meanings of the related art, and are not to be construed as ideal or overly formal, unless expressly defined to the contrary.

It will be apparent to those skilled in the art that various modifications, substitutions and substitutions are possible, without departing from the scope and spirit of the invention as disclosed in the accompanying claims. will be. Therefore, the embodiments disclosed in the present invention and the accompanying drawings are intended to illustrate and not to limit the technical spirit of the present invention, and the scope of the technical idea of the present invention is not limited by these embodiments and the accompanying drawings . The scope of protection of the present invention should be construed according to the following claims, and all technical ideas within the scope of equivalents should be construed as falling within the scope of the present invention.

Claims (21)

Estimating network jitter based on a reception time of the audio information currently received on the specific channel and a reception time of the audio information previously received on the specific channel when the audio information is received on the specific channel, A delay time calculating unit for calculating a packet delay time of the specific channel based on a reception time of the audio information and a reception time of the previously received audio information; And estimating the network jitter by comparing the packet delay time with a reference time. If the packet delay time is greater than or equal to the reference time, it is estimated that a spike has occurred in the specific channel. A jitter estimator for estimating that the specific channel is in a normal operation;
A signal processor for selectively processing the currently received audio information based on the network jitter; And
Extracts audio information generated at the same time on the basis of the generation time of each audio information among the signal-processed audio information received from different channels, mixes the extracted audio information to generate stereoscopic sound in real time The sound-
And a second sound generator for generating a sound. The method according to claim 1,
Wherein the stereo sound generating unit extracts audio information generated based on performance by a different player or musical instrument in different spaces with the audio information generated at the same time. The method according to claim 1,
The jitter estimating unit may calculate the jitter using the average value of the jitters obtained so far, the variance value calculated from the differences between the jitter obtained before the jitter obtained before the jitter is obtained, and the predetermined weight, And estimates the jitter. The method of claim 3,
Wherein the jitter estimation unit uses a difference value between a weight previously used as the weight or the currently obtained jitter and an average value of jitters previously obtained and an operation value of the previously used variance value. delete The method according to claim 1,
Wherein the jitter estimator comprises:
And changes the packet delay time to the reference time if the packet delay time is longer than or equal to the reference time and restores the reference time when the packet delay time obtained using the audio information received later with the specific channel is less than the reference time The reference time adjustment unit
Further comprising: a stereo sound generator for generating stereo sound. The method according to claim 1,
An audio information dividing unit for dividing the currently received audio information in units of intervals according to the presence or absence of an audio signal using the time axis energy of the currently received audio information;
A division information storage unit for storing the audio information divided in units of intervals and sequentially outputting audio information divided in units of intervals according to a sequence order; And
A loss concealment method, and a merge method, and determines whether to process the audio information, which is output in order, by using a compression / decompression method, a loss concealment method,
Further comprising: a stereo sound generator for generating stereo sound. 8. The method of claim 7,
Wherein the signal processing decision unit decides whether or not audio information to be outputted at the present time is present and whether audio information outputted at the previous turn is processed according to the loss concealment method, The loss concealment method, and the merge method according to a result of the determination of whether to perform signal processing using the compression method, the loss concealment method, and the merge method. 8. The method of claim 7,
The signal processing unit,
A compression / non-compression method for processing audio information output at the current time by using the compression / decompression method if the audio information outputted at the current time is present and the audio information output at the previous time is not signal- An uncompression processing unit;
A merging processor for signaling the audio information output at the current time using the merging method if the audio information output at the current time is present and the audio information output at the previous time is signal processed according to the loss concealment method; And
If there is no audio information to be output at the current time, a loss concealment processing unit for processing the audio information output at the current time using the loss concealment method,
And a second sound generator for generating a sound. 10. The method of claim 9,
The compression / non-compression processing unit processes the audio information output at the current time using one of a compression method and an uncompression method depending on whether the audio signal exists or not and whether spikes are generated in the specific channel Wherein the stereo sound generating device is a stereo sound generating device. 11. The method of claim 10,
Wherein the compression / non-compression processing unit is configured to perform the compression / decompression processing on the basis of the difference between the jitter obtained up to now and the jitter obtained before the jitter is acquired when selecting the compression method or the non- Value or the ratio between the network jitter and the audio information divided on the basis of the interval. 10. The method of claim 9,
Wherein the loss concealment processing unit compares the number of consecutive signal processes performed in accordance with the loss concealment method with a reference frequency and outputs the result at the current time using one of a short loss concealment method and a long term loss concealment method And the audio information is signal-processed. 10. The method of claim 9,
Wherein the loss concealment processing unit further uses a linear decreasing scale function for eliminating a buzz sound when signaling the audio information output at the current time. 10. The method of claim 9,
The merging processor extracts sub audio information from the audio information output at the current time and superimposes each sub audio information on the audio information output at the previous time and processes the audio information output at the current time To-be-generated sound. The method according to claim 1,
The stereo sound generator determines whether or not the extracted audio information is received from a larger number of channels than the reference number. If it is determined that the extracted audio information is received from a larger number of channels than the reference number, Wherein the stereophonic sound is generated by mixing the extracted audio information using a concealment method. 8. The method of claim 7,
A receiving information storage unit for storing the audio information received by the specific channel and for combining audio information received by the specific channel with information about the specific channel and generating time information of the audio information received by the specific channel;
A decoding unit decoding audio information selected based on the network jitter among the stored audio information; And
A processing information backup unit for backing up the signal-processed audio information,
Further comprising: a stereo sound generator for generating stereo sound. The method according to claim 1,
A first gain adjuster for adjusting the first gain value of the audio information divided in units of intervals to equalize the volume when the signal processed audio information is divided in units of intervals according to whether an audio signal exists; And
The first gain value and the second gain value are reflected on each of the audio information segments divided in units of intervals, the second gain value of the audio information divided into the interval units including the echo component and the delay component is calculated, The second gain-
Further comprising: a stereo sound generator for generating stereo sound. 18. The method of claim 17,
Wherein the first gain adjustment unit comprises:
A group generating unit for generating audio information groups using information other than audio information including silence among the audio information divided in units of intervals;
A gain calculating unit for calculating a peak value for a volume for each audio information group and calculating a first gain value using the peak value and the reference value; And
The first gain value of the two audio information groups located before and after the audio information group in the time order is compared. If there is a difference value between the two first gain values, the first gain value of the audio information group located before is smoothed A gain comparison unit for adjusting a first gain value of an audio information group located later on the basis of the first gain value,
And a second sound generator for generating a sound. 19. The method of claim 18,
Wherein the stereo sound generating unit generates a mixing control coefficient using the energy correlation between the two audio information groups, and mixes the audio information using the mixing control coefficient. 18. The method of claim 17,
Wherein the second gain reflection unit calculates the second gain value using an all pass filter and a feedback comb filter. Estimating network jitter based on a reception time of the audio information currently received on the specific channel and a reception time of the audio information previously received on the specific channel when the audio information is received on the specific channel, A delay time calculating unit for calculating a packet delay time of the specific channel based on a reception time of the audio information and a reception time of the previously received audio information; And estimating the network jitter by comparing the packet delay time with a reference time. If the packet delay time is greater than or equal to the reference time, it is estimated that a spike has occurred in the specific channel. A jitter estimator for estimating that the specific channel is in a normal operation; A signal processor for selectively processing the currently received audio information based on the network jitter; And extracts audio information generated at the same time based on the generation time of each audio information among signal processed audio information received from different channels and mixes the extracted audio information to generate stereo sound in real time A stereo sound generating device including a stereo sound generating section;
An audio information generating unit for recording performances of different performers or musical instruments in different spaces at the same time to generate respective pieces of audio information;
An encoding unit encoding the generated audio information;
An audio information transmitting unit for dividing the encoded audio information into packets and transmitting the divided audio information to the jitter estimating unit and notifying the generation time of the encoded audio information to the stereo sound generating unit; And
A stereoscopic sound reproducing unit for reproducing the stereo sound in real time,
And a stereo sound reproduction system.

Images