KR100885449B1 - Apparatus for processing mix signal and method thereof - Google Patents

Abstract

The present invention relates to a method and apparatus for processing a mixed signal. The method for processing a mixed signal according to the present invention includes a step in which a remix information providing server coupled with a user terminal through a communication network receives remix information from the user terminal and receives the requested remix information. And transmitting to a user terminal, wherein the remix information includes a mix signal, a source signal included in the mix signal, a mix parameter for controlling the mix signal for each of the source signals, and an upmix for upmixing the mix signal. By providing at least one of a parameter, an icon or an emoticon for representing the source signal, there is an effect that the user can receive the desired remix information from the remix information providing server.

Mix signal, additional information, remix information providing server, billing

Description

Mix signal processing apparatus and method {APPARATUS FOR PROCESSING MIX SIGNAL AND METHOD THEREOF}

1 is a block diagram of an apparatus for encoding a first remix signal according to an embodiment of the present invention.

FIG. 2 is a detailed block diagram of the first remix signal encoding apparatus of FIG. 1 when using a stereo signal. FIG.

3 is a domain for processing a mix signal according to an embodiment of the present invention.

4 is a block diagram of a second remix signal encoding apparatus according to an embodiment of the present invention.

5 is a block diagram of a first remix signal decoding apparatus according to an embodiment of the present invention;

FIG. 6 is a detailed view of the first remix signal decoding apparatus of FIG. 5 when using a stereo signal. FIG.

7 is a block diagram of a second remix signal decoding apparatus according to an embodiment of the present invention.

8 is a block diagram illustrating a combination of a conventional encoding apparatus and a remix signal encoding apparatus according to an embodiment of the present invention.

9 is a block diagram of a second remix signal decoding apparatus according to an embodiment of the present invention used in combination with a conventional decoding apparatus.

10 is a diagram illustrating a remix information providing system according to an embodiment of the present invention.

11 is a block diagram of a remix information providing server according to an embodiment of the present invention.

12 is a module configuration diagram of a remix information providing server according to an embodiment of the present invention.

13 is a flowchart illustrating a method of providing a remix information by a remix information providing server according to an exemplary embodiment of the present invention.

14 is a flowchart illustrating a method in which a user terminal receives remix information according to an exemplary embodiment of the present invention.

15 is a flowchart illustrating a method for transmitting remix information to a remix information providing server by a user according to an exemplary embodiment of the present invention.

<Explanation of symbols for the main parts of the drawings>

103, 206, 403: additional information generation unit 105, 208, 406: additional information encoding unit

505, 607, 707: Remixing unit 503, 605, 703: Additional information decoding unit

1001: user terminal 1002: remix information providing server

The present invention relates to a mixed signal processing method and apparatus. To date, stereo signals are most commonly generated and most widely used by consumers. In recent years, multichannel signals have been increasingly used. However, there is a limitation that the mixed signal is processed in a channel signal unit, not in a source signal unit constituting the mix signal. Therefore, when processing the mixed signal in the channel signal unit, there is a problem that only a specific source signal constituting the mixed signal can not be processed independently. For example, while watching a movie, it is impossible to increase the volume of background music while keeping the volume of actors' voices constant.

An object of the present invention is to provide a signal processing method and apparatus that allows a user to control a mix signal in a unit of a source signal.

Another object of the present invention is to provide a signal processing method and apparatus for allowing a user to receive desired remix information from a remix information providing server.

Another object of the present invention is to provide a signal processing method and apparatus capable of uploading remix information generated by a user to a remix information providing server.

Another object of the present invention is to provide a signal processing method and apparatus capable of receiving or generating a mixed signal in which some source signals are excluded.

In order to achieve the above object, the present invention includes the step of the remix information providing server coupled with the user terminal through the communication network requesting the remix information from the user terminal and transmitting the requested remix information to the user terminal; The remix information may include a mix signal, a source signal included in the mix signal, a mix parameter for controlling the mix signal for each of the source signals, an upmix parameter for upmixing the mix signal, and the source signal. It provides a signal processing method comprising at least one of an icon or an emoticon for.

Hereinafter, with reference to the accompanying drawings, preferred embodiments of the present invention that can specifically realize the above object will be described.

The present invention provides an algorithm that can process a mix signal in an object signal unit. When the mix signal is processed in units of source signals, a wide variety of effects can be produced. For example, while watching a movie, it is possible to increase only the volume of the background music while keeping the volume of the actors' voices constant. In the present invention, an object refers to one or more sources (for example, a piano) constituting a mix signal, and a source signal refers to a signal corresponding to the object. Processing a source signal unit means that in processing a mix signal, characteristics (eg, localization, gain) related to a particular source signal constituting the mix signal are " individually ". It can be modified. By “individually” it is meant that modifying the properties associated with a particular object does not affect the properties of other source signals or only has a small effect that is difficult to perceive. For convenience of explanation, the following description will be given with a mixed signal as an example, but the present invention is not limited to the mixed signal. In the present invention, a small amount of side information is transmitted as well as a typical mix signal format (for example, PCM, MP3, MPEG-AAC). The additional information may be used to re-mix source signals included in the transmitted mixed signal.

1 is a block diagram of a first remixing signal encoder according to an embodiment of the present invention. The first remixing signal encoder includes an additional information generator 103 and an encoder 105.

Referring to FIG. 1, the additional information generator 103 generates additional information 104 using a typical mix signal 101 and a source signal 102 constituting the mixed signal. The mix signal 101 may be a mono, stereo and multi-channel audio signal. The source signal 102 may be some or all of the source signals constituting the mix signal 101. The additional information 104 refers to information used to process the mixed signal in a unit of a source signal. Examples of the additional information 104 may include a gain value and a subband power for each source signal. A detailed definition and generating method for the additional information 104 (??) is described in FIG. The present invention also includes generating the additional information 104 using only the source signal 102 constituting the mix signal. The encoding unit 105 encodes the generated side information 104 to generate an encoded side information signal 106. The mix signal 101 and the side information signal 106 are transmitted to a decoding apparatus.

FIG. 2 is a detailed block diagram of the first remix signal encoding apparatus of FIG. 1 when the mixed signal is a stereo signal. As described above, the mixed signal used in the present invention may be a mono, stereo and multichannel mixed signal, but for convenience, the mixed signal will be described based on the stereo signal 201.

및

는 상기 스테레오 신호를 구성하는 소스 신호들의 합으로 표현될 수 있다. 여기서, n은 타임 인덱스를 의미한다. 따라서, 상기 스테레오 신호(201)는 아래의 [수학식 1]과 같이 표현될 수 있다.The stereo signal 201

And

May be expressed as a sum of source signals constituting the stereo signal. Here, n means time index. Therefore, the stereo signal 201 may be expressed as Equation 1 below.

은 소스 신호들을 나타낸다.

및

는 각각의 소스 신호에 대한 진폭 패닝(amplitude panning) 및 게인(gain)을 결정하는 값이다. 모든

들은 서로 독립적이다. 상기 신호

는 모두 순수한 소스 신호이거나, 또는 순수한 소스 신호에 약간의 잔향(reverberation) 및 효과음 신호성분(sound effect signal components)을 포함할 수 있다. 예를 들면, 특정한 잔향 신호성분은 2개의 소스 신호, 즉, 왼쪽 채널로 믹스된 신호와 오른쪽 채널로 믹스된 신호로 표현될 수 있다.Here, I is the number of source signals included in the stereo signal,

Represents source signals.

And

Is a value that determines the amplitude panning and gain for each source signal. all

They are independent of each other. The signal

Are all pure source signals, or may include some reverberation and sound effect signal components in the pure source signal. For example, the specific reverberation signal component may be represented by two source signals, that is, a signal mixed with the left channel and a signal mixed with the right channel.

M

I)의 소스 신호들이 리믹스 되도록, 상기 소스 신호를 포함하는 스테레오 신호를 수정하는 것이다. 상기 소스 신호들은 서로 다른 게인 팩터들을 가지면서 스테레오 신호로 믹스된다. 리믹스된 스테레오 신호는 아래의 [수학식 2]와 같이 표현될 수 있다. The object of the present invention is M (0

M

The stereo signal including the source signal is modified such that the source signals of I) are remixed. The source signals are mixed into the stereo signal with different gain factors. The remixed stereo signal may be expressed as Equation 2 below.

및

는 리믹스되는 M개의 소스 신호들에 대한 새로운 게인 팩터들이다. 상기

및

는 디코더 단에서 제공될 수 있다. 이 경우에, 부가 정보 생성부(205)는 스테레오 신호(201) 및 M개의 소스 신호(202)를 이용하여 부가 정보(206)를 생성할 수 있다. here,

And

Are new gain factors for the M source signals to be remixed. remind

And

May be provided at the decoder stage. In this case, the additional information generator 205 may generate the additional information 206 using the stereo signal 201 and the M source signals 202.

에 대한 접근 없이, [수학식 1]로 표현되는 통상적인 스테레오 신호가 주어지는 경우에, [수학식 2]로 표현되는 리믹스된 신호를 지각적으로 모방하는 것을 목적으로 한다.As described above, an object of the present invention is to remix the stereo signal in the unit of the source signal when given a conventional stereo signal and some additional information. As in the present invention, it is not possible to completely generate the remixed mixed signal represented by Equation 2 from the mixed signal represented by Equation 1 using a very small amount of additional information. Thus, the present invention provides the respective source signals

It is aimed to perceptually mimic the remixed signal represented by [Equation 2], given a conventional stereo signal represented by [Equation 1] without access to.

및

로 표시된다. 여기서 k는 서브밴드 신호들의 시간 인덱스(time index)이다. 유사하게, M개의 입력 소스 신호들(202)의 서브밴드 신호들(205)은

,

,...,

로 표시된다. 명료한 표현을 위해, 서브밴드(주파수) 인덱스를 사용하지 않았다.Referring to FIG. 2, a stereo signal 201 and M source signals 202 included in the stereo signal 201 are input to a first remix signal encoding apparatus. The stereo signal 201 may be delayed to some extent to be synchronized with the side information and used directly as an output signal. To generate the additional information, the stereo signal 201 and the source signals 202 are decomposed into subband-specific signals 204 and 205 in the time-frequency domain through the filter bank 203. That is, the stereo signal 201 and the source signal are processed in the time-frequency domain, which will be described later with reference to FIG. 3. The subband-specific signal 204 is similarly processed at the center frequency of each subband. At a particular frequency, the subband pair 204 of the stereo signal 201

And

Is displayed. Where k is the time index of the subband signals. Similarly, the subband signals 205 of the M input source signals 202 are

,

, ...,

Is displayed. For clarity, no subband (frequency) index is used.

를 생성한다. 또한, 상기 부가정보생성부(206)는 스테레오 신호(201)의 서브밴드 쌍(204)을 이용하여, 서브밴드별로 게인 팩터

및

를 생성한다. 상기 게인 팩터

및

는 외부에서 직접 주어질 수 있다. 상기 서브밴드별 숏-타임 서브밴드 파워 및 게인 팩터를 이용하여 서브밴드별 부가 정보(207)가 생성된다. 상기 부가정보생성부(206)는 상기 숏-타임 서브밴드 파워 및 게인 팩터들 이외에 상기 스테레오 신호에 관련된 다른 정보를 부가 정보(207)로 생성할 수 있다. 인코딩부(208)는 상기 서브밴드별 부가정보(207)를 이용하여 부호화된 부가 정보 신호(209)를 생성한다. Given the subband signals 205 of the source signals 202, the side information generator 206 may perform short-time subband power for each subband,

Create In addition, the additional information generator 206 uses a subband pair 204 of the stereo signal 201 to obtain a gain factor for each subband.

And

Create The gain factor

And

Can be given directly from the outside. Subband-specific additional information 207 is generated using the short-time subband power and gain factor for each subband. The additional information generator 206 may generate other information related to the stereo signal as additional information 207 in addition to the short-time subband power and gain factors. The encoder 208 generates an encoded side information signal 209 using the subband side information 207.

및

는 고정적이 될 것이다. 만일

및

가 시간 k에 따라 가변적이라면, 상기 게인 팩터들은 시간의 함수로 생성될 것이다. 상기 게인 팩터들은 직접 양자화 및 부호화되지 않고, 먼저 양자화 및 부호화에 더 적합한 다른 값들로 전환될 수 있다. 또한,

는 스테레오 신호(201)의 서브밴드 파워에 상대적인 값으로 정규화될 수 있다. 이것은 스테레오 신호를 효율적으로 부호화하기 위해 통상적인 인코딩 장치가 이용되는 경우에, 본 발명을 상대적으로 변화에 강하도록 만들어준다. 예를 들면,

및

는 아래의 [수학식 3]으로 표현되는 게인 및 데시벨(dB) 단위의 레벨차로 전환되어 전송될 수 있다. For many stereo signals 201, the gain factor

And

Will be fixed. if

And

If is variable over time k, the gain factors will be generated as a function of time. The gain factors are not directly quantized and coded, but may first be converted to other values more suitable for quantization and coding. Also,

Can be normalized to a value relative to the subband power of the stereo signal 201. This makes the present invention relatively resistant to changes when conventional encoding devices are used to encode stereo signals efficiently. For example,

And

May be converted into a level difference in units of gain and decibel (dB) expressed by Equation 3 below, and then transmitted.

는 부가 정보로서 직접 부호화되는 것이 아니라, 아래의 [수학식 4]로 표현되는 스테레오 신호에 상대적으로 정의된 값으로 변환되어 전송될 수 있다. Also,

Is not directly encoded as additional information, but may be converted into a value defined relative to the stereo signal represented by Equation 4 below and transmitted.

는 아래의 [수학식 5]와 같이 계산될 수 있다. To produce short-time subband power, the present invention uses single-pole averaging. In other words,

May be calculated as shown in Equation 5 below.

Here, α∈ [0,1] determines the time-constant of the estimation window (estimation window) which decreases exponentially as shown in Equation 6 below.

는 서브밴드 샘플링 주파수를 나타낸다. 예를 들면, T=40 ms를 이용할 수 있다. 이하에서,

는 숏-타임 평균(short-time averaging)을 나타낸다. 만일

및

가 주어지지 않는다면, 상기

및

는 부가정보생성부(206)에서 생성될 필요가 있다.

이므로,

는 아래의 [수학식 7]과 같이 계산된다.here,

Denotes a subband sampling frequency. For example, T = 40 ms can be used. In the following,

Denotes short-time averaging. if

And

If is not given,

And

Needs to be generated in the additional information generation unit 206.

Because of,

Is calculated as shown in Equation 7 below.

는 아래의 [수학식 8]과 같이 계산된다.Similarly,

Is calculated as shown in Equation 8 below.

3 illustrates a domain for processing a mix signal according to an embodiment of the present invention. As described above, the mix signal and the side information are processed as subband-specific signals in the time-frequency domain as shown in FIG. Subband-specific signals in the time-frequency domain are perceptually derived. For example, a signal for each subband may be generated by using a short time fourier transform (STFT) having a sine wave analysis window and a sine analysis and synthesis window having a length of about 20 ms. In this case, the STFT coefficients may be grouped such that one group has a bandwidth that is about twice the equivalent rectangular bandwidth (ERB).

4 is a block diagram of a second remix signal encoding apparatus according to an embodiment of the present invention. The second remix signal encoding apparatus includes a downmixing unit 402, an additional information generating unit 403, and an encoding unit 406.

Referring to FIG. 4, the downmixing unit 402 generates a sum signal 404 by adding a plurality of source signals 401. Unlike the first remix signal encoding apparatus, the second remix signal encoding apparatus transmits the sum signal 404 instead of the stereo signal. The additional information generator 403 generates the additional information 405 using the source signals 401. The additional information 405 includes subband power and gain factor corresponding to each source signal. In addition, the additional information 405 may include a parameter corresponding to a delay in the remixing unit. Similar to the first remix signal encoding apparatus, the additional information 405 may be converted into another value more suitable for quantization and encoding and transmitted. The encoding unit 406 generates an encoded side information signal 407 using the generated side information 405. The generated sum signal 404 and the side information signal 407 are transmitted to the decoding apparatus. The present invention also includes an encoding apparatus that does not have a downmixing unit 402. In this case, the source signals 401 are not converted to the sum signal 404, and each source signal 401 is transmitted directly.

5 is a block diagram of an apparatus for decoding a first remix signal according to an embodiment of the present invention. The first remix signal decoding apparatus includes a marital information decoding 503 and a remixing unit 505.

Referring to FIG. 5, the channel-specific mix signal 501 and the additional information signal 502 are input to the first remix signal decoding apparatus. The channel-specific mix signal 501 may be a mono, stereo or multichannel mix signal. The side information decoding 503 (??) decodes the side information signal 502 to generate side information 504. The additional information 504 includes a gain factor and a subband power of source signals included in the transmitted mixed signal 501. The remixing unit 505 may input control information 506 directly provided by the user. The remixing unit 505 generates the remixed mix signal 507 by using the channel-specific mix signal 501, the transmitted additional information 504, and the control information 506 directly provided by the user. A detailed description of the method for generating the remixed mix signal will be given later with reference to FIG. 6. The remixed mix signal 507 may be generated as a mix signal having the same number of channels as the number of channels of the transmitted mix signal, or as a mixed signal having a different number of channels.

FIG. 6 is a detailed diagram of the first remix signal decoding apparatus of FIG. 5 when using a stereo signal. As described above, the transmitted mixed signal may be a mono, stereo, and multichannel mixed signal, but is described with reference to the stereo signal 601 for convenience.

및

로 표현된다. 부가(??)정보디코딩(605)는 전송된 부가 정보 신호(602)를 복호화하여, 서브밴드별 부가 정보(606)를 생성한다. 또한, 리믹싱부(607)에 사용자가 제공하는 제어 정보(608)가 입력될 수 있으며, 상기 제어 정보(608)는 서브밴드별로 제공될 수 있다. 전술한 것처럼, 상기 부가 정보(606)는 리믹스 될 M개의 소스 신호에 대한 서브밴드별 게인 팩터(

및

) 및

으로 표현되는 서브밴드 파워를 포함한다. 리믹싱부(607)는 서브밴드별로 생성된 스테레오 신호(604), 전송된 부가 정보(606) 및 사용자가 제공한 제어 정보(608)를 이용하여, 서브밴드별로 리믹스된 스테레오 신호(609),

및

를 생성한다. 상기 리믹스된 스테레오 신호(609)를 생성하는 방법은 아래에서 더욱 상세하게 기술된다. 상기 리믹스된 스테레오 신호(609)는 역필터뱅크(610)를 통해 시간 도메인(time domain)의 스테레오 신호(611),

및

로 변환된다. Referring to FIG. 6, the stereo signal 601 is decomposed into a subband-specific signal 604 in the time-frequency domain through the filter bank 603. As shown in FIG. 6, the subband-specific signal 604 at a particular frequency

And

It is expressed as The additional information decoding 605 decodes the transmitted additional information signal 602 to generate additional information 606 for each subband. In addition, control information 608 provided by a user may be input to the remixing unit 607, and the control information 608 may be provided for each subband. As described above, the additional information 606 is a subband-specific gain factor for M source signals to be remixed.

And

) And

It includes the subband power represented by. The remixing unit 607 uses the stereo signal 604 generated for each subband, the additional information 606 transmitted, and the control information 608 provided by the user, and the stereo signal 609 remixed for each subband,

And

Create The method of generating the remixed stereo signal 609 is described in more detail below. The remixed stereo signal 609 is a stereo signal 611 in the time domain (time domain) through an inverse filter bank 610,

And

Is converted to.

는 서브밴드별 소스 신호

로 교체된다. 즉, 서브밴드별 스테레오 신호(604)는 아래의 [수학식 9]과 같이 표현될 수 있다. A method of generating the remixed stereo signal 609 by the remixing unit 607 is as follows. [Equation 1] and [Equation 2] are also valid for the subband signals 604 and 609. In this case, the source signal

Is the subband-specific source signal

Is replaced by. That is, the subband stereo signal 604 may be expressed as Equation 9 below.

The stereo signal 609 remixed for each subband may be expressed by Equation 10 below.

및

가 주어지면, 아래의 [수학식 11]과 같이 서로 다른 게인들을 가지는 서브밴드별 스테레오 신호가 상기 서브밴드별 스테레오 신호(604)의 선형 조합으로 추정될 수 있다.In order to generate the remixed stereo signal 609, least squares estimation may be used. Per-band stereo signal 604,

And

Given by Equation 11 below, a stereo signal for each subband having different gains may be estimated as a linear combination of the stereo signals 604 for each subband.

,

,

및

는 가중 팩터들(weighting factors)이다. 이때, 생성되는 추정 에러(estimation error)는 아래의 [수학식 12]과 같이 정의될 수 있다.here,

,

,

And

Is the weighting factors. In this case, the generated estimation error may be defined as shown in Equation 12 below.

,

,

및

는 평균제곱오차(mean square error),

및

가 최소가 되도록 서브밴드별로 생성될 수 있다. 이때, 추정 에러,

및

가

및

에 직교(orthogonal)될 때, 상기 평균제곱오차가 최소가 된다는 것을 이용할 수 있다. 생성되는

및

는 아래의 [수학식 13]과 같이 표현될 수 있다.The weighting factors,

,

,

And

Is the mean square error,

And

May be generated for each subband such that is minimized. In this case, the estimation error,

And

end

And

When orthogonal to, it can be used that the mean square error is minimized. Generated

And

May be expressed as Equation 13 below.

,

및

는 직접 생성될 수 있지만,

및

은 전송된 부가 정보(606)(예를 들면,

,

,

) 및 사용자가 제공하는 제어 정보(608)(예를 들면, 게인 팩터

및

)를 이용하여, 아래의 [수학식 14]와 같이 생성될 수 있다. here,

,

And

Can be generated directly,

And

Is transmitted additional information 606 (e.g.,

,

,

) And control information 608 provided by the user (e.g., gain factor

And

) Can be generated as shown in Equation 14 below.

및

가 아래의 [수학식 15]와 같이 생성될 수 있다. Similarly,

And

May be generated as shown in Equation 15 below.

및

는 아래의 [수학식 16]와 같이 표현될 수 있다. here,

And

May be expressed as Equation 16 below.

If the phases of the stereo signals 606 are coherent or nearly synchronized with each other, a value expressed by Equation 17 below approaches 1.

In this case, the weights may be expressed as Equation 18 below.

및

)를 이용하여 실제로 리믹스된 스테레오 신호와 유사하게 들린다. 따라서, 소스 신호를 별도로 생성하지 않고 상기

,

,

및

를 이용하여 믹스 신호를 리믹싱 함으로써, 마치 별도로 생성된 소스 신호에p

,

,

및

를 적용하여 리믹싱을 수행한 효과를 가질 수 있다. The stereo signal 609 remixed for each subband generated as described above is converted into a time-domain stereo signal 611 through the inverse filter bank 610 as described above. The stereo signal 611 is a control information provided by the user (

And

It sounds similar to the stereo signal actually remixed with). Thus, without generating a source signal separately,

,

,

And

By remixing the mix signal using the

,

,

And

It can have the effect of performing remixing by applying.

So far, the focus has been on remixing two-channel stereo signals. However, as described above, the present invention is not limited to stereo signals, but may be extended to remixing multichannel mixed signals, for example, 5.1 channel mixed signals. Those skilled in the art can remix multichannel mix signals, similar to the stereo signals described herein. In this case, Equation 11 may be written as Equation 19 below.

Optionally, certain channels can be left without remixing. For example, for a 5.1 surround channel, it is possible to apply remixing only to the front channel without modifying the two back channels. In this case, a two or three channel remix algorithm is applied to the front channel.

7 is a block diagram of a second remix signal decoding apparatus according to an embodiment of the present invention. The second remix signal decoding apparatus includes a couple information decoding 703, an additional information integrating unit 705, and a remixing unit 707.

Referring to FIG. 7, the sum signal 701 of the source signals and the additional information signal 702 are input to the second remix signal decoding apparatus. Side information decoding 703 decodes side information signal 702 to generate side information 704. The additional information 704 includes a gain factor, a delay constant, a subband power, and the like. The additional information integrator 705 separates the sum signal 701 into a plurality of source signals 706 by using the additional information 704. The remixer 707 may generate the remixed mix signal 709 using the source signals 706. In this case, the remixing unit 707 may generate the remixed mix signal 709 using the parameter 704 transmitted as additional information. In addition, the remixing unit 707 may selectively generate the remixed mix signal 709 using the control information 708 provided by the user.

8 is a block diagram illustrating a combination of a conventional encoding apparatus and a remix signal encoding apparatus according to an embodiment of the present invention. The mixed signal 801 may be encoded by the conventional encoding apparatus 803 and converted into the encoded mixed signal 805. The mix signal 801 may be a channel-specific signal or a source signal. The conventional encoding device 803 includes not only conventional encoding devices such as AAC, MP3 encoder, etc., but also encoding devices to be developed in the future. The remix signal encoding apparatus 804 according to the present invention generates the additional information signal 806 using the mixed signal 801 and the source signal 802 included in the mixed signal. The multiplexer 807 generates the bitstream 808 using the encoded mix signal 805 and the side information signal 806. As described above, the additional information signal 806 may be inserted into an auxiliary data area in a conventional mix signal format to be compatible with conventional devices.

9 is a block diagram of a combination of a conventional decoding device and a remix signal decoding device according to an embodiment of the present invention. The demultiplexer 902 separates the encoded mix signal 903 and the side information signal 904 from the transmitted bitstream 901. The conventional decoding device 905 then decodes the encoded mix signal 903 to produce a mix signal 907 that can be used in the remix signal decoding device 907 according to the present invention. The conventional decoding device 905 includes not only conventional encoding devices such as AAC, MP3 decoder, etc., but also encoding devices to be developed in the future. The mix signal 907 may be a channel-specific signal or a source signal. The remix signal decoding apparatus 907 according to the present invention converts the mixed signal 907 into the remixed mixed signal 908 using at least one of the additional information signal 904 and the control information 909 provided by the user. can do.

The user may be provided with the remix information from a server providing the remix information (hereinafter, referred to as a remix information providing server). The remix information providing server may provide a modified mix signal in which some of the source signals included in the mix signal are excluded. That is, when the mix signal is composed of a plurality of source signals, the remix information providing server may provide a modified mix signal in which one or more source signals of the source signals constituting the mix signal are excluded. Hereinafter, a mix signal in which some source signals are excluded from the source signals included in the mix signal is called a modified mix signal to distinguish the original mix signal. When the remix information providing server provides the modified mix signal, the user may first receive and listen to the modified mix signal before receiving the mixed signal. If the user wants to hear the original mix signal after the source signal is not excluded after listening to the modified mix signal, the remix information providing server may request to transmit the original mix signal including all the source signals.

The remix information providing server may provide the remix information to the user for free, but may also provide some or all of the remix information to the user for a fee. When the remix information providing server provides the remix information to the user for a fee, the user may receive the remix information after paying the fee for using the remix information. The remix information providing server may provide the modified mix signal free of charge and the original mix signal to the user for a fee. 10 to 15 illustrate how a user receives and uses a modified mix signal in which some source signals of a mix signal are missing from the remix information providing server, and the remix information providing server charges the remix information for a fee. If so, the payment method will be described.

10 is a diagram illustrating a remix information providing system according to an exemplary embodiment of the present invention. Referring to FIG. 10, a remix information providing system includes a user terminal 1001, a remix information providing server 1002, a financial institution 1003, and a mobile communication provider 1004. The user needs the mix signal or the source signal and additional information about it to remix the source signal. The additional information includes an encoder mix parameter generated by the encoding apparatus, a blind mix parameter generated by the decoding apparatus, a user mix parameter generated using control information input by the user, and the like. The user may adjust the source signal included in the remixed mix signal by adjusting an icon representing each source signal through a user interface (not shown) included in the user terminal 1001. The user may change the size, position, or sound characteristics of the source signal by adjusting an icon representing the source signal. The icon may include an emoticon expressing a human emotion, a picture or a caricature of a singer or a musical instrument, a picture or a picture of an instrument, or the like. Hereinafter, remixing the mix signal by source signal such as a mix signal, a source signal of each sound source, a mix parameter for controlling the mix signal for each source signal, an upmix parameter for upmixing the mix signal, and an icon for representing the source signal All information necessary for the purpose is called remix information.

The user connects to the remix information providing server 1002 using the user terminal 1001. The user terminal 1001 may perform wired or wireless communication with other user terminals such as a user PC such as a notebook or a desktop, a mobile terminal such as a mobile phone, a personal digital assistant (PDA), a portable multimedia player (PMP), and may reproduce a mixed signal. Device. The user terminal 1001 selects the desired remix information from the remix information providing server 1002. The remix information providing server 1002 retrieves the remix information selected by the user from a database (not shown) and extracts it. The remix information providing server 1002 transmits the extracted remix information to the user terminal 1001. The remix information providing server 1002 may provide a modified mix signal in which a certain source signal is excluded from a specific section of the mix signal or a source signal included in the mix signal. For example, if a user wishes to listen to piano trio music called Tchaikovsky's 'Memories of the Great Artist' consisting of a piano, a violin and a cello combination, the user may first listen to the modified mix signal before listening to the mix signal. . When a user requests the remix information providing server 1002 for a modified mix signal for a piano trio song called 'Memories of the Great Artist', the remix information providing server 1002 causes the 'great artist's memory' to exclude the cello signal. The modified mix signal can be extracted from the database and sent to the user. If the remix information providing server 1002 does not have a modified mix signal for 'memory of the great artist' stored in the database, the source signal included in the mix signal, i.e., the constant source signal among the piano, violin, and cello signals, may be generated. There are many other variations to the mix signal. The remix information providing server 1002 may generate a modified mixed signal by excluding a specific source signal randomly or according to a user's request from among the source signals included in the mixed signal. When the user requests to generate a modified mix signal except for the piano signal among the source signals included in the mix signal, the remix information providing server 1002 generates a modified mix signal including only the violin and cello signals to the user. Can be sent to. The user receives the modified mix signal from the remix information providing server 1002 and listens to it. If the user wants to listen to the mixed signal after listening to the modified mix signal, the user requests the remix information providing server 1002 to transmit a mix signal for the memories of the great artist. The user may listen to the modified mix signal or the mix signal received from the remix information providing server 1002 using the user terminal 1001. The user may use the mixed signal or the like as the user terminal 1001, for example, a call sound of the mobile terminal or a ring tone of the mobile terminal. In addition, the user may use the mix signal as the background music of the homepage. In addition, the user may receive additional information about the mixed signal or the modified mixed signal from the remix information providing server 1002 and adjust the mix signal for each source signal to generate the remixed mixed signal.

In some cases, the remix information providing server 1002 may provide all or part of the remix information for a fee. Basic remix information is provided free of charge, and remix information that is not provided may be provided for a fee. If the use of the remix information is paid, the remix information providing server 1002 requests the user to make a payment. For payment processing, the remix information providing server 1002 may request authentication information from the user terminal 1001 to confirm whether the user is a valid user. The user inputs user information for authentication to the user terminal 1001 in response to a request of the remix information providing server 1002. The remix information providing server 1002 may receive the user information input by the user from the user terminal 1001 and authenticate whether the user is a valid user.

When the user wants to pay the fee for using the remix information using the card, the remix information providing server 1002 transmits the user information to a financial institution 1003 such as a card company or a bank and requests confirmation of user validity. The financial institution 1003 checks whether the user is a valid user who has subscribed to the financial institution by using the user information, and transmits the result to the remix information providing server 1002. When the user is a valid user, the remix information providing server 1002 requests the financial institution 1003 to pay for the remix information fee. The financial institution 1003 withdraws the remix information fee from the user's account and notifies the remix information providing server 1002 that payment has been completed. The financial institution 1003 transfers the withdrawal remix information fee to the account of the remix information providing server 1002 in real time or at regular intervals. When the user wants to pay for the remix information usage fee by the account transfer method, the financial institution 1003 notifies the user that the remix information usage fee has been deposited in the account of the remix information providing server 1002. After the remix information providing server 1002 is notified of the settlement completion from the financial institution 1003, the remix information providing server 1002 extracts the remix information selected by the user and transmits the remix information selected by the user to the user terminal 1001. The user terminal 1001 may receive the remix information from the remix information providing server 1002 and listen to it as it is, but the user may modify and use the remix information. In this case, the user terminal 1001 may include a signal decoding apparatus for processing the remixing. The user may receive the source signal, the mix signal, and additional information together or separately, and remix the source signal using the user terminal 1001. When the user terminal 1001 receives only the mix signal from the remix information providing server 1002, the user terminal 1001 directly generates additional information about the mixed signal, and uses the generated additional information to generate the mixed signal as a source signal. You can also adjust it. The user may adjust the mix signal for each source signal to generate a remixed mix signal and upload the remixed mix signal to the remix information providing server 1002. In addition, the user may generate an icon representing the source signal or modify the icon received from the remix information providing server 1002 and upload it to the remix information providing server 1002. In this case, the remix information includes a remixed mix signal uploaded by a user and an icon uploaded or modified and uploaded. The remix information providing server 1002 stores the remixed mix signal uploaded by the user using the user terminal 1001 in a database (not shown). The remix information providing server 1002 may transmit the remixed mix signal to the second user terminal for a fee or free when another second user requests to transmit the remixed mix signal. The user who uploaded the remix information providing server 1002, the remixed mix signal or icon, etc. to the remix information providing server 1002, and the second user who wants to use the remixed mix signal or icon, etc., mutually according to a predetermined agreement. Remix information fee can be paid. When the remix information providing server 1002 transmits the remixed mix signal to the second user for a fee, the remix information providing server 1002 may request a payment from the second user in the same manner as described above. The remix information providing server 1002 may pay a predetermined amount of the remix information usage fee paid by the second user to the user who uploaded the remixed mix signal or icon to the remix information providing server 1002.

The user may pay for the remix information using the user terminal 1001 such as a mobile phone. When the user inputs the user terminal 1001 number such as a mobile phone to the remix information providing server 1002, the remix information providing server 1002 transmits the user terminal 1001 number of the user to the mobile communication provider 1004, and the user Request an authentication number for authentication. The mobile communication provider 1004 transmits an authentication number to the remix information providing server 1002 and the user terminal 1001 in response to a request of the remix information providing server 1002. The user inputs the authentication number transmitted to the user terminal 1001 to the remix information providing server 1002 using the user terminal 1001. The remix information providing server 1002 determines that the user is a valid user when the authentication number input by the user matches the authentication number received from the mobile communication provider 1004. If the remix information providing server 1002 determines that the user is valid, the remix information providing server 1002 requests the mobile communication provider 1004 to pay the remix information fee. The mobile service provider 1004 adds the remix information fee to the user terminal 1001 fee. When the payment for the remix information fee is completed, the mobile communication provider 1004 notifies the remix information providing server 1002 that the payment is completed. After receiving the notification that the payment is completed, the remix information providing server 1002 transmits the remix information selected by the user to the user terminal 1001. The user generates the remixed mix signal using the remix information transmitted to the user terminal 1001. The user may generate a unique mix signal by adjusting the size, position, or tone of the source signal using the user terminal 1001. The user may upload the remixed mix signal to the remix information providing server 1002 using the user terminal 1001. As described above, the second user may download and use the remixed mix signal uploaded by the other users to the remix information providing server 1002.

11 is a block diagram of a remix information providing server according to an exemplary embodiment of the present invention. Referring to FIG. 11, the remix information providing server 1002 includes a memory system 1001, a central processing unit (CPU) 1104, an input / output device 1108, and a communication device 1109. The central processing unit 1104 includes an arithmetic logic unit (ALU) 1105 for performing calculations, a register 1106 for temporary storage of data and instructions, and a controller for controlling the operation of the remix information providing server 1002. 1107). The memory system 1001 generally includes a high speed main memory 1102 in a form of a storage medium such as random access memory (RAM) or read only memory (ROM), and a floppy disk, hard disk, tape, CD-ROM, flash memory, or the like. A secondary memory 1103 in the form of a long-term storage medium and an apparatus for storing data using electrical, magnetic, optical or other storage media. The main memory 1102 may include a video display memory that displays remix information such as a list or icon of a mixed signal or a modified mixed signal through a display device. The input / output device 1108 may include a keyboard, mouse, for example, a physical transducer such as a touch screen or a microphone. The communication device 1109 may be a device such as a network interface or an RF transmission / reception antenna for performing wired or wireless communication with the user terminal 1001.

12 is a module configuration diagram of a remix information providing server according to an embodiment of the present invention. Referring to FIG. 12, the remix information providing server 1002 includes a transmit / receive processing module 1213, a mix signal, and a modified signal for processing communication with a user terminal 1001, a financial institution 1003, and a mobile communication service provider 1004. Database management module 1214 that stores and manages remix information, including mix signals, etc., mix signal transform module 1215 that generates modified mix signals, and if the remix information is paid, authenticates the user for validity and handles payment. It may include a billing processing module 1216 to. The overall configuration of the remix information providing server through these application modules is as follows. The remix information providing server can use various operating systems (OS) as the OS of the system. The OS provides a high level command to an application program interface (API) 1201 to control the operation of each application module 1204. The remix information providing server includes a high level instruction processing unit 1202 that identifies each corresponding application module 1204 according to the high level instructions provided from the API 1201, decodes the high level instructions, and provides them to the corresponding places. do. The application module controller 1203 controls the operation of the application module 1204 according to the instructions provided from the high level instruction processor 1202. That is, the high level instruction processing unit 1202 identifies whether there is an application module 1204 corresponding thereto according to the high level instruction provided through the API 1201, and if the corresponding application module 1204 exists, The decoded command is recognized by the application module 1204 and transmitted to the corresponding mapping unit or the message transmission is controlled. Accordingly, the application module control unit 1203 is a mapping unit 1205, 1207, 1209, 1211 to the transmission / reception processing module 1213, the database management module 1214, the mix signal transformation module 1215, and the charging processing module 1216. ) And interface units 1206, 1208, 1210, and 1212, respectively.

The transmission / reception processing module mapping unit 1205 receives a high-level command for communicating with the user terminal 1001, the financial institution 1003, or the mobile communication provider 1004 from the high level command processing unit 1202. Mapping is performed at the device level that can be processed by the processing module 1213, and provided to the transmission / reception processing module 1213 through the transmission / reception processing module interface unit 1206. The database management module mapping unit 1207 and the database management module interface unit 1208 store the mix signal or the modified mix signal, and extract the mix signal or the modified mix signal request from the user terminal 1001. The database management module mapping unit 1207 receives a high level command for using the database management module 1214 from the high level command processing unit 1202 and maps it to a device level command, and the database management module interface unit 1208. It is provided to the database management module 1214 through. The mix signal modification module 1215 generates a modified mix signal by excluding a constant source signal included in the mix signal when the mix signal needs to be modified, such as when a modified mix signal is requested from a user. The mixed signal transform module mapping unit 1209 receives a high level command applied through the high level command processor 1202 and maps the modified signal transform module 1215 into a device level command that can be recognized by the modified signal transform module 1215. Device level instructions are provided to the mix signal modifying module 1215 via the mix signal modifying module interface 1210. The billing processing module 1216 is a part that checks whether the user is a valid user and charges a remix information fee when the user is a valid user. In addition, when the second user wants to use the remixed mixed signal generated and uploaded by other users, the charging processing module 1216 may remix between the remix information providing server 1002, the user who uploaded the remixed mixed signal, and the second user. It is the part that handles the information fee. The charging processing module mapping unit 1211 receives the high level command applied through the high level command processing unit 1202 and maps the device level command to be recognized by the charging processing module 1216. The device level command is provided to the charging processing module 1216 through the charging processing module interface 1212.

13 is a flowchart illustrating a method of providing a remix information by a remix information providing server according to an exemplary embodiment of the present invention. Referring to FIG. 13, a user accesses the remix information providing server 1002 using the user terminal 1001. The remix information providing server 1002 receives the remix information from the user terminal 1001 (step 1301). The remix information providing server 1002 may transmit the remix information selected by the user to the user terminal 1001 free of charge, or may transmit it for a fee. The remix information providing server 1002 determines whether the remix information selected by the user is paid or free (step 1303). The remix information providing server 1002 requests payment from the user when the remix information selected by the user is paid (step 1305). For the settlement, the remix information providing server 1002 may perform user authentication to confirm whether the user is a valid user. The remix information providing server 1002 may request the user terminal 1001 to input user authentication information for user authentication, such as a credit card number, an account number, a social security number, and a password. When the user inputs the user authentication information using the user terminal 1001, the remix information providing server 1002 transmits the user authentication information input by the user to the financial institution 1003, the mobile communication provider 1004, and the like. Ask for confirmation. The financial institution 1003 or the mobile communication provider 1004 retrieves user authentication information transmitted from the remix information providing server 1002 from a database (not shown) and notifies the remix information providing server 1002 of the result. If the remix information providing server 1002 determines that the user is valid, the remix information providing server 1002 transmits the payment information to the financial institution 1003 or the mobile communication provider 1004 and requests a payment. The remix information providing server 1002 may include a mix signal modifying unit (not shown). When the payment is completed, the remix information providing server 1002 determines whether the remix information selected by the user is a modified mix signal in which some source signals are excluded (step 1309). If the remix information providing server 1002 is a modified mix signal that the user desires, the mix signal is modified using the mix signal modifying unit or extracts a pre-stored modified mix signal (step 1311). The remix information providing server 1002 provides the modified mix signal to the user terminal 1001 (step 1315). The remix information providing server 1002 extracts previously stored remix information when the remix information selected by the user is not a modified mix signal (step 1313). The remix information providing server 1002 provides the extracted remix information to the user terminal 1001 (step 1315).

14 is a flowchart illustrating a method in which a user terminal receives remix information according to an exemplary embodiment of the present invention. Referring to FIG. 14, a user accesses the remix information providing server 1002 using the user terminal 1001 (step 1401). The user selects desired information from the remix information provided by the remix information providing server 1002 using the user terminal 1001 (step 1403). As described above, the remix information includes a mix signal, a modified mix signal, a source signal, additional information, an icon representing the source signal, an emoticon, and the like. In addition, the remix information includes remix information uploaded by other users to the remix information providing server 1002. The user terminal 1001 determines whether the remix information selected by the user is paid (step 1405). If the selected remix information is paid, the user pays the fee for using the remix information (1407). The fee for using the remix information may vary depending on the type, capacity, bit rate, and the like of the remix information. For example, when a user wants to receive only a specific source signal, the user may pay less fee than when receiving a mix signal including the source signal. When the user receives additional information about the mixed signal or the source signal not included in the mixed signal, the amount to be paid by the user may vary. When the payment is completed, the user receives the remix information from the remix information providing server 1002 (step 1409).

15 is a flowchart illustrating a method of transmitting, by a user, remix information to a remix information providing server according to an exemplary embodiment of the present invention. Referring to FIG. 15, a user receives remix information using the user terminal 1001. The remix information may be transmitted from an encoding device such as a remix information providing server 1002. The user terminal 1001 may include a remix renderer (not shown) for remixing a source signal included in the mix signal using the remix information and the control information. When the user terminal 1001 includes the remix renderer, the user may directly generate the remix information using the mix signal, the source signal, and additional information about the same using the user terminal 1001. The user may modify the remix information stored in the user terminal 1001 using the remix renderer to generate new remix information (step 1501). That is, the user may generate the remixed mixed signal by adjusting the size, position, and tone of the source signal using the remix information that the user has. In addition, the user may modify an icon or emoticon representing the source signal, or directly create an icon or the like. The user transmits the generated remix information to the remix information providing server 1002 and uploads it (step 1503). The remix information providing server 1002 stores the new remix information uploaded by the user in a database, and if the second user wants it, extracts it from the database and transmits it to the second user terminal.

As described above, the signal processing method according to the present invention can adjust the mix signal for each source signal.

In addition, in the signal processing method according to the present invention, the user may receive and use the remix information from the remix information providing server.

In addition, the signal processing method and apparatus according to the present invention may be provided with a modified mix signal in which a specific source signal is excluded from the mix signal.

In addition, in the signal processing method according to the present invention, a user may generate or modify remix information and upload it to the remix information providing server.

Claims (12)

In the signal processing method performed by the remix information providing server connected to the user terminal through a communication network, Receiving a request for remix information from the user terminal; And If the remix information exists, transmitting the remix information to the user terminal, The remix information includes at least one of at least one source signal, a mix signal in which the at least one source signal is mixed, and a mix parameter for controlling the mix signal for each of the at least one source signal, And the mix parameter comprises a gain of the mix signal with respect to the source signal. The method of claim 1, wherein the remix information is And a modified mix signal in which some of the source signals included in the mix signal are excluded. The method of claim 2, Receiving a source signal to be excluded from the mix signal; And Generating a modified mix signal in which the selected source signal is excluded; And the modified mix information is the generated modified mix signal. The method of any one of claims 1 to 3, wherein the transmitting step And when the payment for the remix information is completed. The method of claim 4, wherein the payment is Signal processing method, characterized in that performed using the payment information received from the user terminal. The method of claim 5, wherein the remix information providing server is coupled to a financial institution or a mobile communication provider through a communication network, The payment is Requesting user terminal authentication information from the user terminal; Requesting authentication by transmitting the authentication information received from the user terminal to the financial institution or the mobile communication provider; And And when the user terminal authentication is completed, requesting a payment by transmitting the payment information to the financial institution or the mobile communication provider. A signal processing method performed by a user terminal connected to a remix information providing server through a communication network, Accessing the server and requesting remix information; Transmitting the payment information requested from the server to the server; And When the payment is completed, including the step of receiving the remix information from the server, The remix information includes at least one of at least one source signal, a mix signal in which the at least one source signal is mixed, and a mix parameter for controlling the mix signal for each of the at least one source signal, And the mix parameter comprises a gain of the mix signal with respect to the source signal. A signal processing method performed by a user terminal connected to a remix information providing server through a communication network, Accessing the server; And Uploading remix information to the server; The remix information includes at least one of at least one source signal, a mix signal in which the at least one source signal is mixed, and a mix parameter for controlling the mix signal for each of the at least one source signal, And the mix parameter comprises a gain of the mix signal with respect to the source signal. The method of claim 8, wherein the remix information is Signal processing method, characterized in that generated in the user terminal. In the remix information providing server coupled to the user terminal through a communication network, A transceiver for receiving remix information from the user terminal and transmitting the requested remix information to the user terminal; A database unit for extracting the requested remix information in response to a request of the user terminal; A payment processor configured to process payment for the use of the remix information when the use of the remix information is charged; And A control unit which controls the overall server to transmit the remix information requested by the user terminal to the user terminal,  The remix information includes at least one of at least one source signal, a mix signal in which the at least one source signal is mixed, and a mix parameter for controlling the mix signal for each of the at least one source signal, The mix parameter, the remix information providing server, characterized in that it comprises a gain for the source signal of the mix signal. The method of claim 10, wherein the remix information is A modified mix signal in which some of the source signals included in the mix signal are excluded; The server may further include a mix signal modifying unit configured to generate the modified mix signal by excluding the partial source signal from the mix signal. In the user terminal coupled to the remix information providing server through a communication network, A communication unit for requesting remix information from the remix information providing server and receiving the requested remix information; A user interface for receiving control information from a user; A remix renderer which remixes a source signal included in a mixed signal using the remix information and the control information; And And a controller configured to control the entire user terminal to remix the source signal included in the mix signal. The remix information includes at least one of at least one source signal, a mix signal in which the at least one source signal is mixed, and a mix parameter for controlling the mix signal for each of the at least one source signal, The mix parameter comprises a gain for the source signal of the mix signal.

Images