KR102119240B1 - Method for up-mixing stereo audio to binaural audio and apparatus using the same - Google Patents

Abstract

Disclosed is a method and apparatus for up-mixing stereo audio to binaural audio. A method of up-mixing stereo audio to binaural audio according to an embodiment of the present invention includes generating binaural output by performing binaural encoding based on a high-pitched region and a low-pitched region separated from a stereo signal; Generating stereo wide output by performing stereo wide processing based on the midrange separated from the stereo signal; And combining the stereo signal, the binaural output, and the wide stereo output to generate an upmix stereo output.

Description

How to upmix stereo audio to binaural audio and devices for it {METHOD FOR UP-MIXING STEREO AUDIO TO BINAURAL AUDIO AND APPARATUS USING THE SAME}

The present invention relates to a technique for up-mixing stereo audio to binaural audio, and more particularly, to a technique for up-mixing stereo audio by combining binaural output using high and low tones and wide stereo output using mid-range.

As the multimedia technology has improved, the use of content including multi-channel audio signals such as 7.1 channels, 10.2 channels, 11.1 channels, and 22.2 channels, which are more than 5.1 channels, is increasing. However, since user terminals possessed by users using content can generally reproduce stereo type audio signals such as stereo speakers, headphones, and earphones, high-quality multi-channel audio signals need to be converted into stereo type audio signals. .

Published Korean Patent No. 10-2015-0013073, released on February 4, 2015 (Name: Binaural rendering method and device for multi-channel audio signal)

An object of the present invention is to provide a method of up-mixing an existing stereo file to immersive without performing immersive.

It is also an object of the present invention to reduce the time and cost required to mix stereo files into immersive files.

In addition, the object of the present invention is to improve compatibility with various types of content based on a natural upmix.

The method for up-mixing stereo audio according to the present invention to achieve the above object to binaural audio is performed by performing binaural encoding based on a high-pitched region and a low-pitched region separated from a stereo signal to generate a binaural output. To do; Generating stereo wide output by performing stereo wide processing based on the midrange separated from the stereo signal; And combining the stereo signal, the binaural output, and the wide stereo output to generate an upmix stereo output.

At this time, the binaural output is generated corresponding to a 3D vector for a binaural point located in an 8-channel based 3D cubic composed of 4 up channels and 4 down channels. The positions of the four up channels may be set based on the treble region, and the positions of the four down channels may be set based on the bass region.

At this time, the positions of the four up channels are set using any one treble frequency detected based on the magnitude of the transient in the treble region, and the positions of the four down channels are trans in the bass region. It can be set using any one of the bass frequencies detected based on the magnitude of the shunt.

At this time, the distance between the upper layer of the 3D cubic composed of 4 up channels and the lower layer of the 3D cubic composed of 4 down channels may be set based on the equalizer value of the stereo signal.

At this time, a wide stereo output is generated based on a wide stereo layer corresponding to the mid-range, and the wide stereo layer may correspond to a stereo layer in which an image space is extended corresponding to a reverb value and a delay value.

At this time, a 3D vector may be generated based on a reference listening point located inside the 3D cubic.

At this time, the step of generating the binaural output may generate the binaural output by applying the direction information of the 3D vector to the 3D cubic rotated corresponding to the head tracking information.

At this time, the three-dimensional cubic can be rotated corresponding to at least one rotation parameter of a pan, tilt, and roll.

At this time, the binaural output may include harmonics based on the fundamental frequency of the upper layer.

At this time, the up-mixing method may further include separating the stereo signal into the treble region, the mid-range region, and the bass region by inputting the stereo signal into a treble pass filter, a mid-pass filter, and a bass pass filter, respectively. can do.

In addition, the upmix device according to an embodiment of the present invention performs binaural encoding based on a high-pitched region and a low-pitched region separated from a stereo signal to generate a binaural output, and the mid-tone separated from the stereo signal A processor that performs stereo wide processing based on a region to generate a wide stereo output, and adds the stereo signal, the binaural output, and the wide stereo output to generate an upmix stereo output; And a memory for storing the stereo signal, the binaural output, and the wide stereo output.

At this time, the binaural output is generated corresponding to a 3D vector for a binaural point located in an 8-channel based 3D cubic composed of 4 up channels and 4 down channels. The positions of the four up channels may be set based on the treble region, and the positions of the four down channels may be set based on the bass region.

At this time, the positions of the four up channels are set using any one treble frequency detected based on the magnitude of the transient in the treble region, and the positions of the four down channels are trans in the bass region. It can be set using any one of the bass frequencies detected based on the magnitude of the shunt.

At this time, the distance between the upper layer of the 3D cubic composed of 4 up channels and the lower layer of the 3D cubic composed of 4 down channels may be set based on the equalizer value of the stereo signal.

At this time, a wide stereo output is generated based on a wide stereo layer corresponding to the mid-range, and the wide stereo layer may correspond to a stereo layer in which an image space is extended corresponding to a reverb value and a delay value.

At this time, a 3D vector may be generated based on a reference listening point located inside the 3D cubic.

At this time, the processor may generate the binaural output by applying the direction information of the 3D vector to the 3D cubic rotated corresponding to the head tracking information.

At this time, the three-dimensional cubic can be rotated corresponding to at least one rotation parameter of a pan, tilt, and roll.

At this time, the binaural output may include harmonics based on the fundamental frequency of the upper layer.

At this time, the processor may input the stereo signal into a high-pass filter, a mid-pass filter, and a low-pass filter, respectively, and separate the stereo signal into the high-pitched region, the middle-pitched region, and the low-pitched region.

According to the present invention, it is possible to provide a method of upmixing an existing stereo file to immersive without performing immersive.

In addition, the present invention can reduce the time and cost required to mix stereo files into immersive files.

In addition, the present invention can improve compatibility with various types of content based on a natural upmix.

1 is a view showing a stereo audio up mix structure according to an embodiment of the present invention.
2 is a block diagram showing an upmix device according to an embodiment of the present invention.
3 to 5 are diagrams showing an example of a filter for separating a high-pitched region, a middle-pitched region, and a low-pitched region of a stereo signal according to the present invention.
6 is a view showing a detailed structure for generating a binaural output according to an embodiment of the present invention.
7 is a diagram illustrating an example of an upper layer and a lower layer in an 8-channel based 3D cubic according to the present invention.
8 is a diagram conceptually showing an example of a stereo audio upmix effect according to the present invention.
9 is a view showing the distance between the top and bottom layers in a three-dimensional cubic according to the present invention.
10 is a view showing an example of a three-dimensional vector according to the present invention.
11 is a view showing an example of applying direction information of a 3D vector to a 3D cubic rotated corresponding to head tracking information according to the present invention.
12 is a view showing an example of a rotation parameter according to the present invention.
13 is a view showing a detailed structure for generating a wiper stereo output according to an embodiment of the present invention.
14 is a diagram illustrating an example of extending a stereo image according to the present invention.
15 is a diagram showing an example of a structure in which a 3D cubic upper layer and a lower layer are combined with a wide stereo layer according to the present invention.
16 is a flowchart illustrating a method of up-mixing stereo audio to binaural audio according to an embodiment of the present invention.

If described in detail with reference to the accompanying drawings the present invention. Here, repeated descriptions, well-known functions that may unnecessarily obscure the subject matter of the present invention, and detailed description of the configuration will be omitted. Embodiments of the present invention are provided to more fully describe the present invention to those skilled in the art. Therefore, the shape and size of elements in the drawings may be exaggerated for a more clear description.

Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the accompanying drawings.

1 is a view showing a stereo audio up mix structure according to an embodiment of the present invention.

Referring to FIG. 1, in the stereo audio upmix structure according to an embodiment of the present invention, a stereo signal 110 corresponding to two channels is provided with a high-pass filter 121, a mid-pass filter 122, and a low-pass filter 123. ).

At this time, in the stereo signal 110 input through the treble pass filter 121, only the treble region may pass and be input to the binaural encoder 130. In addition, in the stereo signal 110 input through the mid-pass filter 122, only the mid-range is passed and may be input to the stereo winder 140. Finally, in the stereo signal 110 input through the low-pass filter 123, only the low-pitched region is passed and may be input to the binaural encoder 130 together with the high-pitched region.

At this time, the bass region input to the binaural encoder (Binaural Encoder) 130 does not have a directionality, but since the treble region can have a directionality, the treble region and the bass region are separated, and an immersive effect is achieved. Binaural encoding for the cycle can be performed.

For example, the binaural encoder 130 may generate a 3D layer using 2 stereo channels corresponding to the treble region and 2 stereo channels corresponding to the bass region, and the binaural corresponding to the 3D layer Encoding can be performed.

In this case, the mid-range inputted to the stereo Wider 140 may perform stereo wide processing to expand the stereo image area without performing binaural encoding.

Thereafter, the binaural mixer 150 outputs the stereo signal 110 together with the binaural output output from the binaural encoder 130 and the wide stereo output output from the stereo wiper 140. Combined, you can create an upmix stereo output.

At this time, the upmix stereo output may correspond to a stereo signal or stereo audio with an immersive effect. That is, according to the present invention, it is possible to produce an immersive effect on stereo audio or stereo audio content without performing separate immersive mixing.

Accordingly, it is possible to reduce cost and time than performing immersive mixing in the conventional manner to generate immersive content or immersive audio.

2 is a block diagram showing an upmix device according to an embodiment of the present invention.

Referring to FIG. 2, an upmix device according to an embodiment of the present invention includes a communication unit 210, a processor 220, and a memory 230.

The communication unit 210 serves to transmit and receive information necessary for generating upmix stereo audio through a communication network such as a network. In particular, the communication unit 210 according to an embodiment of the present invention is a stereo signal or content corresponding to a source for generating upmix stereo audio, head tracking information to be input through a head tracking module or a user interface for binaural encoding Etc., and provide upmix stereo audio corresponding to the upmix stereo output.

The processor 220 generates a binaural output by performing binaural encoding based on a high-pitched region and a low-pitched region separated from the stereo signal.

At this time, the stereo signal may be input to the treble pass filter, the mid pass filter, and the low pass filter, respectively, and the stereo signal may be divided into a treble region, a mid region, and a bass region.

For example, referring to FIGS. 3 to 5, the processor 220 may input stereo signals corresponding to two channels to the treble pass filter 300, the mid pass filter 400, and the low pass filter 500, respectively. Can be.

At this time, the treble pass filter 300 corresponds to a filter that passes only the treble region of the input stereo signal range, and may output a stereo signal of the treble region as shown in FIG. 3.

At this time, the mid-pass filter 400 corresponds to a filter that passes only the mid-range in the sound range of the input stereo signal, and may output a stereo signal in the mid-range as shown in FIG. 4.

At this time, the bass pass filter 500 corresponds to a filter that passes only the bass region of the input range of the stereo signal, and may output a stereo signal in the bass region as shown in FIG. 5.

At this time, the treble pass filter 300, the mid pass filter 400, and the low pass filter 500 used in the present invention are not limited to a specific filtering method, and can be operated by applying available or developable technologies in the future. have.

At this time, the binaural output is generated corresponding to a 3D vector for a binaural point located in an 8-channel based 3D cubic composed of 4 up channels and 4 down channels, 4 The positions of the four up channels may be set based on the treble region, and the positions of the four down channels may be set based on the bass region.

At this time, the 8-channel-based 3D cubic corresponds to an element that creates a 3D spatial image. The 3D layer is composed of an upper layer composed of 4 up channels and a lower layer composed of 4 down channels. It may correspond to.

For example, referring to FIG. 6, using a binaural encoder 620 corresponding to a three-dimensional cubic method, corresponding to two channels and a bass region 612 corresponding to a treble region 611 separated from a stereo signal Binaural encoding may be performed corresponding to the two channels.

At this time, the positions of the four up channels can be set using any one treble frequency detected based on the magnitude of the transient in the treble area 611, and the positions of the four down channels are the bass area In 612, it may be set using any one bass frequency detected based on the magnitude of the transient.

At this time, the transient (Transient) may mean a portion of the initial amplitude rise that appears when the sound first starts in the waveform of the sound.

For example, in the present invention, the high frequency region 611 and the low frequency region 612 are searched for frequencies with strong transients, respectively, and the searched frequencies are separated from each other in real time, based on the left channel and the light channel. It is possible to create 4 up channels and 4 down channels. In this case, in order to increase the stereo effect, it is also possible to apply a stereo enhancement to the panning value of the upper layer corresponding to the high-pitched region 611 to generate a natural sound.

Referring to FIG. 7, first, a left and right separation process is performed on the treble frequencies detected in the treble region 611 to obtain positions corresponding to the left channel L and the light channel R, and the position of the left channel is as shown in FIG. 7. The speaker 711 may be disposed, and the speaker 712 may be disposed at the position of the light channel. Thereafter, the speaker 713 is disposed in a position where the left channel L and the light channel R are combined to the'L-(LR)', and the left channel L and the light channel R are corresponding to the'R-(LR)'. By disposing the speaker 714 at the combined position, the upper layer 710 of the three-dimensional cubic can be constructed.

In addition, the bass frequencies detected in the bass region 612 are separated left and right to obtain positions corresponding to the left channel L and the light channel R, and the speaker 721 is disposed at the position of the left channel as shown in FIG. 7. And the speaker 722 can be arrange|positioned at the position of a light channel. Thereafter, the speaker 723 is disposed in a position where the left channel L and the light channel R are combined to the'L-(LR)', and the left channel L and the light channel R are corresponding to the'R-(LR)'. By disposing the speaker 724 at the combined position, the lower layer 720 of the 3D cubic can be constructed.

Accordingly, the binaural output 630 illustrated in FIG. 6 is generated by binaural encoding of 8-channel based audio corresponding to the eight speakers 711 to 714 and 721 to 724 as illustrated in FIG. 7. It may correspond to the output, and may be output in a stereo format corresponding to two channels as shown in FIG. 6. At this time, the two channels corresponding to the binaural output 630 may correspond to the left channel and the right channel, respectively.

That is, as shown in FIG. 8, binaural including a binaural effect corresponding to the 8-channel 820 by binaurally encoding stereo signals of the high-pitched and low-pitched regions that were only 2 channels 810. You can generate output.

At this time, in the embodiment of the present invention, 8-channel-based 3D cubic was used as a 3D layer, but the 3D layer for binaural encoding may not be limited thereto. That is, the upmix device according to an embodiment of the present invention may be configured to include other 3D layers or 3D layers to be developed in the future.

In this case, the distance between the upper layer of the 3D cubic composed of 4 up channels and the lower layer of the 3D cubic composed of 4 down channels may be set based on the equalizer value of the stereo signal.

At this time, the equalizer (Equalizer, EQ) value of the stereo signal is to adjust the spatial range of the sound by adjusting the range, the distance between the upper and lower layers of the three-dimensional cubic shown in Figure 9 (910) is the equalizer value Can be set accordingly. In other words, the image space is adjusted vertically by adjusting the distance between the upper and lower layers (910) by adjusting the hertz (Hz) of the high-layer corresponding to the upper layer or by adjusting the hertz of the lower-layer corresponding to the lower layer. can.

At this time, the 3D vector may be generated based on a reference listening point located inside the cubic.

For example, referring to FIG. 10, the reference listening point 1010 virtually representing the position of the user or the listener may be located in the inner central portion of the 3D cubic 1000 using eight dynamic speakers as each vertex. . At this time, assuming that the binaural point 1020 is located on the upper layer of the 3D cubic 1000 as shown in FIG. 10, the 3D vector 1030 corresponding to the binaural output is shown in FIG. 10. It may be generated in a direction toward the binaural point 1020 from the reference listening point 1010 shown in.

At this time, as illustrated in FIG. 10, when the binaural point 1020 is positioned higher than the reference listening point 1010 on the three-dimensional cubic 1000, the sound output may be formed at the top of the listener. In addition, when the binaural point 1020 is positioned lower than the reference listening point 1010 on the three-dimensional cubic 1000, the sound output may be formed at the bottom of the listener.

As described above, in the present invention, it may be possible to produce more various audios by changing the position of the binaural point 1020 based on the reference listening point 1010 on the three-dimensional cubic 1000.

At this time, although not shown in FIG. 10, the reference listening point 1010 is located inside the 3D cubic 1000, but may be located on a wide stereo layer corresponding to the mid-range of the stereo signal. That is, the wide stereo layer corresponding to the 2-channel based stereo layer may be located between the upper layer and the lower layer of the 3D cubic 1000.

At this time, the binaural output may be generated by applying the direction information of the 3D vector to the rotated 3D cubic corresponding to the head tracking information. That is, since the binaural point is a position set based on the listener's head corresponding to the reference listening point, when the listener's head position or angle is changed, the position of the binaural point on the 3D cubic can also be changed.

For example, it can be assumed that the 3D cubic 1000 shown in FIG. 10 is rotated as shown in FIG. 11 corresponding to the head tracking information. At this time, by applying the direction information of the 3D vector 1030 shown in FIG. 10 to the 3D cubic shown in FIG. 11 as it is, the position of the binaural point changed according to the rotation can be detected.

At this time, the head tracking information corresponds to data tracking the head movement of a user or a listener, and may be obtained corresponding to at least one of a tracking input based on a separate head tracking module and a user input based on a user interface.

For example, if the user or the listener moves the head while wearing the head tracking module directly, the head tracking module may measure and measure the distance or angle of the user's head movement and generate and transmit the head tracking information.

For another example, the head tracking information may be artificially provided by a user or a listener through a user interface. That is, in order to artificially rotate a spatial image by a user or a listener, head tracking information may be input based on a user interface regardless of whether head tracking information is received by the head tracking module. At this time, the user or the listener may input and modify the head tracking information while listening to the mixing process generating the upmix stereo output or the upmix stereo output changing according to the input information.

At this time, the three-dimensional cubic can be rotated corresponding to at least one rotation parameter of a pan, tilt, and roll.

For example, as shown in FIG. 12, when the listener rotates the head corresponding to at least one of a pan, tilt, and roll, this value is obtained as a rotation parameter to obtain a three-dimensional cubic Can be applied to.

As described above, the effect produced by rotating the 3D cubic or moving it up, down, left, and right according to the head tracking information may be mixed with a wide stereo output and a stereo signal in the future to generate an upmix stereo output. Accordingly, it is possible to produce an immersive effect based on head tracking more efficiently than the conventional method of rotating or moving the stereo layer.

At this time, the binaural output may include harmonics based on the fundamental frequency of the upper layer.

At this time, the harmonics corresponds to an overtone in which the frequency is an integer multiple in the sound corresponding to the reference frequency, and may be included in a binaural output and used for mixing to provide a musical naturalness.

In addition, the processor 220 performs stereo wide processing based on the mid-range separated from the stereo signal to generate a wide stereo output.

At this time, the wide stereo output may be generated based on the wide stereo layer corresponding to the mid-range.

For example, referring to FIG. 13, stereo wide processing may be performed corresponding to the wide stereo layer based on the mid region 1310 of the stereo signal input to the stereo wirer 1320. At this time, the wide stereo output 1330 may be output in a stereo format corresponding to two channels as shown in FIG. 13.

At this time, the wide stereo layer corresponds to an element that creates a stereo image, and may correspond to a stereo layer in which the image space is expanded corresponding to a reverb value and a delay value.

For example, referring to FIG. 14, the stereo image area 1400 may be extended based on a reverb 1410 and a delay or pan 1420.

At this time, the reverb 1410 corresponds to the reverberation that the sound originating from the sound source hits the ear more than once by hitting the wall, floor, or ceiling, and the size of the space corresponding to the stereo image area 1400 is forward/backward. Corresponds to the value that can be adjusted with. At this time, the reverb 1410 value may be adjusted based on a pre-delay value corresponding to the time it takes for the reverb 1410 to be heard after the original sound is heard. In addition, in addition to the pre-delay, it is also possible to adjust the value of the reverb 1410 by adjusting the early reflection corresponding to the initial reflection sound as a parameter.

At this time, the delay or the delay in the fan 1420 corresponds to a delay value for the left and right channels, and by adjusting these values differently, the size of the space corresponding to the stereo image area 1400 is turned left/right. Can be adjusted. At this time, the pan (Pan) corresponds to a value that determines the extent to which the sound spreads horizontally, in the present invention, the left or right size of the corresponding space in the stereo image area 1400 by adjusting the delay or the fan 1420 Can be adjusted.

15, the wide stereo layer 1530 according to an embodiment of the present invention may be positioned in combination with a three-dimensional cubic composed of a surround upper layer 1510 and a lower layer 1520. . At this time, the structure shown in FIG. 15 corresponds to an embodiment, and is not limited to a structure in which each layer is combined.

In addition, the processor 220 combines stereo signals, binaural outputs, and wide stereo outputs to generate an upmix stereo output.

That is, the upmix stereo output including the immersive effect can be generated by mixing the immersive element by the binaural output and the extended stereo effect by the wide stereo output together with the stereo signal used as the source. .

In addition, since the present invention can support a natural upmix function based on the processor 220 having the above-described function, compatibility between contents supporting various types of sounds can be improved.

The memory 230 stores stereo signals, binaural outputs, and wide stereo outputs.

In addition, as described above, the memory 230 stores various information generated in the process of generating an upmix stereo output according to an embodiment of the present invention.

According to an embodiment, the memory 230 may be configured independently of the upmix device to support the upmix stereo audio generation function. At this time, the memory 230 may operate as a separate mass storage, and may include a control function for performing the operation.

On the other hand, an upmix device is equipped with a memory and can store information within the device. In one implementation, the memory is a computer-readable medium. In one implementation, the memory may be a volatile memory unit, and in other implementations, the memory may be a non-volatile memory unit. In one embodiment, the storage device is a computer-readable medium. In various different implementations, the storage device may include, for example, a hard disk device, an optical disk device, or some other mass storage device.

Through this upmix device, an existing stereo file can be upmixed to immersively without performing immersive, and the time and cost required to mix the stereo file into an immersive file can be reduced.

16 is a flowchart illustrating a method of up-mixing stereo audio to binaural audio according to an embodiment of the present invention.

Referring to FIG. 16, a method of up-mixing stereo audio to binaural audio according to an embodiment of the present invention is binaural by performing binaural encoding based on a high-pitched region and a low-pitched region separated from a stereo signal. Generate an output (S1610).

At this time, the stereo signal may be input to the treble pass filter, the mid pass filter, and the low pass filter, respectively, and the stereo signal may be divided into a treble region, a mid region, and a bass region.

For example, referring to FIGS. 3 to 5, the processor 220 may input stereo signals corresponding to two channels to the treble pass filter 300, the mid pass filter 400, and the low pass filter 500, respectively. Can be.

At this time, the treble pass filter 300 corresponds to a filter that passes only the treble region of the input stereo signal range, and may output a stereo signal of the treble region as shown in FIG. 3.

At this time, the mid-pass filter 400 corresponds to a filter that passes only the mid-range in the sound range of the input stereo signal, and may output a stereo signal in the mid-range as shown in FIG. 4.

At this time, the bass pass filter 500 corresponds to a filter that passes only the bass region of the input range of the stereo signal, and may output a stereo signal in the bass region as shown in FIG. 5.

At this time, the treble pass filter 300, the mid pass filter 400, and the low pass filter 500 used in the present invention are not limited to a specific filtering method, and can be operated by applying available or developable technologies in the future. have.

At this time, the binaural output is generated corresponding to a 3D vector for a binaural point located in an 8-channel based 3D cubic composed of 4 up channels and 4 down channels, 4 The positions of the four up channels may be set based on the treble region, and the positions of the four down channels may be set based on the bass region.

At this time, the 8-channel-based 3D cubic corresponds to an element that creates a 3D spatial image. The 3D layer is composed of an upper layer composed of 4 up channels and a lower layer composed of 4 down channels. It may correspond to.

For example, referring to FIG. 6, using a binaural encoder 620 corresponding to a three-dimensional cubic method, corresponding to two channels and a bass region 612 corresponding to a treble region 611 separated from a stereo signal Binaural encoding may be performed corresponding to the two channels.

At this time, the positions of the four up channels can be set using any one treble frequency detected based on the magnitude of the transient in the treble area 611, and the positions of the four down channels are the bass area In 612, it may be set using any one bass frequency detected based on the magnitude of the transient.

At this time, the transient (Transient) may mean a portion of the initial amplitude rise that appears when the sound first starts in the waveform of the sound.

For example, in the present invention, the high frequency region 611 and the low frequency region 612 are searched for frequencies with strong transients, respectively, and the searched frequencies are separated from each other in real time, based on the left channel and the light channel. It is possible to create 4 up channels and 4 down channels. In this case, in order to increase the stereo effect, it is also possible to apply a stereo enhancement to the panning value of the upper layer corresponding to the high-pitched region 611 to generate a natural sound.

Referring to FIG. 7, first, a left and right separation process is performed on the treble frequencies detected in the treble region 611 to obtain positions corresponding to the left channel L and the light channel R, and the position of the left channel is as shown in FIG. 7. The speaker 711 may be disposed, and the speaker 712 may be disposed at the position of the light channel. Thereafter, the speaker 713 is disposed in a position where the left channel L and the light channel R are combined to the'L-(LR)', and the left channel L and the light channel R are corresponding to the'R-(LR)'. By disposing the speaker 714 at the combined position, the upper layer 710 of the three-dimensional cubic can be constructed.

In addition, the bass frequencies detected in the bass region 612 are separated left and right to obtain positions corresponding to the left channel L and the light channel R, and the speaker 721 is disposed at the position of the left channel as shown in FIG. 7. And the speaker 722 can be arrange|positioned at the position of a light channel. Subsequently, the speaker 723 is disposed in a position where the left channel L and the light channel R correspond to'L-(LR)', and the left channel L and the light channel R correspond to'R-(LR)'. By disposing the speaker 724 at the combined position, the lower layer 720 of the 3D cubic can be constructed.

Accordingly, the binaural output 630 illustrated in FIG. 6 is generated by binaural encoding of 8-channel based audio corresponding to the eight speakers 711 to 714 and 721 to 724 as illustrated in FIG. 7. It may correspond to the output, and may be output in a stereo format corresponding to two channels as shown in FIG. 6. At this time, the two channels corresponding to the binaural output 630 may correspond to the left channel and the right channel, respectively.

That is, as shown in FIG. 8, binaural including a binaural effect corresponding to the 8-channel 820 by binaurally encoding stereo signals of the high-pitched and low-pitched regions that were only 2 channels 810. You can generate output.

At this time, in the embodiment of the present invention, 8-channel-based 3D cubic was used as a 3D layer, but the 3D layer for binaural encoding may not be limited thereto. That is, the upmix device according to an embodiment of the present invention may be configured to include other 3D layers or 3D layers to be developed in the future.

In this case, the distance between the upper layer of the 3D cubic composed of 4 up channels and the lower layer of the 3D cubic composed of 4 down channels may be set based on the equalizer value of the stereo signal.

At this time, the equalizer (Equalizer, EQ) value of the stereo signal is to adjust the spatial range of the sound by adjusting the range, the distance between the upper and lower layers of the three-dimensional cubic shown in Figure 9 (910) is the equalizer value Can be set accordingly. In other words, the image space is adjusted vertically by adjusting the distance between the upper and lower layers (910) by adjusting the hertz (Hz) of the high-layer corresponding to the upper layer or by adjusting the hertz of the lower-layer corresponding to the lower layer. can.

At this time, the 3D vector may be generated based on a reference listening point located inside the cubic.

For example, referring to FIG. 10, the reference listening point 1010 virtually representing the position of the user or the listener may be located in the inner central portion of the 3D cubic 1000 using eight dynamic speakers as each vertex. . At this time, assuming that the binaural point 1020 is located on the upper layer of the 3D cubic 1000 as shown in FIG. 10, the 3D vector 1030 corresponding to the binaural output is shown in FIG. 10. It may be generated in a direction toward the binaural point 1020 from the reference listening point 1010 shown in.

At this time, as illustrated in FIG. 10, when the binaural point 1020 is positioned higher than the reference listening point 1010 on the three-dimensional cubic 1000, the sound output may be formed at the top of the listener. In addition, when the binaural point 1020 is positioned lower than the reference listening point 1010 on the three-dimensional cubic 1000, the sound output may be formed at the bottom of the listener.

As described above, in the present invention, it may be possible to produce more various audios by changing the position of the binaural point 1020 based on the reference listening point 1010 on the three-dimensional cubic 1000.

At this time, although not shown in FIG. 10, the reference listening point 1010 is located inside the 3D cubic 1000, but may be located on a wide stereo layer corresponding to the mid-range of the stereo signal. That is, the wide stereo layer corresponding to the 2-channel based stereo layer may be located between the upper layer and the lower layer of the 3D cubic 1000.

At this time, the binaural output may be generated by applying the direction information of the 3D vector to the rotated 3D cubic corresponding to the head tracking information. That is, since the binaural point is a position set based on the listener's head corresponding to the reference listening point, when the listener's head position or angle is changed, the position of the binaural point on the 3D cubic can also be changed.

For example, it can be assumed that the 3D cubic 1000 shown in FIG. 10 is rotated as shown in FIG. 11 corresponding to the head tracking information. At this time, by applying the direction information of the 3D vector 1030 shown in FIG. 10 to the 3D cubic shown in FIG. 11 as it is, the position of the binaural point changed according to the rotation can be detected.

At this time, the head tracking information corresponds to data tracking the head movement of a user or a listener, and may be obtained corresponding to at least one of a tracking input based on a separate head tracking module and a user input based on a user interface.

For example, if the user or the listener moves the head while wearing the head tracking module directly, the head tracking module may measure and measure the distance or angle of the user's head movement and generate and transmit the head tracking information.

For another example, the head tracking information may be artificially provided by a user or a listener through a user interface. That is, in order to artificially rotate a spatial image by a user or a listener, head tracking information may be input based on a user interface regardless of whether head tracking information is received by the head tracking module. At this time, the user or the listener may input and modify the head tracking information while listening to the mixing process generating the upmix stereo output or the upmix stereo output changing according to the input information.

At this time, the three-dimensional cubic can be rotated corresponding to at least one rotation parameter of a pan, tilt, and roll.

For example, as shown in FIG. 12, when the listener rotates the head corresponding to at least one of a pan, tilt, and roll, this value is obtained as a rotation parameter to obtain a three-dimensional cubic Can be applied to.

As described above, the effect produced by rotating the 3D cubic or moving it up, down, left, and right according to the head tracking information may be mixed with a wide stereo output and a stereo signal in the future to generate an upmix stereo output. Accordingly, it is possible to produce an immersive effect based on head tracking more efficiently than the conventional method of rotating or moving the stereo layer.

At this time, the binaural output may include harmonics based on the fundamental frequency of the upper layer.

At this time, the harmonics corresponds to an overtone in which the frequency is an integer multiple in the sound corresponding to the reference frequency, and may be included in a binaural output and used for mixing to provide a musical naturalness.

In addition, in the method of upmixing stereo audio to binaural audio according to an embodiment of the present invention, stereo wide processing is performed based on a midrange separated from a stereo signal to generate a wide stereo output (S1620).

At this time, the wide stereo output may be generated based on the wide stereo layer corresponding to the mid-range.

For example, referring to FIG. 13, stereo wide processing may be performed corresponding to the wide stereo layer based on the mid region 1310 of the stereo signal input to the stereo wirer 1320. At this time, the wide stereo output 1330 may be output in a stereo format corresponding to two channels as shown in FIG. 13.

At this time, the wide stereo layer corresponds to an element that creates a stereo image, and may correspond to a stereo layer in which the image space is expanded corresponding to a reverb value and a delay value.

For example, referring to FIG. 14, the stereo image area 1400 may be extended based on a reverb 1410 and a delay or pan 1420.

At this time, the reverb 1410 corresponds to the reverberation that the sound originating from the sound source hits the ear more than once by hitting the wall, floor, or ceiling, and the size of the space corresponding to the stereo image area 1400 is forward/backward. Corresponds to the value that can be adjusted with. At this time, the reverb 1410 value may be adjusted based on a pre-delay value corresponding to the time it takes for the reverb 1410 to be heard after the original sound is heard. In addition, in addition to the pre-delay, it is also possible to adjust the value of the reverb 1410 by adjusting the early reflection corresponding to the initial reflection sound as a parameter.

At this time, the delay or the delay in the fan 1420 corresponds to a delay value for the left and right channels, and by adjusting these values differently, the size of the space corresponding to the stereo image area 1400 is turned left/right. Can be adjusted. At this time, the pan (Pan) corresponds to a value that determines the extent to which the sound spreads horizontally, in the present invention, the left or right size of the corresponding space in the stereo image area 1400 by adjusting the delay or the fan 1420 Can be adjusted.

15, the wide stereo layer 1530 according to an embodiment of the present invention may be positioned in combination with a three-dimensional cubic composed of a surround upper layer 1510 and a lower layer 1520. . At this time, the structure shown in FIG. 15 corresponds to an embodiment, and is not limited to a structure in which each layer is combined.

In addition, in the method of upmixing stereo audio to binaural audio according to an embodiment of the present invention, an upmix stereo output is generated by combining a stereo signal, a binaural output, and a wide stereo output (S1630).

That is, the upmix stereo output including the immersive effect can be generated by mixing the immersive element by the binaural output and the extended stereo effect by the wide stereo output together with the stereo signal used as the source. .

In addition, since the present invention can support a natural upmix function based on the above functions, compatibility between contents supporting various types of sounds can be improved.

In addition, although not shown in FIG. 16, a method of up-mixing stereo audio to binaural audio according to an embodiment of the present invention transmits and receives information necessary for generating up-mix stereo audio through a communication network such as a network. . In particular, for generating upmix stereo audio, stereo signals or contents corresponding to a source, head tracking information to be input through a head tracking module or a user interface for binaural encoding, and the like, and up corresponding to up mix stereo output Mix stereo audio can be provided.

In addition, the method of upmixing stereo audio according to an embodiment of the present invention to binaural audio stores various information generated in the process of generating an upmix stereo output according to an embodiment of the present invention as described above. do.

Embodiments of the present invention may be implemented as a computer-implemented method or a non-transitory computer-readable medium in which instructions executable on a computer are recorded. When computer readable instructions are executed by a processor, computer readable instructions may perform a method in accordance with at least one aspect of the present invention.

As described above, the method and apparatus for up-mixing stereo audio according to the present invention to binaural audio are not limited to the configuration and method of the above-described embodiments, and the above embodiments All or some of the embodiments may be selectively combined to constitute various modifications.

110: stereo signal 121, 300: treble pass filter
122, 400: mid pass filter 123, 500: bass pass filter
130, 620: binaural encoder 140, 1320: stereo winder
150: binaural mixer 210: communication unit
220: processor 230: memory
611: treble signal 612: bass signal
630: binaural output 710, 1510: upper layer
711~714, 721~724: speaker 720, 1520: lower layer
810: 2 channels 820: 8 channels
910: Distance 1000: 3D cube
1010: Reference listening point 1020: Binaural point
1030: 3D vector 1310: Mid-range signal
1330: Wide stereo output 1400: Stereo image area
1410: reverb 1420: delay or fan
1530: Wide stereo layer

Claims (20)

Generating binaural output by performing binaural encoding based on a high-pitched region and a low-pitched region separated from the stereo signal;
Generating stereo wide output by performing stereo wide processing based on the midrange separated from the stereo signal; And
Combining the stereo signal, the binaural output, and the wide stereo output to generate an upmix stereo output that multi-channels the stereo signal.
Including,
The binaural output is generated based on a binaural point located in an 8-channel based 3D cubic composed of 4 up channels and 4 down channels,
The four up channels and the four down channels are generated based on the left channel and the right channel generated by separating left and right frequencies of the strong transients respectively found in the treble region and the bass region,
The up-mix stereo output is a method of up-mixing stereo audio to binaural audio, characterized in that it corresponds to an immersive content type capable of representing the directionality of the treble region based on the binaural point. The method according to claim 1,
The binaural output is
It is generated corresponding to the 3D vector for the binaural point, wherein the positions of the four up channels are set based on the treble region, and the positions of the four down channels are set based on the bass region. How to up-mix stereo audio featuring binaural audio The method according to claim 2,
The positions of the four up channels are set using any one treble frequency detected based on the magnitude of the transient in the treble region,
The method of up-mixing stereo audio to binaural audio is characterized in that the positions of the four down-channels are set using any one bass frequency detected based on the magnitude of a transient in the bass region. The method according to claim 3,
The distance between the upper layer of the three-dimensional cubic composed of the four up channels and the lower layer of the three-dimensional cubic composed of the four down channels is set based on the equalizer value of the stereo signal. How to upmix audio to binaural audio. The method according to claim 1,
The wide stereo output
It is generated based on the wide stereo layer corresponding to the mid-range, the wide stereo layer is a stereo audio characterized in that it corresponds to a stereo layer with an extended image space corresponding to the reverb value and the delay value to binaural audio How to mix up. The method according to claim 2,
The 3D vector
A method of upmixing stereo audio to binaural audio, characterized in that it is generated based on a reference listening point located inside the 3D cubic. The method according to claim 6,
The step of generating the binaural output is
A method of upmixing stereo audio to binaural audio, wherein the binaural output is generated by applying direction information of the 3D vector to the 3D cubic rotated corresponding to head tracking information. The method according to claim 7,
The three-dimensional cubic
A method of upmixing stereo audio to binaural audio, characterized in that it rotates corresponding to at least one rotation parameter of a pan, tilt, or roll. The method according to claim 4,
The binaural output is
A method of upmixing stereo audio to binaural audio, comprising harmonics based on the fundamental frequency of the upper layer. The method according to claim 1,
How to upmix the stereo audio to binaural audio
And inputting the stereo signal into a treble pass filter, a mid pass filter, and a bass pass filter, respectively, and separating the stereo signal into the treble region, the mid region, and the bass region. How to upmix to binaural audio. Binaural encoding is performed based on the treble and bass regions separated from the stereo signal to generate a binaural output, and stereo wide processing is performed based on the midrange separated from the stereo signal to generate a wide stereo output. And a processor generating a multi-channel upmix stereo output by combining the stereo signal, the binaural output, and the wide stereo output. And
Memory for storing the stereo signal, the binaural output and the wide stereo output
Including,
The binaural output is generated based on a binaural point located in an 8-channel based 3D cubic composed of 4 up channels and 4 down channels,
The four up channels and the four down channels are generated based on the left channel and the right channel generated by separating left and right frequencies of the strong transients respectively found in the treble region and the bass region,
The upmix stereo output corresponds to an immersive content type capable of indicating the directionality of the treble region based on the binaural point. The method according to claim 11,
The binaural output is
It is generated corresponding to the 3D vector for the binaural point, wherein the positions of the four up channels are set based on the treble region, and the positions of the four down channels are set based on the bass region. Up-mixing device characterized by. The method according to claim 12,
The positions of the four up channels are set using any one treble frequency detected based on the magnitude of the transient in the treble region,
The position of the four down-channels is set using any one bass frequency detected based on the magnitude of a transient in the bass region. The method according to claim 13,
The distance between the upper layer of the 3D cubic composed of the 4 up channels and the lower layer of the 3D cubic composed of the 4 down channels is set based on the equalizer value of the stereo signal. Mix device. The method according to claim 11,
The wide stereo output
The upmix device is generated based on a wide stereo layer corresponding to the mid-range, wherein the wide stereo layer corresponds to a stereo layer in which an image space is expanded corresponding to a reverb value and a delay value. The method according to claim 12,
The 3D vector
Up-mixing device, characterized in that generated based on the reference listening point located inside the three-dimensional cubic. The method according to claim 16,
The processor
And applying the direction information of the 3D vector to the 3D cubic rotated corresponding to the head tracking information to generate the binaural output. The method according to claim 17,
The three-dimensional cubic
Up-mixing device characterized in that it is rotated in correspondence to at least one rotation parameter of a pan, tilt, and roll. The method according to claim 14,
The binaural output is
And a harmonics based on the fundamental frequency of the upper layer. The method according to claim 11,
The processor
The stereo signal is input to the treble pass filter, the mid pass filter, and the bass pass filter, respectively, and the upmix device, characterized in that to separate the stereo signal into the treble region, the mid region and the bass region.

Images