KR20190114578A - Method for creating multi-layer binaural content and program thereof - Google Patents

Abstract

Disclosed are a method for creating multilayer binaural content and a program therefor. A method for creating multilayer binaural content according to an embodiment of the present invention includes: a step of performing three-dimensional layer binaural encoding corresponding to a three-dimensional binaural layer to generate a three-dimensional layer binaural output and performing audio processing corresponding to a planar layer to generate a planar layer audio output; a step of applying an audio effect corresponding to each of the three-dimensional layer binaural output and the planar layer audio output based on a user input to an audio effect interface; and a step of combining the three-dimensional layer binaural output and the planar layer audio output to which the audio effect is applied to generate multilayer binaural content. It is possible to provide a listening environment suitable for the taste of a user.

Description

Multi-layer binaural content generation method and program therefor {METHOD FOR CREATING MULTI-LAYER BINAURAL CONTENT AND PROGRAM THEREOF}

The present invention relates to a technology for generating multilayer binaural content, and in particular, a technology for maximizing a binaural listening environment based on a binaural output and an audio output to which audio effects according to a user input are respectively applied. It is about.

As multimedia technology is improved, the use of content including multichannel audio signals, such as 7.1 channels, 10.2 channels, 11.1 channels, and 22.2 channels, is increasing. However, since the user terminals possessed by the users using the content can reproduce stereo audio signals such as stereo speakers, headphones, and earphones, high quality multi-channel audio signals need to be converted into stereo audio signals. .

Korean Unexamined Patent Publication No. 10-2015-0013073, published February 4, 2015 (name: method and apparatus for binaural rendering of multi-channel audio signals)

An object of the present invention is to provide a listening environment suitable for a user's taste by providing an interface for individually performing effect editing on various sound elements.

It is also an object of the present invention to preserve sound quality that can be distorted by sudden changes in head tracking data.

It is also an object of the present invention to provide a method for generating binaural content that can maximize the binaural effect by mixing various sound elements.

In order to achieve the above object, the method for generating a multilayer binaural content according to the present invention generates a 3D layer binaural output by performing a 3D layer binaural encoding corresponding to a 3D binaural layer. Generating a planar layer audio output by performing audio processing corresponding to the planar layer; Applying an audio effect corresponding to each of the three-dimensional layer binaural output and the planar layer audio output based on a user input to an audio effect interface; And combining the 3D layer binaural output and the planar layer audio output to which the audio effect is applied to generate the multilayer binaural content.

In this case, the multilayer binaural content generating method may further include applying head tracking data corresponding to each of the 3D binaural output and the planar layer audio output.

In this case, the head tracking data may correspond to any one of sensor input based automatic head tracking data and user input based manual head tracking data for the head tracking interface.

In this case, the head tracking data may be converted into a log formula within a preset tracking limit and applied.

In this case, the step of applying the audio effect may be performed for at least one of a room setting, an equalizer setting, and a plug-in setting.

In this case, the planar layer performs surround layer binaural encoding to generate a surround layer binaural output, and inputs a surround layer and a stereo signal for providing the generated surround layer binaural output to the planar layer audio output. And a proximity stereo layer that receives and generates the planar layer audio output corresponding to the stereo signal.

At this time, the three-dimensional layer binaural output corresponds to a three-dimensional vector of binaural points located on an eight-channel-based three-dimensional cubic composed of four up channels and four down channels. Can be generated.

At this time, generating the multilayer binaural content may include applying an audio effect corresponding to a subwoofer output corresponding to a subwoofer layer based on a user input to the audio effect interface; And generating the multilayer binaural content by summing the subwoofer output to which the audio effect is applied together with the 3D layer binaural output and the planar layer audio output to which the audio effect is applied.

In this case, the method of generating a multilayer binaural content may further include resetting the audio effect in consideration of automation information about a binaural audio source.

In this case, the applying of the audio effect may apply the audio effect in accordance with a preset application range of the entire range of the binaural audio source.

In addition, the multilayer binaural content generating program stored in the computer-readable recording medium according to an embodiment of the present invention, by performing a three-dimensional layer binaural encoding corresponding to the three-dimensional binaural layer, Generating a binaural output and performing audio processing corresponding to the planar layer to generate a planar layer audio output; Applying an audio effect corresponding to each of the three-dimensional layer binaural output and the planar layer audio output based on a user input to an audio effect interface; And generating the multilayer binaural content by adding the 3D layer binaural output and the planar layer audio output to which the audio effect is applied.

In this case, applying the head tracking data corresponding to each of the 3D binaural output and the planar layer audio output may be performed.

In this case, the head tracking data may correspond to any one of sensor input based automatic head tracking data and user input based manual head tracking data for the head tracking interface.

In this case, the head tracking data may be converted into a log formula within a preset tracking limit and applied.

In this case, the audio effect may be applied by setting at least one of a room setting, an equalizer setting, and a plug-in setting.

In this case, the planar layer performs surround layer binaural encoding to generate a surround layer binaural output, and inputs a surround layer and a stereo signal for providing the generated surround layer binaural output to the planar layer audio output. And a proximity stereo layer that receives and generates the planar layer audio output corresponding to the stereo signal.

At this time, the three-dimensional layer binaural output corresponds to a three-dimensional vector of binaural points located on an eight-channel-based three-dimensional cubic composed of four up channels and four down channels. Can be generated.

At this time, applying an audio effect corresponding to a subwoofer output corresponding to a subwoofer layer based on a user input to the audio effect interface; And adding the subwoofer output to which the audio effect is applied together with the 3D layer binaural output and the planar layer audio output to which the audio effect is applied to generate the multilayer binaural content.

At this time, the step of resetting the audio effect may be further performed in consideration of the automation information of the binaural audio source.

In this case, the audio effect may be applied to correspond to a predetermined application range among the entire range of the binaural audio source.

According to the present invention, it is possible to provide a listening environment suitable for a user's taste by providing an interface for individually performing effect editing on various sound elements.

In addition, the object of the present invention is to preserve the sound quality that can be distorted by the abrupt change of the head tracking data.

In addition, the present invention can provide a method for generating binaural content that can maximize the binaural effect by mixing various sound elements.

1 is a block diagram illustrating a multilayer binaural content generation system according to an embodiment of the present invention.
2 is a flowchart illustrating a method of generating multilayer binaural content according to an embodiment of the present invention.
3 is a diagram illustrating a multilayer binaural content generation system according to another embodiment of the present invention.
4 is a diagram showing a detailed structure for generating a three-dimensional layer binaural output according to an embodiment of the present invention.
5 is a view showing an example of an eight-channel based three-dimensional cubic (cubic) according to the present invention.
6 illustrates a detailed structure for generating a surround layer binaural output according to an embodiment of the present invention.
7 is a diagram illustrating an example of a 5-channel surround layer according to the present invention.
8 is a diagram showing a detailed structure for generating a stereo signal according to an embodiment of the present invention.
9 illustrates an example of a proximity stereo layer according to the present invention.
10 illustrates a detailed structure for generating a subwoofer output according to an embodiment of the present invention.
FIG. 11 is a diagram illustrating an example of a structure in which a three-dimensional binaural layer, a planar layer, and a subwoofer layer are combined according to the present invention.
12 to 13 illustrate examples of volume and filter interfaces according to the present invention.
14 to 15 are diagrams showing an example of an audio effect interface according to the present invention.
16 is a diagram illustrating an example of a head tracking interface according to the present invention.
17 is a diagram illustrating an example of head tracking data according to the present invention.
18 is a block diagram illustrating a terminal device for executing a multilayer binaural content generation program according to an embodiment of the present invention.

Hereinafter, the present invention will be described in detail with reference to the accompanying drawings. Here, the repeated description, well-known functions and configurations that may unnecessarily obscure the subject matter of the present invention, and detailed description of the configuration will be omitted. Embodiments of the present invention are provided to more completely describe the present invention to those skilled in the art. Accordingly, the shape and size of elements in the drawings may be exaggerated for clarity.

Hereinafter, exemplary embodiments of the present invention will be described in detail with reference to the accompanying drawings.

1 is a block diagram illustrating a multilayer binaural content generation system according to an embodiment of the present invention.

Referring to FIG. 1, a multilayer binaural content generation system according to an embodiment of the present invention includes a binaural audio source 110, a multilayer decoder 120, a layer 130, and an individual editing module for each layer ( 140, audio mixer 150, and multilayer binaural content 160.

Conventional binaural technology is a method of decoding and providing a binaural encoded binaural output for a multi-channel audio file through a dedicated player. However, since binaural encoding uses a fixed speaker disposed at a certain distance from the listening position, it is difficult to adjust the position of the speaker to increase or decrease the image of the space.

In addition, there is a problem that it is difficult to apply the technology in the case of an audio source that does not have a spatial image, such as music content, as a technology specialized for content that includes video and audio together, such as surround movie content.

In addition, since the binaural content can be played only by using a dedicated player, the efficiency may be reduced in terms of utilization. For example, although the loudness should be delivered to the listeners due to the nature of the music content, only the binaural encoder has a limitation in providing sound effects optimized for the music content.

In addition, since the conventional binaural technology uses only one encoder specialized for an effect mainly used according to the content, it is impossible to apply various effects of the directing effect. For example, since the subwoofer is often not used for the music content, it has hardly been attempted to provide the bass reproduction element according to the subwoofer to the music content through the conventional binaural engine.

Accordingly, the present invention mixes the output including the various binaural sound effects and the output by the audio processing to provide the binaural content including the more dramatic presentation, the final sum of the multi-layered audio By providing the multilayer binaural content 160 in such a way as to provide an environment in which audio effects can be applied according to a user's taste.

To this end, as shown in FIG. 1, the binaural audio source 110 may be decoded into a plurality of layers using the multilayer decoder 120.

In this case, the multilayer decoder 120 corresponds to a multilayer binaural decoder, and the binaural audio source 110 corresponds to an immersive cubic layer and a planar layer corresponding to a 3D binaural layer. And stereo layer and subwoofer layer.

Subsequently, audio effects, head tracking data, and the like for each of the 3D binaural layer, the planar layer, and the subwoofer layer may be edited based on the layer-specific individual editing module 140 according to the present invention. Since such individual editing is performed through user input based on a separate interface, it is possible to generate multilayer binaural content that is most suitable for the user's taste and to personalize the listening environment.

At this time, the user can designate the processing range of the binaural audio source or the binaural sound source through the interface to edit or process only the desired portion.

In this case, when the head tracking data is applied through individual editing, the sound quality may be prevented from being dropped due to a sudden movement by providing an appropriate limit for the head tracking data.

Thereafter, the individual edited layers may be summed through the audio mixer 150 to generate multilayer binaural content 160 that is playable through a general purpose decoder and that is compatible with conventional contents.

For example, for movie content that includes video and audio, three-dimensional layer binaural output, stereo output, and subwoofer output, along with surround layer binaural output that can be generated based on the movement of objects included in the video, are included. Mixing and providing at least one of them may enable more dramatic sound production.

For example, music content containing only audio provides dynamic music by providing a mixture of stereo output or subwoofer output along with a 3D layer binaural output generated based on a 3D binaural layer. You may.

2 is a flowchart illustrating a method of generating multilayer binaural content according to an embodiment of the present invention.

Referring to FIG. 2, in the method for generating multilayer binaural content according to an embodiment of the present invention, a 3D layer binaural output is performed by performing a 3D layer binaural encoding corresponding to a 3D binaural layer. In operation S210, a flat layer audio output is generated by performing audio processing corresponding to the flat layer.

In this case, the 3D binaural layer corresponds to an element for creating a 3D spatial image. For example, referring to FIG. 4, for example, the 3D binaural layer may be 3D using a binaural encoder 420 corresponding to the 3D cubic method. Three-dimensional layer binaural encoding corresponding to a plurality of channels included in the binaural layer may be performed.

In this case, the 3D binaural layer may include four upchannels 411 and four downchannels 412 corresponding to 8-channel-based 3D cubics.

Accordingly, the 3D layer binaural output 430 may correspond to an output generated by binaural encoding 8-channel based audio, and may be output corresponding to two channels as shown in FIG. 4. In addition, two channels corresponding to the 3D layer binaural output 430 may correspond to the left channel and the light channel, respectively.

In this case, although the embodiment shown in FIG. 4 uses an 8 channel based 3D cubic layer as the 3D binaural layer, the 3D binaural layer may not be limited thereto. That is, it may be configured to include other three-dimensional binaural layer applicable to the present invention or three-dimensional binaural layer to be developed in the future.

At this time, the three-dimensional layer binaural output corresponds to a three-dimensional vector of binaural points located on an eight-channel-based three-dimensional cubic composed of four up channels and four down channels. Can be generated.

For example, referring to FIG. 5, an 8-channel based 3D cubic includes four dynamic speakers corresponding to four upchannels 511 to 514 and four dynamic speakers corresponding to four downchannels. It may be a hexahedral structure with each vertex 515 to 518. At this time, since the positions of the eight dynamic speakers 511 to 518 can be changed, the range of the binaural effect generated by the three-dimensional cubic can also be changed dynamically.

In another example, the conventional binaural method may generate immersive sound with eight dynamic speakers by generating three-dimensional cubic using vector base amplitude panning (Vbap) or ambisonics. have. That is, each of the eight dynamic speakers may be given position values for X, Y, and Z, and a vector-based virtual track point based on the midpoint of the 3D cubic may be expressed. In this case, the virtual track point may be represented corresponding to the parameter value included in the head tracking information.

Through such a three-dimensional cubic it is possible to generate a spatial image for the music content containing only audio, it is possible to represent the movement of the sound can provide a more three-dimensional effect.

In this case, the 3D cubic may be generated by changing the positions of the eight dynamic speakers corresponding to the vertices of the 3D cubic according to the size parameter for the 3D binaural layer. In other words, by freely changing the position of the dynamic speaker of the variable method other than the fixed method in accordance with the size parameter can be efficiently generated three-dimensional cubic.

For example, three-dimensional cubics having various ranges may be generated by processing the three-dimensional cubic by setting the size parameter to a constant and multiplying it by a binaural function.

In this case, the 3D vector may be included in the 3D cubic and generated based on a reference listening point corresponding to the center of the 2D plane corresponding to the surround layer.

For example, a reference listening point that virtually represents the location of a user or listener listening to binaural stereo audio is located inside a three-dimensional cubic with eight dynamic speakers at each vertex, but centered on the surround layer. Can be. At this time, assuming that the binaural point is located on the top surface of the 3D cubic, a 3D vector corresponding to the 3D layer binaural output may be generated in a direction from the reference listening point toward the binaural point.

In this case, the planar layer corresponds to a layer having a structure different from that of the 3D binaural layer, and may correspond to an element for making an image corresponding to a surround effect or a stereo effect.

Therefore, the flat layer performs surround layer binaural encoding to generate a surround layer binaural output, and receives a surround signal and a stereo signal that provides the generated surround layer binaural output as a flat layer audio output. It may be any one of a proximity stereo layer that generates a planar layer audio output corresponding to.

For example, referring to FIG. 6, a surround layer binaural encoding corresponding to a surround layer of five or seven channels 610 may be performed using the binaural encoder 620. In this case, as will be described with reference to FIG. 9, 7-channel-based surround layer binaural encoding may be performed by including 2 channels corresponding to a proximity stereo layer in the surround layer.

In this case, the surround layer may correspond to a structure including five speakers 711 to 715, for example, as shown in FIG. 7. In this case, the surround layer binaural output 630 may correspond to a binaural point located on the surround layer. If it is assumed that the listener is listening to the sound at the reference listening point located in the center of the surround layer, the surround layer binaural output 630 by binaural encoding as if the sound is coming from the binaural point on the surround layer. Can be generated. In this case, the surround layer corresponds to an element for creating a surround image corresponding to the surround effect. In FIG. 7, the surround layer is illustrated in the form of a plane for convenience of description, but may not be limited to the plane.

In this case, the surround layer binaural output 630 may be output corresponding to two channels as shown in FIG. 6. In addition, two channels corresponding to the surround layer binaural output 630 may correspond to the left channel and the right channel, respectively.

6 to 7 illustrate a surround layer corresponding to 5 or 7 channels 610, but the channel of the surround layer is not limited to 5 or 7 channels 610. In addition, although the surround layer is illustrated in the form of a rectangular plane in FIG. 7, the surround layer is not limited thereto and may be represented in various forms such as a line thickness, a planar shape, and a distance from a reference listening point.

For another example, referring to FIG. 8, audio processing may be performed corresponding to the proximity stereo layer of the two channels 810 based on the stereo bus 820. That is, the stereo signal 830 corresponding to the planar layer audio output may correspond to the output generated by processing the two channel 810 based stereo audio, and may be output corresponding to the two channels.

In this case, the proximity stereo layer corresponds to an element for creating a stereo image corresponding to the stereo effect, and may be included as a part of the surround layer.

For example, as shown in FIG. 9, a surround stereo layer corresponding to two speakers 911 and 912 is included on a surround layer based on five speakers, so that the layer structure includes a total of seven speakers. It may be indicated.

In this case, as shown in FIG. 9, the proximity stereo layer may be disposed at a distance from the reference listening point 900 positioned on the surround layer. Alternatively, the proximity stereo layer may be used as the left and right side speakers of the reference listening point 900.

In this case, the stereo signal output corresponding to the proximity stereo layer may provide a damping feeling that is difficult to produce by the spatial parameter used for binaural encoding. Accordingly, the binaural stereo output according to an embodiment of the present invention may provide an immersive effect by binaural encoding and at the same time provide a damping feeling.

As such, a flat layer audio output corresponding to a surround layer binaural output or a flat layer audio output corresponding to a stereo signal corresponds to an output that includes only different sound effects when compared to a three dimensional layer binaural output. It may be. That is, the planar layer audio output may include various values than the 3D layer binaural output even if the output is not corresponding to the 3D layer.

In this case, the planar layer may be positioned between four upchannels and four downchannels corresponding to the 3D cubic.

For example, referring to FIG. 11, the surround layer 1120 and the proximity stereo layer 1130 corresponding to the planar layer according to an embodiment of the present invention are three corresponding to the three-dimensional binaural layer 1110. It may be located between four upchannels and four downchannels included in the dimensional cubic.

In this case, the four upchannels may correspond to four speakers positioned at the top of the 3D cubic, and the four downchannels may correspond to four speakers positioned at the bottom of the 3D cubic.

That is, as shown in FIG. 11, the planar layer may be located within the height range of the cube corresponding to the three-dimensional cubic.

Accordingly, each of the speakers included in the surround layer 1120 or the proximity stereo layer 1130 may also be positioned between four upchannels and four downchannels included in the 3D cubic. In this case, in FIG. 11, the plane layer is illustrated in the form of a plane for convenience of description, but the shape of the plane layer according to the embodiment of the present invention may not be limited to the plane.

In addition, the multilayer binaural content generating method according to an embodiment of the present invention applies an audio effect corresponding to each of the 3D layer binaural output and the planar layer audio output based on a user input to the audio effect interface. (S220).

Since typical binaural audio sources are designed for maximum spatial effects, dynamic playback in terms of music is hard to expect. For this reason, if you apply an audio effect to a binaural audio source, the binaural image that provides the spatial effect is likely to be corrupted. For example, when the image of the space is reduced to produce a dynamic sound, the binaural effect may disappear, and the binaural effect may be degraded due to the increase of the bass.

Therefore, in the present invention, by applying the sound effects for each layer constituting the binaural audio source, the user can process the audio effect desired by the user while maintaining the binaural image. In this case, for the convenience of description, the binaural audio source is limitedly described, but the present invention is applied to both the binaural audio source composed of the multilayer and the non-binaural (NON-BINAURL) audio source composed of the multilayer. It may be possible.

In addition, although not shown in FIG. 2, the method for generating multilayer binaural content according to an embodiment of the present invention is based on a user input to an audio effect interface and outputs an audio effect to a subwoofer output corresponding to a subwoofer layer. Can be applied.

In the conventional binaural technology, since only one sound field is provided, an unintentional deformation may occur when an audio effect is applied. Accordingly, in the present invention, various effects may be provided for sound listening by separately applying audio effects for each layer of binaural content composed of multilayers.

For example, if you want to enlarge only the spatial image while maintaining the dynamic sound, you can apply the spatial effect only to the 3D binaural layer without changing the stereo layer.

At this time, the audio effect setting may be performed for at least one of the room setting, the equalizer setting, and the plug-in setting.

For example, referring to FIG. 14, the audio effect interface 1400 according to an embodiment of the present invention may correspond to a mode for adjusting a room size for each layer.

In this case, a mode for setting an audio effect may be changed through a tab menu included in the bottom of the audio effect interface 1400, and a flip button 1450 may also be moved to another effect setting window within each mode.

At this time, in the room size setting mode as illustrated in FIG. 14, the 3D binaural layer menu 1420, the surround layer menu 1430, and the proximity stereo layer menu 1440 are provided together with the visualization interface 1410. can do.

At this time, in the present invention, by providing the visualization interfaces 1410 and 1510 illustrated in FIGS. 14 and 15, the user may check the degree to which the sound changes as the size of the space changes in the sense of hearing and sight.

In this case, the visualization interface 1410 may change a mode in the form shown in FIG. 15 through the visualization flip button 1460.

In this case, the binaural layer menu 1420 and the surround layer menu 1430 illustrated in FIG. 14 may include a menu for distinguishing characteristics of a room by setting a material for reflecting sound. For example, the HARD setting can cause a hard reflection effect, such as cement, and the SOFT setting can cause a soft reflection effect, such as wood.

In addition, the binaural layer menu 1420 and the surround layer menu 1430 illustrated in FIG. 14 may be provided with a predelay fader for setting the size of the room. In this case, 100% of the pre delay value may be set as a reference, or may be set as a time value.

In this case, since the room size does not need to be adjusted in the proximity stereo layer, the width of the stereo may be adjusted through the fader of the proximity stereo layer menu 1440.

In addition, in the case of the subwoofer layer, since the audio effect does not need to be processed, it may be dim out in the room size setting mode.

At this time, by selecting the flip button 1450 illustrated in FIG. 14, the audio effect interface 1500 may be flipped to the room reverb setting mode as illustrated in FIG. 15.

At this time, in the layer-specific menu included in the room reverb setting mode, the reverberation time, that is, the reverb time for maintaining the reverb can be adjusted.

This audio effect interface allows you to set different audio effects for each layer so that you can enjoy binaural content that suits your taste. In addition, it is possible to adjust the size of the space while maintaining the overall sound field of the binaural content, and to select whether to enhance the binaural effect or the dynamic stereo sound.

In addition, the problem that the binaural effect is changed according to the diagram size of the headphone used by the user can be overcome by adjusting the sound effect for each layer.

In this case, the sound effect interface as illustrated in FIGS. 14 to 15 may provide an environment in which the equalizer EQ and the plug-in may be processed layer by layer in addition to the room setting, such as the tabs displayed at the bottom.

In addition, although not shown in FIG. 2, the method for generating a multilayer binaural content according to an embodiment of the present invention may reset an audio effect in consideration of automation information on a binaural audio source.

For example, the copyright holder of the binaural content may not want to distort the intention to convey by changing the sound by the user. Alternatively, the sound of the binaural content whose sound effect is corrected by the user may be worse than the original sound. Accordingly, in order to prevent such a problem, the audio effect may be reset to the state set at the production stage by applying the automation information generated at the authoring stage of the binaural content.

At this time, the audio effect may be applied to correspond to a predetermined application range of the entire range of the binaural audio source.

In addition, although not shown in FIG. 2, the method for generating a multilayer binaural content according to an embodiment of the present invention applies head tracking data corresponding to each of a 3D binaural output and a planar layer audio output. That is, the rotation parameter may be controlled for each layer of the binaural content.

This control can apply rotation parameters only to the layer that needs head tracking, which can help maintain the characteristics of each content.

At this time. The head tracking data may correspond to any one of sensor input based automatic head tracking data and user input based manual head tracking data for the head tracking interface.

In this case, the automatic head tracking data corresponds to data for tracking head movements of a user or a listener, and may be obtained corresponding to a tracking input based on a separate head tracking module.

In addition, manual head tracking data may be obtained corresponding to user input based on the head tracking interface.

For example, if the user or listener moves the head while wearing the head tracking module directly, the acceleration sensor and the 3-axis gyro sensor mounted on the head tracking module can measure the distance or angle of the movement of the user's head to automatically move the head. Can be generated and sent as tracking data.

As another example, manual head tracking information may be artificially assigned by a user or listener through the head tracking interface. That is, a user or a listener may input the head tracking data based on the head tracking interface regardless of whether the head tracking module receives the automatic head tracking data in order to artificially rotate the spatial image. In this case, the user or the listener may manually input and modify the head tracking data while listening to the multilayer binaural content that changes according to the mixing process or the input information for generating the multilayer binaural content.

At this time, the three-dimensional cubic corresponding to the three-dimensional binaural layer may be rotated corresponding to the rotation parameter of at least one of the pan (pan), tilt (tilt) and roll (roll).

In this way, the effect produced by rotating the three-dimensional cubic or moving up, down, left, and right according to the head tracking data may be mixed with the planar layer audio output in the future to generate a binaural stereo output. Accordingly, an immersive effect based on head tracking can be produced more efficiently than a conventional method of rotating or moving a surround layer, a proximity stereo layer, or a subwoofer layer corresponding to a planar layer.

In this case, the head tracking data may be converted into a log formula within a preset tracking limit and applied. That is, it is possible to set the threshold of the head tracking, and to control the head tracking data to reach the threshold in proportion to the movement.

By limiting the range of the head tracking as described above, it is possible to prevent the problem that the musical element disappears or the sound quality is distorted due to excessive head tracking.

For example, referring to FIG. 16, the head tracking interface 1600 may provide a limit range setting menu 1620, a log constant setting menu 1630, and a time limit setting menu 1640.

In this case, when each setting menu is set to ON, the setting menus may be set to be affected by the input head tracking data 1610.

First, the limited range setting menu 1620 may adjust a set angle limited through the fader. This may correspond to the limit setting value in the graph shown in FIG. 17.

In addition, in the log constant setting menu 1630, when head tracking data having a limit angle greater than the limit angle set in the limit range setting menu 1620 is input, the head tracking data may be formed as a log graph of a shape up to a preset time limit. Can be determined.

For example, the head tracking data 1720 processed in FIG. 17 may correspond to a result of applying a log function such as [Equation 1] to the received head tracking data 1710.

[Equation 1]

Y =-(log a (x + 1))

At this time, a value of the log function corresponding to [Equation 1] may be set through the log constant setting menu 1630. At this time, the value of a may be applied to the characteristics of the sound, it is possible to apply the deceleration applied value to x.

In addition, in the time limit setting menu 1640, when head tracking data having a limit angle greater than the limit angle set in the limit range setting menu 1620 is input, it is possible to additionally determine how long the log function ratio is to be applied. That is, the time limit 1730 illustrated in FIG. 17 may be set.

In addition, in the present invention, the volume and filter corresponding to each of the 3D layer binaural output, the planar layer audio output, and the subwoofer output may be adjusted.

For example, referring to FIG. 12, the volume interface 1200 according to an embodiment of the present invention may provide a menu for adjusting the volume for each layer. First, a preset may be selected through the preset and level meter mode selection menu 1210, and then one of the level meter modes, either pre-fader or post-fader, may be selected.

At this time, it is possible to select whether or not to receive the automation. In the case of setting to the locked state, by receiving the automation data, it may be set to a state in which the user cannot change the parameter. On the contrary, when set to the open state, it is possible to adjust the audio effect according to the user's taste regardless of whether or not the automation data is provided.

In this case, the 3D binaural layer menu 1230, the surround layer menu 1240, the proximity stereo layer menu 1250, and the subwoofer layer menu 1260 may be provided to adjust the volume for each layer. . In this case, the 3D binaural layer menu 1230, the surround layer menu 1240, and the proximity stereo layer menu 1250 are used to assign low cut data to the subwoofer (LFE) channel. A menu may be included. In addition, the subwoofer layer menu 1260 may provide only a frequency value that can be set in maximum.

At this time, in the volume interface 1200 according to an embodiment of the present invention, by providing a flip button 1270, a filter interface 1300 based menu capable of adjusting a filter for each layer as shown in FIG. 13 may be provided. It may be.

Therefore, the faders included in the layer-specific menu may be used to adjust the volume or set the filter according to the mode set through the flip buttons 1270 and 1310, respectively.

In this way, the user can freely adjust the volume and filter of the binaural layer so that the user can listen to the binaural content in consideration of the user's situation. This allows you to maximize your binaural effect, maximize your closeness, and more. You can also listen to binaural sound movements by adjusting the subwoofer level.

In addition, the multilayer binaural content generating method according to an embodiment of the present invention generates the multilayer binaural content by adding the 3D layer binaural output and the planar layer audio output to which the audio effect is applied (S230). At this time, multi-layer binaural content with the maximum binaural effect is generated by mixing an immersive element by 3D layer binaural output with a proximity playback element and an object element by flat layer audio output. can do.

In this case, when only the immersive sound is to be configured, binaural content may be generated using only the 3D layer binaural output.

In addition, although not shown in FIG. 2, the method for generating multilayer binaural content according to an embodiment of the present invention includes a three-dimensional layer binaural output and a flat layer audio to which an audio effect is applied to a subwoofer output to which an audio effect is applied. It can be summed up with the output to produce multilayer binaural content.

At this time, the subwoofer output can be summed to maximize the immersive effect corresponding to the multilayer binaural content and to produce a dynamic bass reproduction element.

For example, referring to FIG. 10, a signal of a single channel or two channels 1010 included in a subwoofer layer may be audio processed based on a low frequency effects bus 1020. That is, the subwoofer output 1030 may correspond to an output generated by processing single channel or two channel 1010 based audio, and may correspond to a single channel or two channels as shown in FIG. 10.

For example, the subwoofer layer may correspond to a single channel, such as 5.1 channels, 7.1 channels, and 11.1 channels, or may correspond to two channels, such as 10.2 channels and 22.2 channels.

In this case, the subwoofer layer may be separated from the 3D cubic or planar layer corresponding to the 3D binaural layer.

For example, as shown in FIG. 11, the subwoofer layer 1140 is separated from the 3D cubic 1110, the surround layer 1120, and the proximity stereo layer 1130 corresponding to the 3D binaural layer. Can be located. In this case, the structure shown in FIG. 11 corresponds to one embodiment, and is not limited to a structure in which respective layers are combined.

In addition, although not shown in Figure 2, the method for generating a multilayer binaural content according to an embodiment of the present invention can support various types of sound because it can support the natural upmix and downmix functions based on the above functions. It can improve the compatibility between the supporting content. For example, a surround image expressed through 3D cubic can be downmixed into a surround layer. The surround layer can also be downmixed back to a proximity stereo layer. As such, as the downmix is performed based on the area, the sound quality of the sound may be more effectively preserved.

In addition, although not shown in FIG. 2, the method for generating a multilayer binaural content according to an embodiment of the present invention may transmit and receive information necessary for generating the multilayer binaural content through a communication network such as a network. Particularly, in the present invention, a binaural audio source or content that can be input for generating multilayer binaural content, head tracking data input from a sensor, and information related to user input can be received, and the multilayer binaural content can be received. Can provide.

In addition, although not shown in Figure 2, the method for generating a multilayer binaural content according to an embodiment of the present invention various information generated in the process of generating the multilayer binaural content according to an embodiment of the present invention Save to a separate storage module.

The multilayer binaural content generation method can provide an listening environment suitable for a user's taste by providing an interface for individually editing effects for various sound elements.

In addition, it is possible to preserve sound quality that may be distorted by sudden changes in the head tracking data.

18 is a block diagram illustrating a terminal device for executing a multilayer binaural content generation program according to an embodiment of the present invention.

Referring to FIG. 18, a terminal device for executing a multilayer binaural content generating program according to an embodiment of the present invention includes a communication unit 1810, a processor 1820, and a memory 1830.

The communication unit 1810 transmits and receives information necessary for generating multilayer binaural content through a communication network such as a network. In particular, the communication unit 1810 according to an embodiment of the present invention may receive a binaural audio source or content input for generating multilayer binaural content, head tracking data input from a sensor, and information related to a user input. And may provide multilayer binaural content.

The processor 1820 generates a 3D layer binaural output by performing a 3D layer binaural encoding corresponding to the 3D binaural layer, and performs a audio processing corresponding to the planar layer to generate a planar layer audio output. Create

In this case, the 3D binaural layer corresponds to an element for creating a 3D spatial image. For example, referring to FIG. 4, for example, the 3D binaural layer may be 3D using a binaural encoder 420 corresponding to the 3D cubic method. Three-dimensional layer binaural encoding corresponding to a plurality of channels included in the binaural layer may be performed.

In this case, the 3D binaural layer may include four upchannels 411 and four downchannels 412 corresponding to eight-dimensional three-dimensional cubics.

Accordingly, the 3D layer binaural output 430 may correspond to an output generated by binaural encoding 8-channel based audio, and may be output corresponding to two channels as shown in FIG. 4. In addition, two channels corresponding to the 3D layer binaural output 430 may correspond to the left channel and the light channel, respectively.

In this case, although the embodiment shown in FIG. 4 uses an 8 channel based 3D cubic layer as the 3D binaural layer, the 3D binaural layer may not be limited thereto. That is, it may be configured to include other three-dimensional binaural layer applicable to the present invention or three-dimensional binaural layer to be developed in the future.

At this time, the three-dimensional layer binaural output corresponds to a three-dimensional vector of binaural points located on an eight-channel-based three-dimensional cubic composed of four up channels and four down channels. Can be generated.

For example, referring to FIG. 5, an 8-channel based 3D cubic includes four dynamic speakers corresponding to four upchannels 511 to 514 and four dynamic speakers corresponding to four downchannels. It may be a hexahedral structure with each vertex 515 to 518. At this time, since the positions of the eight dynamic speakers 511 to 518 can be changed, the range of binaural effects generated by the three-dimensional cubic can also be dynamically changed.

In another example, the conventional binaural method may generate immersive sound with eight dynamic speakers by generating three-dimensional cubic using vector base amplitude panning (Vbap) or ambisonics. have. That is, each of the eight dynamic speakers may be given position values for X, Y, and Z, and a vector-based virtual track point based on the midpoint of the 3D cubic may be expressed. In this case, the virtual track point may be represented corresponding to the parameter value included in the head tracking information.

Through such a three-dimensional cubic it is possible to generate a spatial image for the music content containing only audio, it is possible to represent the movement of the sound can provide a more three-dimensional effect.

In this case, the 3D cubic may be generated by changing the positions of the eight dynamic speakers corresponding to the vertices of the 3D cubic according to the size parameter for the 3D binaural layer. In other words, by freely changing the position of the dynamic speaker of the variable method other than the fixed method in accordance with the size parameter can be efficiently generated three-dimensional cubic.

For example, three-dimensional cubics having various ranges may be generated by processing the three-dimensional cubic by setting the size parameter to a constant and multiplying it by a binaural function.

In this case, the 3D vector may be included in the 3D cubic and generated based on a reference listening point corresponding to the center of the 2D plane corresponding to the surround layer.

For example, a reference listening point that virtually represents the location of a user or listener listening to binaural stereo audio is located inside a three-dimensional cubic with eight dynamic speakers at each vertex, but centered on the surround layer. Can be. At this time, assuming that the binaural point is located on the top surface of the 3D cubic, a 3D vector corresponding to the 3D layer binaural output may be generated in a direction from the reference listening point toward the binaural point.

In this case, the planar layer corresponds to a layer having a structure different from that of the 3D binaural layer, and may correspond to an element for making an image corresponding to a surround effect or a stereo effect.

Therefore, the flat layer performs surround layer binaural encoding to generate a surround layer binaural output, and receives a surround signal and a stereo signal that provides the generated surround layer binaural output as a flat layer audio output. It may be any one of a proximity stereo layer that generates a planar layer audio output corresponding to.

For example, referring to FIG. 6, a surround layer binaural encoding corresponding to a surround layer of five or seven channels 610 may be performed using the binaural encoder 620. In this case, as will be described with reference to FIG. 9, 7-channel-based surround layer binaural encoding may be performed by including 2 channels corresponding to a proximity stereo layer in the surround layer.

In this case, the surround layer may correspond to a structure including five speakers 711 to 715, for example, as shown in FIG. 7. In this case, the surround layer binaural output 630 may correspond to a binaural point located on the surround layer. If it is assumed that the listener is listening to the sound at the reference listening point located in the center of the surround layer, the surround layer binaural output 630 by binaural encoding as if the sound is coming from the binaural point on the surround layer. Can be generated. In this case, the surround layer corresponds to an element for creating a surround image corresponding to the surround effect. In FIG. 7, the surround layer is illustrated in the form of a plane for convenience of description, but may not be limited to the plane.

In this case, the surround layer binaural output 630 may be output corresponding to two channels as shown in FIG. 6. In addition, two channels corresponding to the surround layer binaural output 630 may correspond to the left channel and the right channel, respectively.

6 to 7 illustrate a surround layer corresponding to 5 or 7 channels 610, but the channel of the surround layer is not limited to 5 or 7 channels 610. In addition, although the surround layer is illustrated in the form of a rectangular plane in FIG. 7, the surround layer is not limited thereto and may be represented in various forms such as a line thickness, a planar shape, and a distance from a reference listening point.

For another example, referring to FIG. 8, audio processing may be performed corresponding to the proximity stereo layer of the two channels 810 based on the stereo bus 820. That is, the stereo signal 830 corresponding to the planar layer audio output may correspond to the output generated by processing the two channel 810 based stereo audio, and may be output corresponding to the two channels.

In this case, the proximity stereo layer corresponds to an element that creates a stereo image corresponding to the stereo effect, and may be included and represented as a part of the surround layer.

For example, as shown in FIG. 9, a surround stereo layer corresponding to two speakers 911 and 912 is included on a surround layer based on five speakers, so that the layer structure includes a total of seven speakers. It may be indicated.

In this case, as shown in FIG. 9, the proximity stereo layer may be disposed at a distance from the reference listening point 900 positioned on the surround layer. Alternatively, the proximity stereo layer may be used as the left and right side speakers of the reference listening point 900.

In this case, the stereo signal output corresponding to the proximity stereo layer may provide a damping feeling that is difficult to produce by the spatial parameter used for binaural encoding. Accordingly, the binaural stereo output according to an embodiment of the present invention may provide an immersive effect by binaural encoding and at the same time provide a damping feeling.

As such, a flat layer audio output corresponding to a surround layer binaural output or a flat layer audio output corresponding to a stereo signal corresponds to an output that includes only different sound effects when compared to a three dimensional layer binaural output. It may be. That is, the planar layer audio output may include various values than the 3D layer binaural output even if the output is not corresponding to the 3D layer.

In this case, the planar layer may be positioned between four upchannels and four downchannels corresponding to the 3D cubic.

For example, referring to FIG. 11, the surround layer 1120 and the proximity stereo layer 1130 corresponding to the planar layer according to an embodiment of the present invention are three corresponding to the three-dimensional binaural layer 1110. It may be located between four upchannels and four downchannels included in the dimensional cubic.

In this case, the four upchannels may correspond to four speakers positioned at the top of the 3D cubic, and the four downchannels may correspond to four speakers positioned at the bottom of the 3D cubic.

That is, as shown in FIG. 11, the planar layer may be located within the height range of the cube corresponding to the three-dimensional cubic.

Accordingly, each of the speakers included in the surround layer 1120 or the proximity stereo layer 1130 may also be positioned between four upchannels and four downchannels included in the 3D cubic. In this case, in FIG. 11, the plane layer is illustrated in the form of a plane for convenience of description, but the shape of the plane layer according to the embodiment of the present invention may not be limited to the plane.

In addition, the processor 1820 applies an audio effect corresponding to each of the 3D layer binaural output and the planar layer audio output based on a user input to the audio effect interface.

Since typical binaural audio sources are designed for maximum spatial effects, dynamic playback in terms of music is hard to expect. For this reason, if you apply an audio effect to a binaural audio source, the binaural image that provides the spatial effect is likely to be corrupted. For example, when the image of the space is reduced to produce a dynamic sound, the binaural effect may disappear, and the binaural effect may be degraded due to the increase of the bass.

Therefore, in the present invention, by applying the sound effects for each layer constituting the binaural audio source, the user can process the audio effect desired by the user while maintaining the binaural image. In this case, for the convenience of description, the binaural audio source is limitedly described, but the present invention is applied to both the binaural audio source composed of the multilayer and the non-binaural (NON-BINAURL) audio source composed of the multilayer. It may be possible.

In addition, the processor 1820 may apply the audio effect to the subwoofer output corresponding to the subwoofer layer based on a user input to the audio effect interface.

In the conventional binaural technology, since only one sound field is provided, an unintentional deformation may occur when an audio effect is applied. Accordingly, in the present invention, various effects may be provided for sound listening by separately applying audio effects for each layer of binaural content composed of multilayers.

For example, if you want to enlarge only the spatial image while maintaining the dynamic sound, you can apply the spatial effect only to the 3D binaural layer without changing the stereo layer.

At this time, the audio effect setting may be performed for at least one of the room setting, the equalizer setting, and the plug-in setting.

For example, referring to FIG. 14, the audio effect interface 1400 according to an embodiment of the present invention may correspond to a mode for adjusting a room size for each layer.

In this case, a mode for setting an audio effect may be changed through a tab menu included in the bottom of the audio effect interface 1400, and a flip button 1450 may also be moved to another effect setting window within each mode.

At this time, in the room size setting mode as illustrated in FIG. 14, the 3D binaural layer menu 1420, the surround layer menu 1430, and the proximity stereo layer menu 1440 are provided together with the visualization interface 1410. can do.

At this time, in the present invention, by providing the visualization interfaces 1410 and 1510 illustrated in FIGS. 14 and 15, the user may check the degree to which the sound changes as the size of the space changes in the sense of hearing and sight.

In this case, the visualization interface 1410 may change a mode in the form shown in FIG. 15 through the visualization flip button 1460.

In this case, the binaural layer menu 1420 and the surround layer menu 1430 illustrated in FIG. 14 may include a menu for distinguishing characteristics of a room by setting a material for reflecting sound. For example, the HARD setting can cause a hard reflection effect, such as cement, and the SOFT setting can cause a soft reflection effect, such as wood.

In addition, the binaural layer menu 1420 and the surround layer menu 1430 illustrated in FIG. 14 may be provided with a predelay fader for setting the size of the room. In this case, 100% of the pre delay value may be set as a reference, or may be set as a time value.

In this case, since the room size does not need to be adjusted in the proximity stereo layer, the width of the stereo may be adjusted through the fader of the proximity stereo layer menu 1440.

In addition, in the case of the subwoofer layer, since the audio effect does not need to be processed, it may be dim out in the room size setting mode.

At this time, by selecting the flip button 1450 illustrated in FIG. 14, the audio effect interface 1500 may be flipped to the room reverb setting mode as illustrated in FIG. 15.

At this time, in the layer-specific menu included in the room reverb setting mode, the reverberation time, that is, the reverb time for maintaining the reverb can be adjusted.

This audio effect interface allows you to set different audio effects for each layer so that you can enjoy binaural content that suits your taste. In addition, it is possible to adjust the size of the space while maintaining the overall sound field of the binaural content, and to select whether to enhance the binaural effect or the dynamic stereo sound.

In addition, the problem that the binaural effect is changed according to the diagram size of the headphone used by the user can be overcome by adjusting the sound effect for each layer.

In this case, the sound effect interface as illustrated in FIGS. 14 to 15 may provide an environment in which the equalizer EQ and the plug-in may be processed layer by layer in addition to the room setting, such as the tabs displayed at the bottom.

In addition, the processor 1820 may reset the audio effect in consideration of automation information about the binaural audio source.

For example, the copyright holder of the binaural content may not want to distort the intention to convey by changing the sound by the user. Alternatively, the sound of the binaural content whose sound effect is corrected by the user may be worse than the original sound. Accordingly, in order to prevent such a problem, the audio effect may be reset to the state set at the production stage by applying the automation information generated at the authoring stage of the binaural content.

At this time, the audio effect may be applied to correspond to a predetermined application range of the entire range of the binaural audio source.

In addition, the processor 1820 applies head tracking data corresponding to each of the three-dimensional binaural output and the planar layer audio output. That is, the rotation parameter may be controlled for each layer of the binaural content.

This control can apply rotation parameters only to the layer that needs head tracking, which can help maintain the characteristics of each content.

At this time. The head tracking data may correspond to any one of sensor input based automatic head tracking data and user input based manual head tracking data for the head tracking interface.

In this case, the automatic head tracking data corresponds to data for tracking head movements of a user or a listener, and may be obtained corresponding to a tracking input based on a separate head tracking module.

In addition, manual head tracking data may be obtained corresponding to user input based on the head tracking interface.

For example, if the user or listener moves the head while wearing the head tracking module directly, the acceleration sensor and the 3-axis gyro sensor mounted on the head tracking module can measure the distance or angle of the movement of the user's head to automatically move the head. Can be generated and sent as tracking data.

As another example, manual head tracking information may be artificially assigned by a user or listener through the head tracking interface. That is, a user or a listener may input the head tracking data based on the head tracking interface regardless of whether the head tracking module receives the automatic head tracking data in order to artificially rotate the spatial image. In this case, the user or the listener may manually input and modify the head tracking data while listening to the multilayer binaural content that changes according to the mixing process or the input information for generating the multilayer binaural content.

At this time, the three-dimensional cubic corresponding to the three-dimensional binaural layer may be rotated corresponding to the rotation parameter of at least one of the pan (pan), tilt (tilt) and roll (roll).

In this way, the effect produced by rotating the three-dimensional cubic or moving up, down, left, and right according to the head tracking data may be mixed with the planar layer audio output in the future to generate a binaural stereo output. Accordingly, an immersive effect based on head tracking can be produced more efficiently than a conventional method of rotating or moving a surround layer, a proximity stereo layer, or a subwoofer layer corresponding to a planar layer.

In this case, the head tracking data may be converted into a log formula within a preset tracking limit and applied. That is, it is possible to set the threshold of the head tracking, and to control the head tracking data to reach the threshold in proportion to the movement.

By limiting the range of the head tracking as described above, it is possible to prevent the problem that the musical element disappears or the sound quality is distorted due to excessive head tracking.

For example, referring to FIG. 16, the head tracking interface 1600 may provide a limit range setting menu 1620, a log constant setting menu 1630, and a time limit setting menu 1640.

In this case, when each setting menu is set to ON, the setting menus may be set to be affected by the input head tracking data 1610.

First, the limited range setting menu 1620 may adjust a set angle limited through the fader. This may correspond to the limit setting value in the graph shown in FIG. 17.

In addition, in the log constant setting menu 1630, when head tracking data having a limit angle greater than the limit angle set in the limit range setting menu 1620 is input, the head tracking data may be formed as a log graph of a shape up to a preset time limit. Can be determined.

For example, the head tracking data 1720 processed in FIG. 17 may correspond to a result of applying a log function such as [Equation 1] to the received head tracking data 1710.

[Equation 1]

Y =-(log a (x + 1))

At this time, a value of the log function corresponding to [Equation 1] may be set through the log constant setting menu 1630. At this time, the value of a may be applied to the characteristics of the sound, it is possible to apply the deceleration applied value to x.

In addition, in the time limit setting menu 1640, when head tracking data having a limit angle greater than the limit angle set in the limit range setting menu 1620 is input, it is possible to additionally determine how long the log function ratio is to be applied. That is, the time limit 1730 illustrated in FIG. 17 may be set.

In addition, the processor 1820 may adjust the volume and filter corresponding to each of the dimensional layer binaural output, the planar layer audio output, and the subwoofer output.

For example, referring to FIG. 12, the volume interface 1200 according to an embodiment of the present invention may provide a menu for adjusting the volume for each layer. First, a preset may be selected through the preset and level meter mode selection menu 1210, and then one of the level meter modes, either pre-fader or post-fader, may be selected.

At this time, it is possible to select whether or not to receive the automation. In the case of setting to the locked state, by receiving the automation data, it may be set to a state in which the user cannot change the parameter. On the contrary, when set to the open state, it is possible to adjust the audio effect according to the user's taste regardless of whether or not the automation data is provided.

In this case, the 3D binaural layer menu 1230, the surround layer menu 1240, the proximity stereo layer menu 1250, and the subwoofer layer menu 1260 may be provided to adjust the volume for each layer. . In this case, the 3D binaural layer menu 1230, the surround layer menu 1240, and the proximity stereo layer menu 1250 are used to assign low cut data to the subwoofer (LFE) channel. A menu may be included. In addition, the subwoofer layer menu 1260 may provide only a frequency value that can be set in maximum.

At this time, in the volume interface 1200 according to an embodiment of the present invention, by providing a flip button 1270, a filter interface 1300 based menu capable of adjusting a filter for each layer as shown in FIG. 13 may be provided. It may be.

Therefore, the faders included in the layer-specific menu may be used to adjust the volume or set the filter according to the mode set through the flip buttons 1270 and 1310, respectively.

In this way, the user can freely adjust the volume and filter of the binaural layer so that the user can listen to the binaural content in consideration of the user's situation. This allows you to maximize your binaural effect, maximize your closeness, and more. You can also listen to binaural sound movements by adjusting the subwoofer level.

In addition, the processor 1820 generates multilayer binaural content by adding the 3D layer binaural output and the planar layer audio output to which the audio effect is applied.

At this time, multi-layer binaural content with the maximum binaural effect is generated by mixing an immersive element by 3D layer binaural output with a proximity playback element and an object element by flat layer audio output. can do.

In this case, when only the immersive sound is to be configured, binaural content may be generated using only the 3D layer binaural output.

In addition, the processor 1820 may generate the multilayer binaural content by adding the subwoofer output to which the audio effect is applied together with the 3D layer binaural output and the planar layer audio output to which the audio effect is applied.

At this time, the subwoofer output can be summed to maximize the immersive effect corresponding to the multilayer binaural content and to produce a dynamic bass reproduction element.

For example, referring to FIG. 10, a signal of a single channel or two channels 1010 included in a subwoofer layer may be audio processed based on a low frequency effects bus 1020. That is, the subwoofer output 1030 may correspond to an output generated by processing single channel or two channel 1010 based audio, and may correspond to a single channel or two channels as shown in FIG. 10.

For example, the subwoofer layer may correspond to a single channel, such as 5.1 channels, 7.1 channels, and 11.1 channels, or may correspond to two channels, such as 10.2 channels and 22.2 channels.

In this case, the subwoofer layer may be separated from the 3D cubic or planar layer corresponding to the 3D binaural layer.

For example, as shown in FIG. 11, the subwoofer layer 1140 is separated from the 3D cubic 1110, the surround layer 1120, and the proximity stereo layer 1130 corresponding to the 3D binaural layer. Can be located. In this case, the structure shown in FIG. 11 corresponds to one embodiment, and is not limited to a structure in which respective layers are combined.

In addition, the processor 1820 may support a natural upmix and downmix function based on the above functions, thereby improving compatibility between contents supporting various kinds of sounds. For example, a surround image expressed through 3D cubic can be downmixed into a surround layer. The surround layer can also be downmixed back to a proximity stereo layer. As such, as the downmix is performed based on the area, the sound quality of the sound may be more effectively preserved.

As described above, the memory 1830 stores various information generated in the process of generating the multilayer binaural content according to the exemplary embodiment of the present invention.

According to an embodiment, the memory 1830 may be configured independently of the terminal device generating the multilayer binaural content to support the multilayer binaural content generating function. In this case, the memory 1830 may operate as a separate mass storage, and may include a control function for performing an operation.

Meanwhile, the terminal device may be equipped with a memory to store information in the device. In one embodiment, the memory is a computer readable medium. In one implementation, the memory may be a volatile memory unit, and for other implementations, the memory may be a nonvolatile memory unit. In one embodiment, the storage device is a computer readable medium. In various different implementations, the storage device may include, for example, a hard disk device, an optical disk device, or some other mass storage device.

By providing an interface for individually editing effects for various sound elements based on such a terminal device, a listening environment suitable for a user's taste can be provided.

In addition, it is possible to preserve sound quality that may be distorted by sudden changes in the head tracking data.

Embodiments of the present invention may be implemented in a computer-implemented method or a non-transitory computer-readable medium on which computer-executable instructions are recorded. When computer readable instructions are executed by a processor, the computer readable instructions may perform a method according to at least one aspect of the present invention.

As described above, the multilayer binaural content generating method and the program for the same according to the present invention are not limited to the configuration and method of the embodiments described as described above, the embodiments are various modifications can be made All or part of each of the embodiments may be configured to be selectively combined to make it possible.

110: binaural audio source 120: multilayer decoder
130: layer 140: individual editing module for each layer
150: audio mixer 160: multilayer binaural content
310: multilayer binaural content generation module
411: 4 upchannels 412: 4 downchannels
420, 620: binaural encoder 430: three-dimensional layer binaural output
511 to 518: Dynamic speaker 610: 5 or 7 channels
630: surround layer binaural output
711 to 715, 911, 912: Speaker 810: 2 channels
820: Stereo Bus 830: Stereo Signal
900: reference listening point 1010: single channel or two channels
1020: Low Frequency Effects (LFE) bus
1030: subwoofer output 1110: three-dimensional binaural layer
1120: surround layer 1130: proximity stereo layer
1140: subwoofer layer 1200: volume interface
1210: Preset and Level Meter Mode Selection Menu 1220: Tab Menu
1230, 1420: 3D binaural layer menu
1240, 1430: Surround layer menu
1250, 1440: Close-up stereo layer menu
1260: Subwoofer layer menu 1270, 1310, 1450, 1520: Flip button
1300: filter interface 1400, 1500: audio effect interface
1410, 1510: visualization interface
1460, 1530: Visualization Flip Button
1600: head tracking interface 1610: head tracking data
1620: Limit Range Setting Menu 1630: Log Constant Setting Menu
1640: timeout setting menu 1710: received head tracking data
1720: processed head tracking data 1810: communication unit
1820: Processor 1830: Memory

Claims (20)

Generating a 3D layer binaural output by performing a 3D layer binaural encoding corresponding to the 3D binaural layer, and generating a planar layer audio output by performing audio processing corresponding to the planar layer;
Applying an audio effect corresponding to each of the three-dimensional layer binaural output and the planar layer audio output based on a user input to an audio effect interface; And
Creating multi-layer binaural content by combining three-dimensional layer binaural output and flat layer audio output with audio effects applied
Multi-layer binaural content generating method comprising a. The method according to claim 1,
The multilayer binaural content generating method
And applying head tracking data corresponding to each of the three-dimensional binaural output and the planar layer audio output. The method according to claim 2,
The head tracking data is
A method for generating multilayer binaural content, characterized in that it corresponds to any one of sensor input based automatic head tracking data and user input based manual head tracking data for the head tracking interface. The method according to claim 3,
The head tracking data is
A method for generating multilayer binaural content, characterized in that the conversion is applied to a log formula within a preset tracking limit. The method according to claim 2,
Applying the audio effect
And setting at least one of a room setting, an equalizer setting, a plug-in setting, and the like. The method according to claim 1,
The planar layer is
A surround layer that performs surround layer binaural encoding to generate a surround layer binaural output, and provides the generated surround layer binaural output to the planar layer audio output;
And a proximity stereo layer receiving a stereo signal and generating the planar layer audio output corresponding to the stereo signal. The method according to claim 1,
The 3D layer binaural output
Multilayer binaural, characterized in that it is generated corresponding to the three-dimensional vector for the binaural point located on the eight-channel based three-dimensional cubic (cub) consisting of four up channels and four down channels How to create content. The method according to claim 1,
Generating the multilayer binaural content
Applying an audio effect corresponding to a subwoofer output corresponding to a subwoofer layer based on a user input to the audio effect interface; And
And adding the subwoofer output to which the audio effect is applied together with the 3D layer binaural output and the planar layer audio output to which the audio effect is applied to generate the multilayer binaural content. How to create binaural content. The method according to claim 8,
The multilayer binaural content generating method
And resetting the audio effect in consideration of automation information regarding a binaural audio source. The method according to claim 9,
Applying the audio effect
And applying the audio effect corresponding to a preset application range among the entire range of the binaural audio source. Generating a 3D layer binaural output by performing a 3D layer binaural encoding corresponding to the 3D binaural layer, and generating a planar layer audio output by performing audio processing corresponding to the planar layer;
Applying an audio effect corresponding to each of the three-dimensional layer binaural output and the planar layer audio output based on a user input to an audio effect interface; And
Multi-layer binaural content stored on a computer-readable recording medium, comprising the steps of generating a multi-layer binaural content by combining the three-dimensional layer binaural output and the flat layer audio output to which the audio effect is applied. program. The method according to claim 11,
And applying head tracking data corresponding to each of the three-dimensional binaural output and the planar layer audio output. 17. The multilayer binaural content generating program stored in the computer-readable recording medium. The method according to claim 12,
The head tracking data is
A multi-layer binaural content generating program stored in a computer-readable recording medium corresponding to any one of sensor input based automatic head tracking data and user input based manual head tracking data for the head tracking interface. The method according to claim 13,
The head tracking data is
A multi-layer binaural content generating program stored in a computer-readable recording medium, characterized in that the conversion is applied to the log formula within the preset tracking limit. The method according to claim 12,
Multi-stored on a computer-readable recording medium, which applies the audio effect by performing at least one of a room setting, an equalizer setting, and a plug-in setting. Binaural Content Generator. The method according to claim 11,
The planar layer is
A surround layer that performs surround layer binaural encoding to generate a surround layer binaural output, and provides the generated surround layer binaural output to the planar layer audio output;
And a proximity stereo layer that receives a stereo signal and generates the planar layer audio output corresponding to the stereo signal. 2. The multilayer binaural content generating program stored in the computer-readable recording medium. The method according to claim 11,
The 3D layer binaural output
A computer-readable record generated corresponding to a three-dimensional vector of binaural points located on an eight-channel based three-dimensional cubic consisting of four up channels and four down channels Multi-layer binaural content generating program stored on the media. The method according to claim 11,
Applying an audio effect corresponding to a subwoofer output corresponding to a subwoofer layer based on a user input to the audio effect interface; And
And adding the subwoofer output to which the audio effect is applied together with the 3D layer binaural output and the planar layer audio output to which the audio effect is applied to generate the multilayer binaural content. A multilayer binaural content generating program stored on a readable recording medium. The method according to claim 18,
And resetting the audio effect in consideration of automation information for the binaural audio source. The multilayer binaural content generating program stored in the computer-readable recording medium. The method according to claim 19,
And applying the audio effect in accordance with a predetermined application range among the entire range of the binaural audio source.

Images