KR20240008241A - The method of rendering audio based on recording distance parameter and apparatus for performing the same - Google Patents

Abstract

An audio rendering method and an apparatus for performing the same are disclosed. An audio rendering method according to an embodiment includes obtaining a first distance related to recording an audio signal about an audio object, obtaining a second distance that is a distance between the audio object and a listener, and the first distance. And based on the second distance, it may include an operation of rendering the audio signal by applying an effect due to attenuation due to air absorption.

Description

Audio rendering method based on recording distance parameter and device for performing the same {THE METHOD OF RENDERING AUDIO BASED ON RECORDING DISTANCE PARAMETER AND APPARATUS FOR PERFORMING THE SAME}

The disclosure below relates to a recording distance parameter-based audio rendering method and a device for performing the same.

Audio services have evolved from mono and stereo services to multichannel services such as 5.1 and 7.1 channels, 9.1, 11.1, 10.2, 13.1, 15.1, and 22.2 channels. Unlike existing channel-based audio services, object-based audio service technology that considers a single sound source as an object is being developed. Object-based audio services can store, transmit, and play object audio signals and object audio-related information (e.g., location of object audio, size of object audio).

Information required when rendering object-based audio signals includes the relative angle and distance between the audio object and the listener, and acoustic spatial information is additionally used to create object-based audio. There are cases where signals are rendered. This is because acoustic space information is information that allows acoustic transmission characteristics according to space to be better realized. Using acoustic spatial information to implement acoustic propagation characteristics in detail and render object-based audio signals can require very complex calculations. In order to simply implement spatial sound transmission characteristics, a method of rendering object-based audio signals was proposed by dividing them into direct sound, early reflections, and late reverberations. .

The background technology described above is possessed or acquired by the inventor in the process of deriving the disclosure of the present application, and cannot necessarily be said to be known technology disclosed to the general public before this application.

Embodiments can provide a rendering technology that can prevent the effect of attenuation due to air absorption between the sound source and the recording distance from being applied overlappingly by introducing a parameter related to the recording distance.

However, technical challenges are not limited to the above-mentioned technical challenges, and other technical challenges may exist.

An audio rendering method according to an embodiment includes obtaining a first distance related to recording of an audio signal related to an audio object, and obtaining a second distance that is a distance between the audio object and a listener; Based on the first distance and the second distance, the method may include rendering the audio signal by applying an attenuation effect due to air sound-absorption.

The rendering operation includes calculating a third distance based on the difference between the first distance and the second distance, and applying an effect due to attenuation due to air absorption based on the third distance to produce the audio. It may include the operation of rendering a signal.

Calculating the third distance may include calculating the third distance by decreasing the second distance by the first distance.

The operation of calculating the third distance may include calculating the third distance differently based on the difference between the first distance and the second distance.

The operation of calculating the third distance differently includes calculating the third distance as a predetermined value when the second distance is less than or equal to the first distance and calculating the third distance as a predetermined value when the second distance is greater than the first distance. In this case, it may include calculating the third distance by reducing the second distance by the first distance.

The first distance may include the distance between a sound source that is the subject of recording and a recording device.

The rendering operation includes an operation of rendering the audio signal by compensating a tone by an amount equal to the effect of attenuation due to air absorption when the second distance is smaller than the first distance. can do.

An audio rendering device according to an embodiment includes a memory including instructions, a processor electrically connected to the memory, and configured to execute the instructions, and when the instructions are executed by the processor, the processor includes a plurality of Controlling the operations of, wherein the plurality of operations includes obtaining a first distance related to recording of an audio signal related to an audio object, and obtaining a second distance that is a distance between the audio object and a listener. And, based on the first distance and the second distance, an operation of rendering the audio signal by applying an attenuation effect according to air sound-absorption may be included.

The rendering operation includes calculating a third distance based on the difference between the first distance and the second distance, and applying an effect due to attenuation due to air absorption based on the third distance to produce the audio. It may include the operation of rendering a signal.

Calculating the third distance may include calculating the third distance by decreasing the second distance by the first distance.

The operation of calculating the third distance may include calculating the third distance differently based on the difference between the first distance and the second distance.

The operation of calculating the third distance differently includes calculating the third distance as a predetermined value when the second distance is less than or equal to the first distance and calculating the third distance as a predetermined value when the second distance is greater than the first distance. In this case, it may include calculating the third distance by reducing the second distance by the first distance.

The first distance may include the distance between a sound source that is the subject of recording and a recording device.

The rendering operation includes an operation of rendering the audio signal by compensating a tone by an amount equal to the effect of attenuation due to air absorption when the second distance is smaller than the first distance. can do.

1 is a block diagram showing an overview of the components of a renderer according to an embodiment.
FIG. 2 is a diagram for explaining the encoder structure of the renderer shown in FIG. 1.
FIG. 3 is a diagram for explaining renderer steps of the renderer module of the renderer shown in FIG. 1.
Figure 4 is a diagram for explaining the relationship between the recording distance and the attenuation effect due to air absorption.
Figure 5 is a diagram for explaining an example of calculating the distance for applying the air sound absorption effect.
Figure 6 is a diagram for explaining an example of calculating the distance for applying the air sound absorption effect considering the recording distance.
Figures 7 and 8 are diagrams for explaining methods for calculating a distance (eg, a third distance) for applying the air sound absorption effect according to an embodiment.
Figure 9 shows the spectrum of an audio signal before and after applying an audio rendering method according to an embodiment.
Figure 10 is a flowchart for explaining a method of rendering an audio signal according to an embodiment.
Figure 11 is a schematic block diagram of a device according to one embodiment.

Specific structural or functional descriptions of the embodiments are disclosed for illustrative purposes only and may be changed and implemented in various forms. Accordingly, the actual implementation form is not limited to the specific disclosed embodiments, and the scope of the present specification includes changes, equivalents, or substitutes included in the technical idea described in the embodiments.

Terms such as first or second may be used to describe various components, but these terms should be interpreted only for the purpose of distinguishing one component from another component. For example, a first component may be named a second component, and similarly, the second component may also be named a first component.

When a component is referred to as being “connected” to another component, it should be understood that it may be directly connected or connected to the other component, but that other components may exist in between.

Singular expressions include plural expressions unless the context clearly dictates otherwise. As used herein, “A or B”, “at least one of A and B”, “at least one of A or B”, “A, B or C”, “at least one of A, B and C”, and “A Each of phrases such as “at least one of , B, or C” may include any one of the items listed together in the corresponding phrase, or any possible combination thereof. In this specification, terms such as “comprise” or “have” are intended to designate the presence of the described features, numbers, steps, operations, components, parts, or combinations thereof, and are intended to indicate the presence of one or more other features or numbers, It should be understood that this does not exclude in advance the possibility of the presence or addition of steps, operations, components, parts, or combinations thereof.

Unless otherwise defined, all terms used herein, including technical or scientific terms, have the same meaning as commonly understood by a person of ordinary skill in the art. Terms as defined in commonly used dictionaries should be interpreted as having meanings consistent with the meanings they have in the context of the related technology, and unless clearly defined in this specification, should not be interpreted in an idealized or overly formal sense. No.

The term “module” used in this document may include a unit implemented in hardware, software, or firmware, and may be used interchangeably with terms such as logic, logic block, component, or circuit, for example. A module may be an integrated part or a minimum unit of the parts or a part thereof that performs one or more functions. For example, according to one embodiment, the module may be implemented in the form of an application-specific integrated circuit (ASIC).

The term '~unit' used in this document refers to software or hardware components such as FPGA or ASIC, and '~unit' performs certain roles. However, '~part' is not limited to software or hardware. The '~ part' may be configured to reside in an addressable storage medium and may be configured to reproduce on one or more processors. For example, '~part' refers to software components, object-oriented software components, components such as class components and task components, processes, functions, properties, procedures, May include subroutines, segments of program code, drivers, firmware, microcode, circuitry, data, databases, data structures, tables, arrays, and variables. The functions provided within the components and 'parts' may be combined into a smaller number of components and 'parts' or may be further separated into additional components and 'parts'. Additionally, components and 'parts' may be implemented to regenerate one or more CPUs within a device or a secure multimedia card. Additionally, '~ part' may include one or more processors.

Hereinafter, embodiments will be described in detail with reference to the attached drawings. In the description with reference to the accompanying drawings, identical components will be assigned the same reference numerals regardless of the reference numerals, and overlapping descriptions thereof will be omitted.

1 is a block diagram showing an overview of the components of a renderer according to an embodiment.

Referring to FIG. 1, according to one embodiment, the renderer 10 may be an MPEG-I Immersive Audio standard renderer. MPEG (moving picture experts group) is in the process of standardizing MPEG-I Immersive Audio, a standard for rendering audio signals in a 6DoF (degree of freedom) VR (virtual reality) environment. The scope of standardization includes metadata bitstream and real-time rendering technology to effectively render audio signals in a 6DoF VR environment.

As audio in a 6DoF VR environment, channel-based audio, object-based audio, and scene-based audio are used. Contributions were made to metadata and real-time rendering technology to render the above audio signals well, and the initial version of the MPEG-I Immersive Audio standard renderer (e.g. RM0 (reference model 0)) was selected as a standard and CE ( Core experiments are in progress.

The MPEG-I Immersive Audio standard renderer may include a control unit and a rendering unit. The control unit may include a clock module (101), a scene module (103), and a stream management module (107). The rendering unit may include a renderer module 110, a spatializer, and a limiter. The MPEG-I Immersive Audio standard renderer can render object-based audio signals (hereinafter referred to as object audio signals).

The MPEG-I Immersive Audio standard renderer can interface with external systems and components through the control unit.

The clock module 101 may use a clock input (101_1) as an input. The clock input 101_1 may include synchronization signals with an external module and/or a reference time of the renderer itself (eg, a renderer internal clock). The clock module 101 may output current time information of the scene to the scene module 103.

The scene module 103 can process changes in all internal or external scene information. The scene module 103 inputs information (e.g., listener space description format (LSDF) and listener location and dynamic update information (local updates) (103_1)) and bits received from the external interface of the renderer. Information transmitted by the stream 105 (eg, scene updates information) may be used as input. The scene module 103 may include a scene information module 103_3. The scene information module 103_3 may update the current state of all metadata (e.g., acoustic elements, physical objects) related to 6DoF rendering of the scene. The scene information module 103_3 may output current scene information to the renderer module 110.

The stream management module 107 may provide an interface for inputting acoustic signals (eg, audio input 100) for acoustic elements of the scene information module 103_3. The audio input 100 may be a pre-encoded or decoded sound source signal, a local sound source, or a remote sound source. The stream management module 107 may output an audio signal to the renderer module 110. The renderer module 110 may render the sound signal input from the stream management module 107 using the current scene information. The renderer module 110 may include renderer steps for signal processing and rendering parameter processing of an audio signal (eg, render item) to be rendered.

FIG. 2 is a diagram for explaining the encoder structure of the renderer shown in FIG. 1.

Referring to FIG. 2, according to one embodiment, a renderer (eg, renderer 10 of FIG. 1) may include an encoder (eg, encoder 200). The encoder 200 may include an encoder input format (EIF) parser module 210, a scene metadata module 230, and a bitstream generation module 250. The EIF parser module 210 may input directivity information in EIF and/or SOFA (spatial oriented format for audio) format, which are common input formats of the MPEG-I Immersive Audio encoder. The EIF parser module 210 analyzes information in the EIF and/or SOFA format to include elements constituting scene information of the content (e.g., spatial geometric structure information, sound source information (e.g., location, shape, and directivity of the sound source), Acoustic characteristic information of materials and spaces, update information (e.g., movement information)) can be extracted.

The scene metadata module 230 includes a sound source metadata generation module, a multi HOA (higher order ambisonics) metadata generation module, a reverberation parameterization module, Low-complexity early reflections parameterization module, portal generation module, sound source/object mobility analysis module, mesh merge module , a diffraction path analysis module, and an initial reflective surface and array analysis module.

The bitstream generation module 250 may receive the metadata generated by each module of the encoder 200 and the orientation information of the SOFA file, quantize and multiplex it, and generate a bitstream.

Figure 3 is a diagram for explaining the renderer steps of the renderer module.

Referring to FIG. 3, according to one embodiment, FIG. 3 may be a diagram for explaining renderer steps of the renderer module 110 shown in FIG. 1. Each renderer stage can be executed in a predetermined order. At each renderer stage, render items can be selectively disabled or enabled. Each renderer stage can render the activated render item. Below, each renderer step of the renderer module 110 will be described.

The room assigning stage 301 may be a step in which, when a listener enters a room containing acoustic environment information, metadata of the acoustic environment information about the room the listener entered may be applied to each render item. there is.

The reverberation stage 303 may be a stage that generates reverberation according to acoustic environment information of the current space (eg, a room containing acoustic environment information). The reverberation step 303 may be a step of receiving reverberation parameters from a bitstream (bitstream 105 in FIG. 1), attenuating a feedback delay network (FDN) reverberator, and initializing delay parameters.

The portal stage 305 may be a stage for modeling a sound transmission path. Specifically, the portal step 305 may be a step of modeling a partially open sound transmission path (eg, portal) between spaces with different acoustic environment information for late reverberation. In acoustics, a portal is an abstract concept that models the transmission of sound from one space to another through a geometrically defined opening. The portal step 305 may be a step of modeling the entire space where the sound source is located as a uniform volume sound source. The portal step 305 may be a step in which a wall is considered an obstacle and a render item is rendered as a uniform volume sound source according to the shape information of the portal included in the bitstream 105.

The early reflections stage 307 is a stage in which a rendering method can be selected considering the quality of rendering and the amount of computation. The early reflection step 307 may be omitted. Rendering methods that may be selected in the early reflections step 307 may include a high-quality early reflections rendering method and a low-complexity early reflections rendering method. A high-quality early reflection sound rendering method may be a method of calculating early reflection sounds by determining the visibility of an image source with respect to a wall surface that causes early reflections included in the bitstream 105. The low-complexity early reflection sound rendering method may be a method of replacing the early reflection sound section using simple predefined early reflection sound patterns.

The volume sound source discovery stage 309 uses sound lines radiating in all directions to render a sound source (e.g., a volume sound source) having a spatial size including a portal. This may be the step of finding the intersection point for each portal or volume source. Information (e.g., the intersection of a sound line and a portal) found in the volume sound source discovery step 309 may be output to the obstacle step 311 and the uniform volume sound source step 329.

The obstacles stage 311 may provide information about obstacles on a straight path between the sound source and the listener. The obstacle step 311 may be a step of updating a status flag for fade in-out processing at the boundary of the obstacle and an equalizer (EQ) parameter based on the transmittance of the obstacle.

The diffraction stage 313 may be a stage that generates information necessary to generate a diffracted sound source transmitted to the listener from a sound source blocked by an obstacle. For a fixed sound source, a pre-calculated diffraction path can be used to generate information. For moving sound sources, diffraction paths calculated from potential diffraction edges can be used to generate information.

The metadata management stage 315 can reduce the amount of computation in subsequent stages when the render item is distance attenuated or attenuated below the audible range by an obstacle. This may be a step to deactivate the attenuated render item.

The multi-volume sound source stage 317 may be a stage for rendering a sound source that includes a plurality of sound source channels and has a spatial size.

The directivity stage 319 may be a stage in which a directivity parameter (e.g., gain for each band) for the current direction of the sound source is applied to a render item for which directivity information is defined. The directivity step 319 may be a step of additionally applying the gain for each band to the existing EQ (equalizer) value.

The distance stage 321 applies the effects of delay due to the distance between the sound source and the listener, distance attenuation, and attenuation due to air sound-absorption. It may be a step. The distance step 321 may be a step of applying propagation delay to the signal associated with the render item to generate physically accurate delay and Doppler effect. The distance step 321 may be a step of modeling frequency independent attenuation of audio elements due to geometric diffusion of sound source energy by applying distance attenuation. The distance step 321 models the frequency dependent attenuation of audio elements related to the sound-absorption characteristics of air and applies the effect of medium absorption to the object audio signal. You can. The distance step 321 may be a step of calculating the distance between the listener and the audio object using the location of the listener and the location of the audio object. The distance step 321 may be a step of determining the gain of the object audio signal by applying distance attenuation according to the distance between the audio object and the listener.

The equalizer stage 323 may be a stage in which a finite impulse response (FIR) filter is applied to the gain value for each frequency band accumulated by obstacle transmission, diffraction, initial reflection, directivity, distance attenuation, etc.

The fade stage 325 reduces discontinuous distortion that can occur when the activation of a render item changes or the listener suddenly moves in space through fade in-out processing. This may be the step to do so.

The single HOA (higher order ambisonics) stage 327 may be a stage for rendering background sound by one HOA sound source. The single HOA step 327 converts the signal in ESD (equivalent spatial domain) format input from the bitstream 105 into HOA and converts it into a binaural signal through a magnitude least squares (MagLS) decoder. This may be a step. That is, the single HOA step 327 may be a step of converting the input audio into HOA and spatially combining and converting the signal through HOA decoding.

The uniform volume sound source stage 329 may be a stage of rendering a sound source (eg, a uniform volume sound source) that has a spatial size and a single characteristic. The uniform volume sound source step 329 may be a step of imitating the effects of countless sound sources in the volume sound source space through a decorrelated stereo sound source. The uniform volume sound source step 329 may be a step of generating the effect of the blocked sound source based on the information from the obstacle step 311 when the effect of the sound source is partially blocked by an obstacle.

The panner stage 331 may be a stage for rendering multi-channel reverberation. The panner step 331 may be a step of rendering the audio signal of each channel in head-tracking-based global coordinates based on vector based amplitude panning (VBAP).

The multi HOA stage 333 may be a stage that generates 6DoF sound of content in which two or more HOA sound sources are used simultaneously. That is, the multi-HOA step 333 may be a step of 6DoF rendering HOA sound sources with respect to the listener's location using information in the spatial metadata frame. The output of 6DoF rendered HOA sound sources may be 6DoF sound. Like the single HOA step 327, the multi-HOA step 333 may be a step of converting and processing ESD format signals into HOA.

Hereinafter, with reference to FIGS. 4 to 11, an audio rendering method and a device for performing the same according to an embodiment will be described. According to one embodiment, the device 1100 of FIG. 11 may perform an audio rendering method. Device 1100 may include a renderer (e.g., renderer 10 in FIG. 1).

Figure 4 is a diagram for explaining the relationship between the recording distance and the attenuation effect due to air absorption.

Referring to FIG. 4, according to one embodiment, device 1100 may render an audio signal. The device 1100 determines the gain of the object audio signal based on the distance between the audio object (e.g., audio object 490) and the listener (e.g., listener 400) in the rendering unit (e.g., rendering unit in FIG. 1). gain, propagation delay, and medium absorption can be determined. For example, device 1100 may, at a distance step (e.g., distance step 321 in FIG. 3) of a renderer module (e.g., renderer module 110 in FIG. 1), measure the gain, propagation delay, and medium absorption of the object audio. You can decide on at least one of the following.

Device 1100 may calculate the distance between each render item and the listener in the distance step 321 and interpolate the distance between calls to the update routine of the object audio stream based on the constant velocity model. . A render item may refer to any audio element within the rendering process. Device 1100 may apply propagation delay to signals associated with the render item to create physically accurate delay and Doppler effects. Device 1100 may apply distance attenuation to model the frequency-independent attenuation of audio elements due to geometric spread of source energy. The device 1100 may use a model that considers the size of the sound source to attenuate the distance of the sound source that spreads geometrically. Device 1100 may apply medium absorption to the object audio signal by modeling the frequency-dependent attenuation of audio elements related to the absorption properties of air. The device 1100 may determine the gain of the object audio signal by applying distance attenuation according to the distance between the audio object 490 and the listener 400 (e.g., sound source distance 450). The device 1100 can apply distance attenuation due to geometric diffusion of sound source energy using a parametric model that considers the size of the sound source. When playing audio in a 6 DoF environment, the sound level of an audio object may vary depending on the distance (e.g., sound source distance 450), and 1/r (where the sound level decreases in inverse proportion to the distance) , r is the distance between the audio object and the listener), the loudness of the object audio signal can be determined according to the law. For example, the device 1100 may determine the loudness of the object audio signal according to the 1/r law in an area where the distance between the audio object 490 and the listener 400 is greater than the minimum distance and less than the maximum distance. there is. The minimum distance and maximum distance may refer to distances set to apply distance-dependent attenuation, transmission delay, and air absorption effects. For example, the device 1100 uses metadata to record the location of the listener 400 (e.g., 3D spatial information), the location of the audio object 490 (e.g., 3D spatial information), and audio. The speed of the object 490, etc. can be identified. The device 1100 may calculate the distance (eg, sound source distance 450) between the listener 400 and the audio object 490 using the location of the listener 400 and the location of the audio object 490. The size of the audio signal transmitted to the listener 400 may vary depending on the sound source distance 450. For example, the volume of sound transmitted to a listener located 2 m away may be smaller than the volume of sound transmitted to a listener located 1 m away from an audio source (e.g., the location of the audio object 490). . The size of sound in a free sound-field environment decreases at a rate of 1/r. When the distance between the audio object and the listener is doubled, the sound level heard by the listener is about 6 dB. may decrease. The laws of distance and loudness attenuation can also be applied in a 6Dof VR environment. For one audio object, the device 1100 may use a method of reducing the sound volume of the object audio signal when the distance from the listener is far, and increasing it when the distance is closer. For example, if the sound pressure level of the sound heard by the listener 400 is 0 dB when the listener 400 is 1 m away from the audio object 490, if the listener 400 moves away from the audio object 490 by 2 m. , if you change the sound pressure level to -6dB, it may feel like the sound pressure naturally decreases. When the distance between the audio object and the listener is greater than the minimum distance and less than the maximum distance, the device 1100 may determine the gain of the object audio signal according to Equation 1 below.

[Equation 1]

Here, the reference distance may refer to the reference distance, and the current distance may refer to the distance between the audio object and the listener.

The reference distance may mean the distance at which the gain of the object's audio signal becomes 0 dB, and may be set differently for each audio object. The reference distance may be included in metadata of the device 1100. The device 1100 may determine the gain of the object audio signal by considering the air sound absorption effect according to the distance. Medium attenuation may correspond to the frequency-dependent attenuation of a sound source due to geometric energy diffusion. The device 1100 may modify the EQ field in the distance step 321 to model medium attenuation due to air absorption effect. According to the attenuation of the medium due to the air absorption effect, the device 1100 can apply a low-pass filter effect to audio objects that are far away from the listener. The sound level attenuation of the object audio signal due to the air absorption effect may be determined differently for each frequency domain. For example, depending on the distance between the audio object 490 and the listener 400, the attenuation in the high frequency region may be greater than the attenuation in the low frequency region. The attenuation rate may be defined differently depending on the environment such as temperature, humidity, etc. When information such as temperature and humidity of the actual environment is not given, or when calculating the attenuation constant due to air, it is difficult to accurately reflect the attenuation due to actual air absorption. The device 1100 may apply attenuation effects by air absorption using parameters set for air absorption included in metadata.

An audio signal (eg, recorded sound source 410) may be obtained by recording a sound source (eg, original sound source 401) in the field. The original sound source 401 may refer to an object, scene, etc. that may cause sound. In the recording stage, the recording sound source 410 may have complex propagation characteristics depending on the shape and radiation characteristics of the original sound source 401, so the original sound source 401 and the recording It may be necessary to record with an appropriate distance between devices 405 (e.g., a first distance). In the case of a very loud original sound source 401, setting a first distance (eg, recording distance 403) may be necessary to obtain an appropriate recording sound source 410. In the case of the original sound source 401 (e.g., thunder, airplane sound) to which the recording device 405 cannot be approached closely, the recording device 405 is located on the ground and the recording distance 403 is estimated separately ( estimate) can be made.

Since sound may be attenuated in size or change in tone depending on the medium (e.g., air), there may be differences in the size and tone of the sound between the original sound source 401 and the recorded sound source 410. For example, attenuation effects by air absorption (e.g., attenuation of sound volume, change of timbre) corresponding to the recording distance 403 between the original sound source 401 and the recording device 405. (hereinafter referred to as air absorption effect) may be included in the recorded sound source 410.

An object audio signal (eg, playback sound source 430) may be audio played to the listener 400 in a 6DoF environment (eg, VR environment). The device 1100 may generate the playback sound source 430 by rendering the recorded sound source 410 based on the relationship between the audio object 490 and the listener 400 (e.g., distance, presence of an obstacle). For example, the device 1100 renders the recorded sound source 410 by applying an air absorption effect corresponding to a second distance (e.g., the sound source distance 450), which is the distance between the audio object 490 and the listener 400. By doing this, the playback sound source 430 can be generated.

The device 1100 can generate the playback sound source 430 by rendering the recorded sound source 410 based on Equation 2 below.

[Equation 2]

here, is the audio frequency, is the attenuation due to air absorption, α is the attenuation coefficient due to air absorption [dB/m], and s is the distance [m].

In the process of the device 1100 rendering the recorded sound source 410, an air sound absorption effect corresponding to the recording distance 403 may be superimposed on the original sound source 401. In the recording stage, the air absorption effect according to the recording distance 403, which is the distance between the original sound source 401 and the recording device 405, is included in the recording sound source 410, and the device 1100 is connected to the recording sound source 410 by the sound source distance. Since the playback sound source 430 is generated by rendering by applying the air sound absorption effect according to 450, the air sound absorption effect corresponding to the recording distance 403 can be applied over the original sound source 401. The attenuation of the sound volume of the original sound source 401 and the change in tone due to the air absorption effect according to the recording distance 403 may overlap. In order to create an accurate reproduction sound source 430, when the listener 400 is separated from the audio object 490 by the recording distance 403, the original sound source 430 is played without applying the air sound absorption effect corresponding to the recording distance 403. ) needs to be processed so that it can be reproduced as is. In order to prevent and/or compensate for overlapping application of the air sound absorption effect corresponding to the recording distance 403, the device 1100 may define the recording distance 403 as a parameter and use it in the rendering process. For example, the device 1100 can render by adding the recording distance 403 as a parameter to the EIF input of the encoder 200 of FIG. 2 and the syntax of the bitstream.

The device 1100 can render an audio signal by adding two parameter fields (e.g., recDistance, recDUsage) to the attributes of the sound source (e.g., object sound source, channel sound source, HOA sound source) to the EIF. recDistance may be a parameter related to the recording distance (e.g., recording distance 403). recDistance can be defined in the same format as the reference distance. recDUsage may be a parameter indicating how to apply the recording distance. recDUsage includes a parameter (e.g., 0, 1) indicating two distance calculation methods, which will be explained with reference to FIGS. 5 to 11 below, and a parameter (e.g., 0, 1) indicating the application method of the recording distance to be used according to each manufacturer (e.g., manufacturer of the renderer). Example: 2, 3) may be included. Parameters that the device 1100 adds to the EIF for rendering considering the recording distance may be as shown in Table 1 below.

[Table 1]

The device 1100 may render an audio signal by adding a parameter related to the recording distance to the syntax and data structure of the bitstream. For example, the device 1100 may use parameters related to the recording distance (e.g., recDistance, recDUsage) by adding them to the bitstream syntax of the software (SW) used for rendering, as shown in Table 2 below.

[Table 2]

recDistance is a parameter related to the recording distance and may include parameters (e.g. objectSourceRecDistance, hoaSourceRecDistance, channelSourceRecDistance) for the type of sound source (e.g. object sound source, HOA sound source, channel sound source). recDUsage is a parameter regarding how to apply the recording distance, and may include parameters (e.g. objectSourceRecDUsage, hoaSourceRecDUsage, channelSourceRecDUsage) for the type of sound source (e.g. object sound source, HOA sound source, channel sound source). The data structure for each parameter is shown in Table 3 below.

[Table 3]

SourceRecDistance is the recording distance, which is the distance between the sound source and the recording device, and can be used in an air absorption equalizer (EQ). SourceRecDistance has a value between 0.0 and 2 ^noOfBits - 1 (e.g., 1,023.0), and the device 1100 can use Equation 3 below to quantize the value of SourceRecDistance into a floating point value.

[Equation 3]

SourceRecDUsage depends on how the recording distance is applied: a value (e.g. CONSTEQ) corresponding to a method to prevent the air sound absorption effect within the recording distance (e.g. Method A in Figure 7), a method to compensate for the air sound absorption effect within the recording distance (e.g. Example: It may have a value corresponding to method B in Figure 8 (e.g. COMPEQ) or a value depending on the application method of the recording distance to be used according to each manufacturer's renderer (e.g. RESERVED). The value of SourceRecDUsage is shown in Table 4 below.

[Table 4]

Figure 5 is a diagram for explaining an example of calculating the distance for applying the air sound absorption effect.

Referring to FIG. 5, according to one embodiment, in the conventional MPEG-I Immersive Audio Renderer (RM0)-based technology, the recording distance (e.g., the recording distance 403 in FIG. 4) is considered when applying the air sound absorption effect. Instead, the sound source distance (e.g., the sound source distance 450 in FIG. 4) was used as is. The distance from the audio object to the listener ( _e.g. , sound source distance 450) is _d : 3rd street) In this case, the conventional RM0 renderer Using the equation, the sound source distance was used as the third distance (e.g., playback distance (e.g., s in Equation 2)) without considering the recording distance.

Figure 6 is a diagram for explaining an example of calculating the distance for applying the air sound absorption effect considering the recording distance.

Referring to FIG. 6, according to one embodiment, as described with reference to FIG. 4, it may be necessary to render the audio signal (e.g., the recording sound source 410) so that the air absorption effect during the recording process does not overlap. . Therefore, in order to prevent and/or compensate for the overlapping application of the air absorption effect when the sound source distance d _x is within the recording distance d _r , the reproduction distance The calculation may require modification. For example, the device 1100 can use Equation 4, which introduces the recording distance, to derive the reproduction distance for applying the air absorption effect.

[Equation 4]

Figures 7 and 8 are diagrams for explaining methods for calculating a distance (eg, a third distance) for applying the air sound absorption effect according to an embodiment. The distance calculation methods described in FIGS. 7 and 8 can be selectively performed by the device 1100 of FIG. 11 during the rendering process of the object audio signal.

Referring to FIG. 7, according to one embodiment, FIG. 7 may be a diagram for explaining a method (hereinafter referred to as method A) of preventing the air absorption effect within the recording distance. The device 1100 may select method A to render the object audio signal by applying an air absorption effect considering the recording distance. Method A may be a method of calculating the third distance (e.g., reproduction distance d _a (d _x )) by dividing the range of the sound source distance in calculating the distance (e.g., third distance) for applying the air sound absorption effect. there is. Method A sets the playback distance _{d a (} d _x ) to 0 m in the range 710 where _the sound source distance d This may be a method of calculating the reproduction distance d _a (d _x ). When the device 1100 selects method A to render an object audio signal, the original sound source (e.g., the original sound source 401 in FIG. 4) is stored as the object audio signal (e.g., in FIG. 4) within a distance smaller than the recording distance. It can be used as a playback sound source 430).

Referring to FIG. 8, according to one embodiment, FIG. 8 may be a diagram for explaining a method (hereinafter referred to as method B) of compensating for the air absorption effect within the recording distance. The device 1100 may select method B to render the object audio signal by applying an air absorption effect considering the recording distance. Method B may be a method of using Equation 4 for all ranges of sound source distances in calculating the third distance (e.g., reproduction distance d _a (d _x )). For example, in method _B , the playback distance _{d a (} d _x ) calculated according to Equation 4 becomes negative in the range 810 where the sound source distance d Using _a (d _x ) may be a way to compensate for the low-pass filter effect due to air absorption.

For convenience of explanation, it is assumed that the jet sound-source of the battle scene, which is one of the MPEG-I Immersive Audio CfP (call for proposal) test scenes, was acquired by recording at a recording distance of 90m. . For a recording distance of 90 m, the attenuation by air absorption according to Method A and Method B can be shown in Tables 5 and 6 below.

[Table 5]

[Table 6]

In Tables 5 and 6, the air sound absorption attenuation coefficient was the value at a temperature of 20°C, humidity of 40%, and atmospheric pressure of 101.325kPa according to ISO 9613-1. The amount of air absorption attenuation according to the sound source distance d _x can be calculated using Equation 5 below.

[Equation 5]

Figure 9 shows the spectrum of an audio signal before and after applying an audio rendering method according to an embodiment.

Referring to FIG. 9, according to one embodiment, FIG. 9 may be a diagram showing the result of rendering an object audio signal using method B described with reference to FIG. 8 as a spectrum. Regarding the jet sound-source of the battle scene, which is one of the MPEG-I Immersive Audio CfP (call for proposal) test scenes, it is assumed that the recorded sound source was acquired by recording at a recording distance of 90m. The device 1100 may select Method A or Method B to apply the air absorption effect by considering the recording distance. When the device 1100 selects method B to render a recorded sound source, if the sound source distance _d can do. When compensating for the air absorption effect depending on the sound distance, there may be a noticeable difference in the high frequency band (e.g., band above 3 kHz). For example, it can be seen that the spectrum 930 of the high frequency band after compensating for the negative distance is clearly distinguished from the spectrum 910 of the high frequency band before the air absorption effect is compensated for the negative distance.

Figure 10 is a flowchart for explaining a method of rendering an audio signal according to an embodiment. Operations 1030 to 1050 may be substantially the same as the audio signal rendering method used by the device described with reference to FIGS. 4 to 11 (e.g., device 1100 of FIG. 11).

In operation 1010, the device 1100 may obtain a first distance (eg, recording distance) regarding recording of an audio signal related to an audio object.

In operation 1030, the device 1100 may obtain a second distance (eg, sound source distance), which is the distance between the audio object and the listener.

In operation 1050, the device 1100 may render an audio signal by applying an attenuation effect due to air absorption based on the first distance and the second distance.

Operations 1010 to 1050 may be performed sequentially, but are not limited thereto. For example, two or more operations may be performed in parallel.

Figure 11 is a schematic block diagram of a device according to one embodiment.

Referring to FIG. 11 , according to one embodiment, the device 1100 may perform the audio rendering method described with reference to FIGS. 1 to 11 . Device 1100 may include memory 1110 and processor 1130.

The memory 1110 may store instructions (or programs) that can be executed by the processor 1130. For example, the instructions may include instructions for executing the operation of the processor 1130 and/or the operation of each component of the processor 1130.

Memory 1110 may include one or more computer-readable storage media. Memory 1110 includes non-volatile storage elements (e.g., magnetic hard disk, optical disk, floppy disk, flash memory, electrically programmable memories (EPROM), It may include electrically erasable and programmable (EEPROM).

Memory 1110 may be non-transitory media. The term “non-transitory” may indicate that the storage medium is not implemented as a carrier wave or propagated signal. However, the term “non-transitory” should not be interpreted as meaning that the memory 1210 is immovable.

The processor 1130 may process data stored in the memory 1110. The processor 1130 may execute computer-readable code (eg, software) stored in the memory 1110 and instructions triggered by the processor 1130.

The processor 1130 may be a data processing device implemented in hardware that has a circuit with a physical structure for executing desired operations. For example, the intended operations may include code or instructions included in the program.

For example, data processing devices implemented in hardware include microprocessors, central processing units, processor cores, multi-core processors, and multiprocessors. , ASIC (Application-Specific Integrated Circuit), and FPGA (Field Programmable Gate Array).

The operation performed by the processor 1130 may be substantially the same as the audio rendering method according to an embodiment described with reference to FIGS. 1 to 11. Accordingly, detailed description will be omitted.

The embodiments described above may be implemented with hardware components, software components, and/or a combination of hardware components and software components. For example, the devices, methods, and components described in the embodiments may include, for example, a processor, a controller, an arithmetic logic unit (ALU), a digital signal processor, a microcomputer, and a field programmable gate (FPGA). It may be implemented using a general-purpose computer or a special-purpose computer, such as an array, programmable logic unit (PLU), microprocessor, or any other device capable of executing and responding to instructions. The processing device may execute an operating system (OS) and software applications running on the operating system. Additionally, a processing device may access, store, manipulate, process, and generate data in response to the execution of software. For ease of understanding, a single processing device may be described as being used; however, those skilled in the art will understand that a processing device includes multiple processing elements and/or multiple types of processing elements. It can be seen that it may include. For example, a processing device may include multiple processors or one processor and one controller. Additionally, other processing configurations, such as parallel processors, are possible.

Software may include a computer program, code, instructions, or a combination of one or more of these, which may configure a processing unit to operate as desired, or may be processed independently or collectively. You can command the device. Software and/or data may be used on any type of machine, component, physical device, virtual equipment, computer storage medium or device to be interpreted by or to provide instructions or data to a processing device. It can be saved in . Software may be distributed over networked computer systems and stored or executed in a distributed manner. Software and data may be stored on a computer-readable recording medium.

The method according to the embodiment may be implemented in the form of program instructions that can be executed through various computer means and recorded on a computer-readable medium. A computer-readable medium may store program instructions, data files, data structures, etc., singly or in combination, and the program instructions recorded on the medium may be specially designed and constructed for the embodiment or may be known and available to those skilled in the art of computer software. there is. Examples of computer-readable recording media include magnetic media such as hard disks, floppy disks, and magnetic tapes, optical media such as CD-ROMs and DVDs, and magnetic media such as floptical disks. -Includes optical media (magneto-optical media) and hardware devices specifically configured to store and execute program instructions, such as ROM, RAM, flash memory, etc. Examples of program instructions include machine language code, such as that produced by a compiler, as well as high-level language code that can be executed by a computer using an interpreter, etc.

The hardware devices described above may be configured to operate as one or multiple software modules to perform the operations of the embodiments, and vice versa.

As described above, although the embodiments have been described with limited drawings, those skilled in the art can apply various technical modifications and variations based on this. For example, the described techniques are performed in a different order than the described method, and/or components of the described system, structure, device, circuit, etc. are combined or combined in a different form than the described method, or other components are used. Alternatively, appropriate results may be achieved even if substituted or substituted by an equivalent.

Therefore, other implementations, other embodiments, and equivalents of the claims also fall within the scope of the claims described below.

Claims (15)

Obtaining a first distance for recording an audio signal for an audio object;
Obtaining a second distance, which is the distance between the audio object and the listener;
An operation of rendering the audio signal by applying an effect of attenuation due to air absorption based on the first distance and the second distance.
Including,
Audio rendering method.
According to paragraph 1,
The rendering operation is,
calculating a third distance based on the difference between the first distance and the second distance; and
An operation of rendering the audio signal by applying an effect of attenuation according to the air absorption based on the third distance.
Including,
Audio rendering method.
According to paragraph 2,
The operation of calculating the third distance is,
calculating the third distance by decreasing the second distance by the first distance
Including,
Audio rendering method.
According to paragraph 2,
The operation of calculating the third distance is,
An operation of differently calculating the third distance based on the difference between the first distance and the second distance.
Including,
Audio rendering method.
According to paragraph 4,
The operation of calculating the third distance differently is,
If the second distance is less than or equal to the first distance, calculating the third distance as a predetermined value; and
When the second distance is greater than the first distance, calculating the third distance by decreasing the second distance by the first distance.
Including,
Audio rendering method.
According to paragraph 1,
The first distance is,
Distance between the sound source to be recorded and the recording device
Including,
Audio rendering method.
According to paragraph 1,
The rendering operation is,
When the second distance is smaller than the first distance, an operation of rendering the audio signal by compensating the tone by the size of the effect due to attenuation due to the air absorption.
Including,
Audio rendering method.
A computer program combined with hardware and stored in a computer-readable recording medium to execute the method of any one of claims 1 to 7.
memory containing instructions; and
It is electrically connected to the memory and includes a processor for executing the instructions,
When the instructions are executed by the processor, the processor controls a plurality of operations,
The plurality of operations are,
Obtaining a first distance for recording an audio signal for an audio object;
Obtaining a second distance, which is the distance between the audio object and the listener;
An operation of rendering the audio signal by applying an effect of attenuation due to air absorption based on the first distance and the second distance.
Including,
Audio rendering device.
According to clause 9,
The rendering operation is,
calculating a third distance based on the difference between the first distance and the second distance; and
An operation of rendering the audio signal by applying an effect of attenuation according to the air absorption based on the third distance.
Including,
Audio rendering device.
According to clause 10,
The operation of calculating the third distance is,
calculating the third distance by decreasing the second distance by the first distance
Including,
Audio rendering device.
According to clause 10,
The operation of calculating the third distance is,
An operation of differently calculating the third distance based on the difference between the first distance and the second distance.
Including,
Audio rendering device.
According to clause 12,
The operation of calculating the third distance differently is,
If the second distance is less than or equal to the first distance, calculating the third distance as a predetermined value; and
When the second distance is greater than the first distance, calculating the third distance by decreasing the second distance by the first distance.
Including,
Audio rendering device.
According to clause 9,
The first distance is,
Distance between the sound source to be recorded and the recording device
Including,
Audio rendering device.
According to clause 9,
The rendering operation is,
When the second distance is smaller than the first distance, an operation of rendering the audio signal by compensating the tone by the size of the effect due to attenuation due to the air absorption.
Including,
Audio rendering device.

Images