EP4618595A1

EP4618595A1 - Audio signal rendering method, apparatus, device, and storage medium

Info

Publication number: EP4618595A1
Application number: EP22964699.7A
Authority: EP
Inventors: Chenhao HU; Runyu Shi; Bin Wang
Original assignee: Beijing Xiaomi Mobile Software Co Ltd
Current assignee: Beijing Xiaomi Mobile Software Co Ltd
Priority date: 2022-11-07
Filing date: 2022-11-07
Publication date: 2025-09-17
Also published as: EP4618595A4; CN118318455A; WO2024098221A1

Abstract

The present invention provides an audio signal rendering method, an apparatus, a device, and a storage medium. The method comprises: determining a first gain HRIR corresponding to a direct sound signal in an audio object signal to be rendered; determining a second gain HRIR corresponding to at least one reflected sound signal in said audio object signal; on the basis of the first gain HRIR and the second gain HRIR, determining a fused HRIR corresponding to said audio object signal; and rendering said audio object signal on the basis of the fused HRIR. The method of the present invention can ensure the rendering effect, restore the real environment of sound signals as much as possible, improve the sound source orientation rendering experience, and enable the rendering effect to be closer to the real situation.

Description

TECHNICAL FIELD

The present disclosure relates to the field of communication technology, and specifically to a method for rendering audio signal and a communication apparatus, a device and a storage medium.

BACKGROUND

Since spatial audio may provide users a realistic sense of space and orientation, spatial audio processing technology has been widely applied. An effective method to realize the spatial audio is object-based spatial audio technology. The object-based spatial audio technology mainly includes two main parts: encoding/decoding and rendering. The encoding/decoding part is to transmit an audio object signal to be played by a user end and corresponding metadata, and the rendering part is to reproduce the spatial audio according to the received audio object signal and the metadata of which.
In related arts, the method for rendering the audio object signal includes at least one of:

Method 1: calculating an orientation gain parameter of the audio object signal according to sound cone information of the audio object signal, and then rendering the audio object signal based on the orientation gain parameter.
Method 2: training a set of head-related transfer functions (HRTFs) to simulate a change of a sound source transmitted to a human ear at different directions and orientations, and obtaining orientation information of the sound source corresponding to the audio object signal. Based on the orientation information, a rendered audio object signal is obtained by using trained HRTF filtering during rendering.

However, the rendering effects of both Methods 1 and 2 in the related arts are poor, making an orientation sense of the sound source not very realistic. Moreover, Method 2 is not suitable for a personalized adjustment for various types of sound sources, and may only be applied to a narrow range.

SUMMARY

Embodiments of the present disclosure provide a method for rendering audio signal and a communication apparatus, a device and a storage medium, to solve technical problems, such as a poor rendering effect and a narrow range of application, in the methods in related arts.
According to a first aspect, the embodiments of the present disclosure provide a method for rendering audio signal, performed by a signal receiving end, including:

determining a first gain head-related impulse response (HRIR) corresponding to a direct sound signal in an audio object signal to be rendered;
determining a second gain HRIR corresponding to at least one reflected sound signal in the audio object signal to be rendered;
determining, based on the first gain HRIR and the second gain HRIR, a mixed HRIR corresponding to the audio object signal to be rendered; and
rendering the audio object signal to be rendered based on the mixed HRIR.

In the present disclosure, the signal receiving end may determine the first gain HRIR corresponding to the direct sound signal in the audio object signal to be rendered; determine the second gain HRIR corresponding to at least one reflected sound signals in the audio object signal to be rendered; determine, based on the first gain HRIR and the second gain HRIR, the mixed HRIR corresponding to the audio object signal to be rendered; and render the audio object signal to be rendered based on the mixed HRIR. In the present disclosure, the first gain HRIR may represent the gain loss caused by the orientation and the distance of the sound source during the transmission of the direct sound signal. The second gain HRIR may represent at least one of the gain loss caused by the orientation of the sound source, the gain loss caused by the reflection, and the gain loss caused by the transmission distance during the transmission of the reflected sound signal. Therefore, when rendering the audio object signal to be rendered based on the mixed HRIR obtained from the first gain HRIR and the second gain HRIR, the rendering effect may be ensured, the real environment of the sound signal may be restored as much as possible, the experience of sound source orientation rendering may be enhanced, and the rendering effect may be closer to the real situation. In addition, the rendering method of the present disclosure may achieve the personalized adjustment for other types of sound sources, so as to have a wide application range. In the present disclosure, the targeted processing of different sound sources may be realized during the rendering, and the scene information of the scene where the sound source is located may be fully considered, so as to further ensure the rendering effect.
According to a second aspect, the embodiments of the present disclosure provide a communication apparatus, configured in a signal receiving end, including:

a processing module, configured to determine a first gain HRIR corresponding to a direct sound signal in an audio object signal to be rendered,
in which the processing module is further configured to determine a second gain HRIR corresponding to at least one reflected sound signal in the audio object signal to be rendered;
in which the processing module is further configured to determine, based on the first gain HRIR and the second gain HRIR, a mixed HRIR corresponding to the audio object signal to be rendered; and
in which the processing module is further configured to render the audio object signal to be rendered based on the mixed HRIR.

According to a third aspect, the embodiments of the present disclosure provide a communication apparatus including a processor. When a computer program in a memory is called by the processor, the method in the first aspect is implemented.
According to a fourth aspect, the embodiments of the present disclosure provide a communication apparatus including a processor and a memory storing a computer program. When the computer program stored in the memory is executed by the processor, the communication apparatus is caused to implement the method in the first aspect.
According to a fifth aspect, the embodiments of the present disclosure provide a communication apparatus including a processor and an interface circuit. The interface circuit is configured to receive code instructions and transmit the code instructions to the processor; and the processor is configured to run the code instructions to cause the communication apparatus to implement the method in the first aspect.
According to a sixth aspect, the embodiments of the present disclosure provide a communication system including the communication apparatus in the second aspect, or the communication system including the communication apparatus in the third aspect, or the communication system including the communication apparatus in the fourth aspect, or the communication system including the communication apparatus in the fifth aspect.
According to a seventh aspect, the embodiments of the present disclosure provide a computer readable storage medium for storing instructions used by a network device. When the instructions are executed, a terminal is caused to implement the method in the first aspect.
According to an eighth aspect, the embodiments of the present disclosure also provide a computer program product including computer programs. When the computer programs are running on a computer, the computer is caused to implement the method in the first aspect.
According to a ninth aspect, the embodiments of the present disclosure provide a chip system including at least one processor and interface, and the chip system is configured for supporting a network device to perform functions covered by the method in the first aspect, for example, to determine or process at least one piece of data and information covered by the method above. In one possible design, the chip system also includes a memory for saving a computer program and data necessary for a source auxiliary node. The chip system may include a chip, or may include a chip and other discrete elements.
According to a tenth aspect, the embodiments of the present disclosure provide a computer program. When the computer program is running on a computer, the computer is caused to implement the method in the first aspect.

BRIEF DESCRIPTION OF THE DRAWINGS

The above and/or additional aspects and advantages of embodiments of the present disclosure will become apparent and more easily understandable from the following descriptions of the embodiments with reference to the accompanying drawings.

FIG 1 is a schematic diagram of a communication system according to an embodiment of the present disclosure.
FIG 2a is a flowchart of a method for rendering audio signal according to an embodiment of the present disclosure.
FIG 2b is a schematic diagram of a direct sound signal and a reflected sound signal according to an embodiment of the present disclosure.
FIG 3a is a flowchart of a method for rendering audio signal according to another embodiment of the present disclosure.
FIG 3b is a schematic diagram of a sound cone outer angle and a sound cone outer angle according to an embodiment of the present disclosure.
FIG 4 is a flowchart of a method for rendering audio signal according to yet another embodiment of the present disclosure.
FIG 5 is a flowchart of a method for rendering audio signal according to further another embodiment of the present disclosure.
FIG 6 is a flowchart of a method for rendering audio signal according to further another embodiment of the present disclosure.
FIG 7 is a flowchart of a method for rendering audio signal according to further another embodiment of the present disclosure.
FIG 8 is a flowchart of a method for rendering audio signal according to further another embodiment of the present disclosure.
FIG 9 is a flowchart of a method for rendering audio signal according to further another embodiment of the present disclosure.
FIG 10 is a flowchart of a method for rendering audio signal according to further another embodiment of the present disclosure.
FIG 11 is a flowchart of a method for rendering audio signal according to further another embodiment of the present disclosure.
FIG 12 is a flowchart of a method for rendering audio signal according to further another embodiment of the present disclosure.
FIG 13 is a schematic diagram of a communication apparatus according to an embodiment of the present disclosure.
FIG 14 is a schematic diagram of a communication apparatus according to an embodiment of the present disclosure.
FIG 15 is a schematic diagram of a chip according to an embodiment of the present disclosure.

DETAILED DESCRIPTION

Reference will now be made in detail to example embodiments, examples of which are illustrated in the accompanying drawings. The following description refers to the accompanying drawings in which the same numbers in different drawings represent the same or similar elements unless otherwise represented. The implementations set forth in the following description of example embodiments do not represent all implementations consistent with the present disclosure. Instead, they are merely examples of apparatuses and methods consistent with aspects related to the present disclosure as recited in the appended claims.
The terms used in embodiments of the present disclosure are solely for the purpose of describing a particular embodiment and are not intended to limit embodiments of the present disclosure. The terms "a/an" and "the" in a singular form used in embodiments and claims of the present disclosure are also intended to include a plural form, unless the context clearly indicates other meaning. It may also be understood that the term "and/or" as used herein refers to any or all possible combinations of one or more associated listed items.
It may be understood that although the terms first, second, third, etc. may be used to describe various information in the embodiments of the present disclosure, such information should not be limited to these terms. These terms are used only to distinguish information in the same type from one another. For example, without departing from the scope of embodiments of the present disclosure, the first information may also be referred to as the second information, and similarly, the second information may be referred to as the first information. Depending on the context, words "if" and "in case that" used here may be interpreted as "when", "while", or "in response to determining".
The embodiments of the present disclosure are described in detail below. Examples of the embodiments are shown in the accompanying drawings, in which the same or similar labels throughout the embodiments of the present disclosure represent the same or similar elements or elements having the same or similar functions. The embodiments below with reference to the accompanying drawings are exemplary and are intended to explain the present disclosure, but should not be construed as a limitation to the present disclosure.
In order to better understand a method for rendering audio signal disclosed in the embodiments of the present disclosure, a communication apparatus applicable to the embodiments of the present disclosure will be first introduced.
Please refer to FIG 1, FIG 1 is a schematic diagram a communication system according to an embodiment of the present disclosure. The communication system may include, but is not limited to, one signal sending device and one signal receiving device, which may both be a terminal or a network device. The number and form of devices in FIG 1 are only shown as an example, and do not constitute a limitation on the embodiments of the present disclosure. The communication system may include two or more signal sending devices and two or more signal receiving devices in a practical application. The communication system in FIG 1 including one signal sending device 11 and one signal receiving device 12 is shown as an example.
It needs to be noted that the technical solution of embodiments of the present disclosure may be applied to various communication systems, for example, a long term evolution (LTE) system, a 5th generation (5G) mobile communication system, a 5G new radio (NR) system, or other new mobile communication systems in the future.
The network device in the embodiments of the present disclosure is an entity on a network side for sending or receiving signals. For example, the network device 11 may be an evolved NodeB (eNB), a transmission reception point (TRP), a next generation NodeB (gNB) in an NR system, a base station in other mobile communication system in the future, or an access node in a wireless fidelity (WiFi) system. The embodiments of the present disclosure do not limit a specific technology and a specific device form used by the network device. The network device according to embodiments of the disclosure may be composed of a central unit (CU) and distributed units (DUs). The CU may also be referred to as a control unit. The use of CU-DU structure allows to divide a protocol layer of the network device, such as a base station, such that some of the protocol layer functions are placed in the CU for centralized control, and some or all of the remaining protocol layer functions are distributed in the DUs, and the DUs are centrally controlled by the CU.
The terminal in the embodiments of the present disclosure is an entity on a user side for receiving or sending signals, such as a mobile phone. The terminal may also be called a terminal, a user equipment (UE), a mobile station (MS), a mobile terminal (MT), etc. The terminal may be a car with communication function, a smart car, a mobile phone, a wearable device, a Pad, a computer with a wireless receiving and sending function, a virtual reality (VR) terminal, an augmented reality (AR) terminal, a wireless terminal in industrial control, a wireless terminal in self-driving, a wireless terminal in remote medical surgery, a wireless terminal in a smart grid, a wireless terminal in transportation safety, a wireless terminal in a smart city, a wireless terminal in a smart home, etc. The embodiments of the present disclosure do not limit a specific technology and a specific device form used by the terminal.
It may be understood that the communication system in embodiments of the disclosure is to more clearly explain the technical solutions of embodiments of the disclosure, and does not constitute a limitation on the technical solutions in embodiments of the disclosure. Those skilled in the art know that with evolution of the system architecture and emergence of new service scenarios, the technical solution in the embodiments of the disclosure is equally applicable to similar technical problems.
A method for rendering audio signal and a communication apparatus, a device and a storage medium provided in the embodiments of the present disclosure are described in detail below with reference to accompanying drawings.
It should be noted that, in the present disclosure, a method for rendering audio signal provided in any embodiment may be executed alone. Any implementation in the embodiments may also be executed alone, or may be executed in combined with other embodiments, or may be executed in combined with possible implementations in other embodiments, or may be executed with any technical solution in the relevant arts.
FIG 2a is a flowchart of a method for rendering audio signal according to an embodiment of the present disclosure. The method is performed by a signal receiving end. As shown in FIG. 2a, the method for rendering audio signal may include the following steps.
At step S201, a first gain head-related impulse response (HRIR) corresponding to a direct sound signal in an audio object signal to be rendered is determined.
In an embodiment of the present disclosure, the direct sound signal above may be: a signal whose transmission path coincides with a straight line between a sound source and a listener. In other words, it may be understood that the direct sound signal is a signal emitted by the sound source and directly transmitted to the listener's position without being reflected, refracted, or diffracted. Each audio object signal includes one direct sound signal.
FIG 2b is a schematic diagram of a direct sound signal and a reflected sound signal according to an embodiment of the present disclosure. As shown in figures (1) and (2) of FIG 2b, when an orientation of a sound source changes, due to a possibly existing obstacle (such as a wall) in a space where the sound source is located, the listener's ears may receive different sound signals. In figure (1) of FIG 2b, the sound source emits sound towards the listener, and the listener's left ear may receive a relatively stronger direct sound signal, and also receive a reflected sound signal that is reflected once after a sound cone radiation attenuation. In figure (2) of FIG 2b, the sound source emits sound towards a certain area in front of the listener, and the listener's left ear may receive a direct sound signal after the sound cone radiation attenuation, while listener's right ear may receive a reflected sound signal that is reflected once.
In an embodiment of the present disclosure, the first gain HRIR above may be determined based on an orientation gain parameter and a distance gain parameter of the direct sound signal, in which the first gain HRIR may be configured to represent a gain loss caused by an orientation and a distance of the sound source during a transmission process of the direct sound signal.
Further, in an embodiment of the present disclosure, the "determining the first gain HRIR corresponding to the direct sound signal in the audio object signal to be rendered" may include: for each audio object signal to be rendered, determining respectively the first gain HRIR corresponding to the direct sound signal in the audio object signal.
A detailed description of how to determine the first gain HRIR is provided in subsequent embodiments.
At step S202, a second gain HRIR corresponding to at least one reflected sound signal in the audio object signal to be rendered is determined.
In an embodiment of the present disclosure, the reflected sound signal above may be: a signal whose transmission path coincides with a straight line between a mirrored sound source position and a listener. The mirrored sound source position may be: a mirrored image position of a sound source relative to a reflector. How to determine the mirrored sound source position will be explained in detail in subsequent embodiments.
Furthermore, in an embodiment of the present disclosure, the reflected sound signal may be understood as: a signal emitted by the sound source that is transmitted to a listener position after being reflected.
The second gain HRIR may be determined based on at least one of an orientation gain parameter, a reflection gain parameter, or a distance delay gain parameter of the reflected sound signal. The second gain HRIR is configured to represent at least one of following gain losses during a transmission process of the reflected sound signal:

a gain loss caused by an orientation of the sound source;
a gain loss caused by reflection; or
a gain loss and a transmission delay loss caused by a transmission distance.

It needs to be noted that, in an embodiment of the present disclosure, a reason for calculating the first gain HRIR and second gain HRIR above is mainly to render the audio object signal based on the calculated first HRIR and second HRIR in the subsequent steps. This is to ensure that the gain losses in all aspects may be considered during the rendering process, so as to ensure a rendering accuracy, enhance an experience of sound source orientation rendering, and make a rendering effect closer to a real situation.
However, in an embodiment of the present disclosure, the above "determining a second gain HRIR corresponding to at least one reflected sound signal in the audio object signal to be rendered" mainly include: for each audio object signal to be rendered, determining respectively the second gain HRIR corresponding to the reflected sound signal in the audio object signal. Each audio object signal includes at least one reflected sound signal. Based on this, when the second gain HRIR corresponding to each reflected sound signal in the audio object signal is calculated, a large amount of calculation is required. Therefore, in an embodiment of the present disclosure, only a second gain HRIR corresponding to an early reflected sound signal may be calculated, so as to reduce the amount of calculation. In addition, the second gain HRIR corresponding to the early reflected sound signal is sufficient to provide a good sense of direction, and may also guarantee the rendering effect in a large extent.
In an embodiment of the present disclosure, the early reflected sound signal may include at least one of:

a first-order reflected sound signal (i.e., a sound signal reflected only once); or
a second-order reflected sound signal (i.e., a sound signal reflected twice).

In addition, it needs to emphasize that terms "distance gain parameter" and "distance delay gain parameter" mentioned above are different. The distance gain parameter is configured to represent: a loss such as radiation attenuation caused by a distance during a sound signal transmission process. The distance delay gain parameter configured to represent: a loss such as radiation attenuation caused by the distance, and a transmission delay loss caused by the distance during the sound signal transmission process.
At step S203, a mixed HRIR corresponding to the audio object signal to be rendered is determined based on the first gain HRIR and the second gain HRIR.
In an embodiment of the present disclosure, the mixed HRIR may be obtained by performing a weighted summation on the first gain HRIR corresponding to the direct sound signal of the audio object signal to be rendered and the second gain HRIR corresponding to the at least one reflected sound signal of the audio object signal to be rendered.
In an embodiment of the present disclosure, the mixed HRIR corresponding to the audio object signal is determined respectively for each audio object signal to be rendered.
At step S204, the audio object signal to be rendered is rendered based on the mixed HRIR.
According to above content, in an embodiment of the present disclosure, the mixed HRIR is determined based on the first gain HRIR and second gain HRIR corresponding to the audio object signal to be rendered. The first gain HRIR may represent the gain loss caused by the orientation and the distance of the sound source during the transmission of the direct sound signal. The second gain HRIR may represent at least one of the gain loss caused by the orientation of the sound source, the gain loss caused by the reflection, and the gain loss caused by the transmission distance during the transmission of the reflected sound signal. Based on this, when rendering the audio object signal to be rendered based on the mixed HRIR, not only the gain loss caused by the orientation of the sound source during the transmission of the direct sound signal, but also the gain loss caused by the orientation of the sound source during the transmission of the reflected sound signal, the gain loss caused by the reflection, and the gain loss caused by the transmission distance. Therefore, the rendering effect for the audio object signal to be rendered may be ensured, a real environment of the sound signal may be restored as much as possible, the experience of sound source orientation rendering may be enhanced, and the rendering effect may be closer to the real situation.
In an embodiment of the present disclosure, the "rendering the audio object signal to be rendered based on the mixed HRIR" mainly include: based on mixed HRIR of each audio object signal to be rendered, rending each audio object signal to be rendered respectively. It may be seen that the method in the present disclosure performs targeted rendering for each audio object signal to be rendered, that is, the present disclosure realizes targeted processing of different sound sources, so as to further ensure the rendering effect.
Furthermore, in an embodiment of the present disclosure, the reflection gain parameter, the distance gain parameter, and the distance delay gain parameter may all represent scene information of a scene where the sound source is located. Therefore, in the method of the present disclosure, the scene information of the scene where the sound source is located may be fully considered when rendering the signal, so as to ensure the rendering accuracy.
In addition, in an embodiment of the present disclosure, when rendering the audio object signal, it does not rely solely on the orientation gain parameter, but also refers to the reflection gain parameter and the distance gain parameter, so that the method of the present disclosure is not limited to processing an orientated sounding source, but also be applied to perform the personalized adjustment on other types of sound sources (such as, a divergent sounding source), and has a wide applicable range.
In summary, in the method for rendering audio signal provided by the embodiments of the present disclosure, the signal receiving end may determine the first gain HRIR corresponding to the direct sound signal in the audio object signal to be rendered; determine the second gain HRIR corresponding to at least one reflected sound signals in the audio object signal to be rendered; determine, based on the first gain HRIR and the second gain HRIR, the mixed HRIR corresponding to the audio object signal to be rendered; and render the audio object signal to be rendered based on the mixed HRIR. In the present disclosure, the first gain HRIR may represent the gain loss caused by the orientation and the distance of the sound source during the transmission of the direct sound signal. The second gain HRIR may represent at least one of the gain loss caused by the orientation of the sound source, the gain loss caused by the reflection, and the gain loss caused by the transmission distance during the transmission of the reflected sound signal. Therefore, when rendering the audio object signal to be rendered based on the mixed HRIR obtained from the first gain HRIR and the second gain HRIR, the rendering effect may be ensured, the real environment of the sound signal may be restored as much as possible, the experience of sound source orientation rendering may be enhanced, and the rendering effect may be closer to the real situation. In addition, the rendering method of the present disclosure may achieve the personalized adjustment for other types of sound sources, so as to have a wide application range. In the present disclosure, the targeted processing of different sound sources may be realized during the rendering, and the scene information of the scene where the sound source is located may be fully considered, so as to further ensure the rendering effect.
FIG 3a is a flowchart of a method for rendering audio signal according to an embodiment of the present disclosure. The method is performed by a signal receiving end. As shown in FIG. 3a, the method for rendering audio signal may include the following steps.
At step S301, a code stream sent by a signal sending end is received.
At step S302, the code stream is decoded to obtain at least one audio object signal to be rendered and metadata of the at least one audio object signal to be rendered.
In an embodiment of the present disclosure, the metadata includes at least one of:

orientation information of a sound source of the audio object signal to be rendered;
sound cone information of a sound source of the audio object signal to be rendered;
room information of a room where a sound source of the audio object signal to be rendered is located;
spatial position information of a sound source of the audio object signal to be rendered;
orientation information of a listener; or
spatial position information of a listener.

Specifically, in an embodiment of the present disclosure, the above "orientation information of the sound source" may include an orientation angle and/or an orientation direction of the sound source.
The above "sound cone information of the sound source" may include at least one of a sound cone inner angle, a sound cone outer angle, or an outer angle gain value.
Each sound source corresponds to the sound cone inner angle and the sound cone outer angle. FIG 3b is a schematic diagram of a sound cone outer angle and a sound cone outer angle according to an embodiment of the present disclosure. The sound cone inner angle means that for a sound signal within a range of the sound cone inner angle, it is considered that there is no radiation attenuation caused by an orientation. The sound cone outer angle means that for a sound signal outside the range of the sound cone inner angle but within a range of the sound cone outer angle, it is considered that there is the radiation attenuation caused by the orientation, in which the outer angle gain value may be configured to represent a maximum degree of the radiation attenuation caused by the orientation of this part of the signal. In addition, it is considered that a sound signal emitted by the sound source will not be attenuated outside the range of the sound cone outer angle.
The room information above may include at least one of a dimension of the room, a reflection coefficient of an object (such as, a wall) in the room, etc.
The spatial position information of the sound source above may include an absolute position of the sound source and/or a relative position of the sound source relative to the listener.
The orientation information of the listener above may include an orientation angle and/or an orientation direction of the listener.
The spatial position information of the listener may include an absolute position of the listener and/or a relative position of the listener relative to the sound source.
The absolute position and the relative position above may be represented as an absolute coordinate or a relative coordinate respectively. The absolute coordinate and the relative coordinate may be coordinate values in a specific coordinate system, in which the specific coordinate system may be a three-dimensional coordinate system established with a certain point in the room where the sound source is located as an origin.
It needs to be noted that the spatial position information of the sound source and the spatial position information of the listener should be able to determine an absolute position of the listener and the absolute position of the sound source.
In addition, in an embodiment of the present disclosure, the orientation information of the listener and/or the spatial position information of the listener may be directly determined by the listener.
At step S303, the first gain HRIR corresponding to the direct sound signal in the audio object signal to be rendered and the second gain HRIR corresponding to the reflected sound signal in the audio object signal to be rendered are be determined based on the metadata of the audio object signal to be rendered,
A detailed description of this section may be referred to the subsequent embodiments.
In summary, in the method for rendering audio signal provided by the embodiments of the present disclosure, the signal receiving end may determine the first gain HRIR corresponding to the direct sound signal in the audio object signal to be rendered; determine the second gain HRIR corresponding to at least one reflected sound signals in the audio object signal to be rendered; determine, based on the first gain HRIR and the second gain HRIR, the mixed HRIR corresponding to the audio object signal to be rendered; and render the audio object signal to be rendered based on the mixed HRIR. In the present disclosure, the first gain HRIR may represent the gain loss caused by the orientation and the distance of the sound source during the transmission of the direct sound signal. The second gain HRIR may represent at least one of the gain loss caused by the orientation of the sound source, the gain loss caused by the reflection, and the gain loss caused by the transmission distance during the transmission of the reflected sound signal. Therefore, when rendering the audio object signal to be rendered based on the mixed HRIR obtained from the first gain HRIR and the second gain HRIR, the rendering effect may be ensured, the real environment of the sound signal may be restored as much as possible, the experience of sound source orientation rendering may be enhanced, and the rendering effect may be closer to the real situation. In addition, the rendering method of the present disclosure may achieve the personalized adjustment for other types of sound sources, so as to have a wide application range. In the present disclosure, the targeted processing of different sound sources may be realized during the rendering, and the scene information of the scene where the sound source is located may be fully considered, so as to further ensure the rendering effect.
FIG 4 is a flowchart of a method for rendering audio signal according to an embodiment of the present disclosure. The method is performed by a signal receiving end. As shown in FIG. 4, the method for rendering audio signal may include the following steps.
At step S401, an incident angle of the direct sound signal relative to the listener is determined based on the spatial position information of the sound source and/or the spatial position information of the listener.
In an embodiment of the present disclosure, a sound source position and a listener position (the position refers to an absolute position here) may be determined based on the spatial position information of the sound source and/or the spatial position information of the listener, and then the incident angle of the direct sound signal relative to the listener may be determined based on a straight line between the sound source position and the listener position. An angle between d a normal of an incident surface and the straight line connecting the sound source position and the listener position is the incident angle of the direct sound signal relative to the listener.
At step S402, a first HRIR corresponding to the direct sound signal is determined based on the incident angle of the direct sound signal relative to the listener and the orientation information of the listener.
Specifically, in an embodiment of the present disclosure, the first HRIR corresponding to the direct sound signal may be determined from a head-related transfer function (HRTF) database based on the incident angle of the direct sound signal relative to the listener and the orientation information of the listener.
At step S403, an orientation gain parameter of the direct sound signal is determined based on the orientation information of the sound source and the sound cone information.
In an embodiment of the present disclosure, the orientation information of the sound source and the sound cone information may be input into a sound cone model, and the sound cone model outputs the orientation gain parameter of the direct sound signal.
At step S404, a distance gain parameter of the direct sound signal is determined based on the spatial position information of the sound source and/or the spatial position information of the listener.
In an embodiment of the present disclosure, the sound source position and the listener position (the position refers to an absolute position here) may be determined based on the spatial position information of the sound source and/or the spatial position information of the listener, and then the distance gain parameter of the direct sound signal may be determined based on a distance between the sound source position and the listener position.
At step S405, the first gain HRIR is determined based on the orientation gain parameter, the distance gain parameter, and the first HRIR.
Specifically, the first gain HRIR may be obtained by performing a weighted summation on the first HRIR using the orientation gain parameter and the distance gain parameter.
In summary, in the method for rendering audio signal provided by the embodiments of the present disclosure, the signal receiving end may determine the first gain HRIR corresponding to the direct sound signal in the audio object signal to be rendered; determine the second gain HRIR corresponding to at least one reflected sound signals in the audio object signal to be rendered; determine, based on the first gain HRIR and the second gain HRIR, the mixed HRIR corresponding to the audio object signal to be rendered; and render the audio object signal to be rendered based on the mixed HRIR. In the present disclosure, the first gain HRIR may represent the gain loss caused by the orientation and the distance of the sound source during the transmission of the direct sound signal. The second gain HRIR may represent at least one of the gain loss caused by the orientation of the sound source, the gain loss caused by the reflection, and the gain loss caused by the transmission distance during the transmission of the reflected sound signal. Therefore, when rendering the audio object signal to be rendered based on the mixed HRIR obtained from the first gain HRIR and the second gain HRIR, the rendering effect may be ensured, the real environment of the sound signal may be restored as much as possible, the experience of sound source orientation rendering may be enhanced, and the rendering effect may be closer to the real situation. In addition, the rendering method of the present disclosure may achieve the personalized adjustment for other types of sound sources, so as to have a wide application range. In the present disclosure, the targeted processing of different sound sources may be realized during the rendering, and the scene information of the scene where the sound source is located may be fully considered, so as to further ensure the rendering effect.
FIG 5 is a flowchart of a method for rendering audio signal according to an embodiment of the present disclosure. The method is performed by a signal receiving end. As shown in FIG. 5, the method for rendering audio signal may include the following steps.
At step S501, an incident angle of the reflected sound signal relative to the listener is determined based on the spatial position information of the sound source and/or the spatial position information of the listener.
In an embodiment of the present disclosure, the "determining the incident angle of the reflected sound signal relative to the listener based on the spatial position information of the sound source and/or the spatial position information of the listener" may include the following steps.
At step a, a sound source position and a listener position (both positions here are absolute positions) are determined based on the spatial position information.
At step b, a mirrored sound source position of the sound source position relative to a reflector is determined.
In an embodiment of the present disclosure, the reflector may be an object that reflects the sound signal, such as a wall.
In an embodiment of the present disclosure, different reflected sound signals correspond to different mirrored sound source positions.
For the first-order reflected sound signal, its mirrored sound source position is: a mirrored position of the sound source position relative to the reflector. For the second-order reflected sound signal, a method to determine its mirrored sound source position includes: determining a first reflector that reflects the reflected sound signal for the first time and a second reflector that reflects the reflected sound signal for the second time, then determining a first mirrored sound source position of the sound source position relative to the first reflector, then determining a second mirrored sound source position of the first mirrored sound source position relative to the second reflector, and finally determining the second mirrored sound source position as the mirrored sound source position of the second-order reflected sound signal.
At step c, the incident angle of the reflected sound signal relative to the listener is determined based on the mirrored sound source position and the listener position.
An angle between the normal of the incident surface and a straight line connecting the mirrored sound source position and the listener position is the incident angle of the reflected sound signal relative to the listener.
At step S502, a second HRIR corresponding to the reflected sound signal is determined based on the incident angle of the reflected sound signal relative to the listener and the orientation information of the listener.
Specifically, in an embodiment of the present disclosure, the second HRIR corresponding to the reflected sound signal may be determined from the HRTF database based on the incident angle of the reflected sound signal relative to the listener and the orientation information of the listener.
At step S503, at least one of an orientation gain parameter, a reflection gain parameter, or a distance delay gain parameter of the reflected sound signal is determined based on information in the metadata.
Optionally, in an embodiment of the present disclosure, a method to determine the orientation gain parameter of the reflected sound signal based on the information in the metadata may include the following steps.
At step 1, a mirrored listener position of the listener relative to a reflector is determined.
In an embodiment of the present disclosure, the reflector may be an object that reflects the sound signal, such as a wall.
In an embodiment of the present disclosure, different reflected sound signals correspond to different mirrored sound source positions.
For the first-order reflected sound signal, its mirrored listener position is a mirrored position of the listener position relative to the reflector. For the second-order reflected sound signal, a method to determine its mirrored listener position includes: determining a first reflector that reflects the reflected sound signal for the first time and a second reflector that reflects the reflected sound signal for the second time, then determining a first mirrored listener position of the listener position relative to the first reflector, then determining a second mirrored listener position of the first mirrored listener position relative to the second reflector, and finally determining the second mirrored listener position as the mirrored listener position of the second-order reflected sound signal.
At step 2, an exit angle of the reflected sound signal relative to the sound source is determined based on the mirrored listener position and the sound source position.
An angle between the normal of the incident surface and a straight line connecting the mirrored listener position and the sound source position is the exit angle of the reflected sound signal relative to the sound source.
At step 3, the orientation gain parameter of the reflected sound signal is determined based on the exit angle, the orientation information of the sound source and the sound cone information.
In an embodiment of the present disclosure, the exit angle, the orientation information of the sound source and the sound cone information may be input into the sound cone model, and the sound cone model outputs the orientation gain parameter of the reflected sound signal.
Optionally, in an embodiment of the present disclosure, determining the reflection gain parameter of the reflected sound signal based on the information in the metadata may include: determining the reflection gain parameter of the reflected sound signal based on the room information. Specifically, a reflection coefficient of an object that reflects the reflected sound signal may be determined based on the room information, and then the reflection gain parameter of the reflected sound signal may be determined based on the reflection coefficient.
Optionally, in an embodiment of the present disclosure, determining the distance delay gain parameter of the reflected sound signal based on the information in the metadata may include the following steps.
At step 1, a mirrored listener position of the listener relative to a reflector is determined.
At step 2, the distance delay gain parameter is determined based on a distance between the sound source position and the mirrored listener position, or a distance between the mirrored sound source position and the listener position.
At step S504, the second gain HRIR is determined based on the second HRIR and at least one of the orientation gain parameter, the reflection gain parameter, or the distance delay gain parameter.
Specifically, the second HRIR may be determined by performing a weighted summation on the second HRIR using at least one of the orientation gain parameter, the reflection gain parameter, or the distance delay gain parameter.
In summary, in the method for rendering audio signal provided by the embodiments of the present disclosure, the signal receiving end may determine the first gain HRIR corresponding to the direct sound signal in the audio object signal to be rendered; determine the second gain HRIR corresponding to at least one reflected sound signals in the audio object signal to be rendered; determine, based on the first gain HRIR and the second gain HRIR, the mixed HRIR corresponding to the audio object signal to be rendered; and render the audio object signal to be rendered based on the mixed HRIR. In the present disclosure, the first gain HRIR may represent the gain loss caused by the orientation and the distance of the sound source during the transmission of the direct sound signal. The second gain HRIR may represent at least one of the gain loss caused by the orientation of the sound source, the gain loss caused by the reflection, and the gain loss caused by the transmission distance during the transmission of the reflected sound signal. Therefore, when rendering the audio object signal to be rendered based on the mixed HRIR obtained from the first gain HRIR and the second gain HRIR, the rendering effect may be ensured, the real environment of the sound signal may be restored as much as possible, the experience of sound source orientation rendering may be enhanced, and the rendering effect may be closer to the real situation. In addition, the rendering method of the present disclosure may achieve the personalized adjustment for other types of sound sources, so as to have a wide application range. In the present disclosure, the targeted processing of different sound sources may be realized during the rendering, and the scene information of the scene where the sound source is located may be fully considered, so as to further ensure the rendering effect.
FIG. 6 is a flowchart of a method for rendering audio signal according to an embodiment of the present disclosure. The method is performed by a signal receiving end. As shown in FIG. 6, the method for rendering audio signal may include the following steps.
At step S601, a sound source position and a listener position are determined based on the spatial position information.
At step S602, a mirrored sound source position of the sound source position relative to a reflector is determined, in which the reflector is an object reflecting the sound signal.
At step S603, an incident angle of the reflected sound signal relative to a listener is determined based on the mirrored sound source position and the listener position.
For a detailed description of S601 to 603, please refer to the description of above embodiments.
In summary, in the method for rendering audio signal provided by the embodiments of the present disclosure, the signal receiving end may determine the first gain HRIR corresponding to the direct sound signal in the audio object signal to be rendered; determine the second gain HRIR corresponding to at least one reflected sound signals in the audio object signal to be rendered; determine, based on the first gain HRIR and the second gain HRIR, the mixed HRIR corresponding to the audio object signal to be rendered; and render the audio object signal to be rendered based on the mixed HRIR. In the present disclosure, the first gain HRIR may represent the gain loss caused by the orientation and the distance of the sound source during the transmission of the direct sound signal. The second gain HRIR may represent at least one of the gain loss caused by the orientation of the sound source, the gain loss caused by the reflection, and the gain loss caused by the transmission distance during the transmission of the reflected sound signal. Therefore, when rendering the audio object signal to be rendered based on the mixed HRIR obtained from the first gain HRIR and the second gain HRIR, the rendering effect may be ensured, the real environment of the sound signal may be restored as much as possible, the experience of sound source orientation rendering may be enhanced, and the rendering effect may be closer to the real situation. In addition, the rendering method of the present disclosure may achieve the personalized adjustment for other types of sound sources, so as to have a wide application range. In the present disclosure, the targeted processing of different sound sources may be realized during the rendering, and the scene information of the scene where the sound source is located may be fully considered, so as to further ensure the rendering effect.
FIG. 7 is a flowchart of a method for rendering audio signal according to an embodiment of the present disclosure. The method is performed by a signal receiving end. As shown in FIG. 7, the method for rendering audio signal may include the following steps.
At step S701, a mirrored listener position of the listener relative to a reflector is determined.
At step S702, an exit angle of the reflected sound signal relative to the sound source is determined based on the mirrored listener position and a sound source position.
At step S703, an orientation gain parameter of the reflected sound signal is determined based on the exit angle, the orientation information of the sound source and the sound cone information.
For a detailed description of S701 to 703, please refer to the description of above embodiments.
In summary, in the method for rendering audio signal provided by the embodiments of the present disclosure, the signal receiving end may determine the first gain HRIR corresponding to the direct sound signal in the audio object signal to be rendered; determine the second gain HRIR corresponding to at least one reflected sound signals in the audio object signal to be rendered; determine, based on the first gain HRIR and the second gain HRIR, the mixed HRIR corresponding to the audio object signal to be rendered; and render the audio object signal to be rendered based on the mixed HRIR. In the present disclosure, the first gain HRIR may represent the gain loss caused by the orientation and the distance of the sound source during the transmission of the direct sound signal. The second gain HRIR may represent at least one of the gain loss caused by the orientation of the sound source, the gain loss caused by the reflection, and the gain loss caused by the transmission distance during the transmission of the reflected sound signal. Therefore, when rendering the audio object signal to be rendered based on the mixed HRIR obtained from the first gain HRIR and the second gain HRIR, the rendering effect may be ensured, the real environment of the sound signal may be restored as much as possible, the experience of sound source orientation rendering may be enhanced, and the rendering effect may be closer to the real situation. In addition, the rendering method of the present disclosure may achieve the personalized adjustment for other types of sound sources, so as to have a wide application range. In the present disclosure, the targeted processing of different sound sources may be realized during the rendering, and the scene information of the scene where the sound source is located may be fully considered, so as to further ensure the rendering effect.
FIG 8 is a flowchart of a method for rendering audio signal according to an embodiment of the present disclosure. The method is performed by a signal receiving end. As shown in FIG. 8, the method for rendering audio signal may include the following step.
At step S801, a reflection gain parameter of the reflected sound signal is determined based on the room information.
For a detailed description of S801, please refer to the description of above embodiments.
In summary, in the method for rendering audio signal provided by the embodiments of the present disclosure, the signal receiving end may determine the first gain HRIR corresponding to the direct sound signal in the audio object signal to be rendered; determine the second gain HRIR corresponding to at least one reflected sound signals in the audio object signal to be rendered; determine, based on the first gain HRIR and the second gain HRIR, the mixed HRIR corresponding to the audio object signal to be rendered; and render the audio object signal to be rendered based on the mixed HRIR. In the present disclosure, the first gain HRIR may represent the gain loss caused by the orientation and the distance of the sound source during the transmission of the direct sound signal. The second gain HRIR may represent at least one of the gain loss caused by the orientation of the sound source, the gain loss caused by the reflection, and the gain loss caused by the transmission distance during the transmission of the reflected sound signal. Therefore, when rendering the audio object signal to be rendered based on the mixed HRIR obtained from the first gain HRIR and the second gain HRIR, the rendering effect may be ensured, the real environment of the sound signal may be restored as much as possible, the experience of sound source orientation rendering may be enhanced, and the rendering effect may be closer to the real situation. In addition, the rendering method of the present disclosure may achieve the personalized adjustment for other types of sound sources, so as to have a wide application range. In the present disclosure, the targeted processing of different sound sources may be realized during the rendering, and the scene information of the scene where the sound source is located may be fully considered, so as to further ensure the rendering effect.
FIG 9 is a flowchart of a method for rendering audio signal according to an embodiment of the present disclosure. The method is performed by a signal receiving end. As shown in FIG. 9, the method for rendering audio signal may include the following steps.
At S901, a mirrored listener position of the listener relative to a reflector is determined.
At S902, a distance delay gain parameter is determined based on a distance between the sound source position and the mirrored listener position, or a distance between the mirrored sound source position and the listener position.
For a detailed description of S901 to S902, please refer to the description of above embodiments.
In summary, in the method for rendering audio signal provided by the embodiments of the present disclosure, the signal receiving end may determine the first gain HRIR corresponding to the direct sound signal in the audio object signal to be rendered; determine the second gain HRIR corresponding to at least one reflected sound signals in the audio object signal to be rendered; determine, based on the first gain HRIR and the second gain HRIR, the mixed HRIR corresponding to the audio object signal to be rendered; and render the audio object signal to be rendered based on the mixed HRIR. In the present disclosure, the first gain HRIR may represent the gain loss caused by the orientation and the distance of the sound source during the transmission of the direct sound signal. The second gain HRIR may represent at least one of the gain loss caused by the orientation of the sound source, the gain loss caused by the reflection, and the gain loss caused by the transmission distance during the transmission of the reflected sound signal. Therefore, when rendering the audio object signal to be rendered based on the mixed HRIR obtained from the first gain HRIR and the second gain HRIR, the rendering effect may be ensured, the real environment of the sound signal may be restored as much as possible, the experience of sound source orientation rendering may be enhanced, and the rendering effect may be closer to the real situation. In addition, the rendering method of the present disclosure may achieve the personalized adjustment for other types of sound sources, so as to have a wide application range. In the present disclosure, the targeted processing of different sound sources may be realized during the rendering, and the scene information of the scene where the sound source is located may be fully considered, so as to further ensure the rendering effect.
FIG 10 is a flowchart of a method for rendering audio signal according to an embodiment of the present disclosure. The method is performed by a signal receiving end. As shown in FIG. 10, the method for rendering audio signal may include the following step.
At S1001, in response to there being a plurality of audio object signals to be rendered, a plurality of rendered audio object signals are down-mixed.
In summary, in the method for rendering audio signal provided by the embodiments of the present disclosure, the signal receiving end may determine the first gain HRIR corresponding to the direct sound signal in the audio object signal to be rendered; determine the second gain HRIR corresponding to at least one reflected sound signals in the audio object signal to be rendered; determine, based on the first gain HRIR and the second gain HRIR, the mixed HRIR corresponding to the audio object signal to be rendered; and render the audio object signal to be rendered based on the mixed HRIR. In the present disclosure, the first gain HRIR may represent the gain loss caused by the orientation and the distance of the sound source during the transmission of the direct sound signal. The second gain HRIR may represent at least one of the gain loss caused by the orientation of the sound source, the gain loss caused by the reflection, and the gain loss caused by the transmission distance during the transmission of the reflected sound signal. Therefore, when rendering the audio object signal to be rendered based on the mixed HRIR obtained from the first gain HRIR and the second gain HRIR, the rendering effect may be ensured, the real environment of the sound signal may be restored as much as possible, the experience of sound source orientation rendering may be enhanced, and the rendering effect may be closer to the real situation. In addition, the rendering method of the present disclosure may achieve the personalized adjustment for other types of sound sources, so as to have a wide application range. In the present disclosure, the targeted processing of different sound sources may be realized during the rendering, and the scene information of the scene where the sound source is located may be fully considered, so as to further ensure the rendering effect.
The above methods of the present disclosure will be illustrated by examples as follows.
The method in the present disclosure mainly focuses on improving a reverberation module by using a sound cone model to simulate reflected sounds received by two ears under different orientations.
The method for rendering audio signal provided in the present disclosure also belongs to object-based spatial audio technology, and its overall workflow is shown in FIG 11. An innovative part of the present disclosure mainly lies in "audio orientation rendering" in FIG 11. An orientation rendering is performed on an audio object by using orientation-related parameters (e.g., angle information, room information, sound cone information) in the metadata. The orientation rendering technology in the present disclosure generates binaural signals of the audio object under different orientations based on a scene (the room information) where the audio object is located and a position (spatial position of the audio object in other metadata) of the audio object, thereby achieving more realistic sense of sound source orientation.
In the real world, when the sound source orientation changes, the listener may perceive both a change in volume and a difference between the binaural signals. Therefore, a basic idea of this method is to process a direct sound and an early reflected sound using a mirrored source and the sound cone model. A mirrored source method is adopted to calculate a mirrored position of the sound source, and a delayed HRIR after being absorbed by a wall is obtained. It needs to be noted that in a real-time communication scenario, a cost of calculating and processing all reflected sound rays is too costly. A first-order reflection or a second-order reflection is sufficient to provide a good sense of direction, while other reflections may be processed in a reverberation function. Based on a direction of the sound source and positions of the sound source and a receiver, the sound cone model is used to assign different gains to each reflected sound ray. In this way, when an orientation changes, a presented left channel and right channel will be different.
In a rendering section of the present disclosure, the orientation rendering is performed by using orientation rendering parameters (e.g., sound source angle information, room parameter information, and sound cone information of the audio object) along with a spatial position of the audio object. The spatial position of the audio object could be a relative coordinate relative to the listener or an absolute coordinate in the scene. An orientation rendering process is shown in FIG 12. The sound source orientation rendering in the present disclosure includes two parts. One part is to calculate a gain parameter of a direct sound by using the sound cone information, in which a corresponding HRIR of the direct sound is obtained by using a conventional spatial audio rendering technology. The other part is to calculate a reflected sound of the listener position based on the room information and the sound cone information, and during the calculation, a HRIR of the reflected sound and a related gain parameter are obtained according to a position and an orientation of the listener, as well as a position and an orientation of the sound source, in which the gain parameter is used to process the HRIR of the reflected sound. Then, the HRIR of the object with a sense of orientation is obtained by performing a weighted summation on the HRIR of the two parts. A binaural rendering is performed on the audio object signal by using the HRIR of the object. After repeating this process for a plurality of objects, a down-mixing is performed to obtain a final output.
In a specific implementation of processing the reflected sound, first, positions of the mirrored sound source and the mirrored listener are calculates. A connecting line between a real listener and each mirrored sound source may form the reflected sound ray. Then, a gain and a delay of each reflected sound ray are obtained by using the mirrored position and the metadata, in which the gain includes an orientation gain, a reflection gain, and a distance gain. A distance loss of each sound ray is based on a distance between a real listener position and a mirrored sound source or a distance between a real sound source position and a mirrored listener. A reflection loss is based on a reflection coefficient in the metadata. An orientation loss of the reflected sound is based on the sound cone model and an exit angle of the reflected sound ray relative to the sound source orientation. The distance loss and delay are determined based on a distance between the real listener position and the mirrored sound source. Finally, in the HRTF database, the HRIR corresponding to each sound ray is obtained based on an incident angle of the reflected sound ray relative to an orientation of a human head. By summing all the HRIRs and processing them with the calculated gain and delay, a final HRIR may be obtained for the binaural rendering.
The present disclosure proposes a method for rendering audio signal with the sense of the sound source orientation. Based on the related arts, the reverberation algorithm is improved by combining with the sound cone model to achieve the rendering for the sense of the sound source orientation. The technical solution proposed the present disclosure may enhance the experience of the audio source orientation rendering, and make the rendering effect closer to the real situation.
FIG 13 is a schematic diagram of a communication apparatus according to another embodiment of the present disclosure. As shown in FIG 13, the communication apparatus may include:

In summary, in the communication apparatus provided by the embodiments of the present disclosure, the signal receiving end may determine the first gain HRIR corresponding to the direct sound signal in the audio object signal to be rendered; determine the second gain HRIR corresponding to at least one reflected sound signals in the audio object signal to be rendered; determine, based on the first gain HRIR and the second gain HRIR, the mixed HRIR corresponding to the audio object signal to be rendered; and render the audio object signal to be rendered based on the mixed HRIR. In the present disclosure, the first gain HRIR may characterize a gain loss caused by a sound source orientation and a distance of the direct sound signal in a transmission, and the second gain HRIR may characterize at least one of a gain loss caused by the sound source orientation, a gain loss caused by a reflection, and a gain loss caused by a transmission distance of the reflected sound signal in a transmission. Therefore, when the audio object signal to be rendered is rendered based on the mixed HRIR obtained from the first gain HRIR and the second gain HRIR, a good rendering effect may be ensured, which may restore a real environment of a sound signal as much as possible, improve experience of sound source orientation rendering, and make the rendering effect closer to a real situation. In addition, the method for rendering audio signal in the present disclosure may also realize personalized processing of different types of sound sources, and may be applied to a wide range. At the same time, the method may realize targeted processing of different sound sources, and fully consider scene information of a scene where the sound source is located, so as to further ensure the rendering effect.
Optionally, in an embodiment of the present disclosure, the processing module is further configured to:

receive a code stream sent by a signal sending end;
decode the code stream to obtain at least one audio object signal to be rendered and metadata of the at least one audio object signal to be rendered; and
determine, based on the metadata of the audio object signal to be rendered, the first gain HRIR corresponding to the direct sound signal in the audio object signal to be rendered and the second gain HRIR corresponding to the reflected sound signal in the audio object signal to be rendered.

Optionally, in an embodiment of the present disclosure, the metadata includes at least one of:

Optionally, in an embodiment of the present disclosure, the processing module is further configured to:

determine an incident angle of the direct sound signal relative to the listener based on the spatial position information of the sound source and/or the spatial position information of the listener;
determine a first HRIR corresponding to the direct sound signal based on the incident angle of the direct sound signal relative to the listener and the orientation information of the listener;
determine an orientation gain parameter of the direct sound signal based on the orientation information of the sound source and the sound cone information;
determine a distance gain parameter of the direct sound signal based on the spatial position information of the sound source and/or the spatial position information of the listener; and
determine the first gain HRIR based on the orientation gain parameter, the distance gain parameter, and the first HRIR.

determine a sound source position and a listener position based on the spatial position information of the sound source and/or the spatial position information of the listener; and
determine the distance gain parameter of the direct sound signal based on a distance between the sound source position and the listener position.

determine an incident angle of the reflected sound signal relative to the listener based on the spatial position information of the sound source and/or the spatial position information of the listener;
determine a second HRIR corresponding to the reflected sound signal based on the incident angle of the reflected sound signal relative to the listener and the orientation information of the listener;
determine at least one of an orientation gain parameter, a reflection gain parameter, or a distance delay gain parameter of the reflected sound signal based on information in the metadata; and
determine the second gain HRIR based on the second HRIR and at least one of the orientation gain parameter, the reflection gain parameter, or the distance delay gain parameter.

determine a sound source position and a listener position based on the spatial position information;
determine a mirrored sound source position of the sound source position relative to a reflector, in which the reflector is an object reflecting a sound signal; and
determine the incident angle of the reflected sound signal relative to the listener based on the mirrored sound source position and the listener position.

determine a mirrored listener position of the listener relative to a reflector;
determine an exit angle of the reflected sound signal relative to the sound source based on the mirrored listener position and the sound source position; and
determine the orientation gain parameter of the reflected sound signal based on the exit angle, the orientation information of the sound source and the sound cone information.

Optionally, in an embodiment of the present disclosure, the processing module is further configured to:
determine the reflection gain parameter of the reflected sound signal based on the room information.
Optionally, in an embodiment of the present disclosure, the processing module is further configured to:

determine a mirrored listener position of the listener relative to a reflector; and
determine the distance delay gain parameter based on a distance between a sound source position and the mirrored listener position, or a distance between a mirrored sound source position and a listener position.

Optionally, in an embodiment of the present disclosure, the processing module is further configured to:
obtain the mixed HRIR by performing a weighted summation on the first gain HRIR corresponding to the direct sound signal of the audio object signal to be rendered and the second gain HRIR corresponding to the at least one reflected sound signal of the audio object signal to be rendered.
Optionally, in an embodiment of the present disclosure, the processing module is further configured to:
down-mix a plurality of rendered audio object signals.
Optionally, in an embodiment of the present disclosure, the reflected sound signal includes an early reflected sound signal.
Optionally, in an embodiment of the present disclosure, the early reflected sound signal includes at least one of:

a first-order reflected sound signal; or
a second-order reflected sound signal.

Please refer to FIG. 14, which is a schematic diagram of a communication apparatus 1400 according to the embodiments of the present disclosure. The communication apparatus 1400 may be a network device, or a terminal, or a chip, a chip system, a processor, etc. that supports the network device to implement the method, or a chip, a chip system, a processor, etc. that supports the terminal to implement the method. The apparatus is configured to implement the method in the above method embodiments. For details, please refer to the description in the above method embodiments.
The communication apparatus 1400 may include one or more processors 1401. The processor 1401 may be a general-purpose processor or a special-purpose processor. For example, it may be a baseband processor or a central processing unit. The baseband processor is configured to process communication protocols and communication data, and the central processor is configured to control communication apparatus (such as base stations, baseband chips, terminals, terminal chips, DU or CU, etc.) to execute computer programs and process computer program data.
Optionally, the communication apparatus 1400 may also include one or more memories 1402 on which a computer program 1404 is stored. When the computer program 1404 is executed by the processor 1401, the communication apparatus 1400 implements the method in the above method embodiments. Optionally, the memory 1402 may also store data. The communication apparatus 1400 and the memory 1402 may be set separately or integrated together.
Optionally, the communication apparatus 1400 may also include a transceiver 1405 and an antenna 1406. The transceiver 1405 may be called a transceiving unit, a transceiving machine, or a transceiving circuit, etc., to achieve a transceiving function. The transceiver 1405 may include a receiver and a transmitter. The receiver may be called a receiver or a receiving circuit, etc., to achieve a receiving function; and the transmitter may be called a transmitter or a transmitting circuit, etc., to achieve a transmitting function.
Optionally, the communication apparatus 1400 includes one or more interface circuits 1407. The interface circuit 1407 is configured to receive code instructions and transmit the code instructions to the processor 1401. When the code instructions are running on the processor 1401, the communication apparatus 1400 is caused to implement the method in the above method embodiments.
In an implementation, the processor 1401may include a transceiver configured to achieve the receiving and transmitting function. For example, the transceiver may be a transceiving circuit, or an interface, or an interface circuit. The transceiving circuit, the interface, or the interface circuit configured to achieve the receiving and transmitting function may be set separately or integrated together. The transceiving circuit, the interface or the interface circuit may be configured to read and write code/data, or the transceiving circuit, the interface or the interface circuit may be configured to transmit signals.
In an implementation, the processor 1401may store a computer program 1403. When the computer program 1403 is running on the processor 1401, the communication apparatus 1400 is caused to implement the method in the above method embodiments. The computer program 1403 may be solidified in the processor 1401. In this way, the processor 1401may be implemented in hardware.
In an implementation, the communication apparatus 1400 may include a circuit that may achieve the transmitting or receiving or communicating function in the above method embodiments. The processor and transceiver in the present disclosure may be implemented in an integrated circuit (IC), an analog IC, a radio frequency integrated circuit (RFIC), a mixed-signal IC, an application specific integrated circuit (ASIC), a printed circuit board (PCB), an electronic equipment, etc. The processor and transceiver may also be manufactured with various IC process technologies, such as a complementary metal oxide semiconductor (CMOS), nMetal-oxide-semiconductor (NMOS), a positive channel metal oxide semiconductor (PMOS), a bipolar junction transistor (BJT), a bipolar CMOS (BiCMOS), silicon germanium (SiGe), gallium arsenide (GaAs), etc.
The communication apparatus in the above embodiments may be a network device or a terminal, but the scope of the communication apparatus in the present disclosure is not limited to this, and the structure of the communication apparatus may not be restricted by FIG 14. The communication apparatus may be an independent device or part of a larger device. For example, the communication apparatus may be:

(1) an independent IC, or a chip, or a chip system or a subsystem;
(2) a collection including one or more IC, optionally, the IC collection may also include storage components for storing data and computer programs;
(3) an ASIC, such as a modem;
(4) modules embedded in other devices;
(5) a receiver, a terminal, an intelligent terminal, a cellular phone, a wireless device, a handheld phone, a mobile unit, a vehicle-mounted device, a network device, a cloud device, an artificial intelligence device, etc.;
(6) others.

For the case where the communication apparatus may be a chip or a chip system, please refer to FIG 15, which is a block diagram of a chip according to one embodiment of the present disclosure. The chip in FIG 15 includes a processor 1501 and an interface 1502. There may be one or more processors 1501, and there may be one or more interfaces 1502.
Optionally, the chip further includes a memory 1503 for storing necessary computer programs and data.
Those skilled in the art may also understand that the various illustrative logical blocks and steps listed in the embodiments of the present disclosure may be implemented by electronic hardware, computer software, or their combination. Whether such a function is implemented in hardware or software depends on specific applications and design requirements of the overall system. Those skilled in the art may, for each specific application, use a variety of methods to achieve the above function, but such implementation shall not be regarded as going beyond the scope of the protection of the embodiments of the present disclosure.
The present disclosure also provides a readable storage medium for storing instructions. When the instructions are executed by a computer, functions of any one of the above method embodiments are performed.
The present disclosure also provides a computer program product. When the computer program product is executed by a computer, functions of any one of the above method embodiments are performed.
In the above embodiments, the functions may be wholly or partially implemented by software, hardware, firmware, or any combination of them. When implemented by software, the functions may be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer programs. Procedures or functions according to embodiments of the present disclosure are wholly or partially generated when the computer program is loaded and executed on a computer. The computer may be a general purpose computer, a special purpose computer, a computer network, or other programmable device. The computer program may be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer program may be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wire (such as a coaxial cable, a fiber optic, a digital subscriber line (DSL)) or wireless (such as infrared, wireless, microwave). The computer-readable storage medium may be any available medium that may be accessed by a computer, or a data storage device such as a server that integrates one or more of the available media, and a data center. The available medium media be a magnetic medium (such as a floppy disk, a hard disk and a magnetic tape), an optical medium (such as a digital video disk (DVD)), or a semiconductor medium (such as a solid state disk (SSD)).
Those skilled in the art may understand that various numbers such as first and second involved in the present disclosure are distinguished merely for convenience of description, and are not intended to limit the scope of embodiments of the present disclosure, but also to indicate an order of precedence.
The term "at least one" in the present disclosure may also be described as one or more, and the more may be two, three, four, or more, which is not limited in the present disclosure. In the embodiment of the present disclosure, for a technical feature, the technical feature in the technical features are distinguished by terms "first", "second", "third", "A", "B", "C" and "D", etc., and the technical features described by the terms "first", "second", "third", "A", "B", "C" and "D", etc. are not in a sequential order or in an order of size.
Corresponding relationships indicated by tables in the present disclosure may be configured or predefined. Values of information in the tables are only examples, and may be configured as other values, which are not limited in the present disclosure. When the corresponding relationship between information and parameters is configured, it is not always necessary to configure all corresponding relationships indicated in tables. For example, in the tables of the present disclosure, corresponding relationships indicated by some rows may not be configured. For another example, appropriate transformations and adjustments, such as splitting and merging, may be made based on the above tables. Names of parameters shown in headers of the tables may be other names understandable by the communication apparatus, and values or representations of the parameters may be other values or representations understandable by the communication apparatus. When the above tables are implemented, other data structures may be used, for example, arrays, queues, containers, stacks, linear lists, pointers, linked lists, trees, graphs, structures, classes, heaps or hash tables may be used.
Predefined in the present disclosure may be understood as defined, predefined, stored, pre-stored, pre-negotiated, pre-configured, solidified or pre-fired.
Those skilled in the related art may realize that, in combination with units and algorithm steps of the examples described in embodiments of the present disclosure, may be implemented by an electronic hardware or a combination of an electronic hardware and a computer software. Whether the functions are executed by the hardware or the software depends on a specific application and a design constraint of the technical solutions. Those skilled in the art may adopt different methods for each specific application to implement the described functions, but such implementation should not be considered as going beyond the scope of the present disclosure.
Those skilled in the art may clearly understand that, a specific working process of a system, an apparatus and a unit described above may refer to a corresponding process in the above method embodiments, which will not be repeated here.
The above are only implementations of the present disclosure. However, the protection scope of the present disclosure is not limited here. Changes and substitutions that may be easily considered by those skilled in the art shall be contained within the protection scope of the present disclosure. Therefore, the protection scope of the present disclosure shall be subject to the protection scope of claims.

Claims

A method for rendering audio signal, performed by a signal receiving end, comprising:
determining a first gain head-related impulse response (HRIR) corresponding to a direct sound signal in an audio object signal to be rendered;

determining a second gain HRIR corresponding to at least one reflected sound signal in the audio object signal to be rendered;

determining, based on the first gain HRIR and the second gain HRIR, a mixed HRIR corresponding to the audio object signal to be rendered; and

rendering the audio object signal to be rendered based on the mixed HRIR.
The method of claim 1, wherein determining the first gain HRIR corresponding to the direct sound signal and the second gain HRIR corresponding to the reflected sound signal comprises:
receiving a code stream sent by a signal sending end;

decoding the code stream to obtain at least one audio object signal to be rendered and metadata of the at least one audio object signal to be rendered; and

determining, based on the metadata of the audio object signal to be rendered, the first gain HRIR corresponding to the direct sound signal in the audio object signal to be rendered and the second gain HRIR corresponding to the reflected sound signal in the audio object signal to be rendered.
The method of claim 2, wherein the metadata comprises at least one of:
orientation information of a sound source of the audio object signal to be rendered;

sound cone information of a sound source of the audio object signal to be rendered;

room information of a room where a sound source of the audio object signal to be rendered is located;

spatial position information of a sound source of the audio object signal to be rendered;

orientation information of a listener; or

spatial position information of a listener.
The method of claim 3, wherein determining, based on the metadata of the audio object signal to be rendered, the first gain HRIR corresponding to the direct sound signal in the audio object signal to be rendered comprises:
determining an incident angle of the direct sound signal relative to the listener based on the spatial position information of the sound source and/or the spatial position information of the listener;

determining a first HRIR corresponding to the direct sound signal based on the incident angle of the direct sound signal relative to the listener and the orientation information of the listener;

determining an orientation gain parameter of the direct sound signal based on the orientation information of the sound source and the sound cone information;

determining a distance gain parameter of the direct sound signal based on the spatial position information of the sound source and/or the spatial position information of the listener; and

determining the first gain HRIR based on the orientation gain parameter, the distance gain parameter, and the first HRIR.
The method of claim 4, wherein determining the distance gain parameter of the direct sound signal based on the spatial position information of the sound source and/or the spatial position information of the listener comprises:
determining a sound source position and a listener position based on the spatial position information of the sound source and/or the spatial position information of the listener; and

determining the distance gain parameter of the direct sound signal based on a distance between the sound source position and the listener position.
The method of claim 3, wherein determining, based on the metadata of the audio object signal to be rendered, the second gain HRIR corresponding to the reflected sound signal in the audio object signal to be rendered comprises:
determining an incident angle of the reflected sound signal relative to the listener based on the spatial position information of the sound source and/or the spatial position information of the listener;

determining a second HRIR corresponding to the reflected sound signal based on the incident angle of the reflected sound signal relative to the listener and the orientation information of the listener;

determining at least one of an orientation gain parameter, a reflection gain parameter, or a distance delay gain parameter of the reflected sound signal based on information in the metadata; and

determining the second gain HRIR based on the second HRIR and at least one of the orientation gain parameter, the reflection gain parameter, or the distance delay gain parameter.
The method of claim 6, wherein determining the incident angle of the reflected sound signal relative to the listener based on the spatial position information comprises:
determining a sound source position and a listener position based on the spatial position information;

determining a mirrored sound source position of the sound source position relative to a reflector, wherein the reflector is an object reflecting a sound signal; and

determining the incident angle of the reflected sound signal relative to the listener based on the mirrored sound source position and the listener position.
The method of claim 6, wherein determining the orientation gain parameter of the reflected sound signal based on the information in the metadata comprises:
determining a mirrored listener position of the listener relative to a reflector;

determining an exit angle of the reflected sound signal relative to the sound source based on the mirrored listener position and a sound source position; and

determining the orientation gain parameter of the reflected sound signal based on the exit angle, the orientation information of the sound source and the sound cone information.
The method of claim 6, wherein determining the reflection gain parameter of the reflected sound signal based on the information in the metadata comprises:
determining the reflection gain parameter of the reflected sound signal based on the room information.
The method of claim 6, wherein determining the distance delay gain parameter of the reflected sound signal based on the information in the metadata comprises:
determining a mirrored listener position of the listener relative to a reflector; and

determining the distance delay gain parameter based on a distance between a sound source position and the mirrored listener position, or a distance between a mirrored sound source position and a listener position.
The method of claim 1, wherein determining the mixed HRIR corresponding to the audio object signal to be rendered comprises:
obtaining the mixed HRIR by performing a weighted summation on the first gain HRIR corresponding to the direct sound signal of the audio object signal to be rendered and the second gain HRIR corresponding to the at least one reflected sound signal of the audio object signal to be rendered.
The method of claim 1, in response to there being a plurality of audio object signals to be rendered, further comprising:
down-mixing a plurality of rendered audio object signals.
The method of any one of claims 1 to 12, wherein the reflected sound signal comprises an early reflected sound signal.
The method of claim 13, wherein the early reflected sound signal comprises at least one of:
a first-order reflected sound signal; or

a second-order reflected sound signal.
A communication apparatus, configured in a signal receiving end, comprising:
a processing module, configured to determine a first gain head-related impulse response (HRIR) corresponding to a direct sound signal in an audio object signal to be rendered,

wherein the processing module is further configured to determine a second gain HRIR corresponding to at least one reflected sound signal in the audio object signal to be rendered;

wherein the processing module is further configured to determine, based on the first gain HRIR and the second gain HRIR, a mixed HRIR corresponding to the audio object signal to be rendered; and

wherein the processing module is further configured to render the audio object signal to be rendered based on the mixed HRIR.
A communication apparatus, comprising a processor and a memory for storing a computer program, wherein the computer program stored in the memory is executed by the processor, the method of any one of claims 1 to 14 is implemented.
A communication apparatus, comprising a processor and an interface circuit; wherein
the interface circuit is configured to receive code instructions and transmit the code instructions to the processor; and

the processor is configured to run the code instructions to implement the method of any one of claims 1 to 14.
A computer readable storage medium for storing instructions, wherein when the instructions are executed, the method of any one of claims 1 to 14 is implemented.