WO2020039734A1

WO2020039734A1 - Audio reproducing device, audio reproduction method, and audio reproduction program

Info

Publication number: WO2020039734A1
Application number: PCT/JP2019/025199
Authority: WO
Inventors: 哲曲谷地; 一敦大栗
Original assignee: ソニー株式会社
Priority date: 2018-08-21
Filing date: 2019-06-25
Publication date: 2020-02-27
Also published as: DE112019004193T5; CN112567769B; CN112567769A

Abstract

An audio reproducing device comprising a first decoder for decoding a first signal correlated to a spatial frequency into a multiple-channel audio signal, a second decoder for decoding a second signal that includes a band different from the first signal and is correlated to spatial coordinates into a multiple-channel audio signal, and an addition unit for adding together the multiple-channel audio signal decoded by the first decoder and the multiple-channel audio signal decoded by the second decoder.

Description

Audio playback device, audio playback method, and audio playback program

The present disclosure relates to an audio playback device, an audio playback method, and an audio playback program.

Conventionally, in an audio reproducing apparatus, in addition to reproducing an audio signal in a stereo form (two channels), a multi-channel form in which the number of speakers is further increased is known. In the audio signal reproduction using such a multi-channel, it is possible to three-dimensionally reproduce the sound field at the time of sound pickup, and to provide a listener with a realistic sound field.

文献 Patent Document 1 discloses a method of encoding an audio signal using higher-order ambisonics for such audio reproduction using multi-channels.

JP 2012-133366 A

では In such a field, it is desired to improve the quality of reproduced audio.

開示 One object of the present disclosure is to provide an audio reproducing device, an audio reproducing method, and an audio reproducing program for improving the quality of reproduced audio.

The present disclosure, for example,
A first decoder for decoding a first signal associated with a spatial frequency into audio signals of a plurality of channels;
A second decoder that decodes a second signal including a band different from the first signal and corresponding to spatial coordinates into audio signals of a plurality of channels;
An audio playback device comprising: an adder that adds a plurality of channels of audio signals decoded by the first decoder and a plurality of channels of audio signals decoded by the second decoder.

The present disclosure, for example,
A first signal associated with a spatial frequency is decoded into an audio signal of a plurality of channels, and a second signal including a band different from that of the first signal and associated with a spatial coordinate is converted into an audio signal of a plurality of channels. Decode and
An audio playback method for adding a plurality of channels of audio signals decoded based on the first signal and a plurality of channels of audio signals decoded based on the second signal.

The present disclosure, for example,
A first decoding process of decoding a first signal associated with a spatial frequency into audio signals of a plurality of channels;
A second decoding process of decoding a second signal including a band different from the first signal and associated with spatial coordinates into audio signals of a plurality of channels;
An audio reproduction program for causing an information processing apparatus to execute an addition process of adding audio signals of a plurality of channels decoded by the first decoder and audio signals of a plurality of channels decoded by the second decoder.

According to at least one embodiment of the present disclosure, it is possible to improve the quality of audio to be reproduced. The effects described here are not necessarily limited, and may be any of the effects described in the present disclosure. In addition, the contents of the present disclosure are not to be construed as being limited by the illustrated effects.

FIG. 1 is a diagram illustrating a configuration of an audio system as a comparative example. FIG. 2 is a diagram for describing an overview of the audio system according to the first embodiment. FIG. 3 is a diagram illustrating frequency characteristics of the audio system according to the first embodiment. FIG. 4 is a diagram illustrating a configuration of the audio system according to the first embodiment. FIG. 5 is a diagram showing a recording format of a recording signal used in the audio system according to the first embodiment. FIG. 6 is a diagram illustrating a configuration of an audio system according to the second embodiment. FIG. 7 is a diagram illustrating a configuration of an audio system according to the third embodiment. FIG. 8 is a diagram illustrating a configuration of an audio system according to the fourth embodiment. FIG. 9 is a diagram illustrating a configuration of an audio system according to the fifth embodiment. FIG. 10 is a diagram illustrating frequency characteristics of the audio system according to the fifth embodiment. FIG. 11 is a diagram illustrating a configuration of an audio system according to the sixth embodiment. FIG. 12 is a diagram illustrating a recording format of a recording signal used in an audio system according to a modification.

Hereinafter, embodiments and the like of the present disclosure will be described with reference to the drawings. The description will be made in the following order.
<1. Higher Order Ambisonics (About HOA)>
<2. First Embodiment>
<3. Second Embodiment>
<4. Third Embodiment>
<5. Fourth embodiment>
<6. Fifth Embodiment>
<7. Sixth embodiment>
<8. Modification>
The embodiments and the like described below are preferred specific examples of the present disclosure, and the contents of the present disclosure are not limited to these embodiments.

<1. Higher Order Ambisonics (About HOA)>
2. Description of the Related Art In recent years, in the field of audio, development and spread of three-dimensional sound for recording, transmitting, and reproducing spatial information from all around have been advanced. In the field of broadcasting regarding such three-dimensional sound, progress has been remarkable, for example, a three-dimensional multi-channel sound broadcasting using 22.2 channels is planned. In the field of virtual reality, in addition to video that surrounds the entire circumference, audio that reproduces a signal that surrounds the entire circumference is becoming popular.

の中 In such a situation, attention has been paid to a method of expressing three-dimensional audio information called ambisonics that can flexibly cope with an arbitrary recording and reproducing system. In particular, ambisonics whose order is second or higher are called higher order ambisonics (HOA: \ Higher \ Order \ Ambisonics). In three-dimensional multi-channel sound, sound information spreads in the spatial axis in addition to the time axis. In Ambisonics, information is stored by performing spatial frequency conversion (spherical harmonic function conversion) in the angular direction of three-dimensional polar coordinates. are doing. This can be considered to correspond to time-frequency conversion of the audio signal with respect to the time axis. An advantage of this method is that information can be encoded and decoded from an arbitrary microphone array to an arbitrary speaker array without limiting the number of microphones and the number of speakers.

On the other hand, the following problems are raised in the recording / reproduction of the HOA system.
-When HOA encoding an input signal from a spherical or annular microphone set (also referred to as a microphone array), spatial aliasing may occur at a high frequency depending on the number and spacing of microphones and the radius of the array. Sound fields cannot be recorded and expressed correctly above the frequency.
When a microphone set is constructed using an actual microphone, since the frequency calculated from its actual size and the number of microphones is within the audible band, sound quality within a frequency band that can be perceived in HOA recording / reproduction. Deterioration occurs.

First, the details of the HOA method will be described below. Encodings used in the HOA method can be roughly classified into two types. One is a recording base, and the other is an object base. Since the first recording base is targeted in the present embodiment, this will be described.

(Conversion from microphone recording signal to HOA signal)
A certain time frequency ω of the sound signal recorded by the annular or spherical microphone array is converted into HOA signals A _m (ω) and _An ^m (ω) according to the following equations, respectively.

Equation (1) is for a circular microphone array, and Equation (2) is for a spherical microphone array. Here, φ _q and θ _q represent the azimuth and elevation of the q-th microphone, and P _q (ω) represents the sound pressure of the q-th microphone. In Equation (1), J _m (ka) is a Bessel function, m is its order, k is the wave number, and a is the radius of the microphone array. In equation (2), the Bessel function of Equation (1) is the spherical Bessel function, e ^-Imfaiq spherical harmonics _{^{_{Y n m (φ q, θ}}} q) is replaced to. Where the spherical harmonic is

Is defined as P _n ^m is the associated Legendre functions. There are various other definitions of the spherical harmonic function, but any definition will not affect the purpose of the present disclosure, and will be used in the future.

M, n is the order of the HOA. Since this is a conversion of the sound pressure P which is a continuous function of the azimuth and the elevation, the orders m and n exist up to infinity. However, when recording with a spherical microphone array, it is impossible to capture the sound pressure P as a continuous function. Therefore, similar to the sampling theorem at the time frequency, the following relation exists between the reproducible HOA orders M and N and the number Q of microphones.

Equation (4) for a ring, and equation (5) for a sphere.

(Conversion from HOA signal to audio signal)
When converting the HOA signal into an audio signal, in a two-dimensional case, at a certain time frequency ω,

Becomes In the case of three dimensions,

Becomes
Here, R is the radius of the speaker array, α _i , β _i are the elevation and azimuth angles of the i-th speaker, and G _m (R, ω) and G _n ⁰ (R, ω) are the transfer functions of HOA coefficient.

Here, H _m ⁽²⁾ (kR) is a Hankel function of the second kind and h _n ⁽²⁾ (kR) is a Hankel function of the second kind. The conversion formula between the HOA signal and the audio signal differs depending on the shape of the microphone array, the shape of the speaker array, the directivity, and the like. In the following, descriptions as HOA encoding and HOA decoding mean that these various systems are included, and are not limited to any of them.

(Spatial aliasing)
As described above, in recording by a microphone set, the order is finite due to the limitation of the number of microphones. Therefore, if a signal of a higher order is mixed, spatial aliasing occurs. If a signal in which spatial aliasing occurs is encoded and decoded by the HOA method, a signal different from the recorded space will be reproduced. The effect of this aliasing depends on the time frequency and the radius of the microphone. As the time frequency becomes lower and the microphone radius becomes smaller, the higher-order signal of the HOA order becomes smaller. In other words, for the same time frequency, the smaller the radius of the microphone, the smaller the higher-order HOA signal, and the less the aliasing effect. Also, if the microphone radius is the same, the effect of aliasing is reduced in the low frequency band.

In various embodiments described below, an object is to perform high-quality audio reproduction by suppressing the influence of spatial aliasing generated at a particularly high frequency in accordance with the number and spacing of microphones and the radius of the array. .

FIG. 1 is a diagram showing a configuration of an audio system 1 as a comparative example. This comparative example is a conventional form using only the HOA method, and includes a HOA encoder 22 and a HOA decoder 31. Audio signals collected by a plurality of microphones provided in the microphone set 41 are input to the HOA encoder 22. Here, the microphone set 41 includes a plurality of microphones provided in an appropriate arrangement such as a ring, a sphere, and a line. The HOA encoder 22 performs HOA encoding on a plurality of audio signals collected by the microphone set 41, thereby converting the audio signals into a HOA signal represented as a spatial frequency.

The HOA decoder 31 can reproduce the received HOA signal using an arbitrary speaker set 42. Here, the speaker set 42 used includes a plurality of speakers provided in an appropriate arrangement such as a ring, a sphere, a line, and the like. Further, the arrangement of the speaker set 42 does not need to depend on the microphone arrangement of the microphone set 41 that has collected the sound. This is because the HOA signal is expressed in the spatial frequency. By setting the arrangement of the speakers of the speaker set 42 with respect to the HOA decoder 31, it is possible to reproduce the sound field collected. It is possible.

By the way, in the audio system 1 shown in FIG. 1, due to the physical limitations of the microphones in the microphone set 41, information of spatial sound cannot be correctly expressed in a HOA signal, particularly in a high frequency region, resulting in deterioration of sound quality. There is. This problem is not limited to the HOA, and is a problem existing in a recording system using a multi-microphone.

<2. First Embodiment>
FIG. 2 is a diagram for describing an overview of the audio system 1 according to the first embodiment. The audio system 1 according to the first embodiment includes an LPF 21 (Low Pass Filter), a HOA encoder 22, a HOA decoder 31, an HPF 23 (High Pass Filter), a multiplier 33, and an adder 34. The audio system 1 receives a plurality of microphones provided in the microphone set 41 as input and outputs a speaker set 42 in which a plurality of speakers are arranged. In FIG. 2, the audio signal output from the microphone set 41 and input to the HOA encoder 22 via the LPF 21 and the audio signal input to the HPF 23 are equivalent to the number of microphones provided in the microphone set 41. Has the number of channels. The audio signal output from the HOA decoder 31 and output to the speaker set 42 via the adder 34 has the same number of channels as the number of speakers arranged in the speaker set 42. As described above, in the block diagram shown in FIG. 2, for convenience of drawing, there are places where a plurality of channels are indicated by one line.

A plurality of audio signals collected by a plurality of microphones of the microphone set 41 are input to the HOA encoder 22. In the present embodiment, the LPF 21 is used for the plurality of sound signals input from the microphone set 41. , High-frequency components are removed, and the frequency band is limited to a frequency band that can be correctly expressed by the HOA signal. The HOA encoder 22 converts the plurality of audio signals from which the high-frequency components have been removed by the LPF 21 into HOA signals represented as spatial frequency.

The HOA decoder 31 decodes the HOA signal output from the HOA encoder 22 and reproduces the HOA signal using an arbitrary speaker set 42. At this time, in a plurality of audio signals input to the microphone set 41, a high frequency band that cannot be expressed by the HOA encoder 22 is only a high-frequency component via the HPF 23, and after performing gain adjustment by the multiplication unit 33, At 34, the sum is added to the HOA-decoded audio signal and output to the speaker set 42.

FIG. 3 is a diagram illustrating frequency characteristics of the audio system 1 according to the first embodiment. In the frequency characteristics shown in FIG. 3, the low-pass characteristics shown by the solid lines indicate the characteristics of the LPF 21. The high-pass characteristics indicated by the broken lines indicate the characteristics of the HPF 23. By adding the low-pass characteristics of the LPF 21 and the high-pass characteristics of the HPF 23, a flat frequency characteristic is formed from low to high frequencies. These characteristics are just one example, and various characteristics are possible depending on the design method.

(Configuration of audio playback system)
FIG. 4 is a diagram illustrating a configuration of the audio system 1 according to the first embodiment. Although the outline of the audio system 1 has been described with reference to FIG. 2, the audio system 1 is actually divided into a recording device 2 provided on the recording side and a reproducing device 3 provided on the reproducing side. The recording signal recorded by the recording device 2 is recorded on a recording medium or transmitted via communication. The reproduction device 3 reproduces the sound field at the time of recording by reproducing the recording signal recorded on the recording medium or the recording signal transmitted via communication.

In the present embodiment, the input side and the output side have eight channels (ch: channel), the microphone set 41 uses eight microphones m1 to m8, and the speaker set 42 also has eight speakers. s1 to s8 are used. The microphones m1 to m8 and the speakers s1 to s8 are arranged such that the numbers of the subscripts correspond to each other. In FIG. 4, the numbers shown on the lines between the blocks indicate the number of channels.

The recording device 2 located on the recording side of the audio system 1 includes the LPF 21, the HOA encoder 22, the HPF 23, and the encoder 24. The LPF 21, the HOA encoder 22, and the HPF 23 are the same as those described with reference to FIG. The encoder 24 converts the audio signal that has passed through the HPF 23 into a signal corresponding to spatial coordinates. Here, as a method of converting into a signal corresponding to the spatial coordinates, for example, PCM (Pulse Code Modulation) coding, ADPCM (Adaptive Differential Pulse Code Modulation coding, Delta modulation, etc.) A method that depends on coordinates.

On the other hand, the HOA encoder 22 is different from the encoder 24 in that the HOA encoder 22 converts the audio signal input from the LPF 21 into a signal corresponding to a spatial frequency. The HOA signal converted by the HOA encoder 22 can reproduce the sound at the spatial coordinate position by designating the spatial coordinates to be reproduced, that is, the positions of the speakers s1 to s8 in the speaker set 42. .

The HOA signal obtained as a result of conversion by the HOA encoder 22 of the recording device 2 and the high-frequency signal obtained as a result of conversion by the encoder 24 are recorded on a recording medium as a recording signal, or sent to the reproducing device 3 located on the reproducing side. Is sent.

FIG. 5 is a diagram showing a recording format of a recording signal used in the audio system 1 according to the first embodiment. The recording signal has a header section and a data section. The header section is a section in which various meta information necessary for reproducing the recorded audio signal is recorded. In the present embodiment, the meta information to be recorded in the header section is configured to include a sampling rate, a frame length, the number of frames, the number of band divisions, and band information (first band information and second band information) for each band. .

The sampling rate is the sampling rate used at the time of recording, and may be fixed or variable. The frame length is information defining the length of a frame recorded in the data section. Either fixed or variable frame length may be adopted. The number of frames (L) is a number that defines the number of frames forming a chunk that is a unit of one data in the data portion. The number of band divisions is a number indicating the number of bands divided in the audio system 1. In the present embodiment, the number of band divisions is “2” by the LPF 21 and the HPF 23 as described with reference to FIG. 2 ".

The first band information is information relating to conversion on the low band side, that is, the conversion of the HOA encoder 22. In the present embodiment, the first band information is configured to include a cutoff frequency, spatial domain information, signal domain information, compression scheme information, and an order. ing. The cutoff frequency corresponds to the cutoff frequency on the high frequency side of the LPF 21 described in FIG. The spatial domain information includes information indicating that the band is a HOA signal. In addition, information on the collected microphone set 41, for example, information on the arrangement of the microphones m1 to m8 in the microphone set 41, for example, , Spherical, annular, linear, inward, outward, and the like. The signal domain information is information indicating whether it is recorded on the time axis or on the time frequency axis. The compression method information is information indicating the presence or absence of compression and the compression method being used. The order is the order used in the HOA encoder 22.

On the other hand, the second band information is information relating to the conversion on the high frequency side, that is, the encoder 24. In the present embodiment, the second band information includes cutoff frequency, spatial domain information, signal domain information, compression scheme information, and channel information. It is configured. The cutoff frequency corresponds to the cutoff frequency on the low frequency side of the HPF 23 described with reference to FIG. The spatial domain information includes information indicating that the band is a signal encoded by the encoder 24. In addition, information on the collected microphone set 41, for example, information of the microphones m1 to m8 in the microphone set 41 is included. Information on the arrangement, for example, information such as spherical, annular, linear, inward, outward and the like may be included. The signal domain information is information indicating whether it is recorded on the time axis or on the time frequency axis. The compression method information is information indicating the presence or absence of compression and the compression method being used. The channel information includes the number of channels and channel coordinates. The number of channels corresponds to the number of microphones in the microphone set 41 (in this case, “8”). The channel coordinates are coordinates indicating the spatial arrangement of the microphones m1 to m8 in the microphone set 41.

The data section stores signals converted by the HOA encoder 22 and the encoder 24. In the present embodiment, for the first band (low band) and the second band (high band), frame chunks having frames are provided by the number of frames (L). The data recorded in the frame as described above is converted into a sound signal by the HOA decoder 31 or the decoder 32 with reference to the meta information described in the header portion.

In the recording format described above, information common to bands can be combined into one. The recording format described above is merely an example, and the present invention is not limited to this format, and can be configured in various forms.

On the other hand, the playback device 3 located on the playback side of the audio system 1 includes a HOA decoder 31, a decoder 32, a multiplier 33, and an adder 34. The HOA decoder 31 decodes the HOA signal encoded by the HOA encoder 22 and forms an 8-channel audio signal. The decoder 32 combines the signals encoded by the encoder 24 to form an 8-channel audio signal. The adder 34 adds, for each channel, the audio signal formed by the HOA decoder and the audio signal formed by the decoder 32 and appropriately multiplied by the multiplier 33, and outputs the result to the speaker set 42. In the present embodiment, since the number of microphones m1 to m8 of the microphone set 41 and the number of speakers s1 to s8 of the speaker set 42 are the same eight, signals of the corresponding channels are output to the speakers s1 to s8. This makes it possible to reproduce the sound field at the time of sound pickup.

Although the audio system 1 according to the first embodiment has been described above, according to the first embodiment, the number of the microphones m1 to m8 in the microphone set 41 for the HOA signal that is a signal corresponding to the spatial frequency is set. , Spacing, or the effect of spatial aliasing that occurs in accordance with the radius of the array, etc., makes it possible to suppress the deterioration of the audio signal that occurs at a certain frequency or higher, and to collect and reproduce the sound field with high accuracy. .

<3. Second Embodiment>
In the first embodiment, as described with reference to FIG. 4, the number of microphones m1 to m8 of the microphone set 41 and the number of speakers s1 to s8 of the speaker set 42 match. However, it is conceivable that the arrangement of the speaker set 42 cannot be configured in the same manner as the arrangement of the microphone set 41 at the time of sound collection due to the convenience of the reproduction side. The second and third embodiments described below are embodiments in which the number of microphones in the microphone set 41 and the number of speakers in the speaker set 42 do not match.

FIG. 6 is a diagram showing a configuration of the audio system 1 according to the second embodiment. The configurations of the microphone set 41 and the recording device 2 located on the recording side are the same as those described with reference to FIG. 4, and a description thereof will be omitted. The speaker set 42 located on the reproduction side is different from the configuration in FIG. 4 in that the number of speakers s1 to s4 is smaller than the number (eight) of microphones m1 to m8. Further, the reproducing apparatus 3 is different in that a matrix section 35 is provided between the multiplication section 33 and the addition section 34. Then, by specifying the number and positions of the speakers s1 to s4 in the speaker set 42, the HOA decoder 31 outputs audio signals for four channels according to the arrangement of the speakers s1 to s4.

On the other hand, as an audio signal output by decoding by the decoder 32, an audio signal for eight channels corresponding to the microphones m1 to m8 at the time of sound pickup is output. In the present embodiment, the audio signal output from the decoder 32 is mixed by the matrix unit 35 as a conversion unit in accordance with the arrangement of the speakers s1 to s4 of the speaker set 42. Specifically, audio signals collected by three microphones m1, m2, and m8 are mixed as audio signals to be output to the speaker s1. At this time, the audio signals collected by the microphones m2 and m8 are multiplied by a coefficient of 0.25. Similarly, the audio signal output to the speaker s2 is obtained by mixing the audio signals collected by the three microphones m2, m3, and m4 in the matrix unit 35. The audio signal output to the speaker s3 is obtained by mixing the audio signals collected by the three microphones m4, m5, and m6 in the matrix unit 35. The audio signal output to the speaker s4 is obtained by mixing the audio signals collected by the three microphones m6, m7, and m8 in the matrix unit 35.

As described above, the HOA decoder 31 for restoring an audio signal based on a signal corresponding to a spatial frequency has a sound collecting form, that is, regardless of the arrangement form of the microphones m1 to m8, depending on the arrangement form of the speakers s1 to s4. While the sound field can be reproduced, the decoder 32 for restoring the audio signal based on the signal corresponding to the spatial coordinates reproduces the sound field with an audio signal depending on the positions of the microphones m1 to m8. Become. In the present embodiment, in consideration of the difference between the two, a matrix unit 35 is provided as a conversion unit, and the number of channels of the audio signal output from the decoder 32 is converted according to the arrangement of the speakers s1 to s4 of the speaker set 42. This makes it possible to reproduce a sound field according to the arrangement of the speakers s1 to s4 on the reproduction side. Note that the configuration of the matrix unit 35 may be various methods other than those described in the present embodiment, and is not limited to one method.

<4. Third Embodiment>
FIG. 7 is a diagram illustrating a configuration of the audio system 1 according to the third embodiment. The third embodiment is different from the second embodiment described with reference to FIG. 6 in the arrangement of the microphones in the microphone set 41 and the arrangement of the speakers in the speaker set 42. More specifically, the microphone set 41 includes four microphones m1 to m4, and the speaker set 42 includes eight speakers s1 to s8.

In this case, the matrix unit 35 as a conversion unit converts the four-channel audio signals output from the decoder 32 corresponding to the microphones m1 to m4 into eight-channel audio signals corresponding to the speakers s1 to s8. I have. Specifically, the audio signals output to the speakers s1, s3, s5, and s7 directly output the audio signals collected by the microphones m1, m2, m3, and m4 in the corresponding arrangement. On the other hand, the speakers s2, s4, s6, and s8, which do not have correspondingly arranged microphones, are formed by mixing audio signals of a plurality of microphones. For example, the audio signal for the speaker s2 is formed by mixing the audio signals of the microphone m1 and the microphone m2. Although the mixing is performed by multiplying and adding fixed coefficients, the coefficients may be dynamically changed. For example, by distributing a large coefficient in a direction in which the magnitude (level) of the audio signal is large, it is possible to emphasize the sense of direction of the sound field during reproduction.

As described above, even when the number of microphones m1 to m4 is smaller than the number of speakers s1 to s8, the number of channels is converted into the number of channels corresponding to the arrangement of the speakers s1 to s4 on the reproduction side by the matrix unit 35 as a conversion unit. The sound field can be reproduced properly. Note that the conversion unit can not only convert the number of channels, but also convert the audio signal so that when the sound pickup direction of the microphone and the sound emission direction of the speaker are different, they are in an appropriate form. . At that time, the number of microphones and the number of speakers may be the same. The configuration of the matrix unit 35 may be various methods other than the one described in the present embodiment, and is not limited to one method.

<5. Fourth embodiment>
FIG. 8 is a diagram illustrating a configuration of an audio system 1 according to the fourth embodiment. In the fourth embodiment, for example, the audio system 1 according to the first embodiment described with reference to FIG. 4, the downsampling unit 26 as a sampling frequency conversion unit on the recording side, and the delay caused by the downsampling unit 26 are described. The difference is that a delay unit 25 for compensating is provided, and an upsampling unit 37 as a sampling frequency conversion unit is provided on the reproduction side.

Since the processing in the HOA encoder 22 does not include a signal in a high frequency region, it is conceivable that even if the frequency of the input audio signal is reduced, the effect on the sound quality is not so large. In the fourth embodiment, the amount of calculation in the HOA encoder 22 is reduced by performing a down-sampling process on the time axis in the down-sampling unit 26 for the audio signal input to the HOA encoder 22. Further, by performing the downsampling, the data amount of the signal output from the HOA encoder 22 can be reduced, and the storage capacity and the communication amount can be reduced.

For example, when the sampling frequency of an audio signal input from the microphone set 41 is a general sampling frequency of Fs = 44.1 kHz (or 48 kHz), the down-sampling unit 26 uses a sampling frequency half that of the original signal. Fs = 22.05 kHz (or 24 kHz). On the reproduction side, an up-sampling section 37 arranged downstream of the HOA decoder 31 performs up-sampling at the same sampling frequency as that of the decoder 32 side. In this case, an FIR filter is mainly used in many cases. In the present embodiment, the delay unit 25 is provided on the path on the encoder 24 side to compensate for the delay generated in the downsampling unit 26. Instead of compensating for the delay on the recording side, the delay may be compensated on the reproducing side (or the recording side and the reproducing side). When compensating for the delay on the reproduction side, for example, it is conceivable to arrange a delay unit at a stage subsequent to the decoder 32.

As described above, in the third embodiment, the downsampling process on the time axis is performed on the audio signal input to the HOA encoder 22 to reduce the calculation amount in the HOA encoder 22 and to reduce the amount of calculation from the HOA encoder 22. It is possible to reduce the data amount of the output signal. In addition, as the amount of data output from the HOA encoder 22 can be reduced, a larger amount of information (for example, the number of bits) can be assigned to the signal output from the encoder 24. The conversion of the sampling frequency is performed not on the HOA encoder 22 side and the HOA decoder 31 side but on the encoder 24 and the decoder 32 side, or on both the HOA encoder 22 side and the HOA decoder 31 side and the encoder 24 and the decoder 32 side. It may be performed by.

<6. Fifth Embodiment>
FIG. 9 is a diagram illustrating a configuration of an audio system 1 according to the fifth embodiment. In the fifth embodiment, for example, the audio system 1 according to the first embodiment described with reference to FIG. 4 uses one HOA encoder 22, whereas in the fifth embodiment, the audio system 1 is in charge of a low band. This is different from the first embodiment in that a HOA encoder 22a for performing a mid-range operation is provided.

An LPF 21a for removing a high-frequency component of an input audio signal is arranged at a stage preceding the HOA encoder 22a, and a BPF 21b (Band) for extracting a mid-range component of the input audio signal is arranged at a stage prior to the HOA encoder 22b. Pass Filter). With the configuration on the recording side, a HOA decoder 31a for decoding an audio signal encoded by the HOA encoder 22a and a HOA decoder 31a for decoding an audio signal encoded by the HOA encoder 22b are arranged on the reproduction side. . The adder 34 converts the audio signal decoded by the decoder 32 and multiplied by the coefficient by the multiplier 33 and the audio signal decoded by the HOA decoder 31a and the HOA decoder 31b, and outputs the converted signal to the speaker set 42. .

FIG. 10 is a diagram illustrating frequency characteristics of the audio system 1 according to the fifth embodiment. In the frequency characteristic shown in FIG. 10, the low-pass characteristic shown by the solid line indicates the characteristic of the LPF 21a. The mid-pass characteristic indicated by the dashed line indicates the characteristic of the BPF 21b. The high-pass characteristics indicated by broken lines indicate the characteristics of the HPF 23. By adding the low-pass characteristics of the LPF 21a, the mid-pass characteristics of the BPF 21b, and the high-pass characteristics of the HPF 23, a flat frequency characteristic is formed from low to high frequencies. These characteristics are just one example, and various characteristics are possible depending on the design method.

よう As in the fifth embodiment, by providing the

HOA encoders

22a and 22b separately in a plurality of frequency bands, it is possible to vary the order used in the HOA processing of the

HOA encoders

22a and 22b. In particular, in the case of a signal in an extremely low frequency range, the wavelength is sufficiently long, so that the direction of arrival of the sound perceived by humans is insensitive. It is conceivable to reduce the calculation amount by performing the processing.

<7. Sixth embodiment>
FIG. 11 is a diagram illustrating a configuration of an audio system 1 according to the sixth embodiment. In the fourth embodiment described with reference to FIG. 8, the mode in which the delay unit 25 is provided on the recording side to compensate for the delay generated in the downsampling unit 26 has been described. Since the processing is performed separately, a time lag may occur between the bands.

The sixth embodiment shows a configuration for eliminating such a time lag between bands on the reproduction side. For example, a time lag caused by the processing of the encoder 24 and the HOA encoder 22 on the recording side or a time lag caused by the processing of the decoder 32 and the HOA decoder 31 on the reproduction side can be eliminated by providing the delay unit 36 on the reproduction side. Is possible. Also in the sixth embodiment, instead of compensating for the delay on the reproducing side, the delay may be compensated on the recording side (or the recording side and the reproducing side).

<8. Modification>
(Modification of HOA method)
As described above, the first to sixth embodiments have described the embodiments using the HOA method using the HOA encoder 22, the HOA decoder 31, and the like. The signals used in the various embodiments are not limited to the signals encoded by the HOA method, but the signals associated with the spatial frequencies, in other words, the positions to be reproduced when decoding (the positions at which the speakers are installed) ), Various methods can be adopted as long as the signal can reproduce the audio signal at the position. Here, a signal associated with a spatial frequency is referred to as an SF signal.

{On the other hand, the method used by the encoder 24, the decoder 32, and the like uses a signal corresponding to the spatial coordinates. In other words, it can be said that the signal is a signal that can reproduce the audio signal at the picked-up position (spatial coordinates). Here, a signal associated with the spatial coordinates is referred to as an SA signal. Hereinafter, the format of the recording signal when the SA signal and the SF signal are used will be described.

(Modification of recording format)
FIG. 12 is a diagram illustrating a recording format of a recording signal used in the audio system 1 according to a modification. In FIG. 5, the format of the recording signal used in the first embodiment has been described. In this modified example, a description will be given of a recording format when the HOA signal is generalized as a signal (SF signal) associated with a spatial frequency, and a signal in charge of a high band is a signal (SA signal) associated with spatial coordinates (SA signal). .

The recording signal has a header section and a data section. The header section is a section in which various meta information necessary for reproducing the recorded audio signal is recorded. In the present embodiment, the meta information to be recorded in the header section includes a sampling rate, a frame length, the number of frames (L), the number of band divisions (N), and band information (first to N-th band information) for each band. It is configured. For example, when the frequency band is divided into three frequency bands as in the fifth embodiment, first to third band information is provided.

The sampling rate is the sampling rate used at the time of recording, and may be fixed or variable. The frame length is information defining the length of a frame recorded in the data section. Either fixed or variable frame length may be adopted. The number of frames (L) is a number that defines the number of frames forming a chunk that is a unit of one data in the data portion. The number of band divisions is a number indicating the number of bands to be divided in the audio system 1. For example, when the band is divided into three frequency bands as in the fifth embodiment, the number of band divisions is "3". The first to third band information is provided.

Each band information (first to N-th band information) is provided with a first cutoff frequency indicating the lower limit of the assigned frequency band and a second cutoff frequency indicating the upper limit. The time delay information is information indicating delay or advance with respect to another band, and can be used, for example, for setting the delay time in the delay unit 36 described in the sixth embodiment. The spatial domain information is information indicating whether the band is an SF signal or an SA signal, and the reproducing device 3 can determine a decoding method for the band by referring to the spatial domain information. is there. The spatial domain information may include information on the microphone arrangement of the collected microphone set 41 and the like. The signal domain information is information indicating whether it is recorded on the time axis or on the time frequency axis. The compression method information is information indicating the presence or absence of compression and the compression method being used.

As the order / channel information, the order is stored when the SF signal is used, and the channel information is stored when the SA signal is used. The order stored for the frequency band using the SF signal is the order used for the process of forming a signal corresponding to the spatial frequency.

On the other hand, the channel information stored for the frequency band using the SA signal is information stored when the SA signal is used, and as described with reference to FIG. It is configured to include coordinates. The number of channels corresponds to the number of microphones in the microphone set 41 (for example, “4” in the case of the third embodiment shown in FIG. 7). In this case, the channel coordinates are coordinates indicating the spatial arrangement of the microphones m1 to m4 in the microphone set 41. The matrix unit 35 (conversion unit) described in the second embodiment described with reference to FIG. 6 and the third embodiment described with reference to FIG. 7 uses the channel information and the arrangement of speakers in the speaker set 42 based on this channel information. Various conversions such as converting the number of channels of an audio signal can be performed.

The data section stores signals converted for each band. In the present modification, frame chunks having frames are provided for the first to N-th bands by the number of frames (L). The data recorded in such a frame is converted into an audio signal with reference to the meta information described in the header.

Note that, for the above-described recording format, information common to the bands can be combined into one. The recording format described above is merely an example, and the present invention is not limited to this format, and can be configured in various forms.

(Modification of playback environment)
The playback device 3 in each of the above-described embodiments outputs an audio signal to the speaker set 42 including a plurality of speakers. In addition to such a form, the reproducing device 3 may reproduce an audio signal in a virtual environment using headphones, for example. That is, in the above-described embodiment, if the head-related transfer functions from each speaker in the speaker set 42 to both ears of the listener are known, each head-related transfer function is convolved with the audio signal driving each speaker. It can be seen how the sound of each speaker is heard by both ears of the listener. By reproducing the sum of the left and right ears through headphones or the like, a sound field similar to that of a listener using the speaker set 42 can be reproduced.

音 Sound field formation using such a virtual environment can be realized with an electro-acoustic transducer that is driven not only by headphones but also by two or more channels. At that time, if necessary, various corrections such as crosstalk cancellation can be performed on the audio signal reproduced by the electro-acoustic transducer.

The present disclosure can be realized in various forms such as an apparatus, a method, and a program. In addition, the items described in each of the embodiments and the modified examples can be appropriately combined.

The present disclosure can employ the following configurations.
(1)
A first decoder for decoding a first signal associated with a spatial frequency into audio signals of a plurality of channels;
A second decoder that decodes a second signal including a band different from the first signal and corresponding to spatial coordinates into audio signals of a plurality of channels;
An audio reproducing apparatus comprising: an adder that adds a plurality of channels of audio signals decoded by the first decoder and a plurality of channels of audio signals decoded by the second decoder.
(2)
The audio reproduction device according to (1), wherein the audio signal decoded by the second decoder has a higher frequency band than the audio signal decoded by the first decoder.
(3)
The audio reproduction device according to (1) or (2), wherein the first decoder performs decoding based on an arrangement of speakers to be output.
(4)
The audio playback device according to any one of (1) to (3), wherein the first decoder uses a HOA method.
(5)
The audio playback device according to any one of (1) to (4), further including a conversion unit configured to convert audio signals of a plurality of channels output from the second decoder based on an arrangement of speakers to be output. .
(6)
The audio reproduction device according to (5), wherein the conversion unit converts the number of channels of the audio signal output from the second decoder.
(7)
The first signal and the second signal have different sampling frequencies,
The audio playback device according to any one of (1) to (6), further including a sampling frequency conversion unit configured to convert at least one of the first signal and the second signal.
(8)
A plurality of the second decoders are provided for each band,
The audio reproduction device according to any one of (1) to (7), wherein the plurality of second decoders use different orders for decoding.
(9)
The audio playback device according to any one of (1) to (8), further including a delay unit that adjusts a time shift generated between the first decoder and the second decoder.
(10)
Decoding the first signal corresponding to the spatial frequency into audio signals of a plurality of channels;
Decoding a second signal including a band different from the first signal and corresponding to spatial coordinates into an audio signal of a plurality of channels;
An audio reproduction method for adding a plurality of channels of audio signals decoded based on the first signal and a plurality of channels of audio signals decoded based on the second signal.
(11)
A first decoding process of decoding a first signal associated with a spatial frequency into audio signals of a plurality of channels;
A second decoding process of decoding a second signal including a band different from the first signal and corresponding to spatial coordinates into audio signals of a plurality of channels;
An audio reproduction program for causing an information processing device to execute an addition process of adding audio signals of a plurality of channels decoded by the first decoder and audio signals of a plurality of channels decoded by the second decoder.

1: audio system 2: recording device 3: playback device 21 (21a, 21b): LPF
22 (22a, 22b): HOA encoder 23: HPF
24: encoder 25: delay unit 26: downsampling unit 31 (31a, 31b): HOA decoder 32: decoder 33: multiplying unit 34: adding unit 35: matrix unit 36: delay unit 37: upsampling unit 41: microphone set 42 : Speaker set m1 to m8: Microphones s1 to s8: Speaker

Claims

A first decoder for decoding a first signal associated with a spatial frequency into audio signals of a plurality of channels;
A second decoder that decodes a second signal including a band different from the first signal and corresponding to spatial coordinates into audio signals of a plurality of channels;
An audio reproducing apparatus comprising: an adder that adds a plurality of channels of audio signals decoded by the first decoder and a plurality of channels of audio signals decoded by the second decoder.
The audio reproduction device according to claim 1, wherein the audio signal decoded by the second decoder has a higher frequency range than the audio signal decoded by the first decoder.
The audio playback device according to claim 1, wherein the first decoder performs decoding based on an arrangement of speakers to be output.
The audio reproduction device according to claim 1, wherein the first decoder uses a HOA method.
The audio reproduction device according to claim 1, further comprising: a conversion unit configured to convert audio signals of a plurality of channels output from the second decoder based on an arrangement of speakers to be output.
The audio playback device according to claim 5, wherein the conversion unit converts the number of channels of the audio signal output from the second decoder.
The first signal and the second signal have different sampling frequencies,
The audio playback device according to claim 1, further comprising a sampling frequency conversion unit configured to convert at least one of a sampling frequency of the first signal and the second signal.
A plurality of the second decoders are provided for each band,
The audio playback device according to claim 1, wherein the plurality of second decoders use different orders for decoding.
The audio playback device according to claim 1, further comprising: a delay unit that adjusts a time lag between the first decoder and the second decoder.
Decoding the first signal corresponding to the spatial frequency into audio signals of a plurality of channels;
Decoding a second signal including a band different from the first signal and associated with spatial coordinates into an audio signal of a plurality of channels;
An audio reproducing method for adding a plurality of channels of audio signals decoded based on the first signal and a plurality of channels of audio signals decoded based on the second signal.
A first decoding process of decoding a first signal associated with a spatial frequency into audio signals of a plurality of channels;
A second decoding process of decoding a second signal including a band different from the first signal and associated with spatial coordinates into audio signals of a plurality of channels;
An audio reproduction program for causing an information processing device to execute an addition process of adding audio signals of a plurality of channels decoded by the first decoder and audio signals of a plurality of channels decoded by the second decoder.