RU2819779C1 - Low frequency amplification for loudspeakers - Google Patents

Abstract

FIELD: physics.

SUBSTANCE: invention relates to acoustics. Device comprises means for receiving an audio signal, means for converting a signal, means for amplifying low frequencies, comprising an upsampler, a harmonic generator, a device for processing dynamic characteristics, a converter, a filter, a delay element and a mixer, as well as an inverse transformation unit. Harmonic generator comprises an amplifier, a compressor, an adder, as well as a normalization unit, a multiplier, and a loudness correction stage.

EFFECT: elimination of spectral overlap effect.

18 cl, 12 dwg, 2 tbl

Description

CROSS-REFERENCES TO RELATED APPLICATIONS

[0001] This application claims priority to International Application No. PCT/CN2020/080460, filed March 20, 2020, and U.S. Provisional Patent Application No. 63/010,390, filed April 15, 2020, both of which are incorporated by reference herein in their entirety.

TECHNICAL FIELD

[0002] The present invention relates to audio processing and, in particular, to bass enhancement.

BACKGROUND OF THE INVENTION

[0003] Unless otherwise indicated herein, the approaches described in this section are not prior art to the claims herein and should not be considered prior art for inclusion in this section.

[0004] Low frequency effect is a desirable user experience and user evaluation indicator of mobile devices such as mobile phones, multimedia players, tablet computers, laptops, headphones, earbuds, etc. Due to the physical limitations of transducers in mobile devices (eg, diaphragm size, magnet weight, etc.), it is difficult for a mobile device loudspeaker to fully reproduce the acoustic characteristics of the original low-frequency sound. As a consequence, mobile devices often implement technical audio processing solutions (e.g., using software processes, etc.) to improve low-frequency audio. These low-frequency enhancement processes can broadly be called "virtual low-frequency" technical solutions.

SUMMARY OF THE INVENTION

[0005] One problem with existing low frequency amplification systems is that they can be highly computationally complex. Given the above, there may be a need to implement low frequency amplification with reduced computational complexity.

[0006] As discussed in more detail herein, embodiments discuss low frequency enhancement solutions based on the "no fundamental" principle. This principle establishes that psychoacoustically, if a person hears the harmonics of a low-frequency signal rather than the low-frequency signal itself (the fundamental), the listener's brain is able to extrapolate and thus perceive the missing low-frequency signal. Thus, for loudspeakers that are physically unable to reproduce low-frequency signals (low frequencies), a method of psychoacoustic enhancement is to generate harmonics for the low-frequency range to enhance the low-frequency effect.

[0007] The low frequency enhancement solution disclosed herein is less computationally complex than traditional virtual low frequency solutions but achieves a similar result. Thus, in embodiments, computational complexity is reduced. Additionally, reduced computational complexity makes it possible to reduce latency. The solution also includes volume control circuits to adjust the power of the generated harmonics, resulting in a more realistic perception of the resulting volume and greater low-frequency effect appeal.

[0008] The techniques disclosed herein can be used to amplify the output from mid-size speakers and smaller transducers, such as mobile phone speakers, wireless speakers, etc.

[0009] In an embodiment, a computer-implemented audio processing method includes receiving a first signal in a conversion domain. The first transform domain signal is a hybrid complex transform domain signal having multiple bands. At least one of the plurality of bands has a plurality of subbands, and the first signal in the conversion domain has a first plurality of harmonics.

[0010] The method further includes generating a second transform domain signal based on the first transform domain signal. The second transform domain signal is generated by generating harmonics for the first transform domain signal according to a nonlinear process. The second signal in the conversion domain has a second harmonic set that is different from the first harmonic set. The second transform domain signal is further generated by performing volume expansion with respect to the second set of harmonics. The second signal in the transformation domain is a complex-valued signal that has an imaginary part.

[0011] The method further includes generating a third signal in the transform domain by filtering the second signal in the transform domain. The third signal in the transform domain has a plurality of bands, and at least one of the plurality of bands has a plurality of subbands. The method further includes generating a fourth transform domain signal by mixing the third transform domain signal with a delayed version of the first transform domain signal, wherein a given subband of the third transform domain signal is mixed with a corresponding subband of the delayed version of the first transform domain signal.

[0012] According to another embodiment, the device includes a loudspeaker and a processor. The processor is configured to control the device to implement one or more of the methods described herein. The device may further comprise parts similar to those in one or more of the methods described herein.

[0013] In another embodiment, a computer program is stored on a non-transitory computer-readable medium that, when executed by a processor, controls an apparatus to perform processing including one or more methods described herein.

[0014] The following detailed description and accompanying drawings provide further understanding of the nature and advantages of the various implementations.

BRIEF DESCRIPTION OF GRAPHIC MATERIALS

[0015] In FIG. 1 is a block diagram of an audio processing system 100.

[0016] In FIG. 2 is a block diagram of a bass enhancement system 200.

[0017] In FIG. 3 is a block diagram of a harmonic generator 300.

[0018] In FIG. 4 is a block diagram of a harmonic generator 400.

[0019] In FIG. 5 is a block diagram of a harmonic generator 500.

[0020] In FIG. 6 is a graph 600 showing equal loudness curves.

[0021] In FIG. 7 is a graph 700 showing various compression gain factors c .

[0022] In FIG. 8 is a block diagram of a harmonic generator 800.

[0023] In FIG. 9A, 9B, 9C, 9D, 9E and 9F show a set of plots 900a-900f.

[0024] In FIG. 10 is a block diagram of a bass enhancement system 1000.

[0025] In FIG. 11 depicts a mobile device architecture 1100 for implementing the features and processes described herein, according to an embodiment.

[0026] In FIG. 12 is a flowchart of an audio processing method 1200.

DETAILED DESCRIPTION

[0027] This document describes technical solutions related to bass enhancement. In the following description, for purposes of explanation, numerous examples and specific details are set forth in order to provide a thorough understanding of the present invention. However, it will be apparent to one skilled in the art that the present invention, as defined by the claims, may include some or all of the features of these examples, alone or in combination with other features described below, and may further include modifications and equivalents of the features and concepts, described in this document.

[0028] The following description describes in detail various methods, processes and procedures. Although certain steps may be described in some order, this order is primarily for convenience and clarity. A given step may be repeated more than once, may occur before or after other steps (even if those steps are otherwise described in a different order), and may occur in parallel with other steps. The second stage follows the first stage only when the first stage must be completed before the start of the second stage. This situation will be specifically stated if it is not clear from the context.

[0029] The terms “and”, “or” and “and/or” are used herein. Such terms should be read as having an inclusive meaning. For example, "A and B" may mean at least the following: "both A and B", "at least both A and B". As another example, "A or B" may mean at least the following: "at least A", "at least B", "both A and B", "at least both A and B". As another example, "A and/or B" may mean at least the following: "A and B", "A or B". When an exclusive “or” is intended, it will be indicated specifically (for example, “either A or B,” “at most one of A and B”).

[0030] Various processing functions associated with structures such as blocks, elements, components, circuits, etc. are described herein. In general, these structures may be implemented by a processor under control of one or more computer programs.

[0031] In FIG. 1 is a block diagram of an audio processing system 100. The audio processing system 100 generally receives an input audio signal 102, processes the input audio signal 102 according to the bass enhancement processes described herein, and generates an output audio signal 104. The audio processing system 100 includes a signal conditioning system 110, a bass enhancement system 120 , an additional processing system 130 (optional) and a signal deconversion system 140. The audio processing system 100 may include other components that (for the sake of brevity) are not discussed in detail. The components of the audio processing system 100 may be implemented by one or more computer programs executed by a processor.

[0032] Signal conversion system 101 receives input audio signal 102, performs a signal conversion process, and generates a converted audio signal 112. Input audio signal 102 may be a digital time domain signal containing multiple discrete values that correspond to audio (e.g., audio in pulse code modulation (PCM) waveform format). The input audio signal 102 may have a sampling rate of 32 kHz, 44.1 kHz, 48 kHz, 192 kHz, etc. The input audio signal 102 may come from a variety of formats, including the Advanced Television Systems Committee (ATSC) standard for digital audio compression (AC-3, E-AC-3). As a specific example, the input audio signal 102 may originate from a Dolby Digital Plus™ signal at a sampling rate of 48 kHz.

[0033] The signal conversion system 110 may perform various signal conversion processes. In general, the signal conversion process converts the input audio signal 102 from a first signal domain to a second signal domain. For example, the first domain may be a time domain, and the second signal domain may be a frequency domain, a quadrature mirror frequency (QMF) domain, a complex quadrature mirror frequency (CQMF) domain, a hybrid complex quadrature mirror frequency (HCQMF) domain, etc. . The transformation from the first signal region to the second signal region may also be called "analysis", for example, transformation analysis, signal analysis, filter bank analysis, QMF analysis, CQMF analysis, HCQMF analysis, etc.

[0034] In general, QMF information is generated using a filter whose frequency response is a mirror image of the π/2 frequency response of another filter; together these filters are known as a QMF pair. QMF theory also includes filter banks with more than two channels (e.g. 64 channels); these can be called M-channel QMF blocks. QMF theory additionally contains information about M-channel pseudo-QMF blocks, which belong to a class called modulated filter blocks. In general, the information in the "CQMF" domain is the result of applying a complex-modulated discrete Fourier transform (DFT) filter bank to the time domain signal. CQMF is a "complex" signal because it contains complex-valued signals, such as signals containing an imaginary part in addition to a real part. In general, the information in the “HCQMF” domain corresponds to the information in the CQMF domain, in which the CQMF filter bank has been extended to a hybrid structure to obtain an effective non-uniform frequency resolution that better matches the frequency resolution of the human auditory system. In general, the term “hybrid” refers to a structure in which at least one frequency band is divided into subbands.

[0035] In a specific HCQMF implementation, HCQMF information is generated into 77 frequency bands, with the lower CQMF bands further divided into subbands to obtain higher frequency resolution for lower frequencies. According to a further specific implementation, the signal conditioning system 110 converts each channel of the input audio signal 102 into 64 CQMF bands and further divides the lower 3 bands into subbands as follows: the first band is divided into 8 subbands, and each of the second and third bands is divided into 4 subbands. (This hybrid subbanding of the lowest bands is intended to increase the low frequency resolution of these bands.) Signal conditioning system 110 may include Nyquist filters to subband the bands. Then the 77 HCQMF bands correspond to the 61 highest CQMF bands with the addition of 16 sub-bands (8+4+4) of the lowest 3 CQMF bands. Subbands and bands can be numbered from 0 to 76, with number 0 being the lowest frequency subband. Then the other subbands are numbered from 1 to 15, and the remaining bands are numbered from 16 to 76. Then these 77 HCQMF bands can be called "hybrid bands", or "channels", along with their number, for example, hybrid band 0, hybrid band 1, hybrid band 76, channel 0, channel 1, channel 76, etc. Hybrid bands 0–15 may also be called "subbands" along with their number, e.g. subband 0, subband 1, subband 15, etc. Hybrid bands 16–76 may also be called "strips" along with their number, such as band 16, band 17, band 76, etc. Channels 1 and 3 may have bandwidths on the negative frequency axis, but other channels typically do not.

[0036] (It should be noted that the terms "QMF", "CQMF" and "HCQMF" are used somewhat simplistically throughout this document. In particular, the terms "QMF"/"CQMF" may be simplistically used to refer to a DFT filter bank that can contain more than two bands. The term "HCQMF" can be simplistically used to refer to a non-uniform DFT filter bank that can contain more than two bands.)

[0037] As a specific example, the signal conversion system 110 performs HCQMF conversion on the input audio signal 102 to generate a converted audio signal 112 having 77 frequency bands. In this case, the signal region of the converted audio signal 112 may be called an HCQMF region or a hybrid region, and the HCQMF transformation may be called an HCQMF analysis.

[0038] The bandwidth and sampling frequency of the bands will depend on the sampling frequency of the input audio signal 102. For example, when the input audio signal 102 has a sampling frequency of 48 kHz (corresponding to a maximum bandwidth of 24 kHz), the 77-band hybrid structure discussed above results in a frequency 750 Hz sampling for all bands. The 61 highest frequency bands have a bandwidth of 375 Hz; The 8 lowest frequency subbands have a bandwidth of 93.75 Hz; and the subbands with the next lowest frequencies have a bandwidth of 187.5 Hz.

[0039] Bass enhancement system 120 receives the converted audio signal 112, performs low frequency amplification, and generates an enhanced audio signal 122. Overall, bass enhancement system 120 generates harmonics for the converted audio signal 112 for the purpose of psychoacoustically perceiving the missing fundamental harmonic to the listener. Additional details of the bass enhancement system 120 are provided below (eg, with reference to FIG. 2, etc.).

[0040] The post-processing system 130 is optional. If present, the after-processing system 130 receives the amplified audio signal 122, performs further processing on the signal, and generates the processed audio signal 132. Alternatively, the after-processing system 130 may act on the converted audio signal 112 before acting on the bass boost system 120, in which case the boost system 120 The low pass receives as its input the signal output from the post-processing system 130 (instead of receiving the output signal directly from the signal conditioning system 110). As another possibility, the post-processing system 130 may be multiple post-processing systems operating both before and after the bass enhancement system 120. The specific layout of the post-processing system 130 in the audio processing system 100 may vary according to the particular types of post-processing that the post-processing system 130 performs.

[0041] In general, the additional processing system 130 performs additional processing of the input audio signal 102 in the conversion domain. This allows the bass enhancement system 120 to operate in combination with existing audio processing solutions that have been implemented in the conversion field. Examples of additional processing include dialogue enhancement, intelligent frequency response equalization, volume control, spectrum limiting, etc. Dialogue enhancement refers to the enhancement of speech signals (e.g., over sound effects) to improve speech intelligibility. Intelligent frequency response equalization refers to performing dynamic adjustments to the audio tone, for example to ensure consistent spectral balance (also known as "tonal" or "timbral"). Volume control refers to increasing the volume of a quiet sound and decreasing the volume of a loud sound, for example, to reduce the need for the listener to make manual volume adjustments. Spectral clipping refers to limiting selected frequencies or frequency bands, for example to limit the lowest frequencies that are difficult to output from small speakers.

[0042] The deconversion system 140 receives the amplified audio signal 122 (or optionally the processed audio signal 132), performs the deconversion, and generates an output audio signal 104. The deconversion typically converts the signal from the second signal region back to the first signal region. In general, deconversion is a signal conversion process that is the reverse of the process performed by signal conversion system 110. For example, when the signal conditioning system 110 performs the HCQMF transform, the signal deconversion system 140 performs the inverse HCQMF transform. Conversion from the second signal domain back to the first signal domain may also be called "synthesis", such as transform synthesis, signal synthesis, filter bank synthesis, etc.; and the inverse transformation of HCQMF can be called HCQMF synthesis.

[0043] Thus, the output audio signal 104 corresponds to the input audio signal 102 with the addition of bass boost and/or additional signal enhancements. The output audio signal 104 can then be output by a loudspeaker and perceived by the listener in the form of sound.

[0044] As discussed above and in more detail below, the bass enhancement system 120 is suitable for small to medium sized speakers. The processes implemented by the bass enhancement system 120 may be simpler than many existing bass enhancement techniques; Compared to these existing methods, the bass enhancement system 120 has lower computational complexity and allows for short latency while still maintaining audio quality. The bass boost system 120 is well suited for medium-sized speakers, such as those found in televisions or wireless speakers, and is also effective for boosting the bass frequencies of small-sized transducers, such as those found in cell phones, laptops, and tablets. The bass boost system 120, in one mode of operation, not only adds harmonics to the mixed signal, but also adds (dynamically modified) original bass frequencies, i.e., it can operate to produce an internal boost in the bass region.

[0045] In FIG. 2 is a block diagram of a bass enhancement system 200. The bass boost system 200 may be used as the bass boost system 120 (see FIG. 1). For brevity, the description of FIG. 2 focuses on one signal processing path for the purpose of describing the general operating principle of the bass enhancement system 200; additional signal processing paths may also be implemented in variations of the low frequency amplification systems described herein (see, for example, FIG. 10). Additional signal processing pathways will also be briefly described here.

[0046] Bass enhancement system 200 receives converted audio signal 112 (see FIG. 1). As discussed above, the converted audio signal 112 is a hybrid complex signal in the transform domain (eg, a signal in the HCQMF domain) with multiple bands (eg, 77 hybrid bands, with the 3 lowest frequency bands divided into subbands). As a complex signal, the converted audio signal 112 has complex values, such as both real values and imaginary values. Each subrange may be processed in its own processing path, so the following description focuses on the processing of a single subrange (eg, one of subranges 0, 2, 4, 6, etc.). Bass enhancement system 200 includes an upsampler 202 (optional), a harmonic generator 204, a dynamics processor 206 (optional), a converter 208 (optional), a filter 212, a delay element 214, and a mixer 216.

[0047] The upsampler 202 receives the converted audio signal 112, performs upsampling, and generates an upsampled signal 220. As an example, when the input audio signal 102 (see FIG. 1) has a sampling rate of 48 kHz, and the converted audio signal 112 processed to produce 64 bands, each band having a sampling rate of 750 Hz. Upsampler 202 may upsample a selected subband of the converted audio signal 112 to 2, 3, 4, 5, 6, etc. once. A suitable upsampling value is 4, for example, such that the upsampled signal 220 has a sampling rate of 3 kHz when the selected subband of the converted audio signal 112 has a sampling rate of 750 Hz. The upsampled signal 220 is a complex signal in the transform domain. The upsampled signal 220 has a bandwidth that corresponds to the bandwidth of the selected subband of the converted audio signal 112. As an example, when the selected subband 0, having a bandwidth of 93.75 Hz, is input to the upsampler, the upsampled signal 220 also has a bandwidth of 93.75 Hz.

[0048] The upsampler 202 may be implemented by performing CQMF synthesis. As an example, to upsample subband 0 from 750 Hz to 3000 Hz (4x upsampling), the upsampler can implement 4-channel CQMF synthesis, with one input signal being subband 0 and the other 3 input signals being null (nulled) ). The synthesis is performed to store signal 220 as a complex-valued time domain signal.

[0049] Upsampler 202 is optional. In general, upsampler 202 provides additional headroom in harmonic generation (see harmonic generator 204) to allow bandwidth expansion without aliasing (also called aliasing). When processing one or more of the lowest frequency subbands, the upsampler 202 may be omitted. For example, when processing only the lowest band (eg, subband 0), the upsampler 202 may be omitted, since harmonics up to (at least) 6th order can be generated without aliasing. When processing the two lowest subbands (eg, subbands 0 and 2), upsampler 2 may be omitted if only 2nd and 3rd order harmonics are generated. When processing the three lowest bands (eg sub-bands 0, 2 and 4) without aliasing, only 2nd order harmonics can be generated. This is discussed in more detail with reference to harmonic generator 204.

[0050] The harmonic generator 204 receives the upsampled signal 220 (or the selected subband signal of the converted audio signal 112 when the upsampler 202 is omitted) and generates harmonics thereof to produce a signal 222. As mentioned with reference to the upsampler 202, when generating harmonics for signal 222, a harmonic generator 204 expands the bandwidth of its input signal. For example, when subband 0 covers frequencies from 0 to 93.75 Hz, a sampling rate of 750 Hz may be sufficient to prevent aliasing of the generated harmonics. Similarly, when sub-band 2 covers frequencies from 93.75 to 187.5 Hz, a sampling rate of 750 Hz may be sufficient to prevent aliasing of the generated harmonics. However, when sub-band 4 covers frequencies from 187.5 to 281.25 Hz, the harmonics approach the Nyquist frequency of the original signal (with a sampling rate of 750 Hz), so for sub-bands 4, 6, etc. Upsampling is recommended. Signal 222 is a complex transform domain signal. Signal 222 has a bandwidth that is greater than the input bandwidth to harmonic generator 204 due to the addition of harmonic frequencies. For example, when upsampled signal 220 has a bandwidth of 93.75 Hz, signal 222 may have a bandwidth that is greater than 300 Hz.

[0051] The harmonic generator 204 uses a nonlinear process to generate harmonics. In general, the nonlinear process applies different gains to different signal components. Examples of nonlinear processes include multiplication, closed-loop delay loop, rectification, etc., as further described in detail below with reference to FIG. 3, 4, 5 and 8.

[0052] Harmonic generator 204 may also perform volume spreading when generating signal 222. Since the sound pressure level for a fixed volume range (in the background) increases with frequency in the low/mid frequency range (e.g., less than 800 Hz), harmonic generator 204 performs spreading dynamically when generating signal 222. Examples of loudness expansion processes include dynamic compression and loudness equalization. Additional details of volume expansion are provided below with reference to FIG. 6.

[0053] The dynamics processing device 206 receives the signal 222, performs the dynamics processing, and generates a signal 224. The signal 224 is a complex transform domain signal. In general, the dynamics processing device 206 implements dynamics processing by performing compression on the signal 222 to control the ratio of the short-term and tonal components of the signal 224. The dynamics processing device 206 may implement a limiter response time that is relatively longer (for example, 4 times longer). –12 times, for example 8 times longer) than the shutdown time. For example, the limiter's actuation time may be from 140 to 180 ms (eg, 160 ms), and the off time may be from 15 to 25 ms (eg, 20 ms). The dynamics processing apparatus 206 may implement unrelated soft peak detection using a feed-forward topology. The dynamics processor 206 may implement compression that is similar to the compression performed by a harmonic generator (which is described in more detail with reference to FIGS. 3, 4, and 5).

[0054] The dynamics processing device 206 is optional. When dynamics processing device 206 is omitted, transducer 208 receives signal 222 instead of signal 224.

[0055] Converter 208 receives signal 224 (or signal 222 when dynamics processor 206 is omitted), discards the imaginary portion of signal 224, and generates signal 228. In general, discarding the imaginary portion reduces the computational complexity of subsequent analysis filter blocks (e.g., filter 212 ) due to the processing of real-valued signals instead of complex-valued signals. As discussed above, signal 224 is a complex transform domain signal that has complex values, such as both real values and imaginary values. Converter 208 may discard the imaginary part of the signal by taking the real part of the complex-valued signal. Signal 228 represents a real-valued signal in the transform domain.

[0056] Converter 208 is optional and in some embodiments of bass enhancement system 200 may be omitted. When upsampler 202 is omitted, converter 208 must also be omitted in order for the imaginary part to remain in the signal processing path for use by downstream components.

[0057] Filter 212 receives signal 228 (or signal 224 when transducer 208 is omitted, or signal 222 when dynamics processor 206 and transducer 208 are omitted), filters the input signal, and generates signal 230. Signal 230 is a complex-valued signal in transformation areas. In general, filtering splits signal 228 into subbands as one of the input signals to mixer 216. The specifics of filtering will depend on whether upsampling has been performed (see upsampler 202).

[0058] When upsampler 202 is not present, filter 212 may be implemented by feeding an input signal (eg, signal 228) to an 8-channel Nyquist filter bank to generate signal 230, which has hybrid subbands 0–7.

[0059] When upsampler 202 is present, filter 212 may be implemented by a CQMF analysis filter bank and two or more Nyquist filters. The real part of the input signal (for example, signal 228) is fed to the CQMF analysis filter block; the CQMF analysis filter bank has a suitable number of channels to generate a signal 230 having subband signals with a sampling rate of 750 Hz. This corresponding number of channels then depends on the upsampling performed. For example, when 4x upsampling is performed, and thus a 4-channel CQMF analysis filter bank is used in filter 212, each of the three lowest frequency CQMF subband signals is fed into a corresponding Nyquist filter (one of which generates hybrid subbands 0 -7, one of which generates hybrid subbands 8-11, and one of which generates hybrid subbands 12-15). As another example, when 2x upsampling is performed and thus a 2-channel CQMF analysis filter bank is used in filter 212, each of the two CQMF subband signals is fed to a corresponding Nyquist filter (one of which generates hybrid 0- 7, and one of which generates hybrid subbands 8-11). The remaining CQMF channels, if present, are provided to mixer 216 (with an appropriate delay that matches the delay of the Nyquist filters).

[0060] Filter 212 may be implemented by filters that are similar to those used in signal conditioning system 110 (see FIG. 1). For example, a first 8-channel Nyquist analysis filter can generate subbands 0-7, a second 4-channel Nyquist analysis filter can generate subbands 8-11, and a third 4-channel Nyquist analysis filter can generate subbands 12-15.

[0061] Delay element 214 receives converted audio signal 112, implements a delay period, and generates signal 232. Signal 232 corresponds to a delayed version of converted audio signal 112 according to the delay period. The delay element 214 may be implemented using a memory device, a shift register, etc. The delay period corresponds to the processing time of other components in the signal processing chain, such as upsampler 202, harmonic generator 204, dynamics processor 206, converter 208, filter 212, etc. Since some of these other components are optional, the delay period decreases as these optional components are omitted. In one example, the delay period is 961 samples, of which 577 correspond to upsampling and 384 correspond to the remaining components, such as Nyquist filters. As another example, when upsampler 202 is omitted, the delay period is 384 samples.

[0062] Mixer 216 receives signal 230 and signal 232, performs mixing, and generates an amplified audio signal 122 (see FIG. 1). The amplified audio signal 122 is a transform domain signal. Mixer 216 mixes signals on a band basis. For example, signal 230 and signal 232 may each have 77 hybrid bands (e.g., 8+4+4+61 HCQMF bands), and mixer 216 mixes subband 0 of signal 230 with subband 0 of signal 232, mixes subband 1 of signal 230 with subband 1 signal 232, etc. Mixer 216 does not necessarily mix all bands; when generating the amplified audio signal 122, one or more bands of the signal 232 may be skipped. For example, the highest frequency bands (e.g., one or more hybrid bands 16-77) of signal 232 may be passed without mixing.

[0063] Further details of the bass enhancement system 200 are provided below. Referring first to FIG. 3-5, various possibilities for harmonic generator 204 are discussed.

[0064] In FIG. 3 is a block diagram of a harmonic generator 300. The harmonic generator 300 may be used as the harmonic generator 204 (see FIG. 2). In general, the harmonic generator 300 generates each subsequent harmonic by multiplying (eg, using direct signal multiplication) the input signal and the previous harmonics.

[0065] The harmonic generator 300 includes one or more multipliers 302 (two shown: 302a and 302b), two or more gain stages 304 (three shown: 304a, 304b and 304c), two or more compressors 306 (three shown: 306a, 306b and 306c) and two or more adders 308 (three shown: 308a, 308b and 308c). In general, each row of components in the harmonic generator 300 corresponds to one of the generated harmonics, so the number of rows (and the corresponding number of components) can be adjusted to realize the desired number of harmonics. The first processing row includes a gain stage 304a, a compressor 306a, and an adder 308a. The second processing row includes a multiplier 302a, a gain stage 304b, a compressor 306b, and an adder 308b. The third processing row includes a multiplier 302b, a gain stage 304c, a compressor 306c, and an adder 308c. Additional rows may be added to generate additional harmonics, each new row being connected to the previous row in a manner similar to that shown in the figure.

[0066] Harmonic generator 300 receives input signal 320, also designated “x”. Input signal 320 corresponds to the upsampled signal 220 (see FIG. 2) when upsampler 202 is present, or the converted audio signal 112 when upsampler 202 is absent. Input signal 320 is a complex transform domain signal. For example, input signal 320 may correspond to an HCQMF band (eg, hybrid subband 0, hybrid subband 2, hybrid subband 4, hybrid subband 6, etc.). The harmonic generator 300 generates a signal 222 (see FIG. 2).

[0067] Starting with multipliers 302, multiplier 302a receives input signal 320, multiplies input signal 320 by itself, and generates signal 322a, also denoted " ^x2 ". Multiplier 302b receives input signal 320 and signal 322a, multiplies input signal 320 by signal 322a, and generates signal 322b, also designated “ ^x3 ”. It should be noted that the output signal of a given multiplier is provided as an input signal to the multiplier in the next row of processing: signal 322a is provided to multiplier 302b, signal 322b is provided to the multiplier in the next row (shown by the dotted line), etc.

[0068] Referring to gain stages 304, gain stage 304a receives input signal 320, applies gain g ₁ and generates signal 324a. Gain stage 304b receives signal 322a, applies gain g ₂ and generates signal 324b. Gain stage 304c receives signal 322b, applies gain g ₃ and generates signal 324c. If necessary, gain factors g ₁ , g ₂ , g ₃ , etc. can be adjusted, usually in the form of making adjustments for each specific device implementing the harmonic generator 300. In general, the gain g ₁ can be much smaller than other gains (for example, less than 50% relative to other gains). Setting the gain g ₁ to a small value reduces what is called the direct signal corresponding to the original low-frequency harmonic, which is undesirable in small speakers that are physically unable to reproduce any signal in the frequency range of the direct signal. If necessary, the gain g ₁ can be set to zero to eliminate the direct signal.

[0069] Referring to compressors 306, compressor 306a receives signal 324a, performs dynamic compression, and generates signal 326a. Compressor 306b receives signal 324b, performs dynamic compression, and generates signal 326b. Compressor 306c receives signal 324c, performs dynamic compression, and generates signal 326c. Dynamic compression generally follows the equation y ^r , where y corresponds to the input signal (eg, signal 324a) and r is the compression ratio, with r less than 1. The compression ratio r may be different for each harmonic (for example, for each row). For example, the compression ratio r ₁ for compressor 306a may be different from the compression ratio r ₂ for compressor 306b, which may be different from the compression ratio r ₃ for compressor 306c, etc. The compression ratios may be adjusted in the form of settings based on the specific physical characteristics of the device implementing the harmonic generator 300. Additional details of the 306 compressors are provided below in the discussion regarding volume expansion.

[0070] Referring to the adders 308, the adder 308c receives signal 326c (and any output from the adder in any additional row), performs the addition, and generates signal 328b. Adder 308b receives signal 326b and signal 328b, performs an addition, and generates signal 328a. Adder 308a receives signal 326a and signal 328a, performs the addition, and generates signal 222 (see FIG. 2). It should be noted that one of the inputs to this adder is provided by the adder in the following processing row: adder 308c receives the output of the adder in the next processing row (shown by the dotted line), adder 308b receives the output of adder 308c, adder 308a receives the output of adder 308b, etc.

[0071] The harmonic generator 300 processes complex-valued signals, for example, signals with a very small contribution of negative frequencies. Thus, generating harmonics by multiplying a complex-valued signal by itself produces a cleaner output signal than if the input signal were real-valued, for example, resulting in less intermodulation distortion. In the complex-valued case, for an input signal consisting of many frequencies, only the required terms are generated, with the addition of terms from the sums of frequencies, but not terms from the differences of frequencies, as would be the case for real-valued processing. Difference terms, although usually low-frequency, are more unpleasant to hear than sum terms. The sum terms may in fact be desired, for example, when the input signal contains a harmonic series.

[0072] In FIG. 4 is a block diagram of a harmonic generator 400. The harmonic generator 400 may be used as the harmonic generator 204 (see FIG. 2). In general, the harmonic generator 400 generates harmonics by applying a feedback delay loop to an input signal. The harmonic generator 400 includes a multiplier 402, a gain stage 404, an add stage 406, a compressor 408, a delay stage 410, a gain stage 412, and a gain stage 414.

[0073] The harmonic generator 400 receives an input signal 420. The input signal 420 corresponds to the upsampled signal 220 (see FIG. 2) when the upsampler 202 is present, or the converted audio signal 112 when the upsampler 202 is absent. Input signal 420 is a complex transform domain signal. For example, input signal 420 may correspond to an HCQMF band (eg, hybrid subband 0, hybrid subband 2, hybrid subband 4, hybrid subband 6, etc.). A harmonic generator 400 generates a signal 222 (see FIG. 2).

[0074] Multiplier 402 receives input signal 420, multiplies input signal 420 by signal 432, and generates signal 422. Signal 432 may also be referred to as feedback signal 432, and is discussed in more detail below with reference to gain stage 412.

[0075] Gain stage 404 receives input signal 420, applies gain a, and generates signal 424. Gain a may also be referred to as transient gain. The value of the gain a may be adjusted in the form of a tuning parameter based on the particular physical characteristics of the device implementing the harmonic generator 400.

[0076] Adder stage 406 receives signal 422 and signal 424, performs the addition, and generates signal 426. The combination of gain stage 404 and adder stage 406, when added to signal 422, is used to assist in starting a closed-loop loop (for example, when signal 432 is initially zero ) and otherwise contributes to maintaining the operation of the feedback loop.

[0077] Compressor 408 receives signal 426, performs dynamic compression, and generates signal 428. Dynamic compression generally follows the equation y ^r , where y corresponds to the input signal (eg, signal 426) and r is the compression ratio, with r less than 1 The compression ratio may be adjusted in the form of a tuning parameter based on the specific physical characteristics of the device implementing the harmonic generator 400. Additional details of the 408 compressor are provided below in the discussion regarding volume expansion.

[0078] Delay stage 410 receives signal 428, performs a delay operation, and generates signal 430. Delay stage 410 may be implemented using a memory device.

[0079] Gain stage 412 receives signal 430, applies gain g , and generates signal 432. Gain g may also be referred to as feedback gain. As discussed above with respect to multiplier 402, signal 432 is multiplied by input signal 420 to generate harmonics of a theoretically unlimited order.

[0080] Gain stage 414 receives signal 428, applies gain h, and generates signal 222 (see FIG. 2). The gain h can also be called the output gain. The value of the gain h may be adjusted in the form of a tuning parameter based on the specific physical characteristics of the device implementing the harmonic generator 400.

[0081] As with the harmonic generator 300, the harmonic generator 400 generates a direct signal corresponding to the original low frequency harmonic. If necessary, the direct signal can be reduced by adjusting the values of the gain a and the compression ratio r .

[0082] As with the harmonic generator 300, the harmonic generator 400 processes complex-valued signals, and by generating harmonics by multiplying the complex-valued signal by itself, a cleaner output signal is produced than if the input signal were real-valued.

[0083] In FIG. 5 is a block diagram of a harmonic generator 500. The harmonic generator 500 may be used as the harmonic generator 204 (see FIG. 2). The harmonic generator 500 is similar to the harmonic generator 400 (see FIG. 4) except for the addition of a transient gain signal after the compressor. The harmonic generator 500 includes a multiplier 502, a compressor 504, a gain stage 506, an add stage 508, a delay stage 510, a gain stage 512, and a gain stage 514.

[0084] The harmonic generator 500 receives an input signal 520. The input signal 520 corresponds to the upsampled signal 220 (see FIG. 2) when the upsampler 202 is present, or the converted audio signal 112 when the upsampler 202 is absent. Input signal 520 is a complex transform domain signal. For example, input signal 520 may correspond to an HCQMF band (eg, hybrid subband 0, hybrid subband 2, hybrid subband 4, hybrid subband 6, etc.). The harmonic generator 500 generates a signal 222 (see FIG. 2).

[0085] Multiplier 502 receives input signal 520, multiplies input signal 520 by signal 532, and generates signal 522. Signal 532 may also be referred to as feedback signal 532, and is discussed in more detail below with reference to gain stage 512.

[0086] Compressor 504 receives signal 522, performs dynamic compression, and generates signal 524. Dynamic compression generally follows the equation y ^r , where y corresponds to the input signal (eg, signal 522) and r is the compression ratio, with r less than 1 The compression ratio may be adjusted in the form of a tuning parameter based on the specific physical characteristics of the device implementing the harmonic generator 500. Additional details of the 504 compressor are provided below in the discussion regarding volume expansion.

[0087] Gain stage 506 receives input signal 520, applies gain a, and generates signal 526. Gain a may also be referred to as transient gain. The value of the gain a may be adjusted in the form of a tuning parameter based on the particular physical characteristics of the device implementing the harmonic generator 500.

[0088] Adder stage 508 receives signal 524 and signal 526, performs the addition, and generates signal 528. The combination of gain stage 506 and adder stage 508, when added to signal 524, is used to assist in starting a closed-loop loop (for example, when signal 532 is initially zero ) and otherwise contributes to maintaining the operation of the feedback loop.

[0089] Delay stage 510 receives signal 528, performs a delay operation, and generates signal 530. Delay stage 510 may be implemented using a memory device.

[0090] Gain stage 512 receives signal 530, applies gain g , and generates signal 532. Gain g may also be referred to as feedback gain. As discussed above with respect to multiplier 502, signal 532 is multiplied by input signal 520 to generate harmonics of a theoretically unlimited order.

[0091] Gain stage 514 receives signal 524, applies gain h, and generates signal 222 (see FIG. 2). The gain h can also be called the output gain. The value of the gain h may be adjusted in the form of a tuning parameter based on the particular physical characteristics of the device implementing the harmonic generator 500.

[0092] Compared to harmonic generator 300 (see FIG. 3) and harmonic generator 400 (see FIG. 4), harmonic generator 500 avoids the direct signal path by adding input signal 520 later in the cycle (e.g., as a signal 526). In such an arrangement, input signal 520 is passed through multiplier 502 (as opposed to adder 406 in FIG. 4) as part of generating signal 222, so signal 222 does not contain a direct signal.

[0093] As with harmonic generator 300 and harmonic generator 400, harmonic generator 500 processes complex-valued signals, and by generating harmonics by multiplying the complex-valued signal by itself, a cleaner output signal is produced than if the input signal were real-valued.

Volume expansion

[0094] As discussed above, since the sound pressure level for a fixed volume range (in the background) increases with frequency in the low/mid frequency range (eg, less than 800 Hz), harmonic generators (eg, harmonic generator 204 shown in FIG. 2 , the harmonic generator 300 shown in Fig. 3, the harmonic generator 400 shown in Fig. 4, the harmonic generator 500 shown in Fig. 5, etc.) perform expansion in the dynamic characteristics when generating their output signals. When performing volume expansion, the harmonic generators may use compressors (eg, compressors 306 of FIG. 3, compressor 408 of FIG. 4, compressor 504 of FIG. 5, etc.). Examples of loudness expansion processes include dynamic compression and loudness equalization.

Dynamic compression

[0095] Harmonic generators can generate nth order harmonics using an operation corresponding to equation (1):

[0096] In equation (1), n is the harmonic order, y is the output signal, x is the input signal, is a complex exponential function, j is an imaginary number, and phase. The output signal is generated by multiplying the input signal n times by itself. Accordingly, increasing n increases the order of the generated harmonic. (The right side of equation (1) serves later in this document as an illustration of why dynamic expansion clearly results in dynamic compression when the signals have been multiplied by themselves.)

[0097] In FIG. 6 is a graph 600 showing equal loudness curves. In the 600 graph, the x-axis represents the frequency in Hz and the y-axis represents the sound pressure level (SPL) in dB. The graph 600 contains 6 curves 602a, 602b, 602c, 602d, 602e and 602f (collectively, curves 602). Each of the curves 602 corresponds to a background loudness level, which is a logarithmic measure of the perceived amplitude of the sound. Each of the curves 602 may also be referred to as a sound volume equal curve. Curve 602a corresponds to the perceptual threshold, curve 602b corresponds to the loudness of 20 phons, curve 602c corresponds to the loudness of 40 phons, curve 602d corresponds to the loudness of 60 phons, curve 602e corresponds to the loudness of 80 phons, and curve 602f corresponds to the loudness of 100 phons.

[0098] When generating harmonics using the operation described in equation (1), the dynamic characteristics are expanded by a factor n . Given this information, equal sound volume curves 602 assume the relationship of Equation (2):

[0099] In equation (2), the term represents the degree of expansion of the remainder, which is related to the fundamental frequency f and the harmonic order n . Residue expansion ratio usually in the range of 1.1-1.4 depending on the fundamental frequency f and the order of the harmonics n . When harmonics are generated according to equation (1), the required expansion ratio is can be achieved by compressing the output signal from the harmonic generator by a factor . (As an aside, the terms "expansion" and "compression" can generally be used interchangeably, with "compression" being used when the degree is less than 1 and "expansion" being used when the degree is greater than 1. Therefore, the coefficient may be called "compression" due to the divisor n ).

[0100] In graph 600, lines 610 and 612 illustrate an example of volume expansion. Line 610 indicates a volume range from 20 phons to 80 phons for a fundamental frequency of 50 Hz. Line 612 corresponds to generating a 50 Hz 4th order harmonic from 400 Hz with the same volume range. Arrow 614 from 610 to 612 indicates 4th order harmonic generation. The dynamic range of the fundamental frequency SPL (line 610) is approximately 38 dB over the loudness range from 20 to 80 phons, and the dynamic range of the 4th order harmonic SPL (line 612) is approximately 50 dB over the same loudness range. Thus, when generating a 4th order harmonic from a fundamental with a volume of 80 phons and a frequency of 50 Hz, this harmonic must be attenuated by approximately 20 dB. When the fundamental, on the other hand, is 20 phons loud, that harmonic needs to be attenuated by nearly 40 dB, with an increase in the required attenuation of approximately 20 dB.

[0101] The degree of SPL spreading relative to the background, also called loudness spreading, can be approximated according to equation (3):

[0102] In equation (3) is the degree of expansion of the SPL relative to the background, which includes the inverse of the frequency f .

[0103] Residue expansion ratio given by equation (4):

[0104] In equation (4), the expansion ratio of the remainder corresponds to the ratio of the degree of expansion SPL in relation to the background of the fundamental frequency f and the degree of expansion SPL in relation to the background of the harmonic which corresponds to the relationship between the natural logarithm of n (harmonic order) and the natural logarithm of f (fundamental frequency). In other words, the degree of expansion of the remainder defines the coefficient required when generating the nth harmonic based on the fundamental frequency f (in Hz). Equations (3) and (4) are in good agreement with the equal loudness curves shown in Fig. 6, in the volume range of 20-80 backgrounds and at frequencies from 20 to 1000 Hz. When using a harmonic generator 400 (see FIG. 4) or a harmonic generator 500 (see FIG. 5), the required dynamic compression can be achieved with reasonable accuracy using a single simple compressor having a constant degree (for example, such as compressor 408 or compressor 504).

[0105] The compressor may apply dynamic compression using a first order averaging filter to avoid distortion due to sample-based normalization. The first order averaging filter can process the control signal s , which can be calculated according to equation (5):

[0106] In Equation (5), m is the sample number, c is the compression gain, and α is the weighting coefficient between the control signal value for the previous sample and the compression gain value for the current sample. The weighting factor α can also be called the exponential smoothing factor, and it corresponds to a pole in the first-order low-pass system.

[0107] The weighting factor α can be calculated using equation (6):

[0108] In equation (6) sampling rate, and time constant.

[0109] The compression gain c can be calculated using equation (7):

[0110] In equation (7), a and b are polynomial coefficients that apply to each order of magnitude of the sample value m of the input signal x . Applying a compression gain c (or a smoothed version of s from equation (5)) to a signal x of the form c ⋅ x (or c ⋅ x ) corresponds to a rational approximation , which is the absolute value of the signal x subjected to the compression ratio r , multiplied by the sign function of x .

[0111] In FIG. 7 is a graph 700 showing various compression gain factors c . In plot 700, the x-axis represents the input power (input signal x ) in dB, and the y-axis represents the compression gain c in dB. Various curves are shown, with each curve corresponding to a certain compression ratio value r . In particular, 9 values are given for r in the range from 0.5 to 1.0: 0.5, 0.6, 0.65, 0.7, 0.73, 0.77, 0.8, 0.9 and 1.0, each value corresponding to one of the curves on the graph 700 (for example, an r value of 0.5 corresponds to the top curve). It should be noted that the specified gains in FIG. 7 are not accurate; they are merely an illustration of the general concept. Also for graph 700 it can be noted that the gain is limited for low input power and has the form of the ratio . This prevents excessive gain from being applied in circumstances such as transient attacks following quiet periods of the signal. (In contrast, this gain, in combination with the time constant in equation (6), allows most of the energy to pass through the compressor during, for example, shock attacks, which contribute to the perceived "punch" in the low-frequency signal.)

Volume correction

[0112] An alternative approach to achieve loudness expansion is to apply normalization to the input signal in a first step, before generating harmonics, followed by a gain adjustment step. This may be called volume correction.

[0113] In FIG. 8 is a block diagram of a harmonic generator 800. The harmonic generator 800 generally performs loudness correction using normalization of the input signals. Amplitude normalization theoretically allows to prevent dynamic expansion of harmonics (by a factor of n when n ≥2) when generated according to equation (1).

[0114] The harmonic generator 800 includes two or more normalization stages 802 (two shown: 802a and 802b), two or more multipliers 804 (two shown: 804a and 804b), two or more loudness correction stages 806 (two shown: 806a and 806b ), two or more adders 808 (two shown: 808a and 808b), and an adder 810. In general, each row of components in the harmonic generator 800 corresponds to one of the harmonics generated, so the number of rows (and the corresponding number of components) can be adjusted to realize the desired number of harmonics. . The first processing row includes a normalization stage 802a, a multiplier 804a, a loudness correction stage 806a, and an adder 808a. The second processing row includes a normalization stage 802b, a multiplier 804b, a loudness correction stage 806b, and an adder 808b. Additional rows may be added to generate additional harmonics, each new row being connected to the previous row in a manner similar to that shown in the figure.

[0115] The harmonic generator 800 receives an input signal 820. The input signal 820 corresponds to the upsampled signal 220 (see FIG. 2) when the upsampler 202 is present, or the converted audio signal 112 when the upsampler 202 is absent. Input signal 820 is a complex transform domain signal. For example, input signal 820 may correspond to an HCQMF band (eg, hybrid subband 0, hybrid subband 2, hybrid subband 4, hybrid subband 6, etc.). A harmonic generator 800 generates a signal 222 (see FIG. 2).

[0116] Starting with normalization stages 802, normalization stage 802a receives input signal 820, performs normalization, and generates signal 822a. Normalization stage 802b receives input signal 820, performs normalization, and generates signal 822b. Similar to equation (5), each of the normalization stages 802 may perform normalization using a first order smoothing filter to avoid distortion caused by normalization from one sample value to another. Normalization stages 802 may perform normalization in the manner described by equation (8):

[0117] In equation (8) the current sample value m of the normalized version of the input signal x , is the previous sampled value of the normalized version of the input signal, α is the smoothing coefficient, and given by equation (9):

[0118] In equation (9) corresponds to the relationship between the complex value of the current input signal sample and the amplitude (also called the absolute value) of the current input signal sample. The smoothing factor, α , can be adjusted as needed to control the required smoothing time, and is dependent on the dynamic characteristics of the input signal. During attack events (for example, when there is rapidly increasing signal energy), a smaller α is applied than under stationary or decreasing energy conditions to avoid signal clipping.

[0119] Alternatively, a single normalization stage (e.g., 802a) may be used in the harmonic generator, with the output signal (e.g., 822a) provided as an input to each of the multipliers 804.

[0120] Referring to multipliers 804, multiplier 804a receives input signal 820 and signal 822a, multiplies these signals by each other, and generates signal 824a. Multiplier 804b receives signal 822b and signal 824a, multiplies these signals by each other, and generates signal 824b. Signal 824a corresponds to the second harmonic, signal 824b corresponds to the third harmonic, etc. It should be noted that the output signal of a given multiplier is delivered as an input signal to the multiplier in the next row of processing: signal 824a is delivered to multiplier 804b, signal 824b is delivered to the multiplier in the next row (shown by the dotted line), etc.

[0121] Referring to the loudness correction stages 806, the loudness correction stage 806a receives signal 824a, performs loudness correction, and generates signal 826a. Volume correction stage 806b receives signal 824b, performs volume correction, and generates signal 826b. In general, the loudness correction stages 806 apply dynamic expansion and attenuation of the normalized energy of the generated harmonics according to the equal loudness curves depicted in FIG. 6, in order to maintain loudness compared to the fundamental harmonic. To adjust the volume, a correction factor k is specified, and k depends on the order of the harmonic n , the smoothed amplitude of the fundamental harmonic (see equation (8)) and the hybrid band index b . This correction factor k is applied according to equation (10):

[0122] In equation (10) harmonic with volume correction, and normalized harmonic, respectively, for each harmonic.

[0123] As discussed above, low frequency enhancement processes may be performed on one or more hybrid bands (eg, one or more subbands 0, 2, 4, 6, 7, 9, etc.). Each band generates several harmonics, such as the 2nd, 3rd and 4th. If we assume that the center frequency approximates the fundamental frequency in each band, then the SPL to background ratio can be calculated using a single parameter: harmonic order n . As an example, the first hybrid band (eg, sub-band 0) has a center frequency of 46.875 Hz (eg, approximately 47 Hz), and the corresponding values from the ELC curves in FIG. 6 are listed in Table 1.

[0124] In Table 1, the value in parentheses represents the difference in SPL compared to the fundamental. A function representing the difference between the SPL of a harmonic and its fundamental can be calculated according to equation (11):

[0125] In equation (11), K _b,n is the gain value in dB, minimum attenuation value, X is the smoothed input fundamental energy on a logarithmic scale, while – scale parameter of the input energy depending on the harmonic order n . can be calculated according to equation (12):

[0126] The correction factor in the linear scale can be calculated according to equation (13):

[0127] In equations (12) and (13), everything are constants based on the hybrid stripes and can be estimated for optimal fit to the ELC curves depicted in FIG. 6. The parameters listed in Table 2 will result in sufficient accuracy for the first six hybrid bands, and the resulting loudness correction factors are depicted in FIG. 9. For bands 6, 7 and 9, the generated harmonics are in the frequency range from 700 Hz to 2000 Hz, assuming that the ELC curves are flat. Loudness correction stages 806 may calculate loudness correction coefficients using segment linear approximation to reduce computational complexity.

[0128] In FIG. 9A, 9B, 9C, 9D, 9E and 9F show a set of plots 900a-900f. In each plot, the x-axis represents the amplitude of the normalized harmonic signal in the loudness correction stage (eg, the signal 824a input to the loudness correction stage 806a, etc.), and the y-axis represents the correction factor k . Plot 900a corresponds to hybrid band 0, plot 900b corresponds to hybrid band 2, plot 900c corresponds to hybrid band 4, plot 900d corresponds to hybrid band 6, plot 900e corresponds to hybrid band 7, and plot 900f corresponds to hybrid band 9. Lines for three harmonics (2- 1st, 3rd and 4th) are shown in each graph, however in graphs 900d, 900e and 900f the lines overlap as they converge as the hybrid band number increases. Overall, the lines show the loudness correction factors k for the first 6 hybrid bands when using constants based on the hybrid bands listed in Table 2.

[0129] Returning to FIG. 8 and adders 808, adder 808b receives signal 826b (and any signal received from the next row of processing shown in dashed line), adds and generates signal 828b. Adder 808b receives signal 826a and signal 828b, performs an addition, and generates signal 828a. It should be noted that one of the inputs to a given adder is provided by the adder in the next processing row: adder 808b receives the output of the adder in the next processing row (shown by the dotted line), adder 808a receives the output of adder 808b, etc.

[0130] Adder 810 receives input signal 820 and signal 828a, performs an addition, and generates signal 222 (see FIG. 2).

Processing of multiple hybrid strips

[0131] Although the description of the low frequency enhancement system 200 (see FIG. 2) has focused on processing a single hybrid band, similar processing may be performed on multiple hybrid bands. For example, low frequency enhancement system 120 (see FIG. 1) may operate on four hybrid bands (e.g., sub-bands 0, 2, 4, and 6), six hybrid bands (e.g., sub-bands 0, 2, 4, 6, 7 and 9), etc. Each band generates several harmonics (for example, 2nd, 3rd, 4th, etc.).

[0132] In FIG. 10 is a block diagram of a bass enhancement system 1000. The bass boost system 1000 may be used as the bass boost system 120 (see FIG. 1). Bass boost system 1000 is similar to bass boost system 200 (see FIG. 2), with similar components having similar names and reference numbers, with the addition of multiple processing paths explicitly. Each processing path corresponds to processing a hybrid subband signal. As a specific example, 4 processing paths are shown (eg to process hybrid subbands 0, 2, 4 and 6). If necessary, the number of processing paths can be increased or decreased. For example, six processing paths may be used to process hybrid subbands 0, 2, 4, 6, 7, and 9.

[0133] Bass enhancement system 1000 receives converted audio signal 112 (see FIG. 1). As discussed above, the converted audio signal 112 is a hybrid complex signal in the hybrid band conversion domain. Four of the hybrid bands of the converted audio signal 112 are shown as input signals to the bass enhancement system 1000: subband 0 (labeled 1002a), subband 2 (1002b), subband 4 (1002c), and subband 6 (1002d). Each subrange corresponds to one of the processing paths. Bass enhancement system 1000 includes upsamplers 1010 (four shown: 1010a, 1010b, 1010c and 1010d), harmonic generators 1012 (four shown: 1012a, 1012b, 1012c and 1012d), adder 104, dynamics processor 1016 (optional), a converter 1018 (optional), a filter 1022, a delay element 1024, and a mixer 1026.

[0134] Upsampler 1010a receives signal 1002a, performs upsampling, and generates upsampled signal 1030a. Upsampler 1010b receives signal 1002b, performs upsampling, and generates upsampled signal 1030b. Upsampler 1010c receives signal 1002c, performs upsampling, and generates upsampled signal 1030c. Upsampler 1010d receives signal 1002d, performs upsampling, and generates upsampled signal 1030d. Signals 1030a, 1030b, 1030c and 1030d are complex transform domain signals. In other respects, the upsamplers 1010 are similar to those described above with respect to the upsampler 202 (see FIG. 2).

[0135] A harmonic generator 1012a receives the upsampled signal 1030a and generates its harmonics to produce a signal 1032a. A harmonic generator 1012b receives the upsampled signal 1030b and generates its harmonics to produce a signal 1032b. A harmonic generator 1012c receives the upsampled signal 1030c and generates its harmonics to produce a signal 1032c. A harmonic generator 1012d receives the upsampled signal 1030d and generates its harmonics to produce a signal 1032d. Signals 1032a, 1032b, 1032c, and 1032d are complex transform domain signals. Otherwise, the harmonic generators 1012 are similar to the harmonic generator 204 (see FIG. 2). For example, one or more harmonic generators 1012 may be implemented using a harmonic generator 300 (see FIG. 3), a harmonic generator 400 (see FIG. 4), a harmonic generator 500 (see FIG. 5), a harmonic generator 800 (see FIG. see Fig. 8), etc.

[0136] Adder 1014 receives signals 1032a, 1032b, 1032c, and 1032d, performs addition, and generates signal 1034. Signal 1034 is a complex transform domain signal.

[0137] The dynamics processing device 1016 receives the signal 1034, performs the dynamics processing, and generates a signal 1036. The signal 1036 is a complex transform domain signal. Otherwise, the dynamic characteristics processing device 1016 is similar to the dynamic characteristics processing device 206 (see FIG. 2). The dynamic characteristics processing device 1016 is optional. When dynamics processing device 1016 is omitted, transducer 1018 receives signal 1034 instead of signal 1036.

[0138] Converter 1018 receives signal 1036 (or signal 1034 when dynamics processor 1016 is omitted), discards the imaginary part from signal 1036, and generates signal 1040. Signal 1040 is a transform domain signal. In other respects, converter 1018 is similar to converter 208 (see FIG. 2), and its inclusion is optional.

[0139] Filter 1022 receives signal 1040 (or signal 1036 when transducer 1018 is omitted, or signal 1034 when dynamics processor 1016 and transducer 1018 are omitted), performs filtering, and generates signal 1042. Signal 1042 is a transform domain signal. In other respects, filter 1022 is similar to filter 212 (see FIG. 2).

[0140] Delay element 1024 receives signal 1042, implements a delay period, and generates signal 1044. Signal 1044 corresponds to a delayed version of converted audio signal 112 according to the delay period. The delay element 1024 may be implemented using a memory device, a shift register, etc. The delay period corresponds to the processing time of other components in the signal processing chain; since some of these other components are optional, the delay period decreases as optional components are omitted. Otherwise, delay element 1024 is similar to delay element 214 (see FIG. 2).

[0141] Mixer 1026 receives signal 1042 and signal 1044, performs mixing, and generates an amplified audio signal 122 (see FIG. 1). In other respects, mixer 1026 is similar to mixer 216 (see FIG. 2).

[0142] In FIG. 11 depicts a mobile device architecture 1100 for implementing the features and processes described herein, according to an embodiment. The architecture 1100 may be implemented in any electronic device, including, but not limited to: a desktop computer, consumer audiovisual (AV) equipment, broadcast equipment, mobile devices (e.g., smartphone, tablet computer, laptop, wearable device), etc. In the exemplary embodiment shown, architecture 1100 is for a laptop and includes processor(s) 1101, peripheral interface 1102, audio subsystem 1103, speakers 1104, microphone 1105, sensors 1106 (e.g., accelerometers, gyroscopes, barometer, magnetometer, camera), device 1107 location data processing (e.g., GNSS receiver), wireless communications subsystem 1108 (e.g., Wi-Fi, Bluetooth, cellular), and input/output (I/O) subsystem(s) 1109, which contains touch controller 1110 and other controllers input 1111, touch surface 1112, and other input/control devices 1113. Other architectures with more or fewer components may also be used to implement the disclosed embodiments.

[0143] Memory interface 114 is coupled to processors 1101, peripheral interface 1102, and storage device 115 (eg, flash memory, RAM, ROM). Memory 1115 stores software commands and data, including, but not limited to: operating system commands 1116, communication commands 1117, GUI commands 1118, sensor processing commands 1119, telephone commands 1120, electronic messaging commands 1121, web browsing commands 1122. pages, audio processing commands 1123, navigation/GNSS commands 1124, and applications/data 1125. Audio processing commands 1123 contain commands for performing the audio processing described herein.

[0144] In FIG. 12 is a flowchart of an audio processing method 1200. The method 1200 may be performed by a device (eg, laptop, mobile phone, etc.) comprising components of the architecture 1100 depicted in FIG. 11, to implement the functionality of the audio processing system 100 (see FIG. 1), bass enhancement system 200 (see FIG. 2), bass enhancement system 1000 (see FIG. 10), etc., for example , by executing one or more computer programs. In general, method 1200 performs audio signal processing in the complex-valued subband domain (eg, in the HCQMF domain).

[0145] At step 1202, a first signal in the transform domain is received. The first transform domain signal is a hybrid complex transform domain signal having a number of bands. At least one of the bands has a number of subbands. The first signal in the conversion domain has a first set of harmonics. For example, bass enhancement system 200 (see FIG. 2) may receive converted audio signal 112. The first signal in the conversion domain may have 77 hybrid bands numbered 0-76, with bands 0-15 being subbands resulting from separating one or more larger strips. The first signal in the transform domain may be a signal in the CQMF domain. The first transform domain signal may be an HCQMF domain signal generated by dividing (eg, using Nyquist filter banks) a subset of the channels of the CQMF domain signal into subbands to increase frequency resolution for the lowest frequency range.

[0146] At step 1204, a second transform domain signal is generated based on the first transform domain signal. The second transform domain signal is generated by generating harmonics of the first transform domain signal according to a nonlinear process. The second signal in the transform domain has a second set of harmonics that are different from the first set of harmonics, and the second signal in the transform domain is a complex-valued signal having an imaginary part. The second transform domain signal is further generated by performing volume expansion with respect to the second set of harmonics. For example, harmonic generator 204 (see FIG. 2), harmonic generator 300 (see FIG. 3), harmonic generator 400 (see FIG. 4), harmonic generator 500 (see FIG. 5), harmonic generator 800 (see . fig. 8), etc. may generate a second transform domain signal (e.g., signal 222) based on a first transform domain signal (e.g., signal 220, etc.)

[0147] At step 1206, by filtering the second signal in the transform domain, a third signal in the transform domain is generated. The third signal in the conversion domain has a number of bands, and at least one of the bands has a number of subbands. For example, filter 212 (see FIG. 2) may filter signal 228 (or signal 226) to generate signal 230. As another example, filter 1022 (see FIG. 10) may filter signal 1040 to generate signal 1042. The third signal in The conversion region can have 77 hybrid bands, numbered 0–76, with bands 0–15 being subbands resulting from the division of one or more larger bands. The third signal in the transform domain may be a signal in the HCQMF domain.

[0148] At step 1208, a fourth transform domain signal is generated by mixing the third transform domain signal with a delayed version of the first transform domain signal. This subband of the third transform domain signal is mixed with a corresponding subband of a delayed version of the first transform domain signal. For example, mixer 216 (see FIG. 2) may mix signal 230 with delayed signal 232. As another example, mixer 1026 (see FIG. 10) may mix signal 1042 with delayed signal 1044. Input signals may have 77 hybrid bands numbered 0–76, where a given band of an input signal (eg, band 0) is mixed with a corresponding band (eg, band 0) of another input signal.

[0149] Method 1200 may include additional steps corresponding to other functionality of bass enhancement system 200, bass enhancement system 1000, etc., as described herein. For example, the fourth signal in the conversion region may be output by a speaker, such as speakers 1104 (see FIG. 11). As another example, the signals in the transform domain may be upsampled (e.g., using upsampler 202, upsamplers 1010) before generating harmonics at step 1204. As another example, dynamics processing may be applied to the signals in the transform domain, e.g. using the dynamic characteristics processing device 206 or the dynamic characteristics processing device 1016. As another example, generating harmonics may involve performing a multiplication, using a feedback delay loop, etc. As another example, the second transform domain signal may be a plurality of second transform domain signals, each of which corresponds to a hybrid band of the first transform domain signal. As another example, the imaginary portion of the second signal in the transform region may be discarded before generating the third signal in the transform region.

Implementation details

[0150] An embodiment of the invention may be implemented in hardware, executable modules that are stored on a computer-readable medium, or a combination thereof (eg, programmable gate arrays). Unless otherwise indicated, the steps performed by the embodiments need not be inherently related to any particular computer or other device, although this may be the case in some embodiments. In particular, various general purpose machines may be used in conjunction with programs written in accordance with the teachings herein, or it may be more convenient to construct a more specialized device (eg, integrated circuits) to perform the necessary method steps. Thus, embodiments may be implemented in one or more computer programs executing on one or more programmable computer systems, each of which includes at least one processor, at least one storage system (including volatile and nonvolatile storage devices, and/ or storage elements), at least one input device or port, and at least one output device or port. Program code is applied to input data to perform the functions described in this document and generate output information. The output information is applied to one or more output devices in a known manner.

[0151] Each such computer program is preferably stored or loaded onto a storage media or device (e.g., solid-state storage device or media, or magnetic or optical media) readable by a general purpose or special purpose programmable computer for configuring and operating the computer when the storage media or device are read by a computer system to perform the procedures described in this document. The system of the invention can also be considered to be implemented as a computer readable storage medium equipped with a computer program, wherein the storage medium so equipped causes the computer system to operate in a specific and predetermined manner to perform the functions described herein. (Software itself and intangible or volatile signals are excluded to the extent that they constitute non-patentable subject matter.)

[0152] Aspects of the systems described herein may be implemented in a suitable networked audio processing environment using computers to process digital or digitized audio recording files. Portions of an adaptive audio system may include one or more networks that contain any desired number of individual machines, including one or more routers (not shown) that serve to buffer and route data transmitted between computers. Such a network can be built on a variety of network protocols and can be the Internet, a wide area network (WAN), a local area network (LAN), and any combination thereof.

[0153] One or more components, blocks, processes, or other functional components may be implemented by a computer program that controls the operation of a system processor-based computing device. It should also be noted that the various functions disclosed herein may be described using any number of combinations of hardware, firmware and/or data and/or instructions embodied in various machine-readable or computer-readable media, based on characteristics of their behavior, inter-register transfer, logical components and/or other characteristics. Computer-readable media on which such formatted data and/or instructions may be embodied include, but are not limited to, physical (read-only), nonvolatile storage media in various forms, such as optical, magnetic, or semiconductor storage media.

[0154] In the above description, various embodiments of the present invention are illustrated along with examples of how aspects of the present description may be implemented. The above examples and embodiments are not to be considered as mere embodiments, but are presented to illustrate the flexibility and advantages of the present invention as defined in the following claims. Based on the above description and the following claims, other arrangements, embodiments, implementations and equivalents will be apparent to those skilled in the art, and may be embodied without departing from the spirit and scope of the present specification as defined by the claims.

Claims (46)

1. A computer-implemented sound processing method, the method including: receiving a first transform domain signal, wherein the first transform domain signal is a hybrid complex transform domain signal having a plurality of bands, wherein at least one of the plurality of bands has a plurality of subbands, wherein the first transform domain signal has a first plurality of harmonics; generating an upsampled signal in a transform domain by upsampling a first signal in the transform domain, wherein the upsampled signal is a complex valued time domain signal; generating a second transform domain signal based on the upsampled first transform domain signal by: generating a second harmonic set for the upsampled first signal in the transform domain according to a non-linear process, wherein the second signal in the transform domain has a second harmonic set that is different from the first harmonic set; And performing volume expansion with respect to the second plurality of harmonics, wherein the second signal in the transform domain is a complex-valued signal having an imaginary part; filtering the second transform domain signal to divide the second transform domain signal into a plurality of subbands and generate a third transform domain signal, wherein the third transform domain signal has a plurality of bands, wherein at least one of the plurality of bands has a plurality of subbands; And generating a fourth transform domain signal by mixing the third transform domain signal with a delayed version of the first transform domain signal, wherein a given subband of the third transform domain signal is mixed with a corresponding subband of the delayed version of the first transform domain signal. 2. The method of claim 1, wherein the second set of harmonics results in a fourth transform domain signal having perceptually enhanced low frequencies compared to the first transform domain signal. 3. The method according to any of the previous claims, characterized in that the generation of the upsampled signal in the conversion domain is performed according to the synthesis of complex quadrature mirror filtering. 4. Method according to any one of paragraphs. 1–3, characterized in that it additionally includes: performing dynamic processing on the second transform domain signal before generating a third transform domain signal based on the second transform domain signal. 5. Method according to any one of paragraphs. 1–4, characterized in that the plurality of bands of the first signal in the conversion region has a first band, a second band and a third band, wherein the first band is divided into 8 subbands, and the second band is divided into 4 subbands, and the third band is divided into 4 subbands. 6. Method according to any one of paragraphs. 1–5, characterized in that the first signal in the conversion region has 64 bands, the first band being divided into 8 subbands, the second band being divided into 4 subbands, and the third band being divided into 4 subbands. 7. Method according to any one of paragraphs. 1-6, characterized in that the first signal in the conversion domain has a bandwidth of 24 kHz, wherein the first signal in the conversion domain has 64 bands, and the bandwidth in the form of a bandwidth of each band is 375 Hz. 8. Method according to any one of paragraphs. 1–7, characterized in that the nonlinear process includes multiplication of the first signal in the conversion region. 9. Method according to any one of paragraphs. 1-8, characterized in that the nonlinear process involves a feedback delay loop applied to the first signal in the conversion region. 10. Method according to any one of paragraphs. 1–9, characterized in that the generation of the second signal in the conversion region includes: generating a second transform domain signal based on one of a plurality of subbands of the first transform domain signal, wherein one of the plurality of subbands is smaller than all of the plurality of subbands of the first transform domain signal. 11. Method according to any one of paragraphs. 1–9, characterized in that the generation of the second signal in the conversion region includes: generating a plurality of second signals in the transform domain based on two or more of the plurality of subbands of the first signal in the transform domain, wherein two or more of the plurality of subbands are less than all of the plurality of subbands of the first signal in the transform domain, and wherein each of the plurality of second signals in the transform domain transformation corresponds to one of two or more of a plurality of subranges; And generating a second signal in the transform region by summing the plurality of second signals in the transform region. 12. Method according to any one of paragraphs. 1–11, characterized in that it additionally includes: output by the loudspeaker the sound corresponding to the fourth signal in the conversion region. 13. Method according to any one of paragraphs. 1–12, characterized in that the first signal in the conversion region is in the first signal region, wherein the method further includes: receiving an input signal in a second signal region; generating a first signal in the conversion domain by converting an input signal from the second signal domain to the first signal domain; And generating an output signal by converting the fourth signal in a transform region from the first signal region to the second signal region. 14. The method according to claim 13, characterized in that the second transform domain is a time domain, the first signal domain is a hybrid complex quadrature mirror filter (HCQMF) signal domain; wherein generating the first signal in the transform domain includes generating the first signal in the transform domain by performing HCQMF analysis on the input signal; And wherein generating an output signal includes generating an output signal by performing HCQMF synthesis on the fourth signal in the transform domain. 15. Method according to any one of paragraphs. 1–14, characterized in that it additionally includes: discarding the imaginary part from the second signal in the transform region before generating a third signal in the transform region. 16. A non-transitory computer-readable medium storing a computer program that, when executed by a processor, controls a device for executing processing, including the method according to any one of claims. 1–15. 17. A device for processing sound, wherein the device contains: CPU, wherein the processor is configured to control the device to receive the first signal in the transform domain, wherein the first signal in the transform domain is a hybrid complex transform domain signal having a plurality of complex values and a plurality of bands, wherein at least one of the plurality of bands has a plurality of subbands, wherein the first signal in the conversion region has a first plurality of harmonics; wherein the processor is configured to control the device for: generating an upsampled signal in a transform domain by upsampling a first signal in the transform domain, wherein the upsampled signal is a complex valued time domain signal; And generating a second transform domain signal based on the upsampled first transform domain signal by: generating a second harmonic set for the upsampled first signal in the transform domain according to a non-linear process, wherein the second signal in the transform domain has a second harmonic set that is different from the first harmonic set; And performing volume expansion with respect to the second plurality of harmonics, wherein the second signal in the transform domain is a complex-valued signal having an imaginary part; wherein the processor is configured to control the apparatus for filtering the second signal in the transform domain to divide the second signal in the transform domain into a plurality of subbands and generate a third signal in the transform domain, wherein the third signal in the transform domain has a plurality of bands, wherein at least one from a plurality of bands has a plurality of subbands; wherein the processor is configured to control the apparatus to generate a fourth transform domain signal by mixing the third transform domain signal with a delayed version of the first transform domain signal, wherein a given subband of the third transform domain signal is mixed with a corresponding subband of the delayed version of the first signal in the field of transformation. 18. The device according to claim 17, characterized in that it additionally contains: a loudspeaker configured to output the fourth signal in the conversion region in the form of sound.