RU2803638C2 - Processing of spatially diffuse or large sound objects - Google Patents

Abstract

FIELD: computer engineering.

SUBSTANCE: processing audio data. Decorrelation process is performed on the audio signals of large sound objects using the decorrelation component associated with the input interface to create decorrelated audio signals of large sound objects that depend on a certain location of large sound objects and other information with the decorrelated audio signals of large sound objects mutually independent of each other, and the decorrelation process includes adjusting the level of each audio signal by adjusting the level of the corresponding audio gain for each of the audio signals to generate decorrelated audio signals of large sound objects corresponding to the speaker signals supplied for each speaker of the plurality of speakers, the plurality of speakers covering a large spatial area, while the decorrelated audio signals of the large sound objects are mixed with at least one audio signal for other sound objects that are spatially separated by a second distance threshold from the large sound object.

EFFECT: improved accuracy of audio data reproduction.

35 cl, 19 dwg

Description

CROSS REFERENCE TO RELATED APPLICATIONS

[0001] This application claims priority to Spanish Patent Application No. P201331193, filed July 31, 2013, and U.S. Provisional Application No. 61/885,805, filed October 2, 2013, each of which is incorporated herein by reference in its entirety.

FIELD OF TECHNOLOGY TO WHICH THE INVENTION RELATES

[0002] This disclosure relates to audio data processing. In particular, this disclosure relates to the processing of audio data corresponding to diffuse or spatially large audio objects.

BACKGROUND OF THE ART

[0003] Since the advent of sound in films in 1927, there has been a steady advance in the technology used to capture the original intent of a cinematic soundtrack and to reproduce that content. In the 1970s, Dolby introduced a cost-effective means of encoding and distributing mixes with three on-screen channels and a mono surround channel. Dolby provided digital sound for cinema in the 1990s through a 5.1-channel format that provides discrete left, center and right screen channels, a left and right surround matrix, and a subwoofer channel for low-frequency effects. Dolby Surround 7.1, introduced in 2010, increased the number of surround channels by splitting the existing left and right surround channels into four “zones.”

[0004] Audio playback systems for both cinema and home theaters are becoming increasingly versatile and sophisticated. Home theater audio systems include increasingly larger numbers of speakers. As the number of channels increases and speaker placement moves from a flat two-dimensional (2D) matrix to a three-dimensional (3D) matrix that includes elevation, the reproduction of sounds in a playback environment becomes increasingly complex. Improved audio processing techniques would be desirable.

SUMMARY OF THE INVENTION

[0005] Improved methods for processing diffuse or spatially large audio objects are provided. As used herein, the term "audio object" refers to audio signals (also referred to herein as "audio object signals") and associated metadata that may be created or "developed" without reference to any particular playback environment. Associated metadata may include audio object position data, audio object gain data, audio object size data, audio object path data, etc. As used herein, the term "rendering" refers to the process of converting audio objects into speaker input signals for a particular playback environment. . The rendering process may be performed, at least in part, according to associated metadata and according to data from the playing environment. The playing environment data may include indicating a number of speakers in the playing environment and indicating the location of each speaker within the playing environment.

[0006] A spatially large sound object is not intended to be perceived as a point sound source, but should instead be perceived as covering a large spatial area. In some cases, a large sound object must be perceived as surrounding the listener. Such audio effects cannot be achieved by panning alone and may instead require additional processing. To create convincing feature size or spatial diffuseness, a substantial portion of the speaker signals in the playback environment must be mutually independent or at least uncorrelated (eg, independent in terms of first-order cross-correlation or covariance). A sufficiently complex rendering system, such as a movie theater rendering system, may be able to provide such decorrelation. However, less complex rendering systems, such as those designed for home theater systems, may not be able to provide proper decorrelation.

[0007] Some implementations described herein may include identifying diffuse or spatially large audio objects for special processing. A decorrelation process can be performed on audio signals corresponding to large audio objects to create decorrelated audio signals of large audio objects. These decorrelated audio signals of large sound objects can be associated with object locations, which can be stationary or time-varying locations. The association process may be independent of the configuration of the actual playback speakers. For example, decorrelated audio signals of large audio objects can be rendered to virtual speaker locations. In some implementations, the output from such a rendering process may be input to the scene simplification process.

[0008] Accordingly, at least some aspects of this disclosure may be implemented in a method that may include receiving audio data containing audio objects. Audio objects may include audio object signals and associated metadata. The metadata may include at least size data of the audio object.

[0009] The method may include determining, based on the audio object size data, a large audio object having an audio object size that is larger than a threshold size, and performing a decorrelation process on the large audio object audio signals to create decorrelated large audio object audio signals. The method may include associating decorrelated audio signals of large audio objects with object locations. The association process may be independent of the configuration of the actual playback speakers. The configuration of the actual playback speakers can subsequently be used to render decorrelated audio signals of large audio objects to the speakers of the playback environment.

[0010] The method may include receiving decorrelation metadata for a large audio object. The decorrelation process may be performed at least in part according to the decorrelation metadata. The method may include encoding audio data output from the association process. In some implementations, the encoding process may not include encoding decorrelation metadata for the large audio object.

[0011] The object locations may include locations corresponding to at least some of the audio object position data of the received audio objects. At least some of the object locations may be stationary. However, in some implementations, at least some of the object locations may change over time.

[0012] The association process may include rendering decorrelated audio signals of large audio objects according to virtual speaker locations. In some examples, the receiving process may include receiving one or more audio substrate signals corresponding to speaker locations. The method may include mixing decorrelated large audio object audio signals with at least some of the received audio substrate signals or the received audio object signals. The method may include outputting decorrelated large audio object signals as additional audio substrate signals or audio object signals.

[0013] The method may include applying a level control process to decorrelated audio signals of large audio objects. In some implementations, the large audio object metadata may include audio object position metadata, and the level adjustment process may depend, at least in part, on the audio object size metadata and the audio object position metadata of the large audio object.

[0014] The method may include attenuating or removing audio signals of large audio objects after the decorrelation process is performed. However, in some implementations, the method may include storing audio signals corresponding to the point source contribution of a large audio object after the decorrelation process is performed.

[0015] The large audio object metadata may include audio object position metadata. In some such implementations, the method may include calculating contributions from virtual sources within an area or volume of an audio object determined by audio large object position data and audio large object size data. The method may also include determining a set of audio object gain values for each of the plurality of output channels based at least in part on the calculated contributions. The method may include mixing decorrelated audio signals of large audio objects with audio signals for audio objects that are spatially separated by a distance threshold from the large audio object.

[0016] In some implementations, the method may include performing a process of clustering audio objects after a decorrelation process. In some such implementations, the audio object clustering process may be performed after the association process.

[0017] The method may include evaluating audio data to determine a content type. In some such implementations, the decorrelation process may be selectively performed according to the type of content. For example, the amount of decorrelation that must be performed may depend on the type of content. The decorrelation process may include delays, universal filters, pseudo-random filters, and/or reverberation algorithms.

[0018] The methods disclosed herein may be implemented in hardware, firmware, software stored on one or more non-transitory media, and/or combinations thereof. For example, at least some aspects of this disclosure may be implemented in an apparatus that includes an interface system and a logic system. The interface system may include a user interface and/or a network interface. In some implementations, the device may include a memory system. The interface system may include at least one interface between a logic system and a memory system.

[0019] The logic system may include at least one processor, such as a general-purpose single- or multi-chip processor, a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA), or other programmable logic device , a discrete component circuit or transistor logic circuit, discrete hardware components, and/or combinations thereof.

[0020] In some implementations, the logical system may be capable of receiving, through the interface system, audio data containing audio objects. Audio objects may include audio object signals and associated metadata. In some implementations, the metadata includes at least audio object size data. The logic system may be capable of determining, based on the audio object size data, a large audio object having an audio object size that is greater than a threshold size, and performing a decorrelation process on the audio large audio signals to create decorrelated large audio object audio signals. The logic system may be able to associate decorrelated audio signals of large audio objects with object locations.

[0021] The association process may be independent of the configuration of the actual playback speakers. For example, the association process may include rendering decorrelated audio signals of large audio objects according to the locations of virtual speakers. The configuration of the actual playback speakers can subsequently be used to render decorrelated audio signals of large audio objects to the speakers of the playback environment.

[0022] The logic system may be capable of receiving, through the interface system, decorrelation metadata for a large audio object. The decorrelation process may be performed at least in part according to the decorrelation metadata.

[0023] The logic system may be capable of encoding audio data output from the association process. In some implementations, the encoding process may not include encoding decorrelation metadata for the large audio object.

[0024] At least some of the object locations may be stationary. However, at least some of the object locations may change over time. The large audio object metadata may include audio object position metadata. The object locations may include locations corresponding to at least some of the audio object position metadata of the received audio objects.

[0025] The receiving process may include receiving one or more audio substrate signals corresponding to speaker locations. The logic system may be capable of mixing the decorrelated large audio object audio signals with at least some of the received audio substrate signals or the received audio object signals. The logic system may be capable of outputting decorrelated audio signals of large audio objects as additional audio substrate signals or audio object signals.

[0026] The logic system may be able to apply a level control process to decorrelated audio signals of large audio objects. The level adjustment process may depend, at least in part, on audio object size metadata and audio object position metadata of the large audio object.

[0027] The logic system may be able to attenuate or remove the audio signals of large audio objects after the decorrelation process is performed. However, the device may be able to store audio signals corresponding to the point source contribution of a large sound object after the decorrelation process is performed.

[0028] The logic system may be capable of calculating contributions from virtual sources within an area or volume of an audio object determined by the audio large object position data and the audio large object size data. The logic system may be able to determine a set of audio object gain values for each of a plurality of output channels based at least in part on the calculated contributions. The logic system may be capable of mixing decorrelated audio signals of large audio objects with audio signals for audio objects that are spatially separated by a distance threshold from the large audio object.

[0029] The logic system may be able to perform the audio object clustering process after the decorrelation process. In some implementations, the audio object clustering process may be performed after the association process.

[0030] The logic system may be able to evaluate audio data to determine the type of content. The decorrelation process can be selectively performed according to the content type. For example, the amount of decorrelation that must be performed depends on the type of content. The decorrelation process may include delays, universal filters, pseudo-random filters, and/or reverberation algorithms.

[0031] Details of one or more embodiments of the invention described in this specification are set forth in the accompanying drawings and description below. Other features, aspects and advantages will become apparent from the description, drawings and claims. It should be noted that the relative dimensions of the following drawings may not be drawn to scale.

BRIEF DESCRIPTION OF THE DRAWINGS

[0032] Figure 1 depicts an example of a playback environment having a Dolby Surround 5.1 configuration.

[0033] Figure 2 depicts an example of a playback environment having a Dolby Surround 7.1 configuration.

[0034] FIGS. 3A and 3B depict two examples of home theater playback environments that include height speaker configurations.

[0035] FIG. 4A depicts an example graphical user interface (GUI) that illustrates speaker zones at varying elevations in a virtual playback environment.

[0036] FIG. 4B depicts an example of another playback environment.

[0037] FIG. 5 depicts a block diagram that provides an example of audio processing for spatially large audio objects.

[0038] FIGS. 6A through 6F are block diagrams that illustrate example components of an audio processing device capable of processing large audio objects.

[0039] FIG. 7 is a block diagram that depicts an example of a system configured to execute a clustering process.

[0040] FIG. 8 is a block diagram that illustrates an example of a system configured to cluster objects and/or substrates in an adaptive audio processing system.

[0041] FIG. 9 depicts a flow chart that provides an example of a clustering process following a decorrelation process for large audio objects.

[0042] FIG. 10A depicts an example of virtual source locations with respect to the playback environment.

[0043] FIG. 10B depicts an alternative example of virtual source locations relative to the playback environment.

[0044] FIG. 11 is a block diagram that provides examples of components of an audio processing device.

[0045] Like reference numerals and designations in the various drawings indicate like elements.

DESCRIPTION OF IMPLEMENTATION OPTIONS

[0046] The following description is directed to specific implementations for the purpose of describing some of the innovative aspects of this disclosure, as well as examples of contexts in which these innovative aspects may be implemented. However, the principles described here can be applied in various other ways. For example, although various implementations are described with respect to specific playback environments, the principles described herein may be broadly applicable to other known playback environments, as well as playback environments that may be introduced in the future. In addition, the described embodiments may be implemented, at least in part, in various devices and systems as hardware, software, firmware, cloud systems, etc. Accordingly, the principles of this disclosure are not intended to be limited to the embodiments shown in the drawings and/or described herein, but instead have broad applicability.

[0047] Figure 1 depicts an example of a playback environment having a Dolby Surround 5.1 configuration. In this example, the playback environment is a movie theater playback environment. Dolby Surround 5.1 was developed in the 1990s, but this configuration is still widely used in home and theater playback environments. In a movie theater playback environment, projector 105 may be configured to project video images, such as a movie, onto screen 150. Audio data may be synchronized with the video images and processed by audio processor 110. Power amplifiers 115 may provide speaker signals to speakers of playback environment 100.

[0048] The Dolby Surround 5.1 configuration includes a left surround channel 120 for a left surround matrix 122 and a right surround channel 125 for a right surround matrix 127 . The Dolby Surround 5.1 configuration also includes a left channel 130 for the left speaker array 132, a center channel 135 for the center speaker array 137, and a right channel 140 for the right speaker array 142. In a movie theater environment, these channels may be referred to as the left screen channel, the center screen channel, and the right screen channel, respectively. A separate low frequency effects (LFE) channel 144 is provided for subwoofer 145.

[0049] In 2010, Dolby provided an improvement to digital cinema audio with the introduction of Dolby Surround 7.1. FIG. 2 depicts an example of a playback environment having a Dolby Surround 7.1 configuration. Digital projector 205 may be configured to receive digital video data and to project video images onto screen 150. Audio data may be processed by audio processor 210. Power amplifiers 215 may provide speaker signals to speakers of playback environment 200.

[0050] Like Dolby Surround 5.1, the Dolby Surround 7.1 configuration includes a left channel 130 for the left speaker array 132, a center channel 135 for the center speaker array 137, a right channel 140 for the right speaker array 142, and an LFE channel 144 for the subwoofer 145. The Dolby Surround 7.1 configuration includes a left side surround (Lss) matrix 220 and a right side surround (Rss) matrix 225, each of which can be driven by one channel.

[0051] However, Dolby Surround 7.1 increases the number of surround channels by splitting the left and right surround channels of Dolby Surround 5.1 into four zones: in addition to the left side surround matrix 220 and the right side surround matrix 225, separate channels are included for the left rear speakers 224 surround sound (Lrs) and right rear speakers 226 surround sound (Rrs). An increased number of surround zones within the playback environment 200 can significantly improve sound localization.

[0052] In an attempt to create a more immersive environment, some playback environments may be configured with increased numbers of speakers driving increased numbers of channels. In addition, some playback environments may include speakers placed at various elevations, some of which may be "height speakers" configured to produce sound from an area that is higher than the seating area of the playback environment.

[0053] FIGS. 3A and 3B depict two examples of home theater playback environments that include height speaker configurations. In these examples, playback environments 300a and 300b include the main features of a Dolby Surround 5.1 configuration, including a left surround speaker 322, a right surround speaker 327, a left speaker 332, a right speaker 342, a center speaker 337, and a subwoofer 145. However, The playback environment 300 includes an extension of the Dolby Surround 5.1 configuration for height speakers, which may be referred to as a Dolby Surround 5.1.2 configuration.

[0054] FIG. 3A depicts an example of a playback environment having height speakers mounted on the ceiling 360 of a home theater playback environment. In this example, the playing environment 300a includes a height speaker 352 that is in the left upper middle (Ltm) position, and a height speaker 357 that is in the right upper middle (Rtm) position. In the example depicted in FIG. 3B, left speaker 332 and right speaker 342 are Dolby elevated speakers that are configured to reflect sound from the ceiling 360. When properly configured, the reflected sound can be perceived by listeners 365 as if the sound source were on the ceiling 360 However, the number and configuration of speakers are provided as an example only. Some current home theater implementations provide up to 34 speaker positions, and proposed home theater implementations may allow for even more speaker positions.

[0055] Accordingly, the current trend is to include not only more speakers and more channels, but also to include speakers at different heights. As the number of channels increases and speaker placement moves from 2D to 3D, the challenges of positioning and rendering sounds become increasingly complex.

[0056] Accordingly, Dolby has developed various tools, including, but not limited to, user interfaces that increase functionality and/or reduce development complexity for a 3D audio sound system. Some such tools can be used to create audio objects and/or metadata for audio objects.

[0057] FIG. 4A depicts an example graphical user interface (GUI) that illustrates speaker zones at varying elevations in a virtual playback environment. The GUI 400 may, for example, be displayed on a display device according to instructions from a logic system, according to signals received from user input devices, etc. Some such devices are described below with reference to FIG. 11.

[0058] As used herein with reference to virtual playback environments, such as virtual playback environment 404, the term “speaker region” generally refers to a logical structure that may or may not have a one-to-one correspondence with the actual speaker of the playback environment. For example, a “speaker area location” may or may not correspond to a specific speaker location of a movie theater playing environment. Instead, the term "speaker area location" may refer generally to an area of the virtual playback environment. In some implementations, the speaker area of the virtual playback environment may correspond to a virtual speaker, for example, by using a virtualization technology such as Dolby Headphone™ (sometimes referred to as Mobile Surround™), which creates a virtual surround sound environment in real time using a set of two-channel stereo headphones. In the GUI 400, there are seven speaker zones 402a on the first elevation and two speaker zones 402b on the second elevation, for a total of nine speaker zones in the virtual playback environment 404. In this example, speaker zones 1-3 are in the front region 405 of the virtual playback environment 404. The front region 405 may correspond, for example, to an area of a movie theater playing environment in which the screen 150 is located, an area of a home in which a television screen is located, etc.

[0059] Here, speaker zone 4 corresponds generally to speakers in the left region 410, and speaker zone 5 corresponds to speakers in the right region 415 of the virtual playback environment 404. Speaker zone 6 corresponds to the left rear region 412, and speaker zone 7 corresponds to the right rear region 414 of the virtual playback environment 404. Speaker zone 8 corresponds to speakers in the upper region 420a, and speaker zone 9 corresponds to speakers in the upper region 420b, which may be a virtual ceiling region. Accordingly, the locations of speaker zones 1 to 9, which are shown in FIG. 4A, may or may not correspond to the locations of actual speakers in the playback environment. Additionally, other implementations may include more or fewer speaker and/or raised areas.

[0060] In various implementations described herein, a user interface, such as GUI 400, may be used as part of a development tool and/or rendering tool. In some implementations, the development tool and/or rendering tool may be implemented using software stored on one or more non-transitory media. The development tool and/or rendering tool may be implemented (at least in part) by hardware, firmware, etc., such as the logic system and other devices described below with reference to FIG. 11. In some implementations of the development, an associated authoring tool may be used to create metadata for the associated audio data. The metadata may, for example, include data indicating the position and/or path of an audio object in three-dimensional space, speaker area limitation data, etc. The metadata may be created with respect to the speaker areas 402 of the virtual playback environment 404 rather than with respect to a specific placement actual speakers of the playback environment. The rendering tool can receive audio data and associated metadata and can calculate audio gains and speaker inputs for the playback environment. Such audio gains and speaker inputs can be calculated according to an amplitude panning process, which can make the sound appear to be coming from position P in the playback environment. For example, speaker input signals may be provided to speakers 1 to N of the playback environment according to the following equation:

[0061] x _i (t)=g _i x(t) , i= 1,..., N (Equation 1)

[0062] In Equation 1, x _i (t) represents the speaker input signal to be applied to speaker i , g _i represents the gain of the corresponding channel, x(t) represents the audio signal, and t represents time. The gains can be determined, for example, according to the amplitude panning techniques described in section 2 on pages 3-4 of V. Pulkki's "Displacement compensation of virtual sources after amplitude panning" (Audio Engineering Society (AES), International Conference on Virtual, synthetic and entertainment audio), which is incorporated herein by reference. In some implementations, the gains may be frequency dependent. In some implementations, the time delay may be represented by replacing x(t) with x(t-Δt) .

[0063] In some rendering implementations, audio playback data created with references to speaker zones 402 may be mapped to speaker locations of a wide range of playback environments, which may be a Dolby Surround 5.1 configuration, a Dolby Surround 7.1 configuration, a Hamasaki 22.2 configuration, or another configuration. For example, with reference to FIG. 2, the rendering tool may map audio playback data for speaker zones 4 and 5 into a left-side surround matrix 220 and a right-side surround matrix 225 of a playback environment having a Dolby Surround 7.1 configuration. Audio playback data for speaker zones 1, 2, and 3 may be displayed on the left screen channel 230, the right screen channel 240, and the center screen channel 235, respectively. Audio playback data for speaker zones 6 and 7 can be displayed to the left surround back speakers 224 and the right surround back speakers 226.

[0064] FIG. 4B depicts an example of another playback environment. In some implementations, the rendering tool may display audio playback data for speaker zones 1, 2, and 3 for the corresponding speakers 455 of the playback environment 450 screen. The rendering tool may map audio playback data for speaker zones 4 and 5 to left-side surround matrix 460 and right-side surround matrix 465, and can map audio playback data for speaker zones 8 and 9 to left-height speakers 470a and right-height speakers 470b. Audio playback data for speaker zones 6 and 7 may be displayed to the left surround back speakers 480a and the right surround back speakers 480b.

[0065] In some development implementations, development tools can be used to create metadata for audio objects. Metadata may indicate the 3D position of an object, rendering constraints, content type (such as dialogue, effects, etc.), and/or other information. Depending on the implementation, the metadata may include other types of data, such as width data, gain data, path data, etc. Some audio objects may be static, while others may move.

[0066] Audio objects are rendered according to their associated metadata, which generally includes positional metadata indicating the position of the audio object in three-dimensional space at some given point in time. When audio objects are tracked or played in the playback environment, the audio objects are rendered according to positional metadata using the speakers that are present in the playback environment, rather than being output to a predefined physical channel, as is the case with traditional channel-based systems such as Dolby 5.1 and Dolby 7.1.

[0067] In addition to positional metadata, other types of metadata may be necessary to create proper audio effects. For example, in some implementations, metadata associated with an audio object may indicate the size of the audio object, which may also be referred to as "width". Size metadata can be used to indicate the spatial area or volume occupied by an audio object. A spatially large sound object must be perceived as covering a large spatial area, not just as a point sound source having a location defined only by the sound object's position metadata. In some cases, for example, a large sound object must be perceived as occupying a significant part of the playing environment, perhaps even surrounding the listener.

[0068] The human hearing system is very sensitive to changes in the correlation or coherence of signals entering both ears, and maps this correlation to the perceived size of an object attribute if the normalized correlation is less than a value of +1. Thus, in order to create convincing feature size, or spatial diffuseness, a significant portion of the speaker signals in the playback environment must be mutually independent or at least uncorrelated (eg, independent in terms of first-order cross-correlation or covariance). A satisfactory decorrelation process is usually quite complex and typically involves time-varying filters.

[0069] A cinematic soundtrack may include hundreds of objects, each of which has position metadata, size metadata, and possibly other spatial metadata associated with it. In addition, a movie theater sound system may include hundreds of loudspeakers that may be individually controlled to provide satisfactory perception of the locations and sizes of sound objects. In a movie theater, therefore, hundreds of objects may be played by hundreds of loudspeakers, and the mapping from object to loudspeaker signal consists of a very large matrix of panning coefficients. When the number of objects is represented as M and the number of speakers is represented as N, this matrix has up to M*N elements. This creates complications for reproducing diffuse or large objects. To create a convincing feature size, or spatial diffuseness, a significant portion of the N loudspeaker signals must be mutually independent or at least uncorrelated. This generally involves the use of many (up to N) independent decorrelation processes, causing significant CPU overhead to the rendering process. Additionally, the amount of decorrelation can be different for each object, further complicating the rendering process. A sufficiently complex rendering system, such as a commercial theater rendering system, may be able to provide such decorrelation.

[0070] However, less complex rendering systems, such as those designed for home theater systems, may not be able to provide sufficient decorrelation. Some such rendering systems do not have decorrelation capabilities at all. Decorrelation programs that are simple enough to run on a home theater system may introduce artifacts. For example, comb filter artifacts may be introduced if a low complexity decorrelation process is followed by a downmixing process.

[0071] Another potential problem is that in some applications, object-based audio is delivered in the form of a backwards-compatible mix (such as Dolby Digital or Dolby Digital Plus) supplemented with additional information to extract one or more objects from the backwards-compatible mix . A backwards-compatible mix will typically not include decorrelation. In some such systems, object restoration can only work reliably if the backwards-compatible mix was created using simple panning procedures. The use of decorrelation tools in such processes can damage the audio object restoration process, sometimes significantly. In the past, this meant that it was easy to choose not to apply decorrelation to a backward-compatible mix, which would compromise the original intent of that mix, or to accept the deterioration of the object restoration process.

[0072] In order to overcome such potential problems, some implementations described herein include identifying diffuse or spatially large audio objects for special processing. Such methods and devices may be particularly suitable for audio data to be rendered in a home theater environment. However, these methods and devices are not limited to home theater use, but instead have wide application potential.

[0073] Due to their spatially diffuse nature, objects with a large size are not perceived as point sources with a compact and specific location. Thus, multiple speakers are used to reproduce such spatially diffuse objects. However, the precise locations of speakers in the playback environment that are used to reproduce large sound objects are less critical than the locations of speakers used to reproduce compact, small sound objects. Accordingly, high-quality reproduction of large audio objects is possible without prior knowledge of the configuration of the actual playback speakers, which is subsequently used to render decorrelated large audio object signals to the actual speakers of the playback environment. Therefore, decorrelation processes for large audio objects can be performed earlier, prior to the process of rendering audio data for playback in a playback environment, such as a home theater system, for listeners. In some examples, decorrelation processes for large audio objects are performed before audio data is encoded for transmission to such playback environments.

[0074] Such implementations do not require the playback environment renderer to have high complexity decorrelation capabilities, thereby allowing for rendering processes that can be relatively simpler, more efficient, and less expensive. Backward-compatible downmixes can include a decorrelation effect to preserve as much of the author's intent as possible without the need to reconstruct the object for decorrelation on the rendering side. High-quality decorrelation tools can be applied to large audio objects earlier than the final rendering process, for example during the development or post-production process in a recording studio. Such decorrelation means may be robust to downmixing and/or other subsequent audio processing.

[0075] FIG. 5 depicts a block diagram that provides an example of audio processing for spatially large audio objects. The operations of method 500, like other methods described herein, are not necessarily performed in the specified order. In addition, these methods may include more or fewer blocks than are shown and/or described. These methods may be implemented, at least in part, by a logic system, such as logic system 1110 shown in FIG. 11 and described below. Such a logic system may be a component of an audio processing system. Alternatively or in addition, such methods may be carried out via a non-transitory medium having software stored thereon. The software may include instructions for controlling one or more devices to perform, at least in part, the methods described herein.

[0076] In this example, method 500 begins at block 505, which includes receiving audio data including audio objects. The audio data may be received by an audio processing system. In this example, audio objects include audio object signals and associated metadata. Here, the associated metadata includes size data of the audio object. The associated metadata may also include audio object position data indicating the audio object's position in three-dimensional space, decorrelation metadata, audio object gain information, etc. The audio data may also include one or more audio substrate signals corresponding to speaker locations.

[0077] In this implementation, block 510 includes determining, based on the audio object size data, a large audio object having an audio object size that is larger than a threshold size. For example, block 510 may include determining whether the numerical size value of an audio object exceeds a predetermined level. The numerical size value of the sound object may, for example, correspond to the portion of the playing environment occupied by the sound object. Alternatively or in addition, block 510 may include determining whether another type of indication, such as a flag, decorrelation metadata, etc., indicates that the audio object has an audio object size that is greater than a threshold size. While much of the discussion of method 500 involves processing a single large audio object, it should be understood that the same (or similar) processes may be applied to multiple large audio objects.

[0078] In this example, block 515 includes performing a decorrelation process on the LOB audio signals, producing decorrelated LOB audio signals. In some implementations, the decorrelation process may be performed at least in part according to received decorrelation metadata. The decorrelation process may include delays, universal filters, pseudo-random filters, and/or reverberation algorithms.

[0079] Here at block 520, decorrelated large audio object audio signals are associated with object locations. In this example, the association process is independent of the configuration of the actual playback speakers, which can be subsequently used to render decorrelated audio signals of large audio objects to the actual playback speakers of the playback environment. However, in some alternative implementations, the locations of the objects may correspond to the locations of the actual playback speakers. For example, according to some such alternative implementations, the locations of the objects may correspond to the locations of the playback speakers of commonly used playback speaker configurations. If the audio cues are received at block 505, the object locations may correspond to playback speaker locations corresponding to at least some of the audio cues. Alternatively or in addition, the object locations may be locations corresponding to at least some of the sound object position data of the received sound objects. Accordingly, at least some of the object locations may be stationary, while at least some of the object locations may change over time. In some implementations, block 520 may include mixing decorrelated audio signals of large audio objects with audio signals for audio objects that are spatially separated by a threshold distance from the large audio object.

[0080] In some implementations, block 520 may include rendering decorrelated audio signals of large audio objects according to virtual speaker locations. Some such implementations may include calculating contributions from virtual sources within an area or volume of an audio object determined by audio large object position data and audio large object size data. Such implementations may include determining a set of audio object gain values for each of the plurality of output channels based at least in part on the calculated contributions. Some examples are described below.

[0081] Some implementations may include encoding audio data emerging from the association process. According to some such implementations, the encoding process includes encoding audio object signals and associated metadata. In some implementations, the encoding process includes a data compression process. The data compression process can be lossless or lossy. In some implementations, the data compression process includes a quantization process. According to some examples, the encoding process does not include encoding decorrelation metadata for the large audio object.

[0082] Some implementations include performing an audio object clustering process, also referred to herein as a "scene simplification" process. For example, an audio object clustering process may be part of block 520. For implementations that include encoding, the encoding process may include encoding audio data that is output from the audio object clustering process. In some such implementations, the audio object clustering process may be performed after the decorrelation process. Additional examples of processes corresponding to blocks of method 500 including scene simplification processes are provided below.

[0083] FIGS. 6A through 6F are block diagrams that illustrate example components of audio processing systems that have the ability to process large audio objects as described herein. These components may, for example, correspond to logic system modules for an audio processing system, which may be implemented in hardware, firmware, software stored on one or more non-transitory media, or combinations thereof. The logic system may include one or more processors, such as general-purpose single- or multi-chip processors. The logic system may include a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic device, discrete or transistor logic, discrete hardware components, and/or combinations thereof. .

[0084] In FIG. 6A, audio processing system 600 is capable of detecting large audio objects, such as large audio object 605. The detection process may be substantially similar to one of the processes described with reference to block 510 of FIG. 5. In this example, audio large object signals 605 are decorrelated by decorrelation system 610 to create decorrelated large object signals 611. Decorrelation system 610 may perform a decorrelation process, at least in part, according to received decorrelation metadata for large audio object 605. The decorrelation process may include one or more of delays, universal filters, pseudo-random filters, or reverberation algorithms.

[0085] The audio processing system 600 also has the ability to receive other audio signals, which are other audio objects and/or substrates 615 in this example. Here, the other sound objects are sound objects that have a size that is less than a threshold size for characterizing the sound object as a large sound object.

[0086] In this example, the audio processing system 600 has the ability to associate decorrelated audio signals of large audio objects 611 with the locations of other objects. Object locations may be stationary or may change over time. The association process may be similar to one or more of the processes described above with reference to block 520 of FIG. 5.

[0087] The association process may include a mixing process. The mixing process may be based, at least in part, on the distance between the location of a large audio object and the location of another object. In the implementation depicted in FIG. 6A, audio processing system 600 is capable of mixing decorrelated large audio object signals 611 with at least some audio signals corresponding to audio objects and/or substrates 615. For example, audio processing system 600 may be capable of mixing decorrelated audio signals. 611 large sound objects with audio signals to other sound objects that are spatially separated by a distance threshold from the large sound object.

[0088] In some implementations, the association process may include a rendering process. For example, the association process may include rendering decorrelated audio signals of large audio objects according to the locations of virtual speakers. Some examples are described below. After the rendering process, there may be no need to store the audio signals corresponding to the large audio object that were received by the decorrelation system 610. Accordingly, audio processing system 600 may be configured to attenuate or remove audio signals of large audio objects 605 after the decorrelation process is performed by decorrelation system 610. Alternatively, the audio processing system 600 may be configured to store at least a portion of the audio signals of the large audio objects 605 (eg, audio signals corresponding to the point source contribution of the large audio object 605) after the decorrelation process is performed.

[0089] In this example, audio processing system 600 includes an encoder 620 that is capable of encoding audio data. Here, encoder 620 is configured to encode audio data after the association process. In this implementation, encoder 620 has the ability to apply a data compression process to the audio data. The encoded audio data 622 may be stored and/or transmitted to other audio processing systems for subsequent processing, playback, etc.

[0090] In the implementation depicted in FIG. 6B, the audio processing system 600 has level control capability. In this example, the level control system 612 is configured to control the levels of the output signals of the decorrelation system 610. The level adjustment process may depend on the metadata of audio objects in the source content. In this example, the level adjustment process depends at least in part on the size metadata of the audio object and the audio object position metadata of the large audio object 605. Such level adjustment can be used to optimize the distribution of the output of the decorrelation means to other audio objects, such as audio objects and/or substrate 615. The choice may be made to mix the output signals of the decorrelation means to signals of another object that are spatially distant, in order to improve the spatial diffuseness of the resulting rendering.

[0091] Alternatively or in addition, a level control process can be used to ensure that sounds corresponding to the decorrelated large sound object 605 are reproduced only by speakers from a particular direction. This can only be accomplished by adding decorrelation means outputs to objects near the desired direction or location. In such implementations, position metadata of the large sound object 605 is taken into account during the level adjustment process in order to preserve information regarding the perceived direction from which its sounds originate. Such implementations may be suitable for objects of intermediate size, for example, for audio objects that are considered large, but not so large that their size includes the entire playback environment.

[0092] In the implementation depicted in FIG. 6C, audio processing system 600 has the ability to create additional substrate objects or channels during the decorrelation process. Such functionality may be desired, for example, if other audio objects and/or substrates 615 are not suitable or optimal. For example, in some implementations, decorrelated large audio object signals 611 may correspond to virtual speaker locations. If other audio objects and/or substrates 615 do not correspond to positions that are close enough to the desired virtual speaker locations, decorrelated large audio object signals 611 may correspond to new virtual speaker locations.

[0093] In this example, the large audio object 605 is first processed by the decorrelation system 610. Subsequently, additional objects or substrate channels corresponding to the decorrelated blob signals 611 are provided to the encoder 620. In this example, the decorrelated blob signals 611 are subject to level adjustment before being sent to the encoder 620. The decorrelated blob signals 611 may be substrate channel signals and/or signals of sound objects, the latter of which can correspond to static or moving objects.

[0094] In some implementations, the audio signals output to encoder 620 may also include at least some of the original large audio object signals. As noted above, the audio processing system 600 may be able to store audio signals corresponding to the point source contribution of the large audio object 605 after the decorrelation process is performed. This may provide advantages, for example, because different signals may be correlated with each other to varying degrees. Thus, it may be useful to skip at least a portion of the original audio signal corresponding to the large audio object 605 (eg, a point source contribution) and render it separately. In such implementations, alignment of the decorrelated signals and the original signals corresponding to the large audio object 605 may be advantageous.

[0095] One such example is shown in Fig. 6D. In this example, at least some of the original audio object signals 613 are subjected to a first equalization process by the level control system 612a, and the decorrelated audio object signals 611 are subject to an equalization process by the level control system 612b. Here, level control system 612a and level control system 612b provide audio outputs to encoder 620. The outputs of level control system 612b are also mixed with other audio objects and/or substrates 615 in this example.

[0096] In some implementations, audio processing system 600 may be able to evaluate input audio data to determine (or at least approximate) the type of content. The decorrelation process may be based, at least in part, on the type of content. In some implementations, the decorrelation process may be selectively performed according to content type. For example, the amount of decorrelation that must be performed on the input audio data may depend, at least in part, on the type of content. For example, it is generally desirable to reduce the amount of decorrelation for speech.

[0097] One example is shown in Fig. 6E. In this example, the media intelligence system 625 has the ability to evaluate audio signals and approximate the type of content. For example, the media intelligence system 625 may be able to evaluate audio signals corresponding to large audio objects 605 and approximate whether the type of content is speech, music, sound effects, etc. In the example depicted in FIG. 6E, the media intelligence system 625 has the ability to send control signals 627 to control the amount of decorrelation or process the size of an object according to an approximate determination of the content type.

[0098] For example, if the media intelligence system 625 approximately determines that the audio signals of large audio objects 605 correspond to speech, the media intelligence system 625 may send control signals 627 indicating that the amount of decorrelation for these signals should be reduced or that these signals should not be decorrelated. Various methods for automatically determining the probability that a signal is a speech signal can be used. According to one embodiment, the media intelligence system 625 may include a speech probability approximate means that is capable of generating a speech probability value based at least in part on center channel audio information. Some examples are described by Robinson and Winton in “Automated Speech/Other Discrimination for Loudness Tracking” (Society of Audio Engineering, Conference 118 Advance Publication No. 6437, May 2005).

[0099] In some implementations, control signals 627 may indicate the amount of level adjustment and/or may indicate parameters for mixing decorrelated large audio object signals 611 with audio signals for audio objects and/or substrates 615.

[00100] Alternatively or in addition, the amount of decorrelation for a large audio object may be based on "stems", "tags" or other direct indications of the content type. Such direct indications of the content type may, for example, be created by the content creator (eg, during a post-processing process) and transmitted as metadata with associated audio signals. In some implementations, such metadata may be human readable. For example, a human-readable stem or tag may explicitly state, in essence, “this is dialogue,” “this is a special effect,” “this is music,” etc.

[00101] Some implementations may include a clustering process that combines objects that are similar in some respects, such as in terms of spatial location, spatial size, or content type. Some examples of clustering are described below with reference to FIGS. 7 and 8. In the example depicted in FIG. 6F, objects and/or substrates 615a are introduced into a clustering process 630. A smaller number of objects and/or substrates 615b are output from the clustering process 630. Audio data corresponding to objects and/or substrates 615b is mixed with aligned decorrelated large audio object signals 611. In some alternative implementations, the clustering process may follow the decorrelation process. One example is described below with reference to FIG. 9. Such implementations may, for example, prevent dialogue from being mixed into a cluster with unwanted metadata, such as an off-center speaker position or a large cluster size.

Simplifying a Scene by Clustering Objects

[00102] For purposes of the following description, the terms "clustering" and "grouping" or "combining" are used interchangeably to describe the combination of objects and/or substrates (channels) to reduce the amount of data in a block of adaptive audio content for transmission and rendering in an adaptive audio playback system ; and the term "reduce" may be used to refer to the act of performing simplification of an adaptive audio scene by such clustering of objects and substrates. The terms "clustering", "grouping" or "combining" throughout this description are not strictly limited to uniquely assigning an object or substrate channel to just one cluster, but instead an object or substrate channel may be distributed across more than one output substrate or cluster using weights or gain vectors that determine the relative contribution of the target or substrate signal to the output cluster or substrate output.

[00103] In one embodiment, an adaptive audio system includes at least one component configured to reduce the bandwidth of object-based audio content by clustering objects and sensory-transparent simplifications of spatial scenes created by a combination of channel substrates and objects. The object clustering process performed by the component(s) uses specific information about objects, which may include spatial position, object content type, temporal attributes, object size, and/or the like, to reduce the complexity of a spatial scene by grouping similar objects into object clusters , which replace the original objects.

[00104] Additional audio processing for standard audio encoding to distribute and render compelling user experiences based on the original complex substrate and audio tracks is generally referred to as scene simplification and/or object clustering. The main purpose of this processing is to reduce the spatial scene through clustering or grouping techniques that reduce the number of individual audio elements (substrates and objects) that must be provided to the playback device, but which still retain enough spatial information so that the perceived difference between the originally designed content and rendered output have been minimized.

[00105] The scene simplification process may render object-plus-substrate content in reduced bandwidth channels or encoding systems using object information such as spatial position, temporal attributes, content type, size, and/or other appropriate characteristics for dynamic clustering of objects to a reduced number. This process can reduce the number of objects by performing one or more of the following clustering operations: (1) clustering objects into objects; (2) clustering an object with substrates; and (3) clustering objects and/or substrates into objects. Additionally, an object may be distributed across two or more clusters. A process can use temporary information about objects to control the clustering and declustering of objects.

[00106] In some implementations, clusters of objects replace the individual waveforms and metaelements of the composite object data with a single equivalent waveform and set of metadata so that data for N objects is replaced with data for a single object, thereby essentially compressing the object data from N to 1. Alternatively or in addition, an object or substrate channel can be distributed across more than one cluster (for example, using amplitude panning techniques), reducing the object data from N to M, where M < N. The clustering process can use an error metric of based on distortion due to changes in location, volume, or other characteristic of the clustered objects to determine the trade-off between cluster compression and audio degradation of the clustered objects. In some embodiments, the clustering process may be performed synchronously. Alternatively or in addition, the clustering process may be event driven, such as through audio scene analysis (ASA) and/or edge detection events to control object simplification through clustering.

[00107] In some embodiments, the process may leverage knowledge of endpoint rendering algorithms and/or clustering control devices. Thus, specific characteristics or properties of the playback device can be used to inform the clustering process. For example, different clustering schemes may be used for speakers as opposed to headphones or other audio drivers, or different clustering schemes may be used for lossless encoding as opposed to lossy encoding, and so on.

[00108] FIG. 7 is a block diagram that depicts an example of a system configured to execute a clustering process. As depicted in FIG. 7, system 700 includes encoder 704 and decoder stages 706 that process input audio signals to produce reduced-bandwidth output audio signals. In some implementations, portion 720 and portion 730 may be in different locations. For example, portion 720 may correspond to a post-production development system, and portion 730 may correspond to a playout environment, such as a home theater system. In the example depicted in FIG. 7, a portion 709 of the input signals is processed through known compression techniques to create a compressed audio bitstream 705. The compressed audio bitstream 705 may be decoded by a decoder step 706 to create at least a portion of the output signal 707. Such known compression techniques may include analyzing the input audio content 709, quantizing the audio data, and then performing compression techniques such as masking, etc. , over the audio data itself. Compression techniques may be lossy or lossless and may be implemented in systems that can provide the user with a choice of compressed bandwidth such as 192 kbps, 256 kbps, 512 kbps, etc.

[00109] In an adaptive audio system, at least a portion of the input audio comprises input signals 701 that include audio objects, which in turn include audio object signals and associated metadata. Metadata defines the exact characteristics of the associated audio content, such as the spatial position of the object, the size of the object, the type of content, volume, and so on. Any practical number of audio objects (eg hundreds of objects) can be processed through the system for playback. To ensure accurate playback of multiple objects across a wide variety of different playback systems and media, system 700 includes a clustering process or component 702 that reduces the number of objects to a smaller, more manageable number of objects by combining the original objects into fewer groups of objects.

[00110] The clustering process thus builds object groups to create a smaller number of output groups 703 from the original set of individual input objects 701. The clustering process 702 essentially processes the object metadata as well as the audio data itself to create a reduced number of object groups. The metadata can be analyzed to determine which objects are most appropriately combined with other objects at any given time, and the corresponding audio waveforms for the combined objects can be summed together to create a replacement or combined object. In this example, the combined groups of objects are then input to encoder 704, which is configured to generate a bitstream 705 containing audio and metadata for transmission to decoder 706.

[00111] In general, an adaptive audio system employing an object clustering process 702 includes components that generate metadata from a source spatial audio format. System 700 includes an audio processing system portion configured to process one or more bitstreams containing both standard channel-based audio elements and audio object encoding elements. An extension layer containing audio object coding elements may be added to the channel-based audio codec bitstream or to the audio object bitstream. Accordingly, in this example, bitstreams 705 include an enhancement layer that must be rendered for use with existing speaker and driver designs or next generation speakers employing individually addressable drivers and driver definitions.

[00112] Spatial audio content from a spatial audio processor may include audio objects, channels, and position metadata. When an object is rendered, it can be assigned to one or more speakers according to the position metadata and the location of the playback speaker. Additional metadata, such as size metadata, may be associated with the object to change the playback location or otherwise limit the speakers that should be used for playback. Metadata can be generated on the audio workstation in response to the engineer's mixing inputs to provide rendering of cues that control spatial parameters (e.g. position, size, speed, intensity, timbre, etc.) and determine which driver(s) or the speaker(s) in the listening environment play the appropriate sounds during the performance. The metadata may be associated with corresponding audio data at the workstation for packaging and transport by the spatial audio processor.

[00113] FIG. 8 is a block diagram that illustrates an example of a system configured to cluster objects and/or substrates in an adaptive audio processing system. In the example depicted in FIG. 8, object processing component 806, which is capable of performing scene simplification tasks, reads an arbitrary number of input audio files and metadata. The input audio files contain input objects 802 and associated object metadata and may include mattes 804 and associated matte metadata. This input file/metadata thus corresponds to either "matte" or "object" tracks.

[00114] In this example, object processing component 806 has the ability to combine media intelligence/content classification, spatial distortion analysis, and object selection/clustering information to create fewer output objects and substrate tracks. In particular, objects can be clustered together to create new equivalent objects or clusters 808 of objects with associated object/cluster metadata. Objects can also be selected for downmixing into mattes. This is shown in FIG. 8 as the output of downmixed objects 810 input to renderer 816 to combine 818 with mattes 812 to generate output matte objects and associated metadata 820. The output matte configuration 820 (e.g., Dolby 5.1 configuration) is optional. must match the input configuration of the substrate, which for example could be 9.1 for an Atmos cinema. In this example, new metadata is generated for the output tracks by combining metadata from the input tracks, and new audio data is also generated for the output tracks by combining audio from the input tracks.

[00115] In this implementation, the object processing component 806 has the ability to use specific processing of configuration information 822. Such processing of configuration information 822 may include the number of output objects, frame size, and specific media intelligence settings. Multimedia intelligence may include determining parameters or characteristics of objects (or associated with them), such as content type (i.e. dialogue/music/effects/etc.), domains (segment/classification), preprocessing, results acoustic analysis of the scene and other similar information. For example, object processing component 806 may be able to determine which audio signals correspond to speech, music, and/or special effects sounds. In some implementations, the object processing component 806 is capable of determining at least some of these characteristics by analyzing the audio signals. Alternatively or in addition, object processing component 806 may be able to determine at least some of such characteristics according to associated metadata such as tags, labels, etc.

[00116] In an alternative embodiment, audio generation can be delayed by storing a reference to all source tracks as well as simplification metadata (eg, which objects belong to which cluster, which objects should be rendered for substrates, etc.). Such information may, for example, be useful for distributing the scene simplification process between the studio and the encoding location or other similar scenarios.

[00117] FIG. 9 depicts a flow chart that provides an example of a clustering process following a decorrelation process for large audio objects. The blocks of the audio processing system 600 may be implemented by any suitable combination of hardware, firmware, software stored on non-transitory media, etc. For example, the blocks of the audio processing system 600 may be implemented by a logic system and/or other elements, such as those described below with reference to FIG. 11.

[00118] In this implementation, audio processing system 600 receives audio data that includes audio objects O ₁ to O _M . Here, the audio objects include audio object signals and associated metadata including at least audio object size metadata. The associated metadata may also include audio object position metadata. In this example, large object detection module 905 is capable of determining, based at least in part on audio object size metadata, large audio objects 605 that have a size that is greater than a threshold size. Large object detection module 905 may operate, for example, as described above with reference to block 510 of FIG. 5.

[00119] In this implementation, module 910 is capable of performing a decorrelation process on audio large object signals 605 to create decorrelated audio large object signals 611. In this example, module 910 also has the ability to render audio signals of large audio objects 605 for virtual speaker locations. Accordingly, in this example, decorrelated audio large object signals 611 output by module 910 correspond to virtual speaker locations. Some examples of rendering audio object signals for virtual speaker locations will now be described with reference to FIGS. 10A and 10B.

[00120] FIG. 10A depicts an example of virtual source locations with respect to the playback environment. The playback environment may be an actual playback environment or a virtual playback environment. The virtual source locations 1005 and speaker locations 1025 are just examples. However, in this example, the playing environment is a virtual playing environment, and the speaker locations 1025 correspond to the virtual speaker locations.

[00121] In some implementations, virtual source locations 1005 may be spaced evenly in all directions. In the example shown in FIG. 10A, virtual source locations 1005 are spaced evenly along the x, y, and z axes. The virtual source locations 1005 may form a rectangular network of N _x by N _y by N _z virtual source locations 1005 . In some implementations, the value of N may be in the range of 5-100. The value of N may depend, at least in part, on the number of speakers in the playback environment (or expected in the playback environment): it may be desirable to include two or more virtual source locations 1005 between all speaker locations.

[00122] However, in alternative embodiments, virtual source locations 1005 may be spaced in various ways. For example, in some implementations, virtual source locations 1005 may have first uniform spacing along the x and y axes and second uniform spacing along the z axis. In other implementations, virtual source locations 1005 may be spaced unevenly.

[00123] In this example, the volume 1020a of the sound object corresponds to the size of the sound object. The audio object 1010 may be rendered according to the virtual source locations 1005 encompassed by the audio object volume 1020a. In the example depicted in FIG. 10A, audio object volume 1020a occupies part, but not all, of the playback environment 1000a. Larger audio objects may occupy a larger portion (or all) of the playback environment 1000a. In some examples, if the sound object 1010 corresponds to a point source, the sound object 1010 may have a size of zero, and the sound object volume 1020a may be set to zero.

[00124] According to some such implementations, the authoring tool may associate the size of an audio object with decorrelation by indicating (for example, through a decorrelation flag included in the associated metadata) that decorrelation should be enabled when the size of the audio object is greater than or equal to a size threshold, and that decorrelation should be turned off if the size of the audio object is less than the size threshold. In some implementations, decorrelation control (eg, increase, decrease, or disable) may be performed according to user input regarding the size threshold and/or other input values.

[00125] In this example, virtual source locations 1005 are defined within virtual source volume 1002. In some implementations, the volume of the virtual source may correspond to a volume within which audio objects can move. In the example depicted in FIG. 10A, the playback environment 1000a and the virtual source volume 1002a have the same extent so that each of the virtual source locations 1005 corresponds to a location within the playback environment 1000a. However, in alternative embodiments, the playback environment 1000a and the virtual source volume 1002 may not have the same extent.

[00126] For example, at least some of the virtual source locations 1005 may correspond to locations outside the playback environment. FIG. 10B depicts an alternative example of virtual source locations relative to the playback environment. In this example, the virtual source volume 1002b extends outward from the playback environment 1000b. Some of the virtual source locations 1005 within the audio object volume 1020b are inside the playback environment 1000b, and other virtual source locations 1005 within the audio object volume 1020b are outside the playback environment 1000b.

[00127] In other implementations, virtual source locations 1005 may have a first uniform spacing along the x and y axes and a second uniform spacing along the z axis. The virtual source locations 1005 may form a rectangular grid of N _x by N _y by M _z virtual source locations 1005 . For example, in some implementations there may be fewer virtual source locations 1005 along the z-axis than along the x- or y-axis. In some such implementations, the value of N may be in the range of 10-100, while the value of M may be in the range of 5-10.

[00128] Some implementations include calculating gain values for each of the virtual source locations 1005 within the audio object volume 1020. In some implementations, gain values for each channel of the plurality of output channels of the playing environment (which may be an actual playing environment or a virtual playing environment) will be calculated for each of the virtual source locations 1005 within the audio object volume 1020. In some implementations, gain values may be calculated by applying a vector amplitude panning ("VBAP") algorithm, a paired panning algorithm, or a similar algorithm to calculate gain values for point sources located at each of the virtual source locations 1005 within the sound object volume 1020. In other implementations, a shared algorithm may be used to calculate gain values for point sources located at each of the virtual source locations 1005 within the sound object volume 1020. The "shared" algorithm used here is one for which the gain of some given speaker can be expressed as the product of a plurality of coefficients (eg, three coefficients), each of which depends on only one of the virtual source location coordinates 1005. Examples include algorithms implemented in various existing mixing console panning tools, including, but not limited to, Pro Tools™ software and panning tools implemented in digital cinema consoles provided by AMS Neve.

[00129] Referring again to FIG. 9, in this example, the audio processing system 600 also receives the substrate channels B ₁ -B _N as well as a low frequency effects (LFE) channel. The audio objects and substrate channels are processed according to a scene simplification or "clustering" process, for example, as described above with reference to FIGS. 7 and 8. However, in this example, the LFE channel is not entered into the clustering process, but is instead passed to encoder 620.

[00130] In this embodiment, the substrate channels B ₁ -B _N are converted into static audio objects 917 by module 915. Module 920 receives static audio objects 917, in addition to audio objects that large object detection module 905 has determined are not large audio objects. Here, module 920 also receives decorrelated large audio object signals 611 that correspond to the virtual speaker locations in this example.

[00131] In this implementation, module 920 has the ability to render static objects 917, received audio objects, and decorrelated large audio object signals 611 for clusters C ₁ to C _P . In general, module 920 will output fewer clusters than the number of audio objects received. In this implementation, module 920 has the ability to associate decorrelated large audio object signals 611 with appropriate cluster locations, for example, as described above with reference to block 520 of FIG. 5.

[00132] In this example, clusters C ₁ to _CP and LFE channel audio data are encoded by encoder 620 and transmitted to playback environment 925. In some implementations, the playing environment 925 may include a home theater system. The audio processing system 930 is capable of receiving and decoding encoded audio data, as well as rendering the decoded audio data according to the configuration of the actual playback environment speakers 925, e.g., speaker positions, speaker capabilities (e.g., low-pass capability), etc. of the actual playback environment speakers. 925 plays.

[00133] FIG. 11 is a block diagram that provides examples of components of an audio processing system. In this example, the audio processing system 1100 includes an interface system 1105. Interface system 1105 may include a network interface, such as a wireless network interface. Alternatively or in addition, interface system 1105 may include a universal serial bus (USB) interface or other such interface.

[00134] Audio processing system 1100 includes logic system 1110. Logic system 1110 may include a processor, such as a general-purpose single- or multi-chip processor. Logic system 1110 may include a digital signal processor (DSP), application specific integrated circuit (ASIC), field programmable gate array (FPGA) or other programmable logic device, discrete or transistor logic circuit, or discrete hardware components, or combinations listed. Logic system 1110 may be configured to control other components of audio processing system 1100. Although no interfaces between components of the audio processing system 1100 are shown in FIG. 11, the logic system 1110 may be configured with interfaces for communication with other components. Other components may or may not be configured to communicate with each other as appropriate.

[00135] Logic system 1110 may be configured to perform audio processing functionality, including, but not limited to, the types of functionality described herein. In some such implementations, logic system 1110 may be configured to operate (at least in part) according to software stored on one or more non-transitory media. Non-transitory media may include memory associated with logic system 1110, such as random access memory (RAM) and/or read-only memory (ROM). Non-transitory media may include memory of the memory system 1115. Memory system 1115 may include one or more suitable types of non-transitory storage media, such as flash memory, hard disk, etc.

[00136] The display system 1130 may include one or more suitable display types depending on the implementation of the audio processing system 1100. For example, the display system 1130 may include a liquid crystal display, a plasma display, a bistable display, etc.

[00137] User input system 1135 may include one or more devices configured to receive input from a user. In some implementations, the user input system 1135 may include a touch screen that is superimposed on the display of the display system 1130. The user input system 1135 may include a mouse, trackball, gesture sensing system, joystick, one or more GUIs and/or menus presented on the display system 1130, buttons, keyboards, switches, etc. In some implementations, the user input system 1135 may include a microphone 1125: the user may provide voice commands to the audio processing system 1100 through the microphone 1125. The logic system may be configured to recognize speech and to control at least some operations of the audio processing system 1100 according to such voice commands. In some implementations, the user input system 1135 may be considered a user interface and, thus, be part of the interface system 1105.

[00138] Power system 1140 may include one or more suitable energy storage devices, such as a nickel-cadmium battery or a lithium-ion battery. Power system 1140 may be configured to receive power from an electrical outlet.

[00139] Various modifications to the embodiments described in this disclosure may be readily apparent to those of ordinary skill in the art. The general principles defined herein may be applied to other implementations without departing from the spirit or scope of this disclosure. Thus, the claims are not intended to be limited to the embodiments shown herein, but are intended to be within the broadest scope consistent with this disclosure, the principles and new features disclosed herein.

Claims (51)

1. A method for processing audio data, comprising the steps of: receiving, through an input interface to an encoding component of the audio rendering system, audio data comprising audio objects, the audio objects comprising audio object signals and associated metadata, wherein the associated metadata includes at least size data of the audio objects; determining, through the large object determination component based on the sound object size data, a large sound object having a sound object size that is larger than a threshold size, wherein the large sound object is spatially diffuse and requires a plurality of speakers to reproduce the large sound object; performing, by means of a decorrelation component associated with an input interface, a decorrelation process on the audio big object signals to create decorrelated audio big object signals that depend on the determined location of the big audio objects and other information, wherein the decorrelated audio big object signals are mutually independent of each other, and the decorrelation process includes adjusting the level of each audio signal by adjusting the level of a corresponding audio gain for each of the audio signals to generate decorrelated large audio object audio signals corresponding to speaker input signals for each speaker of the plurality of speakers, the plurality of speakers covering a large spatial area, wherein the decorrelated audio signals of the large sound objects are mixed with at least one audio signal for other sound objects that are spatially separated by a second distance threshold from the large sound object. 2. The method of claim 1, further comprising receiving decorrelation metadata for the large audio object, wherein the decorrelation metadata includes an indication that the size of the audio object is larger than a threshold size. 3. The method of claim 1, wherein the large sound object has a plurality of object locations, wherein at least some of the plurality of object locations are one of: stationary locations or locations that change over time. 4. The method of claim 1, wherein the decorrelation process is performed prior to the process of rendering audio data for playback in a playback environment including a home theater system. 5. The method of claim 1, wherein the decorrelation process includes one of a delay process, a universal filter process, a pseudo-random filter process, and a reverberation algorithm process. 6. The method of claim 1, wherein the plurality of speakers have a plurality of speaker locations, wherein the plurality of speaker locations include speaker zones defining virtual speaker locations located in the one or more speaker zones. 7. The method of claim 1, wherein the audio data comprises one or more audio substrate signals corresponding to speaker locations, and the method further comprises outputting decorrelated audio signals of large audio objects as additional audio substrate signals or audio object signals for playback through the plurality of speakers. 8. The method of claim 1, wherein the corresponding audio gain for each of the audio signals comprises a gain determined according to an amplitude panning method. 9. The method of claim 6, further comprising using a rendering tool to display speaker input signals for the one or more speaker zones, respectively. 10. The method of claim 1, further comprising attenuating or removing audio signals of large audio objects after the decorrelation process is performed. 11. The method of claim 1, further comprising storing audio signals corresponding to the large sound object point source contribution after the decorrelation process is performed. 12. The method of claim 1, wherein the large audio object has metadata including audio object position metadata, and the method further comprises: calculating contributions from virtual sources within an area or volume of the sound object determined by the large sound object position metadata for the large sound object and the sound object size data; And determining a set of audio object gain values for each of the plurality of output channels based at least in part on the calculated contributions. 13. The method according to claim 1, further comprising the step of performing a process of clustering audio objects after the decorrelation process. 14. The method of claim 1, further comprising evaluating the audio data to determine a content type, the decorrelation process being selectively performed according to the content type. 15. The method of claim 14, wherein the amount of decorrelation to be performed depends on the type of content. 16. The method of claim 1, wherein the decorrelation process includes a complex time-varying filter algorithm. 17. The method of claim 1, wherein the large audio object has metadata including audio object position metadata, and the method further comprises mixing decorrelated audio signals of the large audio objects with audio signals for audio objects that are spatially separated by a distance threshold from a large sound object. 18. A device for processing audio data, including an audio rendering system, the device comprising: an input interface to an audio rendering system receiving audio data comprising audio objects, the audio objects comprising audio object signals and associated metadata, wherein the associated metadata includes at least size data of the audio objects; a processing component that determines, based on the sound object size data, a large sound object having a sound object size that is larger than a threshold size, wherein the large sound object is spatially diffuse and requires a plurality of speakers to reproduce the large sound object; a decorrelation component coupled to an input interface that performs a decorrelation process on the audio boobies to create decorrelated audio boobies that depend on the specific location of the boonies and other information, wherein the decorrelated audio boobies are mutually independent of each other, and the decorrelation process includes adjusting the level of each audio signal by adjusting the level of the corresponding audio gain for each of the audio signals to generate decorrelated large audio object signals corresponding to speaker input signals for each speaker of a plurality of speakers, the plurality of speakers covering a large spatial area, wherein the decorrelated audio signals of the large sound objects are mixed with at least one audio signal for other sound objects that are spatially separated by a second distance threshold from the large sound object. 19. The apparatus of claim 18, further comprising an input interface that receives decorrelation metadata for the large audio object, wherein the decorrelation metadata includes an indication that the size of the audio object is larger than a threshold size. 20. The apparatus of claim 18, wherein the large sound object has a plurality of object locations, wherein at least some of the plurality of object locations are one of: stationary locations or locations that change over time. 21. The apparatus of claim 18, wherein the decorrelation process is performed prior to the process of rendering audio data for playback in a playback environment including a home theater system. 22. The apparatus of claim 18, wherein the decorrelation process includes one of a delay process, a universal filter process, a pseudo-random filter process, and a reverberation algorithm process. 23. The apparatus of claim 18, wherein the plurality of speakers have a plurality of speaker locations, wherein the plurality of speaker locations include speaker zones defining virtual speaker locations located in the one or more speaker zones. 24. The apparatus of claim 18, wherein the audio data comprises one or more audio substrate signals corresponding to speaker locations, and the device further comprises an audio rendering system configured to output decorrelated audio signals of large audio objects as additional audio substrate signals or audio object signals for playback. through multiple speakers. 25. The apparatus of claim 18, wherein the corresponding audio gain for each of the audio signals comprises a gain determined according to an amplitude panning method. 26. The apparatus of claim 23, further comprising a rendering tool configured to display speaker inputs for one or more speaker zones, respectively. 27. The apparatus of claim 18, further comprising an audio rendering system that attenuates or removes audio signals of large audio objects after the decorrelation process is performed. 28. The apparatus of claim 18, further comprising an audio rendering system storing audio signals corresponding to the point source contribution of the large audio object after the decorrelation process is performed. 29. The device of claim 18, wherein the large audio object has metadata including audio object position metadata, and the device further comprises: an audio rendering system calculating contributions from virtual sources within an area or volume of an audio object defined by the large audio object position metadata for the large audio object and the audio object size data; And an audio rendering system determining a set of audio object gain values for each of the plurality of output channels based at least in part on the calculated contributions. 30. The apparatus of claim 18, further comprising an audio rendering system that performs a process of clustering audio objects after a decorrelation process. 31. The apparatus of claim 18, further comprising an audio rendering system evaluating audio data to determine a content type, the decorrelation process being selectively performed according to the content type. 32. The apparatus of claim 31, wherein the amount of decorrelation to be performed depends on the type of content. 33. The apparatus of claim 18, wherein the decorrelation process includes a complex time-varying filter algorithm. 34. The device of claim 18, wherein the large audio object has metadata including audio object position metadata, and the device further comprises an audio rendering system that mixes decorrelated audio signals of the large audio objects with audio signals for audio objects that are spatially separated by a distance threshold from large sound object. 35. A non-transitory medium having program instructions stored thereon that, when executed on a processing component of an audio rendering system, instructs the audio rendering system to: receiving, through an audio rendering system input interface to an audio rendering system encoding component, audio data comprising audio objects, the audio objects comprising audio object signals and associated metadata, wherein the associated metadata includes at least size data of the audio objects; determine, by the audio rendering system large object detection component based on the audio object size data, a large audio object having an audio object size that is larger than a threshold size, wherein the large audio object is spatially diffuse and requires a plurality of speakers to reproduce the large audio object; perform, through the decorrelation component of the audio rendering system associated with the input interface, a decorrelation process on the audio big object signals to create decorrelated big audio signals that depend on the specific location of the big audio objects and other information, wherein the decorrelated big audio signals are mutually independent of each other each other, and the decorrelation process includes adjusting the level of each audio signal by adjusting the level of the corresponding audio gain for each of the audio signals to generate decorrelated audio signals of large audio objects corresponding to speaker input signals for each speaker of a plurality of speakers, the plurality of speakers covering a large spatial area, wherein the decorrelated audio signals of the large sound objects are mixed with at least one audio signal for other sound objects that are spatially separated by a second distance threshold from the large sound object.