RU2820838C2 - System, method and persistent machine-readable data medium for generating, encoding and presenting adaptive audio signal data - Google Patents

Abstract

FIELD: physics.

SUBSTANCE: invention relates to computer engineering for processing audio data. Technical result is achieved by receiving a bit stream, inside which there is an adaptive audio mix containing a number of monophonic audio streams and metadata associated with each of the audio streams and indicating the playback location of the corresponding monophonic audio stream; and presenting a plurality of monophonic audio streams to a plurality of signals supplied to loudspeakers, corresponding to loudspeakers in a playback environment, wherein the loudspeakers of the loudspeaker array are placed in certain positions within the playback environment, and wherein the metadata elements associated with each corresponding object-based monophonic audio stream indicate an allocation volume to be applied to the monophonic audio stream on the basis of objects, thus the monophonic audio stream on the basis of objects is represented in a series of signals supplied to loudspeakers, with a spatial extent corresponding to the volume of distribution specified by the metadata.

EFFECT: improved sound quality in a room.

4 cl, 11 dwg, 12 tbl

Description

CROSS REFERENCE TO RELATED APPLICATIONS

[0001] This application claims priority to US Provisional Patent Application No. 61/504005, filed July 1, 2011, and US Provisional Patent Application No. 61/636429, filed April 20, 2012; both of these applications are incorporated by reference into this disclosure in their entirety in all respects.

TECHNICAL APPLICATION AREA

[0002] One or more implementations generally relate to audio processing and, more specifically, to object- and channel-based hybrid audio processing for use in cinematic, home, and other environments.

BACKGROUND

[0003] It should not be assumed that subject matter discussed in the background section constitutes prior art solely as a result of its mention in the background section. Likewise, it should not be assumed that a problem mentioned in the background section or associated with the subject matter of the invention in the background section is recognized in the prior art. The subject matter in the background section merely represents various approaches that may themselves constitute inventions.

[0004] Since the introduction of sound in films, there has been a steady development of technology designed to capture the artistic intent of the creator of a motion picture soundtrack and to accurately reproduce it in a movie theater environment. The fundamental role of sound in cinematography is to contribute to the story on screen. Typical cinematography soundtracks contain many different audio elements corresponding to on-screen elements and images, dialogue, noise and sound effects that come from various on-screen elements and combine with background music and environmental effects to create the overall viewing experience. The artistic intent of the creators and producers reflects their desire to reproduce said sounds in a manner that is as close as possible to what is shown on screen in terms of position, intensity, movement and other similar parameters of the sound sources.

[0005] Modern motion picture authoring, distribution and playback suffers from limitations that hinder the creation of truly life-like and immersive sound. Traditional channel-based audio systems deliver audio content in the form of speaker signals to individual speakers in a playback environment such as a stereo or 5.1 system. The advent of digital cinematography has created new standards for film audio, such as combining up to 16 channels of sound, allowing for increased creativity for content creators and a greater reach and realism in the listening experience for audiences. The introduction of 7.1 surround systems provided a new format that increases the number of surround channels by splitting the existing left and right surround channels into four zones, thereby expanding the limits of what sound processing and synthesis operators and mixing operators can do when managing the locations of sound elements in a theater.

[0006] To further improve the user experience, audio playback in virtual 3D environments has become an area of increased design and development efforts. The representation of sound in space uses sound objects, which are sound signals with associated parametric source descriptions for positions of the apparent source (eg, 3D coordinates), width of the apparent source, and other parameters. Object-based audio is increasingly used for many modern multimedia applications such as digital motion pictures, video games, simulations and 3D video.

[0007] Moving beyond traditional loudspeaker signals and channel-based sound as a means of propagating sound through space is critical, and there is significant interest in model-based descriptions of sound that hold promise for giving the listener/distributor freedom select a playback configuration that suits their individual needs or budget, with audio data presented specifically for their chosen configuration. At a high level, there are currently four main formats for spatial description of sound: signals applied to loudspeakers, where sound is described as signals intended for loudspeakers at nominal loudspeaker positions; a signal applied to a microphone, where sound is described as signals captured by virtual or actual microphones in a predefined array; model-based description, in which sound is described in terms of a sequence of sound events at described positions; and binaural, in which sound is described by signals that reach the user's ears. These four description formats are often associated with one or more presentation technologies that convert audio signals into signals that are sent to loudspeakers. Modern presentation technologies include panning, in which the audio stream is converted into signals delivered to loudspeakers using a set of panning laws and known or assumed positions of the loudspeakers (usually presenting the data before distribution); ambiphony, in which microphone signals are converted into feed signals for a scalable array of loudspeakers (usually presenting the data after propagation); WFS (Wave Field Synthesis), in which sound events are converted into corresponding loudspeaker signals to synthesize the sound field (typically data presentation occurs after propagation); and binaural technology, in which binaural L/R (left/right) signals are delivered to L/R ears, typically using headphones, but also using loudspeakers and crosstalk cancellation (data presentation occurs before or after propagation). Among these formats, the loudspeaker format is the most common because it is simple and efficient. The best acoustic results (most accurate, most reliable) are achieved by mixing/monitoring and distributing directly into the signals sent to the loudspeakers, since there is no processing between the content creator and the listener. If the playing system is known in advance, a description of the signals sent to the speakers will usually provide the highest fidelity. However, in many practical applications the losing system is unknown. Model-based description is considered the most adaptable because it makes no assumptions about the data representation technology, and therefore it is most easily applied to any data representation technology. Although model-based description is effective at capturing spatial information, it becomes extremely inefficient as the number of sound sources increases.

[0008] For many years, cinematography systems have been characterized by discrete on-screen channels in the form of left, center, right, and sometimes "inner left" and "inner right" channels. These discrete sources typically have sufficient frequency response and switching power to allow sounds to be accurately placed in different areas of the screen and to allow timbral matching as sounds move, or pan, between locations. Modern developments in enhancing the listener's perception strive to accurately reproduce the location of sounds relative to the listener. In a 5.1 setup, the surround "zones" include an array of speakers, all of which carry the same audio information within each, left surround or right surround zone. These arrays can be effective for "ambient" effects and diffuse ambient sound, but in everyday life many sound effects arise from randomly placed point sources. For example, in a restaurant, it seems that ambient music is playing from all sides, while discrete sounds arise from certain points: a person talking from one point, the knock of a knife on a plate from another. Having the ability to discretely place these sounds around the auditorium can create an enhanced sense of reality without being too noticeable. Also important to the clarity of ambient sound are sounds from above. In the real world, sounds come from all directions, and not always from a single horizontal plane. An additional sense of reality can be achieved if sounds can be heard from above, in other words from the “upper hemisphere”. Current systems, however, do not offer truly accurate audio reproduction across multiple audio types across a range of different playback environments. Much more work will need to be done in the processing, knowledge and configuration of actual playback environments to use existing systems to attempt to accurately reproduce the location of certain sounds and thus render current systems unusable for most applications.

[0009] What is needed is a system that supports multiple on-screen channels, resulting in increased clarity and improved audio-visual coherence for on-screen sounds or dialogue and the ability to accurately position sources anywhere in the surrounding areas, improving audio-visual transition from screen to room. For example, if a character on screen is looking indoors towards a sound source, the sound engineer ("mixing operator") must be able to accurately place the sound so that it matches the character's line of sight, and so that the effect is consistent across all viewers. However, with traditional 5.1 or 7.1 surround mixing, the effect is highly dependent on the listener's seating position, which is unfavorable for most large listening environments. The increased resolution of ambient sound creates new opportunities to use sound in a room-centered manner, as opposed to the traditional approach where content is created with the assumption of a single listener in the “sweet spot.”

[0010] In addition to spatial problems, state-of-the-art multi-channel systems suffer in terms of timbre. For example, when played back by an array of loudspeakers, the timbral quality of some sounds, such as the hiss of steam escaping from a damaged pipe, may suffer. The ability to route specific sounds into a single loudspeaker gives the mixing operator the ability to eliminate distortion in array playback and achieve a more realistic audience experience. Traditionally, surround speakers do not support the same full audio frequency range and level that large on-screen channels support. In the past, this has created difficulties for mixing operators, reducing their ability to freely move wide-range sounds from the screen into the room. As a result, theater owners did not feel the need to upgrade the configuration of the surrounding channels, which prevented the widespread adoption of higher-quality installations.

BRIEF DESCRIPTION OF OPTIONS FOR IMPLEMENTING THE INVENTION

[0011] Systems and methods are described for a cinematic audio format and data processing system that includes a new speaker layer (channel configuration) and an associated spatial description format. An adaptive sound system and a format that supports several data presentation technologies have been defined. Audio streams are transmitted along with metadata that describes the "mix operator's intent" including the desired position of the audio stream. The specified position may be expressed as a named channel (of channels within a predefined channel configuration) or as three-dimensional position information. This format - channels plus objects - combines the best ways to describe the sound picture based on channels and based on models. The audio data for an adaptive audio system contains a number of independent monaural audio streams. Each stream has metadata associated with it that describes whether the stream will be a channel-based stream or an object-based stream. Channel-based streams contain data representation information encoded by the channel name; and object-based streams contain location information encoded through mathematical expressions encoded in additional associated metadata. The original independent audio streams are packaged as a single binary sequence that contains all audio data. This configuration allows sound to be represented according to an allocentric reference frame, in which the presentation of sound location data is based on characteristics of the playing environment (eg room size, room shape, etc.) to match the mixing operator's intent. Object position metadata contains the appropriate allocentric coordinate system information needed to correctly play audio using the positions of available speakers in a room that is set up to play adaptive audio content. This allows audio to be mixed optimally for a specific playback environment, which may differ from the mixing environment experienced by the audio engineer.

[0012] Adaptive Audio improves sound quality in a variety of rooms with benefits such as improved control of room equalization and ambient bass so that the mix operator can freely access speakers (both on-screen and off-screen ) without having to think about timbral matching. Adaptive Audio adds the flexibility and capabilities of dynamic audio objects to traditional channel-based workflows. These audio objects allow creators to control discrete audio elements independent of specific playback speaker configurations, including height speakers. The system also brings new efficiency to the layout process, allowing audio engineers to efficiently capture all their designs and then, during real-time or automatic monitoring, generate 7.1 or 5.1 surround sound versions.

[0013] The adaptive audio system simplifies distribution by distilling the audio essence of an artistic concept into a single track file in a digital cinematography device that can be accurately played back in a wide range of theater configurations. The system ensures optimal reproduction of the artistic concept when the mixing and data presentation tools use the same channel configuration and a single inventory with top-down adaptation to the data presentation configuration, i.e. with downmixing.

[0014] These and other advantages are presented through embodiments of the invention that address the cinematic audio platform, addressing the limitations of current systems and delivering audio experiences that are beyond the reach of systems currently available.

BRIEF DESCRIPTION OF GRAPHIC MATERIALS

[0015] In the following graphics, like reference numerals are used to refer to like elements. Although the following figures depict various examples, one or more implementations are not limited to the examples depicted in the figures.

[0016] FIG. 1 is a top view of an audio production and playback environment using an adaptive audio system according to one embodiment of the invention.

[0017] FIG. 2 illustrates combining channel-based and object-based data to generate an adaptive audio mix according to one embodiment of the invention.

[0018] FIG. 3 is a flowchart illustrating a process flow for creating, packaging, and presenting adaptive audio content data according to one embodiment of the invention.

[0019] FIG. 4 is a block diagram of a data presentation step of an adaptive audio system according to one embodiment of the invention.

[0020] FIG. 5 is a table listing metadata types and associated metadata elements for an adaptive audio system according to one embodiment of the invention.

[0021] FIG. 6 is a diagram that illustrates the layout and final processing for an adaptive audio system according to one embodiment of the invention.

[0022] FIG. 7 is a diagram of one example flowchart of a digital motion picture packaging process using adaptive audio files according to one embodiment of the invention.

[0023] FIG. 8 is a top view of one example layout of proposed speaker locations for use with an adaptive sound system in a typical auditorium.

[0024] FIG. 9 is a front view of one example of proposed speaker locations on a screen for use in a typical auditorium.

[0025] FIG. 10 is a side view of one example layout of proposed speaker locations for use with an adaptive audio system in a typical auditorium.

[0026] FIG. 11 is one example of the arrangement of the top surround speakers and the side surround speakers relative to the origin according to one embodiment of the invention.

DETAILED DESCRIPTION

[0027] Systems and methods for an adaptive audio system and associated audio signal and data format that support multiple data presentation technologies are described. Features for one or more embodiments of the invention described in this disclosure may be implemented in an audio system or audiovisual system that processes source audio information in a mixing, presentation and playback system that includes one or more computers or data processing devices executing software commands. provision. Any of the described embodiments of the invention may be used alone or in conjunction with any other embodiments in any combination. Although various embodiments of the invention may be motivated by various disadvantages in the prior art, which may be discussed or mentioned in one or more places in this specification, embodiments of the invention do not necessarily address any of these disadvantages. In other words, various embodiments of the invention may address various disadvantages, which may be discussed herein. Some embodiments of the invention may only partially address some of the disadvantages or only one disadvantage described in this specification, and some embodiments of the invention may not address any of these disadvantages.

[0028] For purposes of this description, the following terms have the following associated meanings:

[0029] Channel, or audio channel: a monaural audio signal, or audio stream, plus metadata in which the position is encoded as a channel identifier, such as "left front" or "right high surround". A channel object can drive multiple loudspeakers, for example the left surround channels (Ls) will be fed to the Ls array loudspeakers.

[0030] Channel configuration: a predefined set of speaker zones with associated nominal locations, for example, 5.1, 7.1, etc.; 5.1 refers to a six-channel surround audio system containing front left and right channels, a center channel, two surround channels and a super low-frequency channel; 7.1 refers to an eight-channel surround sound system that adds two additional surround channels to the 5.1 system. Example 5.1 and 7.1 configurations include Dolby® surround systems.

[0031] Loudspeaker: An audio transducer or set of transducers that represents audio signal data.

[0032] Loudspeaker Zone: An array of one or more loudspeakers that can be uniquely assigned and that receive a single, for example left surround, audio signal, typically found in a movie theater and specifically intended to be excluded or included in the presentation of object data.

[0033] Speaker channel, or speaker feed channel: an audio channel that is associated with a named speaker, or speaker zone, within a specific speaker configuration. A loudspeaker channel is typically represented using an associated loudspeaker zone.

[0034] Speaker Channel Group: A collection of one or more speaker channels corresponding to the channel configuration (e.g., stereo tracks, mono tracks, etc.)

[0035] Object, or object channel: one or more audio channels with a parametric description of the source, such as the position of the apparent source (eg, 3D coordinates), the width of the apparent source, etc. An audio stream plus metadata in which the position is encoded as a three-dimensional position in space.

[0036] Audio program: a complete set of speaker channels and/or object channels and associated metadata that describe the desired representation of sound in space.

[0037] Allocentric reference frame: A spatial reference frame in which audio objects are defined within the presentation environment with respect to features such as room walls and corners, standard speaker locations, and screen location (eg, the front left corner of the room).

[0038] Egocentric frame of reference: a spatial frame of reference in which objects are defined relative to the viewer's perspective of the listener, and which is often defined relative to angles relative to the listener (eg, 30 degrees to the listener's right).

[0039] Frame: Frames are short, independently decoded segments into which a complete audio program is divided. The size and boundaries of audio frames are usually aligned with video frames.

[0040] Adaptive audio: channel-based and/or object-based audio plus metadata that represents audio data based on the playing environment.

[0041] The cinematic audio format and processing system described in this disclosure, also referred to as an "adaptive audio system", uses a new technology for describing and representing spatial audio data to enhance audience immersion, artistic control, system flexibility and scalability, and ease of installation and maintenance. Embodiments of the cinematic audio platform include several discrete components, including mixing tools, packaging/encoder, decompressing/decoder, final mixing and theater presentation components, new speaker circuits, and networked amplifiers. The system includes recommendations for new channel configurations to be used by creators and film distributors. The system uses a model-based description that supports several characteristic features such as: a single inventory with top-down and bottom-up adaptation to the data presentation configuration, i.e. delayed presentation of data and enabling optimal use of available loudspeakers; improved audio coverage, enabling optimized downmixing to avoid correlation between channels; increased spatial resolution through end-to-end array management (e.g., an audio object is dynamically assigned to one or more speakers within a surround array); and support for alternative ways of presenting data.

[0042] FIG. 1 is a top view of an audio production and playback environment using an adaptive audio system, according to one embodiment of the invention. As shown in FIG. 1, a complete, end-to-end environment 100 contains components for creating content, packaging, distributing and/or playing/presenting data to a large number of end devices and use cases. The system 100 as a whole originates from content captured from and for a number of different use cases that involve the perception of 112 viewers. The content data capture element 102 includes, for example, cinematography, television, live television, user-generated content, recorded content, games, music, and the like, and may include audio/visual or pure audio content. As content moves through the system 100 from the data capture stage 102 to the end-user experience 112, it passes through several key processing steps through discrete system components. These process steps include audio pre-processing 104, proprietary tools and processes 106, encoding by audio codec 108 that collects, for example, audio data, additional metadata and rendering information, and object channels. For successful and secure distribution across a variety of media, a variety of processing effects such as compression (lossy or lossless), encryption, etc. can be applied to the object channels. To reproduce and convey the user's specific perception 112 of adaptive audio, appropriate endpoint-specific decoding and data presentation processes 110 are then applied. Audio experience 112 represents the playback of audio or audiovisual content through appropriate speakers and playback devices, and may represent any environment in which the listener experiences playback of the captured content, such as a movie theater, concert hall, outdoor movie theater, home or premises, listening booth, automobile, game console, headphones or headset, public address system or other playing media.

[0043] This embodiment of system 100 includes an audio codec 108 that is capable of efficiently distributing and storing multi-channel audio programs and may therefore be referred to as a “hybrid” codec. Codec 108 combines traditional channel-based audio data with associated metadata to form audio objects that facilitate the creation and delivery of audio that is tailored and optimized for presentation and playback in environments that may differ from the mixing environment. This allows the audio engineer to encode his or her intent into how the final sound should be heard by the listener, based on the listener's actual listening environment.

[0044] Traditional channel-based audio codecs operate under the assumption that the audio program will be reproduced by an array of loudspeakers located at predetermined positions relative to the listener. To create a complete multi-channel audio program, audio engineers typically mix a large number of individual audio streams (eg dialogue, music, effects) to create the desired overall experience. In audio mixing, decisions are typically made by listening to the sound program being played by an array of speakers placed in predetermined positions, such as specifically a 5.1 or 7.1 system in a particular theater. The final, mixed signal serves as input to the audio codec. Spatially accurate sound fields are only achieved during playback when the speakers are placed in predetermined positions.

[0045] One new form of audio coding, called audio object coding, provides individual audio sources (audio objects) in the form of separate audio streams as input to the encoder. Examples of audio objects include dialogue tracks, individual instruments, individual sound effects, and other point sources. Each sound object is associated with spatial parameters, which may include, but are not limited to, sound position, sound width, and velocity information. For distribution and storage, audio objects and associated parameters are then encoded. The final mixing and presentation of the audio object data is performed on the receiving side of the audio distribution chain as part of the playback of the audio program. This step can be based on knowledge of actual speaker positions, so the result is a sound distribution system that is customizable to suit the individual user's listening environment. These two forms of encoding, channel-based and object-based, work optimally for different input signal conditions. Channel-based audio encoders are generally more effective at encoding input signals containing dense mixtures of different audio sources, as well as for diffuse sounds. In contrast, audio object coders are more efficient at encoding small numbers of highly directional audio sources.

[0046] In one embodiment, the methods and components of the system 100 include an audio encoding, distribution and decoding system configured to generate one or more bitstreams containing both traditional channel-based audio elements and audio object encoding elements. This combined approach provides greater encoding efficiency and data representation flexibility than either channel-based or object-based approaches alone.

[0047] Other features of the described embodiments include extending a predefined channel-based audio codec in a backwards-compatible manner to include audio object coding elements. A new "extension layer" containing audio object coding elements is defined and added to the "main" or "backwards compatible" bitstream layer of the channel-based audio codec. This approach allows legacy decoders to process one or more bitstreams that contain the enhancement layer, while at the same time providing an improved listening experience for users with newer decoders. One example of enhancing user experience involves controlling the presentation of audio object data. An additional benefit of this approach is that audio objects can be added or modified throughout the distribution chain without decoding/mixing/re-encoding the multi-channel audio encoded by the channel-based audio codec.

[0048] With respect to the coordinate system, the spatial effects of audio signals are critical in providing the listener with an immersive experience. Sounds that are intended to originate from a specific area of the visual screen or room must be played through the loudspeaker(s) located at the same relative location. Therefore, the primary element of audio metadata for a sound event in a model-based description is position, although other parameters such as size, orientation, speed, and sound dispersion may also be described. To convey position, model-based 3D spatial description of sound requires a 3D coordinate system. The coordinate system used for the transfer (Euclidean, spherical, etc.) is usually chosen for convenience or compactness, but other coordinate systems may be used to handle the data representation. In addition to a coordinate system, a reference system is required to represent the locations of objects in space. Selecting an appropriate reference frame can be a critical factor in the accurate sound reproduction of position-based systems in many different environments. In an allocentric reference frame, the position of a sound source is determined relative to features within the presentation environment such as room walls and corners, standard speaker locations, and screen location. In an egocentric frame of reference, locations are presented relative to the listener's perspective, such as “in front of me, slightly to the left,” etc. Scientific studies of spatial perception (sound, etc.) have shown that the most universal is the use of an egocentric perspective. However, for cinema, for a number of reasons, the allocentric system is more suitable. For example, the exact location of an audio object is more important when the associated object is on screen. When using an allocentric reference frame for each listening position and for any screen size, the sound will be localized to the same position on the screen, for example, one third to the left of the middle of the screen. Another reason is that mixing operators tend to reason and mix in allocentric terms, and panning aids are arranged in an allocentric reference frame (the walls of the room), and mixing operators expect these aids to be represented that way, e.g. “this sound should be on the screen”, “this sound should be behind the screen” or “from the left wall”, etc.

[0049] Despite the use of an allocentric frame of reference in a cinematic environment, there are some cases for which an egocentric frame of reference may be useful. These cases include voiceovers, e.g. sounds that are not present in the "film space", such as musical accompaniment, which may require a homogeneous egocentric representation. Another case is near-field effects (such as the buzzing of a mosquito in the listener's left ear), which require an egocentric representation. There is currently no means to present such near-field sound data using headphones or near-field loudspeakers. Additionally, sound sources at infinity (and the resulting plane waves) appear to come from a constant egocentric position (e.g., 30 degrees to the left), and such sounds are easier to describe in egocentric terms rather than allocentric ones.

[0050] In some cases, it is possible to use an allocentric reference frame as long as the nominal listening position is determined, although some examples require an egocentric representation, the data of which is still impossible to represent. Although an allocentric frame of reference may be more useful and appropriate, the representation of sound must be extensible, since many new features, including an egocentric representation, may be more desirable in some applications and listening environments. Embodiments of an adaptive audio system include a hybrid approach to spatial description that includes a recommended channel configuration for optimal fidelity and for representing data from diffuse or complex, multi-point sources (e.g., stadium crowd, environment) using an egocentric reference frame plus allocentric audio description model-based - to allow high spatial resolution and scalability.

System components

[0051] With reference to FIG. 1, the original audio content data 102 is first processed in the preprocessing unit 104. The pre-processing unit 104 of the system 100 contains an object channel filtering component. In many cases, sound objects contain separate sound sources, allowing sounds to be panned independently. In some cases, such as when creating audio programs using natural or "production" audio, it may be necessary to extract individual audio objects from a recording that contains multiple audio sources. Embodiments of the invention include a method for extracting independent audio signals from a more complex signal. Undesirable elements to be separated from independent source signals may include, but are not limited to, other independent audio sources and background noise. Additionally, reverberation can be removed to recreate dry sound sources.

[0052] The data front end 104 also includes functionality for source separation and content type detection. The system provides automatic generation of metadata by analyzing the input audio. Positional metadata is obtained from a multichannel recording by analyzing the relative levels of the correlated input signal between pairs of channels. Detection of a content type such as “speech” or “music” can be performed, for example, by extracting and classifying characteristic features.

Authoring tools

[0053] The authoring tool block 106 includes features designed to enhance the authoring of audio programs by optimizing the input and codifying the audio engineer's creative intent, allowing him to create the final audio mix once it is optimized for playback in virtually any playback environment. This is accomplished by using audio objects and positional data that are associated and encoded along with the original audio content. In order to accurately place sounds around the perimeter of the auditorium, the audio engineer must have control over how the sound will ultimately be presented based on the actual constraints and characteristics of the playback environment. An adaptive audio system provides this control by allowing the audio engineer to change the way audio content is developed and mixed through the use of audio objects and positional data.

[0054] Sound objects can be considered groups of sound elements that can be perceived as emanating from a specific physical location, or locations, in the auditorium. Such objects may be stationary, or they may be moving. In adaptive audio system 100, audio objects are driven by metadata that, among other things, detail the location of the audio at a given point in time. When objects are monitored or played in a theater, their data is presented according to positional metadata using the speakers that are available, instead of necessarily being output to a physical channel. A track in a session can be an audio object, and standard panning data can be similar to positional metadata. In this way, content placed on the screen can be effectively panned in the same way as with channel-based content, however, content data placed in surrounding channels can optionally be presented in a separate channel. Although the use of audio objects provides the necessary control over discrete effects, other features of the film soundtrack work more effectively in a channel-based environment. For example, many ambient effects or reverberation actually benefit from being fed into loudspeaker arrays. While they can be treated as objects with a width sufficient to fill an array, it is more useful to retain some channel-based functionality.

[0055] In one embodiment of the invention, the adaptive audio system supports "paths" in addition to audio objects, where paths are effectively submixed channel-based signals, or stems. Depending on the intent of the content creator, they can be obtained for final playback (data presentation) either separately or combined into a single path. These paths can be created in channel-based configurations as varied as 5.1, 7.1, and extend to larger formats such as 9.1 and overhead speaker arrays.

[0056] FIG. 2 illustrates the combination of channel-based and object-based data when generating an adaptive audio mix according to one embodiment of the invention. As shown in process 200, channel-based data 202, which for example may be 5.1 or 7.1 surround data presented in the form of pulse code modulation (PCM) data, is combined with audio object data 204 to form an adaptive audio mix. 208. Sound object data 204 is generated by combining original channel-based data elements with associated metadata that indicate certain parameters related to the location of the sound objects.

[0057] As conceptually shown in FIG. 2, proprietary tools provide the ability to create audio programs that simultaneously contain a combination of speaker channel groups and object channels. For example, an audio program may contain one or more speaker channels, optionally organized into groups (or tracks, such as a stereo or 5.1 track), and descriptive metadata for one or more speaker channels, one or more object channels, and descriptive metadata for one or more object channels. Within an audio program, each group of speaker channels and each object channel may be represented using one or more sample rates. For example, Digital Cinema (D-Cinema) applications support 48 kHz and 96 kHz sample rates, but other sample rates may also be supported. In addition, receiving, storing and editing channels with different sampling rates can also be supported.

[0058] Creating a sound program requires a stage of sound design, which involves combining sound elements as the sum of component sound elements with adjusted levels to create the desired new sound effect. The tools of the author's development of the adaptive sound system allow you to create sound effects as a set of sound objects with relative positions, using the spatial-visual graphical user interface of sound scenography. For example, a visual representation of a sound-generating object (e.g., a car) can be used as a template for assembling sound elements (exhaust noise, tire screeching, engine noise) as object channels containing the sound and the corresponding position in space (near the exhaust pipe, tires and hood). Channels of individual objects can then be linked and processed as a group. The proprietary tool 106 includes several user interface elements that allow the audio engineer to enter control information and view mixing parameters, as well as improve the functionality of the system. The sound design and authoring process is also enhanced by allowing object and speaker channels to be linked and processed as a group. One example is combining an object channel with a discrete, dry audio source with a set of speaker channels that contain an associated reverberated signal.

[0059] The proprietary tool 106 supports the ability to combine multiple audio channels, commonly known as “mixing.” A variety of mixing methods are supported, which may include traditional level-based mixing and volume-based mixing. In level-based mixing, wideband scaling is applied to the audio channels, and the scaled audio channels are then summed. The wideband scaling factors for each channel are chosen so that they control the absolute level of the resulting mixed signal, as well as the relative levels of the mixed channels within the mixed signal. In loudness-based mixing, one or more input signals are modified using frequency-dependent amplitude scaling, where the frequency-dependent amplitude is selected so that it provides the desired perceived absolute and relative loudness while maintaining the perceived timbre of the input sound.

[0060] Proprietary tools allow the creation of speaker channels and speaker channel groups. This allows metadata to be associated with each of the speaker channel groups. Each of the speaker channel groups can be labeled according to the content type. The content type is distributed through a text description. Content types may include, but are not limited to, dialogue, music, and effects. Each of the speaker channel groups may be assigned unique instructions on how to upmix from one channel configuration to another, where upmixing is defined as creating M audio channels from N audio channels, where M>N. The upmix instructions may include, but are not limited to, the following commands: an enable/disable flag indicating whether upmixing is permissible; an upmix matrix for controlling the assignment between each input and output channels; and default unlocks and matrix settings can be assigned based on content type, such as unlocking upmixing for music only. Each of the speaker channel groups may also be assigned unique instructions on how to downmix from one channel configuration to another, where downmixing is defined as creating Y audio channels from X audio channels, where Y>X. The downmix instructions may include, but are not limited to, the following instructions: a matrix for controlling the assignment between each input and output channels; and default matrix settings, which can be assigned based on the type of content, for example, dialogue should be downmixed to the screen; effects must be downmixed off screen. Each speaker channel can also be associated with a metadata flag that disables bass management during data presentation.

[0061] Embodiments of the invention include a feature that allows the creation of object channels and groups of object channels. The invention allows metadata to be associated with each of the object channel groups. Each of the object channel groups can be labeled according to the content type. The content type is distributed through a text description, where content types may include, but are not limited to, dialogue, music, and effects. Each of the object channel groups may be assigned metadata to describe how the object(s) data should be represented.

[0062] Position information is provided to indicate the desired position of the apparent source. Position can be indicated using an egocentric or allocentric reference frame. An egocentric frame of reference is appropriate when the position of the source must be based on the listener. Spherical coordinates are suitable to describe the position in the egocentric system. The allocentric frame of reference is a typical frame of reference for cinematic or other audiovisual presentations, where the position of the source is specified relative to objects in the presentation environment, such as the position of a video monitor screen or the boundaries of a room. Three-dimensional (3D) trajectory information is provided to allow position interpolation, or to enable other data representation solutions such as "mode snap" unlocking. Size information is provided to indicate the desired perceived size of the apparent sound source.

[0063] Spatial quantization is provided through a "snap to nearest speaker" control, which indicates the intent of the audio engineer, or mixing operator, to represent the object data with exactly one speaker (potentially sacrificing spatial accuracy). The limit of acceptable spatial distortion may be specified by elevation and azimuth tolerance thresholds so that if the threshold is exceeded, the snap function will not be performed. In addition to distance thresholds, a fade rate parameter can be specified to control how quickly a moving object will transition, or jump, from one speaker to another when the desired position is between the speakers.

[0064] In one embodiment of the invention, dependent spatial metadata is used for some position metadata. For example, metadata can be automatically generated for a "slave" object by associating it with a "master" object that the slave object must follow. The slave can be given a time delay or relative speed. Mechanisms may also be provided to allow the acoustic center of gravity to be determined for sets, or groups, of objects so that the object can be presented in such a way that it is perceived as moving near another object. In this case, one or more objects may rotate around a certain object, or a certain area, such as a dominant point or pit of a room. The acoustic center of gravity could then be used at the data presentation stage to help determine location information for each and sounds based on objects, even though the final location information would be expressed as location relative to a room, as opposed to location relative to another object.

[0065] When object data is presented, it is, in accordance with the position metadata and location of the playing speakers, assigned to one or more speakers. Additional metadata may be associated with the object to limit the speakers that could be used. The use of restrictions may prohibit the use of specified loudspeakers or only suppress specified loudspeakers (allow less energy into the loudspeaker, or loudspeakers, than could be applied). Sets of speakers subject to limitation may include, by way of non-limiting examples, any named speakers or speaker areas (e.g., L, C, R, etc.), or speaker areas such as front wall, rear wall, left wall , right wall, ceiling, floor, indoor speakers, etc. Likewise, in the course of specifying the desired mixing of multiple audio elements, one or more audio elements can be caused to become inaudible, or "masked", due to the presence of other, "masked" audio elements. For example, if "masked" sound elements are detected, they can be identified via a graphical display.

[0066] As described elsewhere, the audio program description can be adapted to represent data across a wide variety of speaker settings and channel configurations. When an author develops a sound program, it is important to monitor the program in expected playback configurations to ensure that the desired results are achieved. This invention includes the ability to select target playback configurations and monitor the result. In addition, the system can automatically monitor the worst case signal levels (ie, the highest levels) that could be generated in each of the expected playback configurations and provide an indicator if clipping or clipping occurs.

[0067] FIG. 3 is a flowchart illustrating a process flow for creating, packaging, and presenting adaptive audio content data according to one embodiment of the invention. The flow 300 of FIG. 3 is divided into three separate task groups labeled creation/authoring, packaging, and display. In general, the hybrid path and entity model shown in FIG. 2 allows most tasks - sound design, editing, pre-mixing and post-mixing - to be performed in the same way as they are currently performed, without adding unnecessary overhead to modern processes. In one embodiment of the invention, the audio adaptation functionality is provided in the form of software, firmware, or circuitry that is used in conjunction with audio generation and processing hardware, which hardware may be new hardware systems or modifications to existing systems. For example, digital audio workstations can provide modular applications that allow existing panning techniques to remain unchanged during sound design and editing. In this way, it is possible to form both paths and objects for a workstation in a 5.1 workstation or similar installations equipped with surrounding channels. Object and audio metadata is recorded during a session to prepare the pre- and post-mix stages in the duplicate theater.

[0068] As shown in FIG. 3, creation or authoring of tasks involves input by a user, such as in the following example, an audio engineer, of mixing control signals 302 into a mixing console or audio workstation 304. In one embodiment of the invention, metadata is embedded into the surface of the mixing console, allowing channel adjustment controls , panning and sound processing to work with both paths, or stems, and sound objects. Metadata can be edited using the console surface or workstation user interface, and ongoing audio monitoring is accomplished using the presentation and finishing unit (RMU) 306. Path and object audio data and associated metadata are recorded during a finishing session to create a “master copy” that includes the adaptive audio mix 310, and any other final output data 308 (such as a 7.1 or 5.1 surround mix for movie theaters). Existing authoring tools (eg digital audio workstations such as Pro Tools) can be used to allow audio engineers to mark individual audio tracks during a mixing session. Embodiments of the invention extend this concept by allowing the user to mark individual sub-segments within a track to aid in retrieval or quick identification of audio elements. The user interface for the mixing console, which allows metadata to be defined and created, can be implemented through graphical user interface elements, physical controls (such as sliders and buttons), or any combination thereof.

[0069] During the packaging step, the master copy file is wrapped using industry standard MXF wrapping procedures, hashed, and optionally encrypted to ensure the integrity of the audio content when delivered to the digital cinematography data packaging equipment. This step may be performed by digital cinematography processing (DCP) device 312, or any suitable audio processing device, depending on the end playback environment, such as a movie theater 318 equipped with standard surround sound, a movie theater 320 capable of adaptive sound, or any different playing environment. As shown in FIG. 3, the processing device 312 outputs corresponding audio signals 314 and 316 depending on the display environment.

[0070] In one embodiment of the invention, the adaptive audio reference contains an adaptive audio mix along with a standard DCI-compatible pulse code modulation (PCM) mix. The PCM mix can be presented by the data presentation and finishing module in the redundant theater or, if desired, created by a separate mixing run. The PCM audio forms a standard main audio track file in the digital cinematography processing device 312, and the adaptive audio forms an additional track file. The specified track file may be compatible with existing industry standards and may be ignored by DCI-compliant servers that cannot use it.

[0071] In one example of a cinematic playback environment, a DCP containing an adaptive audio track file is recognized by the server as a valid packet and is received by the server and then transmitted as a stream to the cinematic adaptive audio processing device. A system that has both linear PCM and adaptive audio files available can switch between them as needed. For distribution to the demonstration stage, the adaptive audio packaging scheme allows for one type of package to be delivered to the theater. The DCP package contains both the PCM file and the adaptive audio files. To provide secure delivery of movie content, or other similar content, may involve the use of security keys, such as delivery of a key-encrypted message (KDM).

[0072] As shown in FIG. 3, adaptive audio methodology is implemented by allowing the audio engineer to express his intent for the presentation of data and playback of audio content through audio workstation 304. By manipulating certain input controls, the engineer is able to specify where and how audio objects and audio elements are played depending on listening environment. Metadata is generated in the audio workstation 304 in response to the engineer's mixing input 302, providing data representation queues that control spatial parameters (e.g., position, speed, intensity, timbre, etc.) and indicate which speaker(s) , or groups of loudspeakers, in the listening environment play the corresponding sounds during the demonstration. The metadata is associated with the corresponding audio data in the workstation 304 or RMU 306 for the purpose of packaging and transmission via the DCP 312.

[0073] The graphical user interface and software tools that enable engineer control of the workstation 304 include at least a portion of the proprietary tools 106 of FIG. 1.

Hybrid audio codec

[0074] As shown in FIG. 1, system 100 includes a hybrid audio codec 108. This component includes an audio encoding, distribution, and decoding system that is configured to generate a single bitstream containing both traditional channel-based audio elements and audio object encoding elements. A hybrid audio encoding system is built around a channel-based encoding system that is configured to generate a single (unified) bitstream that is simultaneously compatible with (e.g., can be decoded by) a first decoder configured to decode audio data encoded in accordance with the first encoding protocol ( channel-based), and one or more secondary decoders configured to decode audio data encoded in accordance with one or more secondary (object-based) decoding protocols. The bitstream may include both encoded data (in the form of data packets) decoded by the first decoder (and ignored by any of the secondary decoders), and encoded data (eg, other data packets) decoded by one or more secondary decoders (and ignored by the first decoder). The decoded audio and associated information (metadata) from the first and one or more secondary decoders can then be combined such that the data presentation of both channel-based information and object-based information occurs simultaneously to recreate an exact copy of the environment, channels, spatial information and objects represented in a hybrid encoding system (for example, within a three-dimensional space or listening environment).

[0075] Codec 108 generates a bitstream containing encoded audio information and information related to multiple sets of channel (speaker) positions. In one embodiment of the invention, one set of channel positions is fixed and used for the channel-based coding protocol, while another set of channel positions is adaptive and used for the audio object-based coding protocol, and thus the channel configuration for audio subject may vary over time (depending on where the subject is placed in the sound field). Thus, a hybrid audio coding system may carry information about two sets of speaker locations for playback, where one set may be fixed and a subset of the other set. Devices supporting legacy encoded audio information can decode and represent data for audio information from a fixed subset, while a device capable of supporting a larger set can decode and represent data for additional encoded audio information that may be assigned in a time-varying manner to different speakers from a larger set. In addition, the system is independent of the first and one or more secondary decoders simultaneously present in the system and/or device. Therefore, a legacy and/or existing device/system containing only a decoder supporting the first protocol can output a fully compatible sound field whose data is subject to presentation through traditional channel-based playback systems. In this case, the unknown, or unsupported, portion(s) of the hybrid bitstream protocol (ie, audio information represented by the secondary encoding protocol) may be ignored by the system or decoder device supporting the first hybrid encoding protocol.

[0076] In another embodiment of the invention, codec 108 is configured to operate in a mode where the first encoding subsystem (supporting the first protocol) contains a combined representation of all audio field information (channels and objects) presented in both the first and one or more secondary encoder subsystems present in a hybrid encoder. This ensures that the hybrid bitstream includes backward compatibility with decoders that support only the first encoder subsystem protocol, allowing audio objects (typically carried in one or more secondary encoder protocols) to be mapped and presented in decoders that only support the first protocol.

[0077] In yet another embodiment of the invention, codec 108 includes two or more encoding subsystems, where each of these subsystems is configured to encode audio data in accordance with a different protocol and is configured to combine the outputs of the subsystems to generate a hybrid (unified) bitstream format .

[0078] One benefit of embodiments of the invention is the ability to transport a hybrid encoded audio bitstream across a wide variety of content distribution systems, where each distribution system traditionally supports only data encoded in accordance with a first encoding protocol. This eliminates the need to modify/change the protocol of any system and/or transport layer to specifically support a hybrid encoding system.

[0079] Audio coding systems typically use standardized bitstream elements to allow additional (arbitrary) data to be carried within the bitstream itself. This additional (arbitrary) data is typically skipped (i.e., ignored) during decoding of the encoded audio contained in the bitstream, but may be used for purposes other than decoding. Different audio coding standards express these additional data fields using unique nomenclature. Bitstream elements of this general type may include, but are not limited to, auxiliary data, missing fields, data stream elements, padding elements, overhead data, and nested data stream elements. Unless otherwise stated, the use of the expression “arbitrary data” herein does not imply a specific type or format of additional data, but rather should be interpreted as a general expression that covers any or all examples associated with the present invention.

[0080] The data channel provided by the "random" bitstream elements of the first encoding protocol in the hybrid encoding system's combined bitstream may carry one or more secondary (dependent or independent) audio data bitstreams (encoded in accordance with one or more secondary encoding protocols ). One or more secondary audio bitstreams may be broken down into blocks of N discrete values and compressed into "auxiliary data" fields of the first bitstream. The first bit stream is decoded by the corresponding (additional) decoder. In addition, the auxiliary data of the first bitstream may be extracted and recombined into one or more secondary audio bitstreams decoded by a processing device supporting the syntax of the one or more secondary bitstreams, and then combined or independently combined and presented. In addition, it is also possible to reverse the roles of the first and second bitstreams so that the data blocks of the first bitstream are compressed into auxiliary data of the second bitstream.

[0081] The bitstream elements associated with the second encoding protocol also carry and convey information characteristics (metadata) of the underlying audio, which may include, but are not limited to, the desired position, speed, and size of the audio source. This metadata is used during the decoding and data presentation processes to reconstruct the proper (ie, original) position of the associated audio object carried in the applicable bitstream. It is also possible to carry the above-described metadata that applies to audio objects contained in one or more secondary bitstreams present in the hybrid stream in the bitstream elements associated with the first encoding protocol.

[0082] Bitstream elements associated with one, first or second, encoding protocol or both encoding protocols of a hybrid encoding system carry/transmit contextual metadata that identifies spatial parameters (eg, the essence of the properties of the signal itself) and additional information describing the type essence of the underlying audio in the form of special audio classes that are carried in a hybrid encoded audio bitstream. Such metadata may indicate, for example, the presence of speech dialogue, music, dialogue against music, applause, singing, etc., and may be used to adaptively modify the behavior of interconnected pre- and post-processing modules upstream or downstream of the hybrid system. coding.

[0083] In one embodiment of the invention, codec 108 is configured to operate with a shared, or common, bit pool in which the bits available for encoding are “shared” among all or a portion of encoding subsystems supporting one or more protocols. Such a codec can distribute available bits (from a common, "shared" bit pool) among encoding subsystems in order to optimize the overall audio quality of a unified bitstream. For example, during a first period of time, the codec may assign more available bits to the first encoding subsystem, and fewer available bits to the remaining subsystems, while during a second period of time, the codec may assign fewer available bits to the first encoding subsystem, and more available bits - other coding subsystems. The decision on how to distribute bits between encoding subsystems may depend, for example, on the results of a statistical analysis of the shared bit pool and/or on an analysis of the audio content encoded by each of the subsystems. The codec can assign bits from a shared pool such that the unified bitstream constructed by compressing the pins of the encoding subsystems maintains a constant frame length/bit rate for a given period of time. Also, in some cases, it is possible to change the frame length/bit rate over a given period of time.

[0084] In an alternative embodiment of the invention, codec 108 generates a unified bitstream including data encoded in accordance with the first encoding protocol, configured and transmitted as an independent substream of the encoded data stream (which will be decoded by a decoder supporting the first encoding protocol), and the data , encoded in accordance with the second protocol, are transmitted as an independent or dependent substream of the encoded data stream (a stream that will be ignored by a decoder supporting the first protocol). More generally, in one class of embodiments of the invention, a codec generates a unified bitstream including two or more independent or dependent substreams (where each substream includes data encoded according to a different, or identical, encoding protocol).

[0085] In yet another alternative embodiment of the invention, codec 108 generates a unified bitstream including data encoded in accordance with the first encoding protocol, configured and transmitted with a unique bitstream identifier (which will be decoded by a decoder supporting the first encoding protocol associated with unique bitstream identifier), and data encoded in accordance with the second protocol, configured and transmitted with a unique bitstream identifier, which a decoder supporting the first protocol will ignore. More generally, in one class of embodiments of the invention, a codec generates a unified bitstream including two or more substreams (where each substream includes data encoded in accordance with a different, or identical, encoding protocol, and where each substream carries a unique bitstream identifier). The methods and systems described above for creating a unified bitstream include the ability to convey (to the decoder) an unambiguous signal about what interleaving and/or protocol was used in the hybrid bitstream (for example, whether AUX, SKIP, DSE or the described subflow approach).

[0086] The hybrid encoding system is configured to support deinterleaving/decompressing and reinterleaving/recompressing bit streams supporting one or more secondary protocols into a first bit stream (supporting the first protocol) at any processing point throughout the media delivery system. The hybrid codec is also configured to have the ability to encode input audio streams at different sampling rates into a single bitstream. This provides a means for efficiently encoding and distributing audio sources containing signals with inherently different bandwidths. For example, dialogue tracks typically have significantly less bandwidth than music and effects tracks.

Data presentation

[0087] In one embodiment of the invention, an adaptive audio system allows multiple (eg, up to 128) tracks to be packaged, typically as a combination of paths and objects. The basic audio data format for an adaptive audio system includes multiple independent monaural audio streams. Each stream has metadata associated with it that indicates whether the stream is a channel-based stream or an object-based stream. Channel-based streams contain data representation information encoded by a channel name, or label; and object-based streams contain location information encoded through mathematical expressions encoded in additional associated metadata. The original independent audio streams are then packaged into a single binary sequence that contains all the audio data in an ordered manner. This configuration of adaptive data allows the audio data to be represented according to an allocentric frame of reference, in which the final presentation location of the audio data is based on the playback environment so that it matches the mixing operator's intent. Thus, the origin of a sound may be indicated in the playback room's reference frame (eg, the middle of the left wall) rather than from a specific labeled speaker or group of speakers (eg, the left surround). Object position metadata contains the appropriate allocentric reference frame information needed to correctly play audio using the positions of available speakers in a room that is prepared to play adaptive audio content.

[0088] The data representation operator receives a bitstream encoding audio tracks and processes the content according to the signal type. Paths are fed into arrays, which will potentially require different latency and adjustment processing than individual objects. The process supports the presentation of data from specified paths and objects to multiple (up to 64) speaker outputs. FIG. 4 is a flow diagram of a data presentation step of an adaptive audio system according to one embodiment of the invention. As shown in system 400 of FIG. 4, multiple input signals, such as up to 128 audio tracks that include adaptive audio signals 402, are generated by certain components of the creation, authoring, and packaging stages of the system 300, such as the RMU 306 and the data processing unit 312. These signals contain channel-based paths and objects that are used by the data presentation operator 404. Channel-based audio (paths) and objects are input to a level control device 406, which provides control of the output levels, or amplitudes, of the various audio components. Some audio components may be processed by the array equalization component 408. The adaptive audio signals are then passed through a processing component 410 in Circuit B, which generates multiple (eg, up to 64) output signals to the speakers. In general, chain B signals refer to the signals processed by power amplifiers, signal splitters, and loudspeakers, as opposed to the contents of chain A, which constitute the soundtrack on film.

[0089] In one embodiment of the invention, presentation operator 404 runs a presentation algorithm that makes the best possible intelligent use of the surrounding speakers in the theater. By improving the power switching and frequency response of the surround speakers, and by maintaining the same monitoring reference level for each output channel, or speaker, in a theater, objects panned between the on-screen and surround speakers can maintain their SPL level and have more close timbral matching, which is important, without increasing the overall sound pressure level in the cinema. An array of suitably specified surround speakers will typically have sufficient headroom to reproduce the maximum available dynamic range within a 7.1 or 5.1 surround soundtrack (i.e. 20 dB above the reference level), but it is unlikely that a single surround speaker will had the same level headroom as a large multi-position on-screen loudspeaker. As a result, there are likely to be cases where an object placed in an ambient field will require greater sound pressure than would be achievable using a single ambient loudspeaker. In these cases, the presentation operator will distribute the sound over an appropriate number of loudspeakers in order to achieve the required sound pressure level. Adaptive Audio improves the quality and power handling of surrounding speakers for improved fidelity. It provides support for surround speaker bass management through the use of optional rear woofers, which allow each surround speaker to achieve improved power handling while potentially using smaller speaker enclosures. It also allows for side surround speakers to be added closer to the screen than current practice, in order to provide a smooth transition of objects from the screen to the surroundings.

[0090] By using metadata to indicate location information of audio objects along with certain data representation processes, system 400 provides content creators with a comprehensive, flexible way to go beyond the limitations of existing systems. As defined above, current systems create and distribute sound that is fixed at specific speaker locations with limited knowledge of the type of content being conveyed in the audio entity (the portion of the audio that is being played). Adaptive audio system 100 provides a new, hybrid approach that includes capabilities for both audio specific to speaker locations (left channel, right channel, etc.) and object-based audio elements that contain generalized spatial information that may include, but are not limited to, location, size, and speed. This hybrid approach provides a balance between precision (provided by fixed speaker locations) and flexibility in data representation (generalized audio objects). The system also provides additional useful information about the audio content, which the content creator pairs with the audio entity at the time the content is created. This information provides significant, detailed information about the characteristic properties of sound, which can be used in extremely powerful ways during data presentation. Said characteristic properties may include, but are not limited to, content type (dialogue, music, effect, noise effect, background/environment, etc.), characteristic properties in space (3D position, 3D size, speed), and data representation information ( reference to speaker location, channel weights, gain, bass management information, etc.).

[0091] The adaptive audio system described in this disclosure provides significant information that can be used to represent data to a widely varying number of endpoints. In many cases, the best data presentation technique used is highly dependent on the device at the endpoint. For example, home theater systems and sound bars can contain 2, 3, 5, 7 or even 9 individual speakers. Many other types of systems such as televisions, computers and music hardware contain only two speakers, and almost all traditionally used devices have a binaural headphone output (PC, laptop, tablet, cell phone, music player, etc.). However, for the traditional audio currently distributed (monaural, stereo, 5.1, 7.1 channels), endpoint devices often need to make simplified decisions and trade-offs to present data and reproduce audio that is currently distributed in channel-specific form /speakers. In addition, there is little or no information conveyed regarding the actual content that is being distributed (dialogue, music, ambience), and there is little or no information about the content creator's intent for the sound to be reproduced. However, the adaptive audio system 100 provides this information and potentially access to audio objects that can be used to create a next generation immersive user experience.

[0092] The system 100 allows the content creator to embed the spatial intent of the mix in the bitstream using metadata such as position, size, speed, etc. metadata through a unique and compelling metadata and adaptive audio transmission format. This allows for much greater flexibility when reproducing sound in a space. From a spatial perspective, adaptive audio allows you to tailor a mix to the exact position of the speakers in a given room, avoiding the spatial distortion that occurs when the geometry of the playback system is not identical to the original design. In modern audio reproduction systems, where only the audio for the speaker channel is transmitted, the intent of the content creator is unknown. System 100 uses metadata passed throughout the creation and distribution pipeline. An adaptive audio playout system may use this metadata information to play the content in a manner that is consistent with the original intent of the content creator. Moreover, the mix can adapt to the exact hardware configuration of the playback system. Currently, in presentation equipment such as televisions, home theaters, sound bars, portable music player hardware modules, etc., there are many different possible speaker configurations and types. When these systems currently transmit channel-specific audio information (ie, left and right channel audio or multi-channel audio), the system must process the audio so that it is appropriately matched to the capabilities of the data presentation equipment. One example is standard stereo audio sent to a soundbar containing more than two speakers. In modern audio reproduction, where only audio is transmitted for speaker channels, the intent of the content creator is unknown. By using metadata passed throughout the creation and distribution process, an adaptive audio playout system can use this information to play content in a way that is consistent with the original intent of the content creator. For example, some soundbars contain side-mounted additional speakers designed to create a sense of envelopment. For adaptive audio, spatial information and content type (such as environmental effects) can be used by the soundbar to send only the appropriate audio to specified side additional speakers.

[0093] The adaptive audio system allows unlimited interpolation of speakers in the system in all front/rear, left/right, top/bottom, near/far sizes. In modern audio reproduction systems, there is no knowledge of how to process sound when it may be desirable to position the sound so that it is perceived by the listener as being between two loudspeakers. Currently, for sound that is assigned only to a specific loudspeaker, a spatial quantization factor is introduced. In the case of adaptive sound, the spatial location of the sound can be known accurately, and it can be reproduced accordingly by the sound reproduction system.

[0094] With respect to the presentation of data by headphones, the creator's intention is realized by aligning head-related transfer functions (HRTFs) with position in space. When sound is played through headphones, spatial virtualization can be achieved by applying a transfer function related to the head that processes the sound by adding perceptual properties that create the perception of the sound being played in three-dimensional space rather than through headphones. The accuracy of spatial reproduction depends on the selection of an appropriate HRTF, which can vary based on several factors including spatial position. The use of spatial information provided by the adaptive audio system may result in the selection of a single HRTF, or a constantly changing number of HRTFs, in order to significantly enhance the perception of the playback.

[0095] The spatial information conveyed by an adaptive audio system can not only be used by the content creator to create an immersive entertainment experience (movie, television program, music, etc.), but the spatial information can also indicate where the listener is located relative to such physical objects, such as buildings or geographic points of interest. This could allow the user to interact with a virtualized audio experience that is related to the real world, i.e. with additional reality.

[0096] Embodiments of the invention enable spatial upmixing by performing advanced upmixing by reading metadata only when audio data of objects is not available. Knowing the position of all objects and their types allows the upmixer to better distinguish elements in tracks on a channel-based basis. To create a high-quality upmix with little or no audible distortion, existing upmix algorithms have to infer information such as the type of audio content (speech, music, environmental effects) as well as the location of various elements in the audio stream. In many cases, the information provided may be incorrect or inappropriate. For adaptive audio, additional information available from metadata related to, for example, audio content type, spatial position, speed, audio object size, etc., can be used by the upmixing algorithm to create a high-quality reproduction result. The system also spatially correlates audio and video by precisely positioning the audio object on the screen in relation to visible elements. In this case, an immersive audio/video playback experience is possible, especially on large screens, if the reproduced location of some audio elements in space corresponds to the image elements on the screen. One example is dialogue in a film or television program that spatially coincides with the person or character speaking on screen. For regular audio, based on speaker channels, there is no easy way to determine where in space dialogue should be located to match the location of the person or character on screen. For audio information available through adaptive audio, such audiovisual alignment can be achieved. Visual positional and spatial audio alignment can also be used for non-role/non-dialogue objects such as cars, trucks, animations, etc.

[0097] The system 100 facilitates spatial masking processing because the spatial intent of the mix, available through adaptive audio metadata, means that the mix can be tailored to any speaker configuration. However, there is a risk of downmixing objects in the same or nearly the same location due to limitations of the playing system. For example, an object that is intended to be panned into the left rear channel may be downmixed into the left front channel if there are no surrounding channels, but if, at the same time, a louder element appears in the left front channel and is downmixed the object will be masked and disappear from the mix. Using adaptive audio metadata, spatial masking can be predicted by the presentation operator, and the spatial and/or loudness downmix parameters for each object can be adjusted so that all audio elements of the mix remain perceptually identical to those in the original mix. Because the view operator understands the spatial relationship between the mix and the playback system, it is able to “lock” objects to nearby speakers instead of creating a ghost image between two or more speakers. Although the spatial representation of the mix may be somewhat distorted, it also allows the presentation operator to avoid unintentional ghosting. For example, if the angular position of the left speaker at the mixing stage does not correspond to the angular position of the left speaker in the playback system, using the snap to the nearest speaker function can avoid the playing system from playing a constant ghost image of the left channel of the mixing stage.

[0098] In terms of content processing, adaptive audio system 100 allows a content creator to create individual audio objects and add content information that can be communicated to the playback system. This allows for greater flexibility in how audio is processed before playback. In terms of content processing and data presentation, the adaptive audio system allows processing to be tailored to the type of object. For example, dialog enhancement can only be applied to dialog objects. Dialogue enhancement refers to a method of processing audio that contains dialogue so that the audibility and/or intelligibility of the dialogue is increased and/or improved. In many cases, the audio processing that is applied to dialogue is inappropriate for non-dialogue audio content (ie, music, environmental effects, etc.) and may result in unwanted audible distortion. For adaptive audio, the audio object may contain only dialogue in one of the content fragments, and it may be marked accordingly so that the data representation solution can selectively apply dialogue enhancement to only the dialogue content. Additionally, if the audio object is only dialogue (and not, as is often the case, a mixture of dialogue and other content), then dialogue gain processing can handle dialogue exclusively (thus limiting any processing performed on any other content). Likewise, bass management (filtering, attenuation, boosting) can be targeted to specific objects based on their type. Bass management refers to selectively highlighting and processing only the bass (or even lower) frequencies in a specific piece of content. In modern sound systems and delivery mechanisms, this process is “blind”, i.e. applies to the entire sound. For adaptive audio, specific audio objects for which bass management is appropriate can be identified by the metadata, and data representation processing can be applied accordingly.

[0099] Adaptive audio system 100 also provides dynamic range compression and object-based selective upmixing. Traditional audio tracks are as long as the content itself, whereas an audio object can only appear within the content for a limited amount of time. Metadata associated with an object may contain information about its average and peak signal amplitude, as well as its onset or rise time (especially for short-lived material). This information could allow the compression device to better adapt its compression and timing constants (rise, release, etc.) to better suit the content. For selective upmixing, content creators can choose to indicate in the adaptive audio bitstream whether the object should be upmixed or not. This information allows the adaptive audio data operator and the upmix operator to discern which audio elements can be safely upmixed while not interfering with the creator's intent.

[00100] Embodiments of the invention also allow the adaptive audio system to select a preferred presentation algorithm from a number of available presentation algorithms and/or surround sound formats. Examples of available data representation algorithms include: binaural, stereo dipole, ambiophonic, wavefield synthesis (WFS), multi-channel panning, raw stems with position metadata. Other algorithms include dual balance and vector-based amplitude panning.

[00101] The binaural distribution format uses a two-channel representation of the sound field based on the signal present in the left and right ears. Binaural information can be created through in-channel recording or synthesized using HRTF models. Playing a binaural presentation is typically done through headphones or by using crosstalk suppression. Playback through an arbitrary speaker circuit would require signal analysis to determine the associated sound field and/or signal source(s).

[00102] Stereo dipole presentation is a transaural crosstalk reduction process to make binaural signals suitable for playback through stereo speakers (eg, + and - 10 degrees from center).

[00103] Ambiophony is a distribution format and reproduction method that is encoded in a four-channel form called the B format. The first channel, W, is an omnidirectional pressure signal; the second channel, X, is a directional pressure gradient containing anterior and posterior information; the third channel, Y, contains left and right, and Z contains top and bottom. These channels define a first-order discrete value for the full sound field at a given point. Ambiophony uses all available loudspeakers to recreate a sampled (or synthesized) sound field within an array of loudspeakers so that when some of the loudspeakers push, others drag.

[00104] Wavefield synthesis is a method of representing data for audio reproduction based on the precise construction of the wavefield by secondary sources. WFS is based on Huygens' principle and is implemented as arrays of loudspeakers (tens or hundreds) that surround the listening space in a ring and act in a coordinated, phased manner to recreate each individual sound wave.

[00105] Multi-channel panning is a distribution format and/or method of presenting data and may be referred to as channel-based audio. In this case, the sound is displayed as a number of discrete sources for playback through an equal number of speakers located at certain angles relative to the listener. The content creator/mixer can create virtual images by panning signals between adjacent channels to create a sense of direction; to create a perception of the direction and properties of the environment, primary reflections, reverberation, etc. can be mixed into several channels.

[00106] Raw stems with position metadata is a distribution format that can also be referred to as object-based audio. In this format, distinct close-mic audio sources are presented along with position and environment metadata. Virtual source data is represented based on metadata, playing equipment, and listening environment.

[00107] The adaptive audio format is a hydride of the multi-channel panning format and the raw stem format. The data presentation method in the present embodiment of the invention is multi-channel panning. For audio channels, data presentation (panning) occurs at the time of authoring, while for objects, data presentation (panning) occurs at playback.

Metadata and Adaptive Audio Transmission Format

[00108] As set forth above, metadata is generated at the creation stage for the purpose of encoding certain position information for audio objects and to accompany an audio program to assist in the presentation of audio program data and, in particular, to describe an audio program in a manner that allows audio program data to be represented. programs for a wide selection of playing equipment and playback environments. Metadata is generated for a given program and the editors and mixing operators who create, assemble, edit, and process audio during composition. An important characteristic of the adaptive audio format is the ability to control how audio will be translated into systems and playback environments that differ from the mixing environment. In particular, a given theater may have fewer capabilities than a mixing environment.

[00109] The adaptive audio data representation operator aims to make the best use of available hardware to recreate the intent of the mixing operator. Additionally, adaptive audio authoring tools allow the mix operator to preview and adjust how mix data will be presented in different playback configurations. All metadata values may vary depending on the playback environment and speaker configuration. For example, based on the configuration or playback mode, a different mixing level may be specified for a given audio element. In one embodiment of the invention, the list of conditional playback modes is expandable and includes the following modes: (1) playback based on channels only: 5.1, 7.1, 7.1 (upper), 9.1; and (2) discrete speaker playback: 3D, 2D (no overhead).

[00110] In one embodiment, the metadata controls, or dictates, various features of the adaptive audio content and is organized based on various types, including: program metadata, audio metadata, and data presentation metadata (for channels and objects). Each metadata type includes one or more metadata elements that provide values for the characteristics referenced by the identifier (ID). FIG. 5 is a table that lists metadata types and associated metadata elements for an adaptive audio system, according to one embodiment of the invention.

[00111] As shown in table 500 of FIG. 5, the first type of metadata is software metadata that includes metadata elements defining frame rate, track count, extensible channel description, and mixing step description. The frame rate metadata element describes the frame rate of audio content in units of frames per second (fps). The raw audio format does not require the inclusion of audio framing or metadata because the audio is delivered as complete tracks (the duration of a reel or entire movie) rather than segments of audio (the duration of an object). The raw format does not require carrying all the information needed to unlock the adaptive audio encoder to frame the audio data and metadata, including the actual frame rate. Table 1 shows the ID, example values, and description of the frame rate metadata element.

TABLE 1

[00112] The track count metadata element indicates the number of audio tracks in a frame. One example of an adaptive audio decoder/processor can support up to 128 audio tracks simultaneously, while an adaptive audio format will support any number of audio tracks. Table 2 shows the ID, example values, and description of the track count metadata element.

TABLE 2

[00113] Channel-based audio can be assigned to non-standard channels, and the "expandable channel description" metadata element allows mixes to use new channel positions. The following metadata, shown in Table 3, must be generated for each extension channel.

TABLE 3

[00114] The "mixing stage description" metadata element specifies the frequency at which a particular loudspeaker generates half of the bandwidth power. Table 4 shows the ID, example values, and description of the Mix Stage Description metadata element, where LF is the low frequency, HF is the high frequency, and the 3 dB point is the edge of the loudspeaker passband.

TABLE 4

[00115] As shown in FIG. 5, the second type of metadata is audio metadata. Each channel-based or object-based audio element consists of an audio entity and metadata. An audio entity is a monophonic audio stream carried by one or more audio tracks. Associated metadata describes how the audio entity is stored in memory (audio metadata, such as sample rate), or how its data is to be represented (data representation metadata, such as the desired position of the audio source). In general, the audio tracks are continuous throughout the duration of the audio program. The program editor, or mixing operator, is responsible for assigning audio elements to tracks. Track usage is expected to be sparse, i.e. the average number of simultaneously used tracks can be only 16-32. In a typical implementation, audio will be transmitted efficiently using a lossless encoder. However, alternative implementations are possible, such as transmission of unencoded audio data or lossy encoded audio data. In a typical implementation, the format consists of up to 128 audio tracks, where each track has a single sampling rate and a single encoding system. Each track lasts for the duration of the movie (there is no explicit reel support). Assigning objects to tracks (time compression) is the responsibility of the content creator (mixing operator).

[00116] As shown in FIG. 3, audio metadata includes the elements "sampling rate", "bit depth" and "encoding systems". Table 5 shows the ID, example values, and description of the sample rate metadata element.

TABLE 5

[00117] Table 6 shows the ID, example values, and description of the bit depth metadata element (for PCM and lossless compression).

TABLE 6

[00118] Table 7 shows the ID, example values, and description of the encoding system metadata element.

TABLE 7

[00119] As shown in FIG. 5, the third type of metadata is data representation metadata. Data representation metadata specifies values that help the data representation operator match as closely as possible to the original mixing operator's intent, regardless of the playback environment. The set of metadata elements for channel-based audio and object-based audio is different. The first metadata field of the data view makes a choice between two types of audio - channel-based audio and object-based audio, as shown in Table 8.

TABLE 8

[00120] Data representation metadata for channel-based audio includes a position metadata element that indicates the position of the audio source as one or more speaker positions. Table 9 shows the ID and values of the position metadata element for the channel-based case.

TABLE 9

[00121] The data presentation metadata for channel-based audio also includes a "data presentation control" element that specifies certain characteristics regarding playback of channel-based audio, as shown in Table 10.

TABLE 10

[00122] For object-based audio, the metadata includes elements similar to those for channel-based audio. Table 11 presents the IDs and values for the object position metadata element. The position of an object is described in one of three ways: three-dimensional coordinates; plane and two-dimensional coordinates; or a line and a one-dimensional coordinate. The data presentation method may be adapted based on the type of position information.

TABLE 11

[00123] The IDs and values of the Object Metadata Presentation Control metadata elements are shown in Table 12. These values provide additional controls, or optimizations, for object-based audio data presentation.

TABLE 12

[00124] In one embodiment of the invention, the metadata described above and illustrated in FIG. 5 are generated and stored as one or more files that are associated, or indexed, with the corresponding audio content so that the audio streams are processed by an adaptive audio system interpreting the metadata generated by the mixing operator. It should be noted that the metadata described above is one example of a set of IDs, values and definitions, and may include other or additional metadata elements for use in an adaptive audio system.

[00125] In one embodiment of the invention, two (or more) sets of metadata elements are associated with each of the channel-based and object-based audio streams. A first set of metadata is applied to a number of audio streams for the first playing environment conditions, and a second set of metadata is applied to a number of audio streams for the second playing environment conditions. The second, or subsequent, set of metadata elements replaces the first set of metadata elements for a given audio stream based on the conditions of the playing environment. Conditions may include factors such as room size, shape, composition of materials within the room, current occupancy and density of people in the room, ambient noise characteristics, ambient light characteristics, and any other factors that may affect the sound or even the mood of the room. playback environment.

Layout and finishing

[00126] The data presentation stage 110 of the adaptive audio processing system 100 may include audio composition steps that result in the creation of the final mix. In one cinematic application, the three main categories of sound used in mixing a motion picture sound recording are dialogue, music, and effects. Effects consist of sounds that are not dialogue or music (eg, ambient noise, background/production noise). Sound effects can be recorded, or synthesized, by a sound engineer, or they can be taken from effect libraries. One subset of effects that involve special sources of noise (such as footsteps, doors, etc.) is known as noise effects and is performed by sound designers. Different types of sounds are labeled and panned accordingly by the sound engineers.

[00127] FIG. 6 illustrates an example flow of a layout process in an adaptive audio system according to one embodiment of the invention. As shown in diagram 600, all of the individual audio components of the music, dialogue, sound effects, and effects are mixed together in the dub theater during the final mix 606, and the re-recording mixer(s) 604 use the premixes (also known as the "minus mix"). along with individual audio objects and positional data to create stems as a way to group, for example, dialogue, music, effects, noise effects and background sounds. In addition to shaping the final 606 mix, the music and all effects stems can be used as the basis for creating versions of the film dubbed in other languages. Each stem consists of a channel-based path and several audio objects with metadata. To form the final mix, stems are combined. Using object panning information from both the audio workstation and the mixing console, presentation and post processing unit 608 presents audio data to speaker locations in the duplicating theater. This view of the data allows mixing operators to hear how channel-based paths and audio objects fit together, and also provides the ability to present the data in different configurations. The mixing operator can use conditional metadata, which is the default data for meaningful profiles, to control how content data is presented to surrounding channels. This way, mixing operators retain full control over how the movie is played across all scaled environments. After each of the re-recording steps 604 and the final mixing step 606, or after both of these steps, a monitoring step may be included to allow the mix operator to listen to and evaluate the intermediate content generated during each of these steps.

[00128] During the final processing session, the specified stems, objects and metadata are brought together in an adaptive audio package 614, which is produced by the master copy operator 610. This package also contains a backwards-compatible (legacy 5.1 or 7.1) 612 surround sound mix for cinemas. The presentation/post-processing unit (RMU) 608 can optionally present this output signal data, thereby eliminating the need for any additional flow steps in generating existing channel-based output data. In one embodiment of the invention, audio files are packaged using standard Material Communications Format (MXF) encapsulation. The adaptive audio mix master file can also be used to generate other outputs, such as custom multi-channel or stereo mixes. Intelligent profiles and conditional metadata enable controlled views of data that can significantly reduce the time required to create such mixes.

[00129] In one embodiment of the invention, a packaging system may be used to create a digital film package for output data including an adaptive audio mix. Audio track files may become chained together, causing synchronization errors with adaptive audio track files. Some countries require that track files be added during the packaging phase, such as adding Hearing Impaired (HI) tracks or Visually Impaired Description (VI-N) tracks to the main audio track file.

[00130] In one embodiment of the invention, the speaker array in the playback environment may include any number of surround speakers positioned and labeled in accordance with surround sound standards. Any number of additional speakers to accurately represent object-based audio content data may also be placed based on the conditions of the playback environment. These additional speakers can be installed by the audio engineer, and this setting is presented to the system in the form of a setup file, which is used by the system to present these object-based adaptive audio components to a specific speaker, or speakers, within the overall speaker array. Said setup file contains at least a list of speaker designations and channel assignments to individual speakers, information regarding speaker grouping and dynamic assignment based on the relative position of the speakers in the playback environment. The specified mapping execution time is used by the "snap to" feature of the system, which presents object-based point source audio content data to a specific loudspeaker that the audio engineer intends to be closest to the perceived location of the sound.

[00131] FIG. 7 is a diagram of one example flowchart for packaging a digital film copy using adaptive audio files according to one embodiment of the invention. As shown in diagram 700, audio files, including both adaptive audio files and 5.1 or 7.1 surround audio files, are input to wrapper/encryption block 704. In one embodiment of the invention, when a digital film package is created at block 706, the PCM MXF file (with associated additional tracks attached) is encrypted using SMPTE specifications in accordance with current practice. The adaptive audio MXF is packaged as an auxiliary track file and optionally encrypted using the SMPTE specification using a symmetric content control key. This single DCP 708 can then be delivered to any Digital Cinema Initiatives (DCI) compliant server. In general, any installations that are not suitably equipped will simply ignore the secondary track file containing the adaptive audio audio track, and will use the existing main audio track file for standard playback. Installations equipped with appropriate adaptive audio processing devices will be capable of receiving and reproducing the adaptive audio audio track where applicable, reverting to the standard audio track when necessary. The wrapper/encryption component 704 may also deliver the input signal directly to the KDM distribution unit 710 to generate an appropriate security key for use in the digital theater server. Other movie elements, or files, such as subtitles 714 and images 716 may be wrapped and encrypted along with the audio files 702. In this case, some processing steps may be included, such as compression 712 in the case of image files 716.

[00132] In terms of content management, adaptive audio system 100 allows a content creator to create individual audio objects and add information about the content to be delivered to the playback system. This allows for significantly greater flexibility in managing audio content. From a content management perspective, adaptive audio techniques enable several distinctive features. These features include changing the language of the content simply by replacing the dialog object to save space, download efficiency, geographical adaptation of playback, etc. Films, television and other entertainment programs are typically distributed throughout the world. This often requires that the language of a piece of content change depending on where it is played (French for films shown in France, German for TV programs shown in Germany). Nowadays, this often requires the creation, packaging and distribution of a completely independent audio track. In the case of adaptive audio and its inherent concept of audio objects, the dialogue for a piece of content may be an independent audio object. This allows you to easily change the language of the content without updating or changing other elements of the audio track such as music, effects, etc. This can apply not only to foreign languages, but also to language unsuitable for some viewers (eg children's television programs, airplane movies, etc.), targeted advertising, etc.

Installation and Hardware Considerations

[00133] The adaptive audio file format and associated processing devices allow for changes in the installation, calibration and maintenance of theater equipment. With the introduction of a much larger number of potential output signals of loudspeakers, each of which is adjusted and balanced separately, there is a need for intelligent and prompt automatic correction of the amplitude-frequency response of the room, which can be carried out through the possibility of manual adjustment of any automatic correction of the amplitude-frequency response of the room . In one embodiment of the invention, the adaptive audio system uses an optimized equalizer with a 1/12-octave frequency band. For more precise sound balance in a cinema, up to 64 output signals can be processed. The system also allows for scheduled monitoring of individual loudspeaker outputs from the cinema processing unit output all the way to the sound being reproduced in the auditorium. Local or network alerts can be generated to ensure that appropriate action is taken. A flexible presentation system can automatically remove a damaged loudspeaker or amplifier from the playback chain and bypass it, allowing the display to continue.

[00134] The cinematic processing device can be connected to the digital cinema server via existing 8xAES main audio connectors and an Ethernet connection for streaming adaptive audio data. Playing ambient 7.1 or 5.1 content uses existing PCM connections. The adaptive audio data stream is transmitted over an Ethernet network to the cinematic processing device for decoding and presentation of the data, and communication between the server and the cinematic processing device allows audio to be identified and synchronized. If there is any problem with the playback of the adaptive audio track, the audio reverts back to PCM audio for Dolby Surround 7.1 or 5.1.

[00135] Although embodiments of the invention have been described in relation to 5.1 and 7.1 surround sound systems, it should be appreciated that many other current and future surround sound configurations, including 9.1, 11.1, may be used in combination with these embodiments of the invention. and 13.1ff.

[00136] The adaptive audio system is designed for both content creators and movie distributors to decide how data should be presented in different playback speaker configurations. The ideal number of speaker output channels used will vary according to room size. Recommended speaker placement therefore depends on many factors such as size, composition, seating configuration, environment, average auditorium sizes, etc. Examples, or samples, of speaker configurations and arrangements are presented in this disclosure for purposes of illustration only and are not intended to limit the scope of any of the claimed embodiments of the invention.

[00137] The recommended speaker layout for an adaptive audio system remains compatible with existing cinematography systems, which is vital in order not to compromise playback of existing channel-based 5.1 and 7.1 formats. In order to preserve the adaptive audio engineer's intent and the intent of the 7.1 and 5.1 content mix operators, the positions of existing on-screen channels should not be changed too radically in an attempt to enhance, or emphasize, the introduction of new speaker locations. Unlike using all 64 available output channels, adaptive audio format data can be accurately presented in theaters to speaker configurations such as 7.1, even allowing the format (and its benefits) to be used in existing theaters without making changes to amplifiers or speakers .

[00138] Depending on the theater design, different speaker locations may have different efficiencies, so there is currently no industry-specific ideal number or placement of channels. Adaptive audio is supposed to be truly adaptable and capable of playing accurately in a variety of auditoriums, whether they have a limited number of playback channels or many channels with highly flexible configurations.

[00139] FIG. 8 is a top view 800 of one example layout of proposed speaker locations for use with an adaptive audio system in a typical algorithm, and FIG. 9 is a front view 900 of this example layout of proposed speaker locations on an auditorium screen. The preferred starting position in this case is 2/3 of the way back from the screen to the back wall on the center line of the screen. Standard on-screen speakers 801 are shown in their normal positions relative to the screen. A study of the perception of screen plane elevation has shown that additional speakers 804 behind the screen, such as the left center (Lc) and right center (Rc) screen speakers (at the left sub and right sub channel locations in 70mm film formats), may be useful in creating smoother panning across the screen. Therefore, the optional loudspeakers listed are recommended, especially in auditoriums with screens greater than 12 m (40 ft) wide. All on-screen speakers should be angled so that they are aimed in the direction of the original position. The recommended placement of the 810 subwoofer behind the screen should remain the same, including maintaining an asymmetrical placement of the cabinet relative to the center of the room to avoid generating standing waves. Additional 816 subwoofers can be located at the rear of the theater.

[00140] The surround speakers 802 should be individually wired to the rear of the amplifier rack and should, where possible, be individually amplified by a designated power amplification channel corresponding to the loudspeaker handling power according to the data sheet. Ideally, loudspeaker surrounds should be specified to handle increased SPL for each individual loudspeaker and, where possible, wider frequency response. As a rule of thumb, for an average sized theater the surround speaker spacing should be between 2-3 m (6 ft 6 in - 9 ft 9 in), with left and right surround speakers placed symmetrically. However, it is believed that, as opposed to using absolute distances between speakers, spacing between surrounding speakers is most effective at the angles subtended from a given listener between adjacent speakers. For optimal playback throughout the auditorium, the angular distance between adjacent speakers should be 30 degrees or less when measured from each of the four corners of the main listening area. Good results can be achieved at intervals of up to 50 degrees. For each loudspeaker zone, where possible, loudspeakers should maintain equal line spacing near the seating zone. Line spacing outside the listening area, such as between the front row and the screen, may be slightly larger. FIG. 11 is one example of the arrangement of the top surround speakers 808 and the side surround speakers 806 relative to a home position according to one embodiment of the invention.

[00141] The additional side surround speakers 806 must be mounted closer to the screen than is currently recommended in practice - starting at approximately one-third of the distance to the rear of the auditorium. When playing Dolby Surround 7.1 or 5.1 soundtracks, these speakers are not used as surround speakers, but allow for smooth transitions and better tonal consistency when panning objects from the on-screen speakers to the surrounds. To maximize the sense of space, surround arrays should be positioned as low as those used in practice, subject to the following restrictions: vertical placement of surround speakers at the front of the array should be close enough to the height of the acoustic center of the on-screen speaker and high enough to maintain good coverage across seating area in accordance with the directionality of the loudspeaker. The vertical placement of surrounding speakers should be such that they form a straight line from front to back and (generally) angled upward so that the relative elevation of the surrounding speakers above the audience is maintained toward the rear of the theater as the seating elevation increases, as shown in FIG. 10, which is a side view of one example layout of proposed speaker locations for use with an adaptive audio system in a typical auditorium. In practice, this is most easily achieved by selecting an elevation for the frontmost and rearmost surround speakers and placing the remaining speakers on a line between these points.

[00142] In order to provide optimal coverage for each speaker throughout the seating area, side surround speakers 806, rear surround speakers 816, and top surround speakers 808 must be aimed in the direction of the theater reference position according to certain spacing guidelines, position, angle, etc.

[00143] Adaptive audio system and format embodiments for cinematography achieve enhanced levels of presence and audience engagement compared to current systems, offering powerful new authoring tools for mixing operators and new cinematography processing engine that feature flexible presentation data that optimizes the sound quality and ambient soundtrack effects for each room speaker placement and performance. In addition, the system maintains backward compatibility and minimizes the impact on modern production and distribution workflows.

[00144] While embodiments of the invention have been described with respect to examples and implementations in a cinematic environment where adaptive audio content is associated with film content for use in digital cinematography processing systems, it should be noted that embodiments of the invention also can be implemented in non-cinematic environments. Adaptive audio content, including object-based audio and channel-based audio, can be used in combination with some related content (linked audio, video, graphics, etc.), or it can constitute stand-alone audio content. The playback environment can be anything from headphones or near-field monitors to small and large rooms, cars, outdoor areas, concert halls, etc.

[00145] Features of the system 100 may be implemented in a suitable network-based computer-based audio processing environment for processing digital or digitized audio files. Parts of an adaptive audio system may include one or more networks that contain any desired number of separate machines, including one or more routers (not shown) that serve to buffer and forward data transmitted between computers. Such a network can be built on various network protocols and can be the Internet, a wide area network (WAN), a local area network (LAN), or any combination thereof. In one embodiment of the invention, where the network includes the Internet, one or more machines may be configured to access the Internet through browser programs.

[00146] One or more components, blocks, processes, or other functionalities may be implemented through a computer program that controls the execution of a processor-based system by a computing device. It should also be noted that the various functions disclosed herein may be described using any number of combinations of hardware, firmware, and/or as data and/or instructions embodied in various computer-readable or computer-readable storage media. , based on the characteristics of their behavior, register transfer, logical components, etc. Computer-readable storage media in which such formatted data and/or instructions may be embodied include, but are not limited to, physical (persistent), non-volatile storage media in various forms, such as optical, magnetic or semiconductor storage media.

[00147] Unless the context clearly requires otherwise, throughout this specification and claims, the words “comprise,” “comprising,” and the like. should be interpreted in an inclusive sense, as opposed to an exclusive or exhaustive sense; that is, in the sense of “including by way of non-limiting example.” Words that use a singular or plural form also respectively include a plural or singular form. In addition, the expressions “in this disclosure,” “in accordance with this disclosure,” “above,” “below,” and similar words refer to this application as a whole and not just to any parts of this application. When the word "or" is used to refer to a list of two or more elements, the word covers all of the following interpretations of the word: any of the elements in the list, all of the elements in the list, all of the elements in the list, and any combination of the elements in the list.

[00148] Although one or more implementations have been described by way of examples and in terms of specific embodiments of the invention, it should be understood that one or more implementations are not limited to the disclosed embodiments of the invention. On the contrary, they are intended to cover various modifications and similar circuits, as will be apparent to those skilled in the art. Therefore, the scope of the appended claims should be given the broadest interpretation so as to cover all such modifications in similar circuits.

Claims (6)

1. A method for presenting sound signals, including: receiving a bitstream within which is embedded an adaptive audio mix comprising a plurality of mono audio streams and metadata associated with each of the audio streams indicating a playback location of the corresponding mono audio stream, wherein at least some of the plurality of mono audio streams are identified as channel-based audio and others of a number of monophonic audio streams are identified as object-based audio, and wherein the playback location of the channel-based monophonic audio stream comprises a speaker designation in the speaker array, and the playback location of the object-based monophonic audio stream comprises a location in three-dimensional space, and wherein each monophonic object-based audio stream represent at least one specific speaker of the speaker array; And representation of a number of monophonic audio streams into a number of signals applied to loudspeakers corresponding to speakers in a playback environment, wherein the speakers of the speaker array are placed at specified positions within the playback environment, and wherein metadata elements associated with each corresponding monophonic audio stream based on objects indicate the volume allocations for application to the object-based mono audio stream, such that the object-based mono audio stream is presented to a series of signals applied to loudspeakers at a spatial extent corresponding to the allocation volume indicated by the metadata. 2. The method of claim 1, wherein the metadata elements associated with each monophonic object-based audio stream further indicate spatial parameters controlling playback of a corresponding audio component comprising one or more of: audio position, audio width, and audio speed. 3. The method of claim 1, wherein the playback location for each of the plurality of monophonic object-based audio streams comprises a position in space relative to a screen in the playback environment or a surface that surrounds the playback environment, wherein the surface comprises a front plane, a back plane , the left plane, the right plane, the top plane and the bottom plane, and/or are independently defined with respect to either an egocentric frame of reference or an allocentric frame of reference, wherein the egocentric frame of reference is defined relative to the listener in the playback environment, and wherein the allocentric frame of reference is defined with respect to one from the characteristics of the playback environment. 4. A non-transitory computer-readable storage medium containing a sequence of commands, wherein when executed by the audio signal processing system, the sequence of commands causes the system to implement the method according to claim 1.