JP2022028896A - Screen-relative rendering of audio and encoding and decoding of audio for such rendering - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide screen-relative rendering of audio, and encoding and decoding of audio for such rendering.

SOLUTION: An object processing subsystem 9 receives speaker channels, object channels, and screen-related metadata of an audio program delivered from a decoder 7. The object processing subsystem 9 performs warping on the object channels and outputs to a rendering subsystem 11 the resulting object channels and/or mixes. The warping is determined by a warping degree parameter indicated by the screen-related metadata and/or a warping degree parameter provided by a controller 10. The rendering subsystem 11 maps to the object channels and/or the speaker channels that can use the mixes.

SELECTED DRAWING: Figure 3

COPYRIGHT: (C)2022,JPO&INPIT

Description

Cross-reference to related applications This application claims the priority benefit of US Provisional Patent Application No. 61 / 904,233 filed on November 14, 2013. The content of the application is incorporated herein by reference in its entirety.

Technical Field of the Invention The present invention relates to the encoding, decoding and rendering of an audio program with corresponding video content (eg, the soundtrack of a movie or other audiovisual program). In some embodiments, the program is an object-based audio channel that includes at least one audio object channel and screen-related metadata, typically also a speaker channel. Is. The screen-related metadata is rendered at various positions (at least in part by the screen-related metadata) of the playback system with respect to the display screen of the sound source indicated by the program (eg, the object indicated by the object channel). Supports screen-relative rendering.

Embodiments of the invention relate to one or more aspects of an audio content generation and delivery pipeline (eg, a pipeline for generating and delivering audio content in an audiovisual program).

Such a pipeline implements the generation of an audio program, typically audio content and an encoded audio program that presents the metadata corresponding to the audio content. The generation of audio programs may include audio production activities (capturing and recording audio) and optionally also "post-production" activities (manipulating recorded audio). Live broadcasts necessarily require that any authoring decision be made during audio production. In the generation of movies and other non-real-time programs, many authoring decisions may be made during post-production.

The audio content generation and delivery pipeline optionally implements program remixing and / or remastering. In some cases, the program diverts the content for alternative use cases and may require additional processing after the content is generated. For example, a program originally generated for playback in a movie theater may be modified (eg, remixed) to be more suitable for playback in a home environment.

Audio content generation and delivery pipelines typically include an encoding stage. Audio programs may require encoding to allow delivery. For example, a program intended for home playback is typically data compressed to allow for more efficient delivery. The encoding process reduces the complexity of the spatial audio scene and / or reduces the data rate of the individual audio streams of the program and / or multiple channels of audio content (eg, compressed audio content) and It may include a step of packaging the corresponding metadata into a bitstream with the desired format.

The audio content generation and delivery pipeline includes decoding and rendering stages (typically implemented by a playback system that includes a decoder). Ultimately, the program is presented to the end consumer by rendering the audio description into a loudspeaker signal based on the playback equipment and environment.

A typical embodiment of the present invention comprises an audio program (eg, a movie or audio and image content) such that the position of the auditory image is presented reliably in a manner consistent with the position of the corresponding visual image. Allow other programs' soundtracks to be played.

Traditionally, in a movie theater mixing room (or other audiovisual program authoring environment), a display screen (referred to in this article as a "reference" screen to distinguish it from an audiovisual program playback screen). The position and size of is consistent with the front wall of the mixing environment, and the left and right edges of the reference screen are aligned with the positions of the left and right main screen loudspeakers. The additional center screen channel is generally located in the center of the reference screen / wall. In this way, the front wall spread, the front loudspeaker position and the screen position are consistently co-located. Typically, the reference screen is about as large as the room, and the left, center, and right loudspeakers are close to the left, center, and right edges of the reference screen. This arrangement is similar to the typical arrangement of screens and front speakers in the expected movie theater playback position. For example, FIG. 1 is a diagram of the front wall of such a movie theater, with display screens S, left and right front speakers (L and R) and front center speakers (C) mounted on (or near) the front wall. Has been done. During the playback of the movie, the visual image B is displayed on the screen S, during which the accompanying sound "A" is emitted from the speakers of the playback system (including speakers L, R, C). For example, the image B may be an image of a sound source (eg, a bird or a helicopter), and the sound "A" may be a sound intended to be perceived as coming from that sound source. The movie is when the front speakers are coplanar with screen S, the left front and right front speakers (L and R) are on the left and right edges of screen S, and the center front speaker is near the center of screen S. It is assumed that sound A is authored and rendered so that sound A is perceived as originating from a sound source position that matches (or nearly matches) the position on screen S where image B is displayed. FIG. 1 assumes that the screen S is at least substantially acoustically transparent and the speakers L, C, R are located behind the screen S (but at least substantially within the plane of the screen S). ing.

However, during playback in the consumer's home (or by a mobile user's portable playback device), the size and position of the front speakers (or headset speakers) of the playback system relative to each other and relative to the playback system's display screen. It does not have to match that of the front speakers and display screen of the program authoring environment (eg cinema mixing room). In such playback cases, the width of the playback screen is typically significantly smaller than the distance that separates the left and right main speakers (left and right front speakers or headsets, such as a pair of headphone speakers). It is possible that the screen is not centered with respect to the main speakers, or even in a fixed position (for example, for mobile users wearing headphones and holding the display). This can create a noticeable discrepancy between perceived audio and visual.

For example, FIG. 2 shows a home theater system display screen (S') mounted on (or near) the front wall, left and right front speakers (L'and R'), and front center speakers (C'). It is a figure of the front wall (W') of the room with the speaker. During playback of the same movie (according to the system of FIG. 2) of the example of FIG. 1, the visual image B is displayed on the screen S'while the accompanying sound A is (speakers L', R', C'. Is emitted from the speakers of the playback system (including). The movie is rendered (by the movie theater playback system) so that sound A is perceived as coming from a sound source position that matches (or nearly matches) the position on the movie theater screen where image B is displayed. And assumed to be authored for reproduction. However, when the movie is played by the home theater system of FIG. 2, the sound A will be perceived as coming from a sound source position close to the left front speaker L'. The sound source position does not coincide with or almost coincide with the position on the home theater screen S'where the image B is displayed. This is because the front speakers L', C'and R'of the home theater system are different in size and position than those the front speakers of the program authoring system have with respect to the reference screen of the program authoring system. By having against screen S'.

In the examples of FIGS. 1 and 2, the expected cinema reproduction system is assumed to have a well-defined relationship between its loudspeakers and the screen. Thus, the desired relative position of the content creator for the displayed image and the corresponding audio source can be reliably reproduced (during playback in the cinema). For playback in other environments (eg in a home audio / video room), the expected relationship between the loudspeakers and the screen is typically not preserved and therefore (desired by the content creator). The relative position of the displayed image and the corresponding audio source is typically not well reproduced. The relative position of the displayed image and the corresponding audio source that is actually achieved during playback (other than in a cinema with the expected relationship between the loudspeaker and the screen) is the loudspeaker and display of the playback system. Based on the actual relative position and size of the screen.

For sounds that are rendered to be perceived in the on-screen position during playback of the audiovisual program, the optimal auditory image position is independent of the listener position. For sounds that are rendered to be perceived in the off-screen position (at a non-zero distance in the direction perpendicular to the plane of the screen), the parallax at the auditory perceived position of the sound source depends on the listener position. There is a possibility of parallax errors. Methods have been proposed that attempt to minimize or eliminate such parallax errors based on known or expected listener positions.

It is known to use a high-end playback system (eg, in a movie theater) to render an object-based audio program (eg, an object-based program that indicates a movie soundtrack). For example, an object-based audio program, which is a movie soundtrack, has many different sound elements (audio objects), dialogs, and noises that correspond to images on the screen to create the intended overall auditory experience. And may indicate sound effects emanating from different locations on the screen (or relative to the screen) as well as background music and ambient effects, which may be indicated by the speaker channel of the program. Accurate reproduction of such programs requires that the sound be reproduced in a manner that corresponds as closely as possible to what the content creator intended in terms of audio object size, position, intensity, movement and depth. ..

Object-based audio programs represent a significant improvement over traditional speaker channel-based audio programs. Speaker channel-based audio is more restricted than object channel-based audio in terms of spatial reproduction of certain audio objects. The audio channels of a speaker channel-based audio program consist only of speaker channels (no object channels), and each speaker channel is typically for a specific, individual speaker in the listening environment. Determine the speaker feed.

Various methods and systems have been proposed for generating and rendering object-based audio programs. During the generation of an object-based audio program, typically any number of loudspeakers are used to play the program, the loudspeakers used for playback (typically in a movie theater). It is assumed that the speakers are located in any position in the playback environment and are not necessarily in the (nominal) horizontal plane or any other predetermined arrangement known at the time of program generation. Typically, the object-related metadata contained in a program uses, for example, a three-dimensional array of speakers to capture at least one object in the program at an apparent spatial position (in a three-dimensional volume) or along a trajectory. Shows the rendering parameters for rendering. For example, the object channel of a program may have corresponding metadata showing a three-dimensional trajectory of the apparent spatial position in which the object (indicated by the object channel) is rendered. Trajectories are a sequence of "floor" positions (in the plane of a subset of speakers that are supposed to be located on the floor in the playback environment or in other horizontal planes) and at least one other other in the playback environment, respectively. It may include a sequence of "above floor" positions (determined by driving a subset of speakers that are supposed to be located in a horizontal plane). An example of rendering an object-based audio program is described, for example, in Patent Document 1 assigned to the assignee of the present application.

The advent of object-based audio program rendering has significantly increased the amount of audio data processed and the complexity of rendering that must be performed by the rendering system. This is partly because object-based audio programs can show many objects (each with corresponding metadata) and can be rendered for playback by a system that includes many loudspeakers. be. It has been proposed to limit the number of object channels contained in an object-based audio program so that the intended rendering system has the ability to render the program. For example, Brett Crockett, Alan Seefeldt, Nocolas Tsingos, Rhonda Wilson and Jeroen Breebaart, entitled "Scene Simplification and Object Clustering for Rendering Object-Based Audio Content," filed December 21, 2012. US Provisional Patent Application No. 61 / 745,401, which was transferred to the transferee of the present invention, clusters the input object channels to generate the clustered object channels to be included in the program. And / or the number of object channels in an object-based audio program by mixing the audio content of the input object channel with the speaker channels to produce a mixed speaker channel that can be included in the program. Describes methods and devices for such limitation. Some embodiments of the invention are speaker channels for generating object-based programs for delivery to a playback system (with screen-related metadata) or for delivery to a playback system. It is believed that it may be performed in the context of such clustering (eg, in a mixing or remix facility) for use in the generation of base programs.

PCT International Application No. PCT / US 2011/028783, International Publication No. 2011/119041A2, published September 29, 2011

Throughout this disclosure, the "warping" of at least one channel (eg, an object channel or speaker channel) of an audio program is assumed to have the corresponding video content of the program (for example). For example, the program may be the soundtrack of a movie or other audiovisual program), processing (or audio data) the audio content (audio data) of each said channel to produce distorted audio content. Replacing each said channel with at least one other audio channel indicating distorted audio content). When the distorted audio content is rendered to produce a speaker feed, and the speaker feed is used to drive the playback speaker, the sound emitted by the speaker is (the content creator sees a reference screen, eg, a reference screen, for example. At least one audio element intended to be perceived in at least one predetermined position with respect to the screen of a cinema theater, and in a perceived distorted position (which may be fixed and change over time). (May be). The distorted position is "warped" in the sense that it is a predetermined position with respect to the display screen of the playback system (rather than with respect to the reference screen envisioned by the content creator). Typically, each distorted position is (in this article referred to as "screen-related" metadata) by the metadata provided with the audio program (eg, included in the audio program). Determined, at least in part, for the display screen of the playback system (sometimes referred to as the "playback screen"). Each distorted position is between screen-related metadata and other data indicating the playback system configuration (eg, the position or position and size and / or size and / or position of the speaker and display screen of the playback system. It can be determined by the relationship (data showing the relationship (s). The distorted position (s) may match the actual playback screen, but it is not necessary. Some embodiments of the invention allow a smooth transition between on-screen and / or off-screen distorted positions that change during reproduction (relative to the reproduction screen).

In this paper, the expression "off-screen distortion" of at least one channel of the program refers to the distorted position of at least one corresponding audio element (determined by the audio content of said at least one channel). "Distortion" of at least one channel of the type that is at a non-zero depth to the playback screen (ie, has a non-zero distance from the playback screen at least substantially perpendicular to the plane of the playback screen). Represents.

In the first class of embodiments, the invention is a method of rendering an audio program (eg, an object-based audio program): (a) determining at least one warping degree parameter. (For example, by parsing the program and identifying at least one of the distortion parameters indicated by the program's screen-related metadata, or by specifying at least one of the distortion parameters for the playback system. (By configuring the playback system to perform rendering, including by); (b) distortion of the audio content of at least one channel of the program, at least in part, corresponding to said channel. At the stage of performing distortion to the degree determined by the degree parameter, each of the distortion degree parameters indicates the maximum degree of distortion performed by the playback system on the corresponding audio content of the program (eg,). , A non-binary value indicating the maximum degree). In some embodiments of the first class, step (a) determines at least one off-screen distortion parameter (eg, parses the program and at least one indicated by the screen-related metadata of the program). The off-screen distortion parameter comprises at least one characteristic of off-screen distortion for the corresponding audio content of the program by the playback system, including the step (by identifying the one off-screen distortion parameter). The distortion performed in (b) includes, at least in part, the off-screen distortion determined by at least one of the off-screen distortion parameters. For example, the off-screen distortion parameter is the depth (in the plane of the playback screen) the distortion method or degree or maximum distortion of the distorted position of the audio element (at least substantially parallel to the plane of the playback screen). It may be controlled as a function of at least the distance from the playback screen in a substantially vertical direction). In some embodiments, the distortion parameter determined in step (a) is in the plane of the playback screen (at a depth that is at least substantially perpendicular to the playback screen) with respect to the corresponding audio content of the program. It indicates the maximum degree of distortion performed at least in substantially parallel planes and is thus an off-screen distortion parameter. In another embodiment, step (a) comprises determining at least one distortion parameter and at least one off-screen distortion parameter that is not a distortion parameter. In some embodiments, the program exhibits at least two objects, step (a) comprising independently determining at least one distortion degree parameter for each of at least two of the objects, step (b). ) Is a step of independently performing distortion on the audio content indicating each of the objects, at least in part, to the extent determined by the at least one distortion parameter corresponding to each of the objects. include.

In the second class of embodiments, the invention is a method of generating (or decoding) an object-based audio program. The method determines at least one distortion parameter for at least one audio object, and screen-related metadata that indicates the object channel (indicating the object) and each of the distortion parameters for the object. Includes steps to include in the program. Each of the distortion parameters indicates the maximum degree of distortion performed by the playback system on the corresponding object (eg, in a plane parallel to the plane of the playback screen) (eg, non-binary indicating the maximum degree). A value (for example, a scalar value with any value out of many values within a given range). For example, the distortion parameter can be a full distortion from a minimum (indicating that distortion should not be performed) (for example, the audio element position defined by the program to be on the far right of the reference screen). It may be a floating point value within the range up to a certain maximum value that indicates that it should be distorted to the distorted position on the right edge of the playback screen. Here, the range is where an intermediate degree of distortion (eg 50% of full distortion) is performed (eg, the audio element position defined by the program to be at the right edge of the reference screen with the right edge of the playback room. Includes at least one intermediate value (greater than the minimum but less than the maximum) indicating that it should be distorted to an intermediate distorted position between the right edges of the playback screen. In this context, full distortion may represent a distortion of the perceived position of the audio element in the plane of the playback screen such that the distorted position coincides with the playback screen, to an intermediate degree (or). Distortion (less than full distortion) represents distortion of the perceived position of the audio element in the plane of the playback screen such that the distorted position coincides with an area larger than the playback screen (and includes the playback screen). May be good.

In some embodiments of the second class, the screen-related metadata indicates at least one said distortion parameter for each of at least two objects in the program, and each said strain parameter corresponds to each other. Shows the maximum degree of distortion performed on an object. For example, the distortion parameter may indicate a different maximum degree of distortion in or parallel to the plane of the playback screen for each object represented by a different object channel. In another example, the distortion parameter is the different maximum degree of distortion in the plane of the playback screen or in the vertical direction parallel to the plane and the plane of the playback screen for each object indicated by a different object channel. It may indicate a different degree of maximum strain within or in the horizontal direction parallel to the plane.

In some embodiments of the second class, the screen-related metadata exhibits at least one characteristic of off-screen distortion performed by the playback system on the corresponding audio content of the program (eg, playback). At least one (shown as a function of the distance between each plane, at least substantially perpendicular to the plane of the playback screen), how and / or the degree to which the distortion is performed in a plane that is at least substantially parallel to the plane of the screen. It also shows off-screen distortion parameters. In some such embodiments, the screen-related metadata indicates one said off-screen distortion parameter for each of at least two objects represented by the program, and each said off-screen distortion parameter corresponds to each other. Shows at least one characteristic of off-screen distortion that should be performed on the object to be performed. For example, a program can include off-screen distortion parameters for each object indicated by different object channels, which off-screen distortion parameters are performed for each corresponding object. (Ie, the metadata can specify different types of off-screen distortion for the object (s) corresponding to each object channel). In some embodiments, the at least one off-screen distortion parameter is at least substantially parallel to the plane of the playback screen (at a depth that is at least substantially perpendicular to the playback screen) with respect to the program's corresponding audio content. It indicates the maximum degree of distortion to be performed in a plane, and thus the off-screen distortion parameter is a distortion degree parameter.

In a third class of embodiment, the invention is:
(A) The stage of generating an object-based audio program;
(B) A speaker that includes at least one set of speaker channels intended for playback by a loudspeaker located in place with respect to the playback screen in response to the object-based audio program. At the stage of generating a channel-based program, the generation of the set of speaker channels determines the audio content of the object-based audio program, at least in part, by at least one distortion parameter. Each of the distortion parameters is performed by the playback system on the corresponding audio content of the object-based audio program (eg, in a plane parallel to the plane of the playback screen), including the steps to distort to the extent that it is done. Indicates the maximum degree of distortion to be performed (eg, a non-binary value indicating the maximum degree (for example, a scalar value having any value among many values within a predetermined range)), a stage, and a stage. Including, the method.

In some embodiments of the third class, step (b) is a step of generating the speaker channel based audio program to include two or more selectable sets of speaker channels. Containing, at least one of those sets represents the undistorted audio content of the object-based audio program, and the generation of at least one of those sets is the object-based audio program. The other of the set is intended for playback by a loudspeaker located in place with respect to the playback screen, including the step of distorting the audio content of the (using the distortion parameter). .. In some embodiments of the third class, step (b) determines at least one off-screen distortion parameter (eg, parsing the object-based program to the object-based program. The off-screen distortion parameter comprises a step (by identifying at least one of the off-screen distortion parameters indicated by screen-related metadata), wherein the off-screen distortion parameter is the corresponding audio of the object-based audio program by the playback system. Demonstrating at least one characteristic of off-screen distortion for content, step (b) includes off-screen distortion determined by at least one of the off-screen distortion parameters, at least in part.

In some embodiments of the third class, the object-based audio program exhibits at least one said distortion parameter (or at least one said distortion parameter and at least one off-screen distortion parameter). Including screen-related metadata, step (b) parses the object-based audio program and the at least one of the distortion parameters (or the at least one of the distortion parameters and the off-screen distortion). Includes the step of identifying the parameter).

The generation of the speaker channel-based program (according to a third class of embodiment) is not configured to perform decoding and rendering of an object-based audio program (but of speaker channel based). Supports screen rendering with playback systems (which can decode and render programs). Typically, speaker channel-based programs are generated by a remix system that has (or assumes) knowledge of a particular playback system speaker and screen configuration. Typically, the object-based program (in response to which the speaker channel-based program is generated) is a well-configured playback system (which can decode and render the object-based program). Includes screen-related metadata that supports screen-to-screen rendering of said object-based program by.

In a fourth class of embodiment, the invention is a method of rendering a speaker channel-based program that includes at least one set of speaker channels that represent distorted content. The program is generated by processing an object-based audio program. The process distorts the audio content of the object-based audio program to the extent determined by at least one distortion parameter, at least in part, to the set of speaker channels that represent the distorted content. Includes by generating. Here, each of the distortion parameters is the maximum amount of distortion performed by the playback system on the corresponding audio content of the object-based audio program (eg, in a plane parallel to the plane of the playback screen). Indicates the degree (eg, a non-binary value indicating the maximum degree (eg, a scalar value having any value among many values within a predetermined range)). How to render this:
(A) A step of parsing the speaker channel-based program to identify the speaker channels of the speaker channel-based program, including each said set of speaker channels exhibiting distorted content. ,
(B) Playback in response to at least some of the speaker channels of the speaker channel based program, including at least one said set of speaker channels indicating distorted content. It involves generating a speaker feed to drive a loudspeaker that is positioned in place with respect to the screen.

In some embodiments of the fourth class, the speaker channel-based program is generated by processing the object-based audio program, which processing is the object-based. Off-screen distortion of the audio content of the audio program, at least in part, to the extent determined by the at least one distortion parameter, to the corresponding audio content of the object-based program. Includes performing with at least one off-screen distortion parameter that exhibits at least one characteristic of screen distortion.

In some embodiments of the fourth class, the speaker channel based audio program comprises two or more selectable sets of speaker channels, at least one of which is the object. The other of those sets showing the undistorted audio content of the base audio program is one said set of speaker channels showing the distorted content, stage (b). ) Contains the step of selecting one of the sets, which is the set of one of the speaker channels indicating the distorted content.

In some embodiments, the method of the invention generates an object-based audio program containing screen-related metadata (eg, in an encoder), decodes it (eg, in a decoder), and / or renders it. including. The object-based program has the corresponding video content (which may be, for example, the soundtrack of a movie or other audiovisual program), at least one audio object channel, screen-related metadata and typical. It also includes speaker channels. Screen-related metadata includes metadata corresponding to at least one of the object channels (and optionally metadata corresponding to each of at least one of the speaker channels). During the rendering and playback of an object-based program, the processing of screen-related metadata (typically with data showing the relationship (s) between the speakers of the playback system and the screen) is on-screen. Allows and thus distorts the perceived position of an audio element (eg, an audio element intended by the content creator to be perceived in place on a movie screen during playback in a movie theater). The position has a predetermined size and position with respect to the actual size and position of the display screen of the reproduction system. The distorted position does not have to match the actual display screen of the playback system, and typical embodiments of the present invention are on-screen and off-screen of audio elements that change position during program playback. -Allows smooth transitions between perceived positions on the screen.

In some embodiments, an object-based audio program is generated, decoded, and / or rendered. The program also includes at least one audio object channel, optionally at least one speaker channel (eg, a set of speaker channels or a "bed"), where each object channel is an audio object or audio. Represents a collection of objects (eg, a mixture or cluster), with at least one object channel having the corresponding screen-related metadata (eg, including). The speaker channel bed may be a normal mix of speaker channels of the type that can be included in a normal speaker channel based broadcast program that does not include object channels (eg 5.1 channel mix). The method may include encoding audio data indicating each said object channel (and optionally the set of speaker channels) to generate an object-based audio program. In response to an object-based audio program produced by a typical embodiment of this class, the rendering stage produces a speaker feed that shows the mixture of each speaker channel and the audio content of each object channel. It may be generated.

Aspects of the invention are for implementing (eg, programmed) systems or devices configured to implement any embodiment of the method of the invention and any embodiment or stage thereof of the method of the invention. Includes a computer-readable medium (eg, a disk) that stores the code (eg, in a non-temporary way). For example, the system of the invention may be a programmable general purpose processor, digital signal processor or microprocessor that performs any of the various actions on the data, including embodiments or stages of the methods of the invention. It is programmed by the firmware and / or configured in some other way or may include it. Such general purpose processors are programmed (and / or otherwise configured) to perform embodiments (or stages thereof) of the methods of the invention in response to input devices, memory, and presented data. It may or may be a computer system including a processing circuit (which has been).

In certain classes of embodiments, the invention presents at least one audio object channel (typically a set of object channels) and at least one speaker channel (typically a set of speaker channels). A system configured to generate object-based audio programs. Each audio object channel represents an object or collection of objects (eg, a mixture or cluster) and typically contains the corresponding object-relational metadata. The set of speaker channels may be a normal mixture of speaker channels of the type that can be included in a normal speaker channel-based broadcast program that does not include object channels (eg, 5.1 channel mix). In response to an object-based audio program produced by a typical embodiment of the system, the spatial rendering subsystem is a speaker feed showing a mixture of speaker channels and the audio content of each object channel. May be generated.

In certain classes of embodiments, the invention comprises at least one frame (including audio content) or other segment (eg, non-temporary) of an audio program produced by any embodiment of the method of the invention. It is an audio processing unit (APU) that includes a buffer memory (buffer) to be stored. If the program is an object-based audio program, the stored segments typically include speaker channel beds and object channel audio content, as well as the corresponding screen-related metadata. In another class of embodiments, the invention includes a buffer memory (buffer) that stores at least one frame or other segment of a speaker channel based audio program (eg, in a non-temporary manner). APU. Here, the segment comprises the audio content of at least one set of speaker channels produced as a result of performing distortion of the audio content of an object-based audio program according to embodiments of the present invention. .. The segment may include audio content of at least two selectable sets of speaker channels in a speaker channel based program, wherein at least one of those sets is a practice of the present invention. Produced as a result of morphological distortion.

A typical embodiment of the invention is a real-time, encoded, object-based audio bitstream for transmission (or otherwise delivery) to an external rendering system (eg, device). Configured to implement generation.

It is a figure of the front wall (W) of a movie theater in which the display screen (S) and the left and right front speakers (L and R) and the front center speaker (C) are installed on the front wall (or near). The front wall (W') of the room where the display screen (S') of the home theater system, the left and right front speakers (L'and R') and the front center speaker (C') are installed on the front wall (or near). ). FIG. 3 is a block diagram of an embodiment of a system configured to perform an embodiment of the method of the invention. FIG. 3 is a diagram of a playback environment including a playback system display screen (playback screen S') and speakers (L', C', R', Ls and Rs). FIG. 4 is a diagram of the reproduction environment of FIG. 4, showing an embodiment in which the parameter EXP has a value different from that of the embodiment described with reference to FIG. FIG. 4 is a diagram of the reproduction environment of FIG. 4, showing an embodiment in which the parameter EXP has a value different from that of the embodiment described with reference to FIGS. 4 and 4A. FIG. 3 is a block diagram of elements of a system configured to perform another embodiment of the invention.

<Notation and nomenclature>
Throughout the present disclosure, including the claims, the expression performing an action "on" a signal or data (eg, filtering, scaling, transforming, or applying gain to the signal or data) is to the signal or data. Perform the operation either directly or against a processed version of the signal (eg, for the version of the signal that has undergone preliminary filtering or preprocessing prior to performing the operation). Used in a broad sense to indicate that.

Throughout this disclosure, including claims, the expression "system" is used in a broad sense to refer to an appliance, system or subsystem. For example, a subsystem that implements a decoder may be referred to as a decoder system, and is a system that includes such a subsystem (eg, a system that produces X output signals in response to multiple inputs. , The subsystem produces M of the inputs, and the other X-M inputs are received from an external source) may also be referred to as a decoder system.

Throughout this disclosure, including claims, the term "processor" is programmable or otherwise programmable (eg, using software or firmware) to perform operations on data (eg, audio or video or other image data). Used in a broad sense to refer to a configurable system or device. Examples of processors were programmed to perform pipelined processing on field programmable gate arrays (or other configurable integrated circuits or chipsets), audio or other sound data. Includes digital signal processors, programmable general purpose processors or computers and programmable microprocessor chips or chipsets configured in and / or other ways.

Throughout the present disclosure, including the claims, the expressions "audio processor" and "audio processing unit" are used broadly to represent a system configured to process audio data interchangeably. Examples of audio processing units include, but are not limited to, encoders (eg, transcoders), decoders, codecs, pre-processing systems, post-processing systems and bitstream processing systems (sometimes referred to as bitstream processing tools).

Throughout this disclosure, including claims, the expression "metadata" (eg, in the expression "screen-related metadata") is separate from the corresponding audio data (the audio content of the bitstream, which also contains the metadata). Refers to different data of. Metadata is associated with audio data and that at least one feature or characteristic of the audio data (eg, what type (s) of processing has already been performed on or performed on that audio data). Indicates the trajectory of the object that should or is indicated by its audio data). The association of metadata with audio data is time-synchronous. Thus, the current (most recently received or updated) metadata should include the result of audio data processing of the type in which the corresponding audio data simultaneously has the characteristics shown and / or is shown. Can be shown.

Throughout this disclosure, including claims, the term "combined" or "combined" is used to mean a direct or indirect connection. Thus, when the first device is coupled to the second device, the connection may be through a direct connection or through an indirect connection via another device and connection.

Throughout this disclosure, including claims, the following expressions have the following definitions:

Speakers and loudspeakers are used interchangeably to represent transducers that emit arbitrary sound. This definition includes loudspeakers implemented as multiple transducers (eg, woofers and tweeters).

Speaker feed: An audio signal applied directly to a loudspeaker or an audio signal applied to an amplifier and loudspeaker in series.

Channel (or "audio channel"): A monophonic audio signal. Such a signal can typically be rendered to be equivalent to applying the signal directly to a loudspeaker in the desired or nominal position. The desired position may be static or dynamic, as is typically the case with physical loudspeakers.

Audio program: Describes one or more audio channels (at least one speaker channel and / or at least one object channel) and optionally associated metadata (eg, desired spatial audio presentation). A set of metadata).

Speaker channel (or "speaker feed channel"): The specified speaker within the speaker configuration associated with or defined for the specified loudspeaker (in the desired or nominal position). • The audio channel associated with the zone. The speaker channel is rendered to be equivalent to applying the audio signal directly to the designated loudspeaker (at the desired or nominal position) or to the speakers in the designated speaker zone. ..

Object channel: An audio channel that represents the sound produced by an audio source (sometimes referred to as an audio "object"). Typically, the object channel determines the parametric audio source description (for example, metadata indicating the parametric audio source description is included within or provided with the object channel). The source description is the sound emitted by the source (as a function of time), the apparent position of the source as a function of time (eg, 3D spatial coordinates) and optionally at least one additional parameter that characterizes the source (as a function of time). For example, the apparent source size or width) may be determined.

Object-based audio program: A collection of one or more object channels (and optionally at least one speaker channel) and optionally associated metadata (eg, by object channels). A source of sound that is shown by a metadata or object channel that shows the trajectory of the audio object that emits the sound, or otherwise shows the desired spatial audio presentation of the sound that is shown by the object channel. An audio program that also contains (metadata) that indicates the identification information of at least one audio object.

Rendering: The process of converting an audio program to one or more speaker feeds or converting an audio program to one or more speaker feeds and using the speaker feeds to one or more loudspeakers. The process of converting to sound. (In the latter case, rendering is sometimes referred to in this paper as "rendering by" loudspeakers.) The audio channel is applied directly to the physical loudspeakers in the desired location (desired location). "In") can be rendered trivially. Alternatively, one or more audio channels may be rendered using one of a variety of virtualization techniques designed to be substantially equivalent (to the listener) to such trivial rendering. can. In this latter case, each audio channel may be converted into one or more speaker feeds that should be added to the loudspeakers (s) in known positions that are generally different from the desired position. , The sound emitted by the loudspeakers in response to the feed will be perceived as coming from the desired position. Examples of such virtualization techniques are binaural rendering through headphones (eg using Dolby headphone processing to simulate surround sound up to 7.1 channels for headphone wearers) and wave field synthesis. )including.

<Detailed description of the embodiment of the invention>
Examples of embodiments of the system of the invention (and methods performed by the system) are described with reference to FIGS. 3, 4, and 5.

FIG. 3 is a block diagram of an example of an audio processing pipeline (audio data processing system), wherein one or more of the elements of the system are configured based on an embodiment of the present invention. The system includes the following elements combined together as shown: capture unit, production unit 3 (which includes the encoding subsystem), delivery subsystem 5, decoder 7, object processing subsystem 9. , Controller 10 and rendering subsystem 11. The variant omits one or more of the elements or includes an additional audio data processing unit for the system in the figure. Typically, elements 7, 9, 10 and 11 are included in a playback system (eg, an end-user home theater system).

The capture unit 1 is typically configured to generate a PCM (time domain) sample with audio content and output the PCM sample. Those samples show multiple streams of audio captured by a microphone. The production unit 3 is configured to accept the PCM sample as input and output an object-based audio program indicating the audio content. Programs are typically encoded (eg, compressed) audio bitstreams or include them. The encoded bitstream data indicating the audio content is sometimes referred to as "audio data" in this paper. When the encoding subsystem of the production unit 3 is configured according to a typical embodiment of the present invention, the object-based audio program output by the unit 3 is a plurality of speaker channels (speakers) of audio data. • Channel “bed”) and multiple object channels of audio data and metadata (screen-related metadata for each object channel and optionally screen-related metadata for each speaker channel). (Including) (ie, including).

In a typical implementation, unit 3 is configured to output the object-based audio program generated there.

In another implementation, unit 3 is combined to generate a speaker channel-based audio program (including speaker channels but not object channels) in response to an object-based audio program. , Including the configured remix subsystem, the unit 3 is configured to output the speaker channel based audio program. The remix subsystem 6 of the system of FIG. 5 is according to an embodiment of the invention according to the invention in response to an object-based audio program (“OP”) generated by the encoder 4 (FIG. 5). In another example of a remix subsystem combined and configured to generate a speaker channel based audio program (program "SP" that includes speaker channels but not object channels) according to embodiments. be.

The delivery subsystem of FIG. 3 is a speaker channel generated by the unit 3 and output from the unit 3 (for example, an object-based audio program or a speaker channel generated in response to the object-based audio program). -It is configured to store and / or transmit (eg, broadcast) a base audio program). For simplicity, it is generated by unit 3 and output from unit 3 (unless it is clear from the context, description or reference that the program generated by unit 3 and output from unit 3 is a speaker channel based audio program). We describe (and refer to it) the system of Figure 3 assuming that the program to be done is an object-based audio program.

In a typical embodiment of the system of FIG. 3, the subsystem 5 implements delivery of an object-based audio program to the decoder 7. For example, the subsystem 5 may be configured to store a program (eg, on a disk) and provide the stored program to the decoder 7. Alternatively, the subsystem 5 may be configured to send the program to the decoder 7 (eg, through a broadcast system or internet protocol or other network).

The decoder 7 is coupled and configured to accept (receive or read) a program delivered by the delivery subsystem 5 and decode the program. If the program is an object-based program and the decoder 7 is configured according to a typical embodiment of the invention, the output of the decoder 7 in a typical operation will include:
A stream of audio samples showing the bed of the program's speaker channels (and optionally a corresponding stream of screen-related metadata); and a stream of audio samples showing the program's object channels and screen-related meta. The corresponding stream of data.

The object processing subsystem 9 is coupled to receive the decoded speaker channels, object channels and object-relational metadata of the delivered program (from the decoder 7). A subsystem 9 uses screen-related metadata to an object channel (or to a selected subset of an object channel, or to at least one mixture (eg, a cluster) of some or all of an object channel). It is coupled and configured to perform distortion and output the resulting object channel and / or mixture to the rendering subsystem 11. The subsystem 9 typically outputs object channels to the subsystem 11 and / or object-related metadata corresponding to the mixture (which is parsed by the decoder 7 from the program delivered by the subsystem 5 and is parsed by the decoder 7). (It is presented to the subsystem 9 from) is also output to the rendering subsystem 11. The subsystem 9 is typically configured to pass the decoded speaker channels from the decoder 7 unchanged (to the subsystem 11).

If the program delivered to the decoder 7 is a speaker channel based audio program (generated from an object-based program based on an embodiment of the invention), the subsystem 9 will be (later detailed). A simple speaker channel selection configured to implement distortion according to the present invention by selecting some of the speaker channels of the program (as described in) and presenting the selected channels to the rendering subsystem 11. It may be implemented as a system (or replaced by such a selection system).

The distortion performed by the subsystem 9 may be controlled, at least in part, by the data presented to the subsystem 9 by the controller 10 (eg, in response to user manipulation of the controller 10 during system setup). Such data may indicate the characteristics of the reproduction system speaker and display screen (eg, may indicate the relative size and position of the reproduction system screen and the reproduction system speaker), and. / Or at least one distortion degree parameter and / or at least one off-screen distortion parameter may be included. The distortion performed by subsystem 9 is typically at least one distortion degree parameter and / or at least one off-screen distortion parameter and / as indicated by the screen-related metadata of the program (delivered to decoder 7). Alternatively, it is determined by at least one distortion degree parameter and / or at least one off-screen distortion parameter presented by the controller 10 to the subsystem 9.

The rendering subsystem 11 of FIG. 3 is configured to render the audio content determined by the output of the subsystem 9 for reproduction by a speaker (not shown) of the reproduction system. Subsystem 11 uses the rendering parameters output from subsystem 9, for example, the spatial position and level values indicated by the object-related metadata output from subsystem 9, and the object output from subsystem 9. • Configured to map the audio object determined by the channel (or mix) to the available speaker channels. The rendering subsystem 11 also receives a bed of speaker channels that has been passed through by the subsystem 9. Typically, the subsystem 11 is an intelligent mixer and is configured to determine the speaker feed for available speakers. The decision maps one or more objects (or mixes) to each of several individual speaker channels and maps those objects (or mixes) to each corresponding speaker in the program's speaker channel bed. Includes by mixing with the "bed" audio content indicated by the channel.

Typically, the output of the subsystem 11 is a set of speaker feeds that are presented to a reproduction system loudspeaker (eg, the speaker shown in FIG. 4) to drive those speakers.

One aspect of the invention is an audio processing unit (APU) configured to perform any embodiment of the method of the invention. Examples of APUs include encoders (eg transcoders), decoders, codecs, pre-processing systems (pre-processing units), post-processing systems (post-processing units), audio bitstream processing systems and combinations of such elements. It is not limited to that. Examples of APUs are the production unit 3, the decoder 7, the object processing subsystem 9 and the rendering subsystem 11 of FIG. Implementations of all these exemplary APUs configured to perform certain embodiments of the methods of the invention are considered and described herein.

In certain classes of embodiments, the invention comprises at least one frame or other segment (eg, non-temporary) of an audio program (including audio content) generated by any embodiment of the method of the invention. An APU that contains a buffer memory (buffer) to store (in some way). If the program is an object-based audio program, the stored segments typically include speaker channel beds and object channel audio content, as well as the corresponding screen-related metadata. An example of such an APU includes an encoding subsystem 3B (configured to generate an object-based audio program according to an embodiment of the invention) and a buffer 3A coupled to the subsystem 3B. It is an implementation of the production unit 3 of. Here, buffer 3A contains at least one frame or other segment of the object-based audio program (including, for example, the bed of the speaker channel and the audio content of the object channel and the corresponding screen-related metadata). Remember (in a temporary way). Another example of such an APU is the implementation of decoder 7 in FIG. 3, which includes buffer 7A and a decoding subsystem 7B (which is coupled to buffer 7A). Here, buffer 7A is at least one frame or other segment of the object-based audio program delivered from subsystem 5 to decoder 7 (speaker channel beds and object channel audio content as well as corresponding screens. Remember (including relational metadata) (for example, in a non-temporary way). The decoding subsystem 7B is configured to parse the program and execute any necessary decoding on the program.

In another class of embodiments, the invention includes a buffer memory (buffer) that stores at least one frame or other segment of a speaker channel based audio program (eg, in a non-temporary manner). APU. Here, the segment comprises the audio content of at least one set of speaker channels produced as a result of performing distortion of the audio content of an object-based audio program according to embodiments of the present invention. .. The segment may include audio content of at least two selectable sets of speaker channels in a speaker channel based program, wherein at least one of those sets is a practice of the present invention. Produced as a result of morphological distortion. An example of such an APU is configured to generate an encoding subsystem 3B, a speaker channel based audio program according to an embodiment of the invention; which is also an object base produced by unit 3. 3 is an implementation of the production unit 3 of FIG. 3 including buffer 3A coupled to subsystem 3B) and by performing distortions on the audio content of the audio program of. Here, the buffer 3A contains at least one frame or other segment of a speaker channel-based audio program (including audio content of at least two selectable sets of speaker channels; where, of those sets). At least one stores (eg, in a non-temporary manner) the audio content of an object-based audio program (produced as a result of performing distortions according to embodiments of the present invention). .. Another example of such an APU is the implementation of decoder 7 in FIG. 3, which includes buffer 7A and a decoding subsystem 7B (which is coupled to buffer 7A). Here, the buffer 7A is delivered from the unit 3 to the decoder 7 via the subsystem 5, at least one frame or other frame of the speaker channel based audio program produced by the exemplary embodiment of the unit 3. Remember the segment (for example, in a non-temporary way). The decoding subsystem 7B is configured to parse the program and execute any necessary decoding on the program. Another example of such an APU is configured to generate a speaker channel based audio program according to a subsystem 6B according to an embodiment of the invention; it is generated by the encoder 4 of FIG. , Typically by performing distortions on the audio content of an object-based audio program, including screen-related metadata) and the buffer 6A coupled to the audio processing subsystem 6B. 5 is an implementation of the remix subsystem 6 of FIG. Here, buffer 6A includes at least one frame or other segment of the speaker channel based audio program generated by subsystem 6B (including audio content of at least two selectable sets of speaker channels; Here, at least one of those sets stores (eg, in a non-temporary way) (produced as a result of distortion according to embodiments of the invention).

In a typical embodiment of the present invention, the reproduction environment is along the width along the x-axis, the depth along the y-axis (perpendicular to the x-axis), and the z-axis (perpendicular to the x-axis and the y-axis, respectively). It is assumed to be a unit cube with a vertical height. The position where the audio element (sound source) represented by the audio program (ie, the audio object represented by the object channel or the sound source represented by the speaker channel) is rendered is in Cartesian coordinates (x, y) in this unit cube. Identified using, z). Each of the x and y coordinates has a range within the interval [0,1]. For example, FIG. 4 is a diagram of a playback environment (room) including a playback system display screen (playback screen S') and speakers (L', C', R', Ls and Rs). The reproduction screen S'of FIG. 4 has a width W1 along the x-axis, the center of which is located along the central vertical axis of the front wall of the room (the plane where y = 0). The rear wall of the room (which has a width of W2) is a plane with y = 1. The front speakers L', C'and R'are located near the front wall of the room, the left surround speakers Ls are located near the left wall of the room (the plane where x = 0), and the right surround speakers. Rs is located near the right wall of the room (the plane where x = 1).

Typically, the z-coordinate of the playback environment is assumed to have a fixed value (nominally corresponding to the level of the user's ear in the playback system). Alternatively, in order to render the object (or other sound source) at a position that is perceived to be below or above the ear level, the z coordinate of the rendering position is (eg, the width equal to 1 in the room, the depth equal to 1 and Fluctuations (over the interval [-1,1]) can be tolerated if they are assumed to have a height equal to 2.

In some embodiments, screen parameterization and / or distortion is achieved using all or part of the following parameters (these may be determined during authoring and / or encoding and are delivered. It may be indicated by the screen-related metadata of the program to be done).
• Position of the audio element (eg object) with respect to the reference screen;
The degree of on-screen distortion (eg, a parameter that indicates the maximum degree of distortion performed in or parallel to the plane of the playback screen). Authoring may typically specify the distortion as a binary decision, and the encoding process modifies the binary decision to a continuous (or almost continuous) variable ranging from no distortion to full (maximum) distortion. It is possible that it may be;
The desired off-screen distortion (eg, how or the degree to which distortion in a plane at least substantially parallel to the plane of the playback screen should be performed as a function of a distance at least substantially perpendicular to the plane of the playback screen. One or more parameters shown). Authoring defines a parameter (s) indicating how or the degree to which distortion should be performed as the perceived distorted position of the audio element moves away from the playback screen in a direction perpendicular to the playback screen plane. Can be done. In some cases, such parameters are not delivered with the program (instead can be determined by the regeneration system);
• Reference screen width for the reference room (or for the reference L / R speakers used during authoring). Typically, this parameter is equal to 1.0 for cinemas (ie, for audiovisual programs authored for playback in cinemas);
• Reference screen center position for the reference room (or for the reference L / R speakers used during authoring). Typically, this parameter is equal to (0.5,0,0.5) for cinemas.

In some embodiments, screen parameterization and / or distortion is achieved using all or part of the following parameters (these are typically achieved by a playback system, eg, during a home theater setup): Will be determined).
-Playback screen width for the playback room (or for the playback system L / R speakers). For example, this parameter may have a default value of 1.0 (for example, if the end user does not specify a playback screen size, the playback system assumes that the playback screen matches the playback room width. In effect, the distortion will be nullified);
The desired off-screen distortion (eg, how or the degree to which distortion in a plane at least substantially parallel to the plane of the playback screen should be performed as a function of a distance at least substantially perpendicular to the plane of the playback screen. One or more parameters shown). In some embodiments, the playback system (eg, controller 10 of the embodiment of FIG. 3) is perceived distorted audio elements from the playback screen surface (at least substantially perpendicular to the playback screen surface). As a function of the distance between positions, it is configured to allow custom settings that indicate how or how much distortion should be performed. In a typical embodiment, the program's screen-related metadata shows a fixed function or default function (which can be replaced, for example, by a user-specified alternative function during playback system setup). For example, it is expected to include at least one off-screen distortion parameter to indicate it). This, at least in part, determines how the distortion should be performed as a function of the perceived distorted position distance of the audio element from the surface of the playback screen.
-Playback screen aspect ratio (for example, with a default value of 1.0);
-Playback screen center position (for example, with the default value (0.5,0,0.5)).

In some embodiments, distortion is achieved using other parameters (instead of or in addition to some or all of the parameters described above) that may be indicated by the screen-related metadata of the delivered program. To. For example, for each channel of the program (object channel or speaker channel) (or for each of some of the channels of the program), one or more of the following parameters can be provided.

1. 1. Distortion enabled. This parameter indicates whether processing should be performed to distort the perceived position of at least one audio element determined by the channel. This parameter is typically a binary value that indicates whether distortion should be performed. An example is the "apply_screen_warping" value described below.

2. 2. Degree of distortion (eg, one or more floating point values or one or more other nons, each with any of a range [0,1] or many different values in another predetermined range. Binary parameters). Such distortion parameters (s) typically distort from an in-plane (or parallel to) position on the reference screen to an in-plane (or parallel to) position on the playback screen. The controlling function is modified to determine the maximum degree of distortion to be performed within (or parallel to) the plane of the playback screen. The distortion parameter (or set of parameters) is along (or parallel to) an axis along which the width of the playback screen is (eg, x-axis) and along an axis (eg, z-axis) along which the height of the playback screen is. It can be different for distortion (or parallel to the axis).

3. 3. Depth distortion (eg, one or more parameters with any floating point value in a given range [1, N], eg N = 2). Such parameters (s) (sometimes referred to in this article as "off-screen distortion parameters") typically modify the function that controls the distortion of the off-screen audio element to the playback screen. Controls the degree of distortion or maximum distortion of the audio element rendering position as a function of the distance (depth) from the surface of. For example, rendering of an audio element that is intended to be perceived as "flying" from the playback screen (at the front of the playback room) to the rear of the playback room and vice versa. You can control the degree of distortion (at least substantially parallel to the plane of the playback screen) of the sequence of positions.

For example, in one class of embodiment, distortion is achieved using screen-related metadata contained in an audio program (eg, an object-based audio program). Here, the screen-related metadata indicates at least one degree of distortion to be performed by the playback system (eg, the maximum degree of distortion to be performed in or parallel to the plane of the playback screen). Indicates one non-binary value (eg, a scalar value with any value out of many values that are continuously variable or within a given range). For example, a non-binary value should have a maximum value (full distortion is performed, for example, the audio element position defined by the program to be on the right edge of the reference screen is distorted to the distorted position on the right edge of the playback screen. It may be a floating point value in the range (indicating that) to a minimum value (indicating that distortion should not be performed). In one example, a non-binary value at the midpoint of the range will be half-distorted (50% distorted) (for example, the right edge of the playback room at the audio element position defined by the program to be at the right edge of the reference screen). And should be distorted to an intermediate distorted position between the right edge of the playback screen).

In some embodiments of this class, the program is an object-based audio program that contains such metadata for each object channel of the program. The metadata indicates the maximum degree of distortion to be performed for each corresponding object. For example, the metadata can indicate different maximum degrees of distortion within or parallel to a plane of the playback screen for each object represented by a different object channel. As another example, the metadata shows that for each object represented by a different object channel, the distortion is different in the plane of the playback screen or in the vertical direction parallel to that plane (eg, parallel to the z-axis in FIG. 4). It is possible to indicate the maximum degree and different maximum degrees of horizontal distortion (eg, parallel to the x-axis in FIG. 4) in or parallel to the plane of the reproduction screen.

In some embodiments of this class, the audio program also includes screen-related metadata that exhibits at least one characteristic of off-screen distortion (and the distortion is achieved using the screen-related metadata). (The property, for example, indicates how or the degree to which distortion is performed in a plane that is at least substantially parallel to the plane of the reproduction screen, as a function of a distance that is at least substantially perpendicular to the plane of the reproduction screen.) In such an embodiment, the program is an object-based audio program containing such metadata for each object channel of the program. The metadata exhibits at least one characteristic of off-screen distortion to be performed for each corresponding object. For example, a program can contain such metadata for each object channel that indicates the type of off-screen distortion to be performed for each corresponding object (ie, the metadata is for each object. You can specify different types of off-screen distortion for the objects that correspond to the channel).

Next, an example of how to process an audio program to implement distortion according to an embodiment of the present invention will be described.

In an exemplary method, the screen-related metadata of the audio program is at least one indicated by the channel in or parallel to the plane of the playback screen (for each channel in which the audio content should be distorted). Includes at least one distortion parameter with a non-binary value that indicates the maximum degree of distortion to be performed by the playback system for the audio element. Thereby, the audio element indicating that the program should be rendered at various positions with respect to the reference screen (in the plane of the reference screen and) is distorted with respect to the playback screen (in the plane of the playback screen and). Rendered at various positions. Preferably, one or two such distortion parameters are included for each channel. Controls how much distortion should be applied to at least one audio element indicated by the channel in the horizontal direction (eg, along the x-axis in FIG. 4) (ie, the maximum degree of distortion to be applied). What indicates a distortion factor (eg, the value XFACTOR described below) and / or how much distortion should be applied to at least one audio element indicated by the channel in the vertical direction (eg along the z-axis in FIG. 4) (eg, along the z-axis in FIG. 4). That is, it indicates a distortion factor that controls (the maximum degree of distortion to be applied). The program's screen-related metadata also shows off-screen distortion parameters (eg, the value EXP described below) for each channel. It controls at least one characteristic of off-screen distortion to be performed as a function of distance (at the distorted position of the corresponding audio element) perpendicular to the plane of the playback screen. For example, the off-screen distortion parameter controls how or the degree or maximum distortion of the distorted position of an audio element as a function of depth perpendicular to the plane of the playback screen (distance along the y-axis in Figure 4). You may.

In an exemplary embodiment, the program's screen-related metadata also includes a binary value for the program (or for each segment of the program's sequence of segments) (referred to herein as apply_screen_warping). .. If the value of apply_screen_warping (for the program or its segment) indicates "off", then the playback system does not apply any distortion to the corresponding audio content. For example, audio content (eg, music or ambient sound) that should be rendered with perceived positions within the plane of the playback screen (or in line with the playback screen), but does not need to be closely tied to the visual sense. For audio content), distortion can be disabled in this way. If the value of apply_screen_warping (for the program or its segment) indicates "on", the playback system applies the distortion to the corresponding audio content as follows: The parameter apply_screen_warping is not an example of the "distortion degree" parameter of the type used and / or generated under the present invention.

The following description is an audio object where the program is an object-based program and each strained channel has an undistorted position (which may be a time-varying position) determined by the program. Suppose it is an object channel that indicates. Those skilled in the art will appreciate that the speaker channel of a program is of the speaker channel if the speaker channel presents at least one audio element with an undistorted position (which may be a time-varying position) determined by the program. It will be clear how to modify this description to implement the distortion. The following description states that the playback environment is as shown in FIG. 4 and that the playback system responds to the program (for speakers L', C', R', Ls and Rs shown in FIG. 4). It is also assumed that it is configured to generate five speaker feeds.

In this exemplary embodiment, the playback system (eg, subsystem 9 of the system of FIG. 3) is rendered from the program (eg, from the screen-related metadata of the program) at a distorted position to be determined by the playback system. The next value that indicates the undistorted position of the object (which should be)
Xs ＝ (x－RefSXcenterpos) / RefSWidth
To decide. Where x is the undistorted object position along the horizontal (x or "width") axis with respect to the left edge of the reference screen, and RefSXcenterpos is the position of the reference screen center point along the horizontal axis. , RefSWidth is the width (along the horizontal axis) of the reference screen.

The playback system (eg, subsystem 9 of the system of FIG. 3) uses the program's screen-related metadata (and other data indicating the playback system configuration) to the following values:
Xwarp ＝ Xs * SWidth ＋ SXcenterpos
YFACTOR = y ^EXP and
X'＝ x * YFACTOR ＋ (1-YFACTOR) * [XFACTOR * Xwarp ＋ (l－XFACTOR) * x)]
Is configured to generate. Where Xwarp is the raw (unscaled) distorted object position along the horizontal (x or "width") axis with respect to the left edge of the playback system display screen ("play screen"), where Xs is. The distorted object position along the horizontal axis with respect to the center point of the playback screen, SXcenterpos is the position of the center point of the playback screen along the horizontal axis, and SWidth is the position of the center point of the playback screen (along the horizontal axis). T) width,
YFACTOR is a depth distortion factor that indicates the degree of distortion along the horizontal (width) axis as a function of the position along the depth axis (y-axis in FIG. 4) perpendicular to the surface of the playback screen, and y is the depth. A distorted object position along an axis, EXP is a given (eg user-selected) constant that is an example of the "off-screen distortion" parameter in this article.
X'represents a distorted object position along the horizontal axis with respect to the left edge of the playback screen (a scaled version of the raw distorted object position Xwarp) (hence the distorted object position in the horizontal plane of the playback environment). Is a point with coordinates X', y), XFACTOR is the width axis distortion parameter indicated by the program's screen-related metadata, which can be determined during program authoring, mixing, remixing or encoding. be. XFACTOR is an example of the "distortion degree" parameter in this paper.

Distortion of an undistorted object position (determined by the program) along the vertical (z or "height") axis to a distorted position along the vertical axis with respect to the playback screen is the aspect of the reference screen. It can be performed in a manner determined by a trivial modification of the above equation, replacing the horizontal or x-axis reference with a vertical or z-axis reference, taking into account the ratio and the aspect ratio of the playback screen. can.

The parameter XFACTOR has values in the range 0 to 1 (including both ends) (ie, one of at least three values in this range, typically one of many). The value of XFACTOR controls the degree to which distortion is applied along the horizontal axis. If XFACTOR = 1, full distortion is performed along the horizontal axis (so that the distorted position is on the playback screen, even if the undistorted position of the object is off the playback screen). .. If XFACTOR = 1/2 (or any other value less than 1), a reduced amount of distortion is performed along the x-axis (so that the undistorted position of the object is far off the playback screen. If so, for example, if it is in the position of the left front playback speaker, the distorted position can also be off the playback screen, for example, halfway between the left front speaker and the left edge of the playback screen). For various reasons, it may be useful to set the XFACTOR to a value less than 1 but greater than 0. For example, if distortion is desired, but full distortion to a small playback screen is considered undesirable, or if the audio object position (for example, for diffuse sources) is only loosely tied to the display screen size. Is so.

The parameter YFACTOR is used to control the degree of distortion (along the horizontal and / or vertical axis) as a function of the distorted position of the audio object along the depth axis, and the value of the parameter YFACTOR is It is a function of the distorted position of the object along the depth axis. In the above example, this function is an exponential function. In an alternative embodiment, another function that is a variant of this exemplary exponential function or otherwise different from this exemplary exponential function is used to determine YFACTOR (eg, YFACTOR is the depth axis). It may be the cosine of the distorted object position y along, or the power of the cosine). In the above example where YFACTOR = y ^EXP , when EXP is greater than 0 (as expected to be a typical choice), it is not distorted in the front of the playback room (ie, on the playback screen). The degree of distortion (in the x and / or z directions perpendicular to the depth axis) of a sound with position is that of a sound with an undistorted position (ie, on the rear wall of the playback room) far from the front of the room. Greater than the degree of distortion (in the direction perpendicular to the depth axis). If EXP is greater than 0 and y = 0 (ie, the distorted and undistorted position of the object is in the plane of the playback screen at the front of the playback room), YFACTOR = 0. And the distorted position (X') along the horizontal "width" axis is determined by the undistorted position (x) along the width axis and the parameters XFACTOR and Xwarp. If EXP is greater than 0 and y = 1 (ie, the distorted and undistorted position of the object is at the rear of the playback room), then YFACTOR = 1 and the horizontal "width". The distorted position (X') along the axis is equal to the undistorted position (x) along the width axis, thus effectively distorting the object (along the width axis) in this case. It will not be executed.

As a more specific example, the audio object A1 in FIG. 4 has an undistorted position (and thus a distorted position) in the plane of the playback screen S'at the front of the playback room (ie, y =). y1 = 0). If EXP is greater than 0, then YFACTOR = 0 to perform horizontal axis distortion for object A1, and distortion is the distorted position of object A1 at some position X'= x1, y that matches the playback screen S'. Place at = 0 (eg, as shown in Figure 4). The audio object A2 of FIG. 4 has an undistorted position (and thus a distorted position) between the front and rear walls of the reproduction room (at 0 <y2 <1). If EXP is greater than 0, then YFACTOR is greater than 0 to perform horizontal axis distortion for object A2, and distortion is the distorted position of object A2 at some position along the line segment between points T1 and T2. '= x2, y = y2 (for example, as shown in Figure 4). Since the distance between points T1 and T2 is W3 (as shown in FIG. 4) and EXP is greater than 0, W3 satisfies W1 <W3 <W2. Here, W1 is the width of the screen S'and W2 is the width of the playback room. The specific value of EXP determines the value of W3. W3 is the width range in which the object at the depth y = y2 can be mapped to the playback screen S'by distortion. If EXP is greater than 1, distortion puts the distorted position of object A2 in a position between curves C1 and C2 (shown in FIG. 4). Here, the distance (W3) between the curves C1 and C2 is a function that increases exponentially with the depth parameter y (as shown in FIG. 4), where the distance W3 has a larger value for y. It grows more rapidly (as the value of y increases) and slower as y has a smaller value (as the value of y increases).

In other embodiments (discussed below with reference to FIG. 4A) that are variants of the exemplary embodiment described with reference to curves C1 and C2 of FIG. 4, EXP is equal to 1. Thus, distortion places the distorted position of object A2 between two curves (eg, curves C3 and C4 in FIG. 4A). Here, the distance between the curves C3 and C4 is a linearly increasing function of the depth parameter y. In other embodiments (discussed below with reference to FIG. 4B) that are variants of the exemplary embodiment described with reference to curves C1 and C2 of FIG. 4, EXP is greater than 0 but less than 1 and thus distorted. Places the distorted position of object A2 between two curves (eg, curves C5 and C6 in FIG. 4B). Here, the distance between the curves C5 and C6 is a logarithmic increasing function of the depth parameter y. The distance between the curves increases more rapidly as y has a smaller value (as the value of y increases) and slower as y has a larger value (as the value of y increases). .. Embodiments with an EXP of 1 or less are expected to be typical. For, in such an embodiment, the effect of distortion is faster with increasing values of y (ie, with increasing distance of the distorted position from the screen) than if EXP is greater than 1. Because it will decrease. When EXP is less than 1, the effect of distortion decreases rapidly as the distorted position begins to move away from the screen, and gradually decreases more slowly as the distorted position moves further away from the screen, and finally. The distorted position reaches the rear wall where the distortion is not performed.

Then, in response to an object-based program, speaker channel-based audio (including speaker channels but not object channels) that includes a distorting step (using screen-related metadata, for example). -Describe an embodiment of another class in which a program is generated. The speaker channel-based audio program comprises at least one set of speaker channels, which distorts (and / or off) the audio content of the object-based program, at least in part, by the distortion parameter. Generated as a result of distortion to a determined degree (using screen distortion parameters) and intended for reproduction by loudspeakers located in predetermined positions with respect to the reproduction system display screen. In some embodiments of this class, speaker channel-based audio programs are generated to include two or more selectable sets of speaker channels, at least one of which is distorted. It is intended for playback by a loudspeaker that is produced as a result and is located at predetermined locations with respect to the playback system display screen. Speaker channel-based program generation is not configured to perform decoding and rendering of object-based audio programs (although speaker channel-based programs can be decoded and rendered) playback. Supports system-to-screen rendering. Typically, speaker channel-based programs are generated by a remix system that has (or assumes) knowledge of a particular playback system speaker and screen configuration. Typically, the object-based program (in response to which the speaker channel-based program is generated) is a well-configured playback system (which can decode and render the object-based program). Includes screen-related metadata that supports screen-to-screen rendering of said object-based program by.

Embodiments of this class are particularly useful when it is desired to implement anti-screen rendering but the available playback system (s) are not configured to render object-based programs. To implement anti-screen rendering of an audio program that contains only speaker channels (not object channels), an object-based program that supports anti-screen rendering first follows an embodiment of the invention. Generated. A speaker channel-based audio program (which supports anti-screen rendering) is then generated in response to the object-based program. A speaker channel-based audio program may contain at least two selectable sets of speaker channels, and the playback system may include those sets of speaker channels to implement anti-screen rendering. It may be configured to render one of the selected ones.

Typical speaker channel configurations envisioned by speaker channel-based programs are stereo (for playback with two speakers) and 5.1 surround sound (for playback with five full-range speakers). including. In such a channel configuration, the speaker channel (audio signal) is, by definition, associated with the loudspeaker position, and the perceived position where the audio element (indicated by the audio content of the channels) is rendered. , Typically, it is determined based on the assumed speaker position in the reproduction environment or the assumed speaker position with respect to the reference listening position.

In some embodiments where speaker channel-based audio programs are generated (in response to an object-based program), the screen-related distortions (once possible) by the screen-related metadata of the object-based program. The scaling feature is utilized to generate speaker channels (of speaker channel based programs) associated with loudspeakers that have predetermined positions with respect to the playback screen. Typically, the size and shape and location of a particular playback screen is envisioned by the system that produces the speaker channel based program. For example, in response to an object-based program, a speaker channel-based program should be generated to contain the following two sets of speaker channels (and optionally other speaker channels): Can:
The first set of normal left (L) and right (R) front speaker channels for rendering audio elements to the perceived position determined for the reference screen (eg in a cinema mixing facility); And the "left screen" (Lsc) for rendering the same audio element to the perceived position determined for the left and right edges of the expected playback display screen (eg, in the remix stage of a remix facility or mixing facility). And a second set of left and right front speaker channels, which may be referred to as the "right screen" (Rsc), where the playback screen and playback system front speakers have a given relative size, shape and position. It is supposed to have).

Typically, the channels of the speaker channel-based program produced as a result of the distortion (eg Lsc and Rsc channels) are the more between the image displayed on the playback screen and the corresponding rendered sound. It can be rendered to allow close proximity matches.

By selecting and rendering the normal left (L) and right (R) front speaker channels, the playback system will perceive the audio element it determines to have an undistorted position. You can render the selected channel. By selecting and rendering the "Left Screen" (Lsc) and "Right Screen" (Rsc) speaker channels, the playback system is in a position where the audio elements it determines are distorted (relative to the playback screen). The selected channel can be rendered so that it is perceived as having. However, the distortion is not performed by the playback system, but at the time of generation of the speaker channel-based program (typically in response to an object-based program containing screen-related metadata).

Some embodiments of this class are: the stage of generating an object-based program with screen-related data (at the time and place of mixing); and then (for example, to generate a recording for home use). Generate speaker channel-based programs from object-based programs using screen-related data (at the time and place of the "remix", which can be in the same position where the original mixing was done). A step, including by performing a screen-related distortion; then a step of delivering a speaker channel-based program to the reproduction system. A speaker channel-based program can include multiple selectable sets of channels. The plurality of selectable sets is a speaker channel (when rendered) that is generated without performing distortion and represents at least one audio element that is perceived to be in at least one undistorted position. For example, the first set of (L and R channels) generated in the usual way) and the same audio element (when rendered) that is generated as a result of distorting the content of an object-based program, but Includes at least one additional set of speaker channels (eg, Lsc and Rsc channels) that are perceived to be in at least one different (ie, distorted) position. Alternatively, the speaker channel-based program is generated as a result of the distortion and indicates the channel (eg, Lsc and) showing at least one audio element perceived in at least one distorted position (when rendered). It contains only one set of Rsc channels) and not another set of channels (eg L and R channels) that indicate (when rendered) the same audio element perceived in an undistorted position.

A speaker channel-based program generated from an object-based program based on an exemplary embodiment contains five forward channels: left (L), left screen (Lsc), center (C), right. Screen (Rsc) and right (R). Lsc and Rsc channels are generated by performing distortions using screen-related metadata in object-based programs. To render and play speaker channel-based programs, the playback system may select and render the L and R channels to drive the front speakers on the left and right edges of the playback screen. Lsc and Rsc channels may be selected and rendered to drive the front speakers away from the left and right edges of the playback screen. For example, Lsc and Rsc channels may be generated with the assumption that they will be used to render audio elements with forward speakers at +30 and -30 degrees azimuths with respect to the expected user position. The L and R channels are said to be used to render audio elements using front speakers at +15 and -15 degree azimuths (on the left and right edges of the playback screen) for the expected user position. It may be generated by assumption.

For example, the system of FIG. 5 includes an encoder 4 configured to generate an object-based audio program (“OP”) containing screen-related metadata based on an embodiment of the invention. The encoder 4 may be implemented in the mixing facility or in the mixing facility. The system of FIG. 5 responds to an object-based audio program generated by encoder 4 with a speaker channel-based audio program (“SP”) that includes speaker channels but does not contain object channels. Also includes a remix subsystem 6 coupled and configured to produce (according to certain embodiments of the invention). The subsystem 6 may be implemented in a remix facility, in a remix facility, or as a remix stage of a mixing facility (eg, a mixing facility in which the encoder 4 is also mounted). The audio content of a speaker channel-based program SP is the audio content of at least two selectable sets of speaker channels (eg, one set containing channels L and R discussed above, and Lsc and Rsc discussed above. The subsystem 6 is distorted using the screen-related metadata of the program OP (and typically not shown by the screen-related metadata) based on embodiments of the present invention. At least one of those sets (eg, using other control data indicating the type and / or degree of) as a result of distorting the audio content of the object-based program OP (generated by encoder 4). It is configured to generate channels Lsc and Rsc). The speaker channel-based program SP is output from the subsystem 6 to the delivery subsystem 5. The subsystem 5 can be identical to the subsystem 5 discussed above for the system of FIG.

Embodiments of the invention may be implemented in hardware, firmware or software or a combination thereof (eg, as a programmable logical array). For example, the system of FIG. 3 (or its subsystem 3 or subsystems 7, 9, 10, 11) was, for example, programmed in properly programmed (or otherwise configured) hardware or firmware. It may be implemented as a general purpose processor, digital signal processor or microprocessor. Unless otherwise noted, the algorithms or processes included as part of the invention are not inherently relevant to any particular computer or other device. In particular, various general-purpose machines may be used with programs written according to the teachings of the present application, or it is better to build more specialized equipment (eg, integrated circuits) to perform the required method steps. It can be convenient. As such, the present invention relates to one or more programmable computer systems (eg, computer systems that implement the system of FIG. 3 (or its subsystem 3 or subsystems 7, 9, 10, 11)). It may be implemented in one or more computer programs running above. Each computer system has at least one processor, at least one data storage system (including volatile and non-volatile memory and / or storage elements), at least one input device or port and at least one output device or port. .. Program code is applied to the input data to perform the functions described in this article and generate output information. The output information is applied to one or more output devices in a known manner.

Each such program may be implemented in any desired computer language for communicating with a computer system, including machine, assembly or high-level procedural, logical or object-oriented programming languages. In either case, the language may be a compiled language or an interpreted language.

For example, when implemented by a sequence of computer software instructions, the various functions and stages of embodiments of the invention may be implemented by a multithreaded software instruction sequence performed in suitable digital signal processing hardware. Well, in that case, the various devices, stages and functions of the embodiment may correspond to parts of the software instructions.

Each such computer program is preferably stored or downloaded and stored on a storage medium or device (eg, semiconductor memory or media or magnetic or optical media) readable by a general purpose or dedicated programmable computer. Configure and operate your computer to perform the procedures described in this article when the medium or device is read by your computer system. The system of the present invention may be implemented as a computer-readable storage medium configured with a computer program (ie, storing the computer program), and the storage medium so configured is described in the computer system. Operate in a specific predefined way to perform the functions described in.

Although the implementations have been described using exemplary individual embodiments as examples, it is understood that the implementations of the present invention are not limited to the disclosed embodiments. Rather, it is intended to cover various modifications and similar configurations that will be apparent to those of skill in the art. Therefore, the scope of the accompanying claims should be given the broadest interpretation to include any such amendments and similar configurations.

Some aspects are described.
[Aspect 1]
How to render an audio program:
(A) At the stage of determining at least one distortion degree parameter;
(B) At the stage of performing distortion to the audio content of at least one channel of the program, at least in part, to the extent determined by the distortion parameter corresponding to the channel. The distortion degree parameter includes a step indicating the maximum degree of distortion performed by the playback system on the corresponding audio content of the program.
Method.
[Aspect 2]
The step (a) includes determining at least one off-screen distortion parameter, wherein the off-screen distortion parameter is at least one of the off-screen distortion for the corresponding audio content of the program by the playback system. The method according to aspect 1, wherein the distortion characteristic and performed in step (b) comprises, at least in part, an off-screen distortion determined by at least one said off-screen distortion parameter.
[Aspect 3]
The off-screen distortion parameter distorts the undistorted position of the audio element along a width axis that is at least substantially parallel to the plane of the playback screen, at least substantially perpendicular to the plane of the playback screen. The method of aspect 2, wherein the audio element is controlled as a function of the distance of the distorted position to be rendered.
[Aspect 4]
Determining the value Xs and the value at which the distortion indicates the undistorted position along the width axis of the audio element to be rendered at some distorted position along the width axis.
Xwarp
YFACTOR = y ^EXP and
X'＝ x * YFACTOR ＋ (1-YFACTOR) * [XFACTOR * Xwarp ＋ (l－XFACTOR) * x)]
Including the decision of
Xwarp represents the raw, distorted position of the audio element along the width axis with respect to the edge of the playback screen.
EXP is an off-screen distortion parameter
YFACTOR indicates the degree of distortion along the width axis as a function of the distorted position y of the audio element along the depth axis at least substantially perpendicular to the plane of the playback screen.
X'represents the distorted object position of the audio element along the width axis with respect to the edge of the playback screen.
XFACTOR is one of the distortion degree parameters,
The method according to any one of aspects 1 to 3.
[Aspect 5]
The program is an object-based audio program, and step (a) comprises parsing the program and identifying at least one of the distortion parameters indicated by the screen-related metadata of the program. The method according to any one of aspects 1 to 4.
[Aspect 6]
The program exhibits at least two objects, step (a) includes step (a) of independently determining at least one distortion degree parameter for each of the objects, step (b):
It comprises performing distortion independently for each audio content of the object channel, at least in part, to the extent determined by the at least one distortion degree parameter corresponding to each of the objects.
The method according to aspect 5.
[Aspect 7]
1. the method of.
[Aspect 8]
How to generate an object-based audio program:
(A) At the stage of determining at least one distortion degree parameter for at least one object;
(B) At the stage of including the object channel indicating the object and the screen-related metadata indicating the respective distortion degree parameters for the object into the program, each of the distortion degree parameters is the object by the reproduction system. Indicates the maximum degree of distortion performed against, including steps,
Method.
[Aspect 9]
The program shows at least two objects, the screen-related metadata shows at least one of the distortion parameters for each of at least two of the objects, and each of the distortion parameters corresponds to each other. The method according to aspect 8, wherein the degree of maximum distortion performed on the object to be performed is shown.
[Aspect 10]
The step (a) includes determining at least one off-screen distortion degree parameter for the at least one object, wherein the off-screen distortion parameter is performed on the object by the reproduction system. 8. The method of aspect 8 or 9, wherein the screen-related metadata that exhibits at least one characteristic of screen distortion and is included in the program indicates the respective off-screen distortion parameters.
[Aspect 11]
The off-screen distortion parameter distorts the undistorted position of the object along a width axis that is at least substantially parallel to the plane of the playback screen, at least substantially perpendicular to the plane of the playback screen. 10. The method of aspect 10, wherein the object is controlled as a function of the distance of the distorted position to be rendered.
[Aspect 12]
Determining the value Xs, which indicates the undistorted position along the width axis of the object, where the distortion should be rendered at some distorted position along the width axis, and the value.
Xwarp
YFACTOR = y ^EXP and
X'＝ x * YFACTOR ＋ (1-YFACTOR) * [XFACTOR * Xwarp ＋ (l－XFACTOR) * x)]
Including the decision of
Xwarp represents the raw, distorted position of the audio element along the width axis with respect to the edge of the playback screen.
EXP is an off-screen distortion parameter
YFACTOR indicates the degree of distortion along the width axis as a function of the distorted position y of the audio element along the depth axis at least substantially perpendicular to the plane of the playback screen.
X'represents the distorted object position of the audio element along the width axis with respect to the edge of the playback screen.
XFACTOR is one of the distortion degree parameters,
The method according to any one of aspects 8 to 11.
[Aspect 13]
One of aspects 8-12, wherein each of the distortion parameters is a non-binary value indicating the maximum degree of distortion performed by the playback system on the corresponding audio content of the program. the method of.
[Aspect 14]
(A) The stage of generating an object-based audio program;
(B) A speaker that includes at least one set of speaker channels intended for playback by a loudspeaker located in place with respect to the playback screen in response to the object-based audio program. At the stage of generating a channel-based program, the generation of the set of speaker channels determines the audio content of the object-based audio program, at least in part, by at least one distortion parameter. Each of the distortion parameters indicates the maximum degree of distortion performed by the playback system on the corresponding audio content of the object-based audio program. Including the method.
[Aspect 15]
Step (b) includes generating the speaker channel based audio program so that the speaker channel based audio program contains two or more selectable sets of speaker channels. , At least one of those sets represents the undistorted audio content of the object-based audio program, and the generation of at least one of those sets is of the object-based audio program. The other of the set comprises a loudspeaker located at a predetermined position with respect to the playback screen, comprising distorting the audio content, at least in part, to the extent determined by the distortion parameter. 14. The method of aspect 14, intended for reproduction by.
[Aspect 16]
Step (b) includes determining at least one off-screen distortion parameter, wherein the off-screen distortion parameter is off to the corresponding audio content of the object-based audio program by the playback system. 13. The method of aspect 14 or 15, wherein step (b) comprises at least one characteristic of screen distortion and at least partially includes off-screen distortion determined by at least one of the off-screen distortion parameters.
[Aspect 17]
The off-screen distortion is an undistorted position of an audio element along a width axis that is at least substantially parallel to the plane of the playback screen, at least substantially perpendicular to the plane of the playback screen. 16. The method of aspect 16, comprising distorting to a degree controlled by the off-screen distortion parameter as a function of the distance of the distorted position to be rendered.
[Aspect 18]
The distorting step is a value that indicates the undistorted position of the audio object to be rendered at a distorted position along the width axis, at least substantially parallel to the plane of the playback screen. Xs determination and value
Xwarp
YFACTOR = y ^EXP and
X'＝ x * YFACTOR ＋ (1-YFACTOR) * [XFACTOR * Xwarp ＋ (l－XFACTOR) * x)]
Including the decision of
Xwarp represents the raw, distorted position of the object along the width axis with respect to the edge of the playback screen.
EXP is an off-screen distortion parameter
YFACTOR indicates the degree of distortion along the width axis as a function of the distorted position y of the object along the depth axis at least substantially perpendicular to the plane of the playback screen.
X'represents the distorted object position of the object along the width axis with respect to the edge of the playback screen.
XFACTOR is one of the distortion degree parameters,
The method according to any one of aspects 14 to 17.
[Aspect 19]
The object-based audio program includes screen-related metadata indicating the at least one distortion degree parameter, and step (b) parses the object-based audio program and said screen-related metadata. 13. The method of any one of embodiments 14-18, comprising identifying the respective strain degree parameters indicated by.
[Aspect 20]
One of aspects 14-19, wherein each of the distortion parameters is a non-binary value indicating the maximum degree of distortion performed by the playback system on the corresponding audio content of the program. the method of.
[Aspect 21]
A method of rendering a speaker channel-based program that includes at least one set of speaker channels that represent distorted content, said speaker channel-based program processing an object-based audio program. The process distorts and distorts the audio content of the object-based audio program, at least in part, to the extent determined by at least one distortion parameter. Each of the distortion parameters is performed by the playback system on the corresponding audio content of the object-based audio program, including by generating the set of speaker channels that represent the content. The maximum degree of distortion is shown and the rendering method is:
(A) A step of parsing the speaker channel-based program to identify the speaker channels of the speaker channel-based program, including each said set of speaker channels exhibiting distorted content. ;
(B) Positioned in place with respect to the playback screen in response to at least some of the speaker channels of the speaker channel based program, including the set of at least one speaker channel indicating distorted content. Including the stage of generating speaker feed to drive loudspeakers
Method.
[Aspect 22]
The speaker channel-based program was generated by processing the object-based audio program, which is an off-screen of the audio content of the object-based audio program. Distortion, at least in part, the at least one distortion parameter and at least one off-screen distortion parameter that exhibits at least one characteristic of off-screen distortion for the corresponding audio content of the object-based program. 21. The method of aspect 21, comprising performing to a degree determined by use.
[Aspect 23]
The speaker channel-based audio program comprises two or more selectable sets of speaker channels, at least one of which is an undistorted audio program of the object-based audio program. The other one of those sets representing the content is one said set of speaker channels showing the distorted content, and step (b) is the one said set of the speaker channels showing the distorted content. 21. The method of aspect 21 or 22, comprising the step of selecting one of the above sets, which is a set.
[Aspect 24]
The one of aspects 21 to 23, wherein each of the distortion parameters is a non-binary value indicating the maximum degree of distortion performed by the playback system on the corresponding audio content of the program. the method of.
[Aspect 25]
With a first subsystem configured to parse a multi-channel audio program and identify the channels of the program;
A processing subsystem coupled to the first subsystem, determined by at least one distortion parameter corresponding to the channel, at least in part, for the audio content of at least one channel of the program. Each said distortion parameter is the maximum amount of distortion performed by the playback system on the corresponding audio content of the program. Show the degree,
system.
[Aspect 26]
The distortion includes, at least in part, an off-screen distortion determined by at least one off-screen distortion parameter, the off-screen distortion parameter being off to the corresponding audio content of the program by the playback system. 25. The system according to aspect 25, which exhibits at least one characteristic of screen distortion.
[Aspect 27]
The off-screen distortion is an undistorted position of an audio element along a width axis that is at least substantially parallel to the plane of the playback screen, at least substantially perpendicular to the plane of the playback screen. 26. The system of aspect 26, comprising distorting to the extent controlled by the off-screen distortion parameter as a function of the distance of the distorted position to be rendered.
[Aspect 28]
Determining the value Xs and the value at which the distortion indicates the undistorted position along the width axis of the audio element to be rendered at some distorted position along the width axis.
Xwarp
YFACTOR = y ^EXP and
X'＝ x * YFACTOR ＋ (1-YFACTOR) * [XFACTOR * Xwarp ＋ (l－XFACTOR) * x)]
Including the decision of
Xwarp represents the raw, distorted position of the audio element along the width axis with respect to the edge of the playback screen.
EXP is an off-screen distortion parameter
YFACTOR indicates the degree of distortion along the width axis as a function of the distorted position y of the audio element along the depth axis at least substantially perpendicular to the plane of the playback screen.
X'represents the distorted object position of the audio element along the width axis with respect to the edge of the playback screen.
XFACTOR is one of the distortion degree parameters,
The system according to any one of aspects 25 to 27.
[Aspect 29]
The program is an object-based audio program, and the first subsystem is configured to parse the program and identify at least one distortion parameter indicated by the program's screen-related metadata. The system according to any one of aspects 25 to 28.
[Aspect 30]
The program exhibits at least two objects, the first subsystem is configured to independently determine at least one distortion parameter for each of the objects, and the processing subsystem is at least partial. 29 is configured to independently perform distortion on the audio content indicating each of the objects to the extent determined by the at least one distortion degree parameter corresponding to each of the objects. Described system.
[Aspect 31]
The one of aspects 25 to 30, wherein each of the distortion parameters is a non-binary value indicating the maximum degree of distortion performed by the playback system on the corresponding audio content of the program. System.
[Aspect 32]
With a first subsystem configured to generate object-based audio programs;
A speaker channel base that includes at least one set of speaker channels intended for playback by loudspeakers located in place with respect to the playback screen in response to the object-based audio program. A system comprising a second subsystem coupled to the first subsystem, which is configured to generate the program of the object-based audio program. The audio content is configured to generate the set of speaker channels, including at least in part by distorting to the extent determined by at least one distortion parameter. Is a system that indicates the maximum degree of distortion performed by the playback system on the corresponding audio content of the object-based audio program.
[Aspect 33]
The second subsystem causes the speaker channel-based audio program to be generated such that the speaker channel-based audio program contains two or more selectable sets of speaker channels. Constructed, at least one of those sets represents the undistorted audio content of the object-based audio program, and the generation of at least one of those sets is of the object-based. The other of the set is positioned in the predetermined position with respect to the playback screen, comprising distorting the audio content of the audio program, at least in part, to the extent determined by the distortion parameter. 32. The system according to aspect 32, which is intended for reproduction by loudspeakers.
[Aspect 34]
The second subsystem is configured to generate the set of speaker channels, including by performing off-screen distortion of the audio content of the object-based audio program. The system according to aspect 32 or 33, wherein the off-screen distortion is determined, at least in part, by at least one off-screen distortion parameter exhibiting at least one characteristic of the off-screen distortion.
[Aspect 35]
The second subsystem distorts the undistorted position of the object along a width axis that is at least substantially parallel to the plane of the playback screen in response to the off-screen distortion parameter. 34. The system of aspect 34, wherein the object is configured to control as a function of the distance of the distorted position to be rendered, at least substantially perpendicular to the plane of.
[Aspect 36]
The second subsystem is an undistorted position along the width axis of the audio element to be rendered at a distorted position along the width axis, at least substantially parallel to the plane of the playback screen. Determine the value Xs that indicates the value
Xwarp
YFACTOR = y ^EXP and
X'＝ x * YFACTOR ＋ (1-YFACTOR) * [XFACTOR * Xwarp ＋ (l－XFACTOR) * x)]
Is configured to perform the distortion, including by determining
Xwarp represents the raw, distorted position of the object along the width axis with respect to the edge of the playback screen.
EXP is an off-screen distortion parameter
YFACTOR indicates the degree of distortion along the width axis as a function of the distorted position y of the object along the depth axis at least substantially perpendicular to the plane of the playback screen.
X'represents the distorted object position of the object along the width axis with respect to the edge of the playback screen.
XFACTOR is one of the distortion degree parameters,
The system according to any one of aspects 32 to 35.
[Aspect 37]
The object-based audio program contains screen-related metadata indicating the at least one distortion degree parameter, and the second subsystem parses the object-based audio program and said screen-related. The system according to any one of aspects 32 to 36, configured to identify each said strain parameter indicated by the metadata.
[Aspect 38]
13. System.
[Aspect 39]
A system that renders a speaker channel-based program that contains at least one set of speaker channels that represent distorted content, said speaker channel-based program processing an object-based audio program. The process distorts and distorts the audio content of the object-based audio program, at least in part, to the extent determined by at least one distortion parameter. Each of the distortion parameters is performed by the playback system on the corresponding audio content of the object-based audio program, including by generating the set of speaker channels that represent the content. Indicates the maximum degree of distortion, the system is:
A first configured to parse the speaker channel based program to identify the speaker channel of the speaker channel based program, including said set of each of the speaker channels showing distorted content. With one subsystem;
A loudspeaker positioned in place with respect to the playback screen in response to at least some of the speaker channels of the speaker channel-based program, including the set of at least one speaker channel indicating distorted content. Includes a rendering subsystem coupled to said first subsystem that is configured to generate speaker feeds to drive speakers.
system.
[Aspect 40]
The speaker channel-based audio program comprises two or more selectable sets of speaker channels, at least one of which is an undistorted audio program of the object-based audio program. The other one of those sets representing the content is one said set of speaker channels showing the distorted content, and the first subsystem is one of the speaker channels showing the distorted content. 39. The system of aspect 39, wherein one of the set, which is the set, is configured to be selected for rendering by the rendering subsystem.
[Aspect 41]
The system of aspect 39 or 40, wherein each of the distortion parameters is a non-binary value indicating the maximum degree of distortion performed by the reproduction system on the corresponding audio content of the program.
[Aspect 42]
With buffer memory;
An audio processing unit comprising at least one processing subsystem coupled to the buffer memory.
The buffer memory is at least one segment of an object-based audio program, the segment comprising at least one object channel audio content representing at least one object, and at least one segment. Screen-related metadata indicating at least one distortion degree parameter for the object, each said distortion degree parameter indicating the maximum degree of distortion to be performed on the object by the reproduction system. It remembers the metadata and
The processing subsystem uses at least a portion of the screen-related metadata to render the object-based audio program or generate the object-based audio program or decode the object-based audio program. Combined and configured to perform at least one of
Audio processing unit.
[Aspect 43]
The program shows at least two objects, the screen-related metadata shows at least one of the distortion parameters for each of at least two of the objects, and each of the distortion parameters corresponds to each other. 42. The audio processing unit according to aspect 42, which indicates the degree of maximum distortion performed on the object.
[Aspect 44]
The segment of the object-based audio program stored in the buffer memory indicates at least one off-screen distortion parameter for the at least one object, and the off-screen distortion parameter is the reproduction. 42 or 43, wherein the screen-related metadata contained in the program exhibits the respective off-screen distortion parameters, indicating at least one characteristic of the off-screen distortion to be performed on the object by the system. Audio processing unit.
[Aspect 45]
The off-screen distortion parameter distorts the undistorted position of the object along a width axis that is at least substantially parallel to the plane of the playback screen, at least substantially perpendicular to the plane of the playback screen. 44. The audio processing unit of aspect 44, wherein the object is controlled as a function of the distance of the distorted position to be rendered.
[Aspect 46]
Determining the value Xs, which indicates the undistorted position along the width axis of the object, where the distortion should be rendered at some distorted position along the width axis, and the value.
Xwarp
YFACTOR = y ^EXP and
X'＝ x * YFACTOR ＋ (1-YFACTOR) * [XFACTOR * Xwarp ＋ (l－XFACTOR) * x)]
Including the decision of
Xwarp represents the raw, distorted position of the audio element along the width axis with respect to the edge of the playback screen.
EXP is an off-screen distortion parameter
YFACTOR indicates the degree of distortion along the width axis as a function of the distorted position y of the audio element along the depth axis at least substantially perpendicular to the plane of the playback screen.
X'represents the distorted object position of the audio element along the width axis with respect to the edge of the playback screen.
XFACTOR is one of the distortion degree parameters,
The audio processing unit according to any one of aspects 42 to 45.
[Aspect 47]
The audio processing unit according to any one of aspects 42 to 46, wherein the audio processing unit is an encoder and the processing subsystem is configured to generate the object-based audio program.
[Aspect 48]
The audio processing unit according to any one of aspects 42 to 46, wherein the audio processing unit is a decoder and the processing subsystem is configured to decode the object-based audio program.
[Aspect 49]
One of aspects 42-48, wherein each of the distortion parameters is a non-binary value indicating the maximum degree of distortion performed by the playback system on the corresponding audio content of the program. Audio processing unit.
[Aspect 50]
With buffer memory;
An audio processing unit comprising at least one processing subsystem coupled to the buffer memory.
The buffer memory stores at least one segment of a speaker channel-based audio program, which segment is intended for playback by a loudspeaker located in place with respect to the playback screen. Containing the audio content of at least one set of speaker channels in the speaker channel-based program, the set of speaker channels was generated in response to an object-based audio program. The generation comprises distorting the audio content of the object-based audio program to a degree determined by at least one distortion parameter, at least in part. Indicates the maximum degree of distortion performed by the playback system on the corresponding audio content of the object-based audio program.
The processing subsystem is configured to perform at least one of rendering of the speaker channel based audio program or decoding of the speaker channel based audio program.
Audio processing unit.
[Aspect 51]
At least one segment of the speaker channel-based audio program stored in the buffer memory contains audio content of two or more selectable sets of speaker channels of those sets. At least one represents the undistorted audio content of the object-based audio program, and at least one other of their set is at least partially determined by the at least one distortion degree parameter. 50. The audio processing unit according to aspect 50, which is generated in response to the object-based audio program, including by distorting the audio content of the object-based audio program to some extent.
[Aspect 52]
The audio processing according to aspect 50 or 51, wherein the set of speaker channels is generated by a process comprising performing off-screen distortion determined by at least one off-screen distortion parameter, at least in part. unit.
[Aspect 53]
The off-screen distortion is an undistorted position of an audio element along a width axis that is at least substantially parallel to the plane of the playback screen, at least substantially perpendicular to the plane of the playback screen. 52. The audio processing unit of aspect 52, comprising distorting to the extent controlled by the off-screen distortion parameter as a function of the distance of the distorted position to be rendered.
[Aspect 54]
The set of speaker channels is an undistorted position along the width axis of the audio object to be rendered at some distorted position along the width axis, at least substantially parallel to the plane of the playback screen. Determining the value Xs to indicate the value and the value
Xwarp
YFACTOR = y ^EXP and
X'＝ x * YFACTOR ＋ (1-YFACTOR) * [XFACTOR * Xwarp ＋ (l－XFACTOR) * x)]
It was generated by a process that involved performing distortions, including the determination of
Xwarp represents the raw, distorted position of the object along the width axis with respect to the edge of the playback screen.
EXP is an off-screen distortion parameter
YFACTOR indicates the degree of distortion along the width axis as a function of the distorted position y of the object along the depth axis at least substantially perpendicular to the plane of the playback screen.
X'represents the distorted object position of the object along the width axis with respect to the edge of the playback screen.
XFACTOR is one of the distortion degree parameters,
The audio processing unit according to any one of aspects 50 to 53.
[Aspect 55]
The object-based audio program contains screen-related metadata indicating the at least one distortion parameter, and the set of speaker channels identifies each of the distortion parameters indicated by the screen-related metadata. The audio processing unit according to any one of aspects 50 to 54, which is generated by a process including a step of parsing the object-based audio program.
[Aspect 56]
The audio processing unit according to any one of aspects 50 to 55, wherein the audio processing unit is a decoder.
[Aspect 57]
One of aspects 50-56, wherein each of the distortion parameters is a non-binary value indicating the maximum degree of distortion performed by the playback system on the corresponding audio content of the program. Audio processing unit.

Claims (5)

A processing method for rendering audio programs:
At the stage of receiving the distortion degree parameter;
The stage of performing distortion on the audio content of at least one audio object in the program, at least in part, to the extent determined by the distortion parameter and the position of at least one playback speaker. The distortion parameter indicates the maximum degree of distortion performed by the playback system on the audio object, and the possible values for the degree determined by the distortion parameter, at least in part, are no distortion. Includes, said maximum degree of distortion and one or more intermediate values between no distortion and said maximum degree of distortion, including steps.
The execution of the distortion on the audio object is based on an index parameter indicating whether or not the distortion is performed on the audio object.
Method. The method of claim 1, wherein the distortion parameter is a non-binary value indicating the maximum degree of distortion performed by the reproduction system on the audio object. A system for rendering an audio program, comprising one or more processors configured to perform the method of claim 1 or 2. A storage medium having a software program adapted for execution on a processor for performing the method step of claim 1 or 2 when executed on a computing device. A computer program product having executable instructions for performing the method according to claim 1 or 2 when executed on a computer.

Images