JP2022513184A - Dual-ended media intelligence - Google Patents

Abstract

The method of encoding audio content is to perform content analysis of the audio content, generate classification information indicating the content type of the audio content based on the content analysis, and encode the audio content and the classification information in a bitstream. And output the bitstream. A method of decoding audio content from a bit stream containing audio content and classification information about the audio content, wherein the classification information indicates content classification of the audio content, the method is the bit. It comprises receiving a stream, decoding the audio content and the classification information, and selecting a post-processing mode for performing post-processing of the decoded audio content based on the classification information. Choosing a post-processing mode can involve calculating one or more control weights for post-processing of decoded audio content based on the classification information.

Description

The present disclosure relates to a method of encoding audio content into a bitstream and a method of decoding audio content from a bitstream. The present disclosure relates specifically to methods such that classification information indicating the content type of audio content is transmitted in a bitstream.

The perceived benefits of audio signal post-processing can be improved if the audio signal processing algorithm is aware of the content being processed. For example, the accurate detection of a dialog by a dialog improver is improved if there is a measured high reliability of the dialog in the current audio frame. Also, to maintain the timbre of the music, the virtualizer may be disabled in the presence of the music content, or to maintain the timbre of the speech, the music should be timbre-matched in the presence of the dialog in the movie. Dynamic equalizers designed to allow (eg, Dolby® Volume Intelligent Equalizer) may be disabled.

Typically, users may be required to switch profiles such as "movie" or "music" to get the best settings on the playback device, which many users recognize. Often requires access to advanced settings or UI that may not be or may not be comfortable.

One approach to tackle this problem is to use content analysis tools (for example, Dolby's Media Intelligence) to determine how likely it is that some content type is in the audio stream. Will detect features in the audio signal.

Today's playback devices, such as mobile phones, that can play a wide variety of content, including movies and music, use content analysis tools (eg, Dolby's Media Intelligence) to capture certain content types in audio streams. You can determine the confidence value for existence. Content analysis tools can return confidence values (confidence scores) for the presence of "music," "utterances," or "background sound effects." The confidence values can then be used in combination to return an algorithm steer weight, which can be used to control certain post-processing features (eg, their intensity).

The method described above is a "single-ended" solution that can be performed in a decoder or in a separate post-processing library that captures PCM audio data. This single-ended implementation can be effective in manipulating post-processing algorithms, but it adds a significant amount of computation to the playback device, so the real-time nature of content analysis is a feature available on the playback device. Can be limited to.

Therefore, there is a need for improved methods and devices for content-aware processing of audio content.

The present disclosure provides a method of encoding audio content and a method of decoding audio content, each of which has the characteristics of an independent claim.

One aspect of this disclosure relates to how audio content is encoded. This method may include performing content analysis of audio content. Content analysis may be performed, for example, by applying Dolby's media intelligence tools. Content analysis may also be performed on each of the plurality of contiguous windows, each of which comprises a predetermined number of contiguous (audio) frames. Here, the content analysis may be based on one or more calculations of certainty / reliability based on determinable features within the audio content. These calculations may be dynamic and can be adjusted to amplify or deamplify a particular certainty. In more general terms, content analysis may be adaptive and / or pre-trained with predetermined audio content. Content analysis may use look-ahead buffers to reduce latency. Additional or alternative, encoding latency may be introduced to accept the processing time required for content parsing. Further, the content analysis may be executed in a plurality of passes. The method may further include generating classification information indicating the content type of the audio content based on (results in) the content analysis. The generation of classification information may also be based on the detection of scene transitions in audio content (or manual instructions for scene transitions). For example, the rate of change of confidence values contained in the classification information may be higher (ie, greater than in steady state) if scene transitions are detected / indicated. The method may further include encoding audio content and classification information, such as confidence values, into a bitstream. The encoded audio content and the encoded classification information may be multiplexed. This method may further include outputting a bitstream.

In the context of the present disclosure, the "content type" of audio content is content that can be played on a playback device and can be distinguished by the human ear by the audio characteristics of one or more of the content types. Means type. For example, music can be distinguished from speech or noise because it involves different audio frequency bandwidths, different power distributions of audio signals across different frequencies, different tone durations, different types and numbers of basic and dominant frequencies, and so on. ..

By performing content analysis on the encoder side and encoding the resulting classification information in the bitstream, the computational load on the decoder can be significantly reduced. In addition, the encoder's excellent computing power can be used to perform more complex and more accurate content analysis. Apart from addressing the different computational powers of the encoder and decoder, the proposed method provides the decoder side with additional flexibility in the audio post-processing of the decoded audio. For example, post-processing may be customized according to the device type of the device that implements the decoder and / or the user's personal preference.

In some embodiments, the content analysis may be based, at least in part, on metadata about the audio content. This provides, for example, additional control over content analysis by the content creator. At the same time, by providing appropriate metadata, the accuracy of content analysis can be improved.

Another aspect of the disclosure relates to further methods of encoding audio content. This method may include receiving user input related to the content type of audio content. User input may include, for example, a manual label or a manual trust value. The method may further include generating classification information indicating the content type of the audio content based on user input. This method may further include encoding audio content and classification information into a bitstream. For example, the label or trust value may be encoded in the bitstream. This method may further include outputting a bitstream. This method provides, for example, additional control over content analysis by the content creator.

In some embodiments, the user input may include one or more labels indicating that the audio content is of a given content type, and one or more trust values, each trust. The value is associated with each content type and gives an indicator of the certainty that the audio content is each content type. This allows the encoder user to be given additional control over the post-processing performed on the decoder side. This makes it possible, for example, to ensure that the content creator's artistic intent is preserved by post-processing.

Another aspect of the disclosure relates to further methods of encoding audio content. Audio content may be provided in a stream of audio content as part of an audio program. The method may include receiving a service type instruction indicating the service type (eg, audio program type) of the audio content. The service type may be, for example, a music service or a news (newscast) service / channel. The method may further include performing content analysis of the audio content, at least in part, based on service-type instructions. The method may further include generating classification information indicating the content type of the audio content based on (results in) the content analysis. The confidence value as an example of the classification information may be provided directly by the content creator along with the audio content. For example, whether or not to take into account the trust value provided by the content creator may depend on the service type instruction. This method may further include encoding audio content and classification information into a bitstream. This method may further include outputting a bitstream.

By considering service-type instructions, encoders can be assisted in performing content analysis. In addition, the encoder-side user can be given additional control over the decoder-side audio post-processing, which ensures that, for example, the content creator's artistic intent is preserved by the post-processing. enable.

In some embodiments, the method may further include determining whether the service type of audio content is a music service, based on service type instructions. This method further generates classification information (content type "music") indicating that the content type of the audio content is music content in response to the determination that the service type of the audio content is a music service. May include doing. This is equivalent to setting the confidence value for the content type "music" to the maximum possible value (eg 1) and setting any other confidence value to zero.

In some embodiments, the method may further include determining whether the service type of audio content is a newscast service, based on service type instructions. This method further adapts the content analysis so that it is more likely that the audio content is spoken content in response to the determination that the service type of the audio content is a newscast service. It may include that. This is to adapt one or more calculations (calculation algorithms) of the content analysis to increase the certainty / reliability of the spoken content (content type "speech") in the result of the content analysis, and / or It can be achieved by adapting the one or more calculations of content analysis to reduce the certainty / reliability of content types other than spoken content.

In some embodiments, service-type instructions may be provided on a frame-by-frame basis. Another aspect of the disclosure relates to further methods of encoding audio content. Audio content may be provided on a file-by-file basis. This method may be performed on a file-by-file basis. The file may contain metadata for each audio content. The metadata may include markers, labels, tags, and the like. This method may include, at least in part, performing content analysis of the audio content based on the metadata about the audio content. The method may further include generating classification information indicating the content type of the audio content based on (results in) the content analysis. This method may further include encoding audio content and classification information into a bitstream. This method may further include outputting a bitstream.

By considering file metadata, the encoder can be assisted in performing content analysis. In addition, the encoder-side user can be given additional control over the decoder-side audio post-processing, which ensures that, for example, the content creator's artistic intent is preserved by the post-processing. Make it possible.

In some embodiments, the metadata may include a file content type indication indicating the file content type of the file. File content types can be music files (file content type "music files"), newscast files / clips (file content type "newscast files"), or dynamic (non-static or mixed source) content. A file containing (for example, a musical genre of a movie that transitions frequently between a spoken scene and a music / song scene, for example once every few minutes; file content type "dynamic content"). You may. The file content type may be the same (uniform) for the entire file, or it may vary from part to part of the file. Content analysis may then be based, at least in part, on file content type instructions.

In some embodiments, the method may further include determining if the file content type of the file is a music file, based on the file content type indication. The method may further include generating classification information indicating that the content type of the audio content is music content in response to the determination that the file content type of the file is a music file. ..

In some embodiments, the method may further include determining if the file content type of the file is a newscast file, based on the file content type indication. This method also adapts the content analysis so that it is more likely that the audio content is spoken content in response to the determination that the file content type of the file is a newscast file. May include. This is to adapt one or more calculations (calculation algorithms) of the content analysis to increase the certainty / reliability of the spoken content in the content analysis, and / or the certainty about the content type other than the spoken content. It may be achieved by adapting the one or more calculations above to reduce the likelihood / reliability.

In some embodiments, the method may further include determining if the file content type of the file is dynamic content, based on the file content type indication. This method further includes adapting the content analysis to allow higher transition rates between different content types in response to the determination that the file content type of the file is dynamic content. good. For example, content types may be allowed to transition more frequently (ie, more frequently than in steady state) between content types, for example, between music and non-music. Further, classification information smoothing (time smoothing) may be disabled for dynamic content (ie, dynamic file content).

In some embodiments, in any of the above aspects or embodiments, the classification information may include one or more confidence values. Each confidence value may be associated with each content type and may give an indication of the likelihood that the audio content is of each content type.

In some embodiments, in any of the above aspects or embodiments, the content type comprises one or more of musical content, spoken content, or effect (eg, background effect) content. May be good. The content type may further include crowd noise / cheers.

In some embodiments, the method according to any of the above aspects or embodiments may further comprise encoding the instructions for scene transitions in the audio content into a bitstream. The scene transition instructions may include one or more scene reset flags, each of which indicates a scene transition. The scene transition may be detected by the encoder or may be provided externally by, for example, the content creator. In the former case, this method would include the step of detecting a scene transition in the audio content, and in the latter case, it would include the step of receiving a (manual) instruction for the scene transition in the audio content. By showing the scene transitions in the bitstream, audible artifacts on the decoder side that can result from improper post-processing across the scene transitions can be avoided.

In some embodiments, the method according to any of the above aspects or embodiments may further include smoothing of the classification information (time smoothing) prior to encoding. For example, confidence values may be smoothed over time. Smoothing may be context-sensitively disabled, for example during scene transitions, for audio content flagged as dynamic (non-static) according to control input / metadata. By smoothing the classification information, the stability / continuity of the audio post-processing on the decoder side can be improved.

In some embodiments, the method according to any of the above aspects or embodiments may further comprise quantizing the classification information prior to encoding. For example, the confidence value may be quantized. Thereby, the bandwidth required for transmitting the classification information in the bitstream can be reduced.

In some embodiments, the method according to any of the above aspects or embodiments may further comprise encoding the classification information into a particular data field within a packet of a bitstream. The bitstream may be, for example, an AC-4 (Dolby® AC-4) bitstream. The particular data field may be a Media Intelligence (MI) data field. The MI data field may include any, some, or all of the following fields: b_mi_data_present, music_confidence, speech_confidence, effects_confidence, b_prog_switch, b_more_mi_data_present, more_mi_data.

Another aspect of the disclosure relates to audio content and a method of decoding audio content from a bitstream containing classification information about the audio content. The classification information may indicate the content classification of the audio content. Content classification may be based on content analysis and optionally user input related to, for example, the content type of audio content (where both content analysis and user input provision are performed by the encoder. ). This method may include receiving a bitstream. The method may further include decoding audio content and classification information. The method may further include selecting a post-processing mode for performing post-processing of the decoded audio content based on the classification information. In other words, the decoding method may select post-processing of the decoded audio content based on the classification information.

By providing classification information to the decoder, the decoder does not have to analyze the content, which greatly reduces the computational load on the decoder. In addition, the decoder is given additional flexibility to be able to determine a suitable post-processing mode based on the classification information. In doing so, additional information such as device type and user preferences may be considered.

In some embodiments, the decoding method may further include calculating one or more control weights for post-processing of the decoded audio content based on the classification information.

In some embodiments, the choice of post-processing mode may be further based on user input.

In some embodiments, the audio content is channel-based. For example, the audio content may be audio content of two or more channels. Post-processing of the decoded audio content may include upmixing the channel-based audio content into the upmixed channel-based audio content. For example, 2-channel based audio content may be upmixed to 5.1 channel, 7.1 channel or 9.1 channel audio content. This method applies a virtualizer to upmixed channel-based audio content and virtualized upmixed channels for virtualization for a speaker array of the desired number of channels. -It may further include obtaining base audio content. For example, virtualization may provide upmixed 5.1-channel, 7.1-channel, or 9.1-channel audio content to a two-channel speaker array, such as headphones. However, virtualization puts upmixed 5.1 channel audio content into a 2-channel or 5.1-channel speaker array and upmixed 7.1-channel audio content into a 2-channel, 5.1-channel or 7.1-channel speaker array. The array may provide upmixed 9.1-channel audio content to 2-channel, 5.1-channel, 7.1-channel, or 9.1-channel speaker arrays.

In some embodiments, the method may further include calculating one or more control weights for post-processing of the decoded audio content based on the classification information.

In some embodiments, the classification information (encoded in the bitstream received by the decoder) may contain one or more trust values, each trust value associated with its own content type. And gives an indicator of the certainty that the audio content is its respective content type. Control weights may be calculated based on confidence values.

In some embodiments, the method further comprises routing the output of the virtualization device to a speaker array and calculating the respective control weights for the upmixer and the virtualization device based on the classification information. You may.

In some embodiments, this method applies a crossfader to channel-based audio content and virtualized upmixed audio content after applying a virtualizer to produce the output of the crossfader. It may further include routing to a speaker array. In this embodiment, the method may further include calculating the respective control weights for the upmixer and crossfader based on the classification information.

In some embodiments, the control weights may be for controlling modules other than upmixers, crossfaders or virtualizers. Similarly, several alternative methods of calculating control weights are possible. The numbers and types of control weights, as well as embodiments relating to their calculation methods, are described below in connection with the following other aspects of the present disclosure. However, these embodiments are not limited to the following aspects of the present disclosure and are applicable to any method of decoding the audio content disclosed herein.

Another aspect of the disclosure relates to further methods of decoding audio content from a bitstream containing audio content and classification information about the audio content. The classification information may indicate the content classification of the audio content. This method may include receiving a bitstream. The method may further include decoding audio content and classification information. The method may further include calculating one or more control weights for post-processing of the decoded audio content based on the classification information. The control weights may be control weights for post-processing algorithms / modules and may be referred to as algorithmic maneuvering weights. Control weights can control the strength of each post-processing algorithm.

In some embodiments, the classification information may include one or more confidence values, each confidence value is associated with a respective content type, and the certainty that the audio content is that respective content type. Give an index. Control weights may be calculated based on confidence values.

In some embodiments, the control weight may be a control weight for each module (algorithm) for post-processing of the decoded audio content. Modules (algorithms) for post-processing include: (intelligent / dynamic) equalizers, (adaptive) virtualization devices, surround processing modules, dialog improvers, upmixers, and crossfaders. Alternatively, it may include a plurality.

In some embodiments, the control weights are a control weight for an equalizer, a control weight for a virtualizer, a control weight for a surround processor, a control weight for a dialog improver, and an upmixer. It may contain one or more of the control weights of and for the crossfader. The equalizer may be, for example, an intelligent equalizer (IEQ). The virtualization device may be, for example, an adaptive virtualization device.

In some embodiments, the calculation of control weights may depend on the device type of the device performing the decoding. In other words, the calculation may be endpoint-specific or personalized. For example, the decoder side may implement a set of endpoint-specific processes / modules / algorithms for post-processing, and the parameters (control weights) for these processes / modules / algorithms are endpoint-specific. It may be determined based on the confidence value in the above manner. Thereby, the individual capabilities of each device can be taken into account when performing audio post-processing. For example, different post-processing can be applied depending on the mobile device and soundbar device.

In some embodiments, the calculation of control weights may be further based on user input. User input may override or partially override the confidence-based calculation. For example, virtualization may be applied to the utterance if the user so desires, or stereo expansion, upmixing, and / or virtualization may be applied for the PC user if the user so desires.

In some embodiments, the calculation of control weights may be further based on the number of channels of audio content. Also, the calculation of control weights may be further based on one or more bitstream parameters (eg, parameters carried by the bitstream and extractable from the bitstream).

In some embodiments, this method performs content analysis of the audio content to obtain one or more additional confidence values (eg, confidence values for content types that are not considered by the encoder side). It may include deciding. This content analysis may proceed in the same manner as described above for the encoder side. The calculation of control weights may then be further based on one or more additional confidence values.

In some embodiments, the control weights may include control weights for the virtualizer. Control weights for the virtualization device are calculated so that the virtualization device is disabled if the classification information indicates that the content type of the audio content is or is likely to be music. May be done. This may be the case, for example, if the confidence value for music exceeds a given threshold. As a result, musical tones can be saved.

In some embodiments, the control weights for the virtualization machine may be calculated so that the coefficients of the virtualization machine scale between pass-through and full virtualization. For example, the control weight for a virtualizer can be calculated as 1-music_confidence * {1-max [effects_confidence, speech_confidence] ^ 2}.

In some embodiments, the control weights for the virtualizer may further depend on the number of channels in the audio content (ie, the channel count) or other bitstream parameters (s) or other bitstream parameters (s). For example, it may be determined based on it). For example, control weights (weighting factors) for virtualization may only be determined based on confidence values for stereo content, and for all multichannel content other than stereo content (ie, ie). , For channels greater than 2, fixed control weights (eg, equal to 1) may be applied.

In some embodiments, the control weights may include control weights for the dialog improver. The control weights for the dialog improver enable dialog enhancement by the dialog improver if the classification information indicates that the content type of the audio content is or is likely to be an utterance. / May be calculated to be improved. This may be the case, for example, if the confidence value for the utterance exceeds a given threshold. Thereby, the dialog enhancement can actually be constrained to the section of audio content where it is useful, while at the same time saving computing power.

In some embodiments, the control weights may include control weights for a dynamic equalizer. The control weight for the dynamic equalizer is disabled if the classification information indicates that the content type of the audio content is or is likely to be utterance. It may be calculated as follows. This may be the case, for example, if the confidence value for the utterance exceeds a given threshold. This avoids unwanted changes in the timbre of the utterance.

In some embodiments, the method may further include smoothing of control weights (time smoothing). Smoothing may be context-sensitively disabled, for example in scene transitions, for audio content flagged as dynamic (non-static) according to control inputs / metadata and the like. By smoothing the control weights, the stability / continuity of audio post-processing can be improved.

In some embodiments, the smoothing of control weights may depend on the particular control weight to be smoothed. That is, the smoothing may differ between at least two control weights. For example, there may be no or little smoothing for dialog improver control weights, and / or stronger smoothing for virtualizer control weights.

In some embodiments, the smoothing of control weights may depend on the device type of the device performing the decoding. For example, the smoothing of virtualization device control weights may differ between the mobile phone and the TV set.

In some embodiments, the method may further include applying a non-linear mapping function to the control weights in order to increase the continuity (eg, stability) of the control weights. This may involve applying a mapping function (such as a sigmoid function) to the control weight that maps a value closer to the domain range boundary of the control weight closer to the image range boundary. Thereby, the stability / continuity of the audio post-processing can be further improved.

Another aspect of the disclosure relates to a method of decoding audio content from a bitstream containing two-channel audio content and classification information about the two-channel audio content. The bitstream may be, for example, an AC-4 bitstream. The classification information may indicate the content classification of the 2-channel audio content. This method may include receiving a bitstream. The method may further include decoding two-channel audio content and classification information. This method may further include upmixing the two-channel audio content into upmixed 5.1-channel audio content. This method may further include applying the virtualization device to the upmixed 5.1 channel audio content for 5.1 virtualization for a 2-channel speaker array. The method may further include applying crossfaders to 2-channel audio content and virtualized upmixed 5.1-channel audio content. This method may further include routing the output of the crossfader to a two-channel speaker array. Here, the method may include calculating the respective control weights for the virtualization device and / or the crossfader based on the classification information. Virtualization and crossfaders can operate under the control of their respective control weights.

Another aspect of the disclosure relates to a further method of decoding audio content from a bitstream containing two-channel audio content and classification information for the two-channel audio content. The bitstream may be, for example, an AC-4 bitstream. The classification information may indicate the content classification of the 2-channel audio content. This method may include receiving a bitstream. The method may further include decoding two-channel audio content and classification information. This method further includes applying an upmixer to the 2-channel audio content to upmix the 2-channel audio content into upmixed 5.1-channel audio content. good. This method may further include applying the virtualization device to the upmixed 5.1 channel audio content for 5.1 virtualization of the 5 channel speaker array. This method may further include routing the output of the virtualization device to a 5-channel speaker array. Here, the method may include calculating the respective control weights for the upmixer and / or the virtualization device based on the classification information. The upmixer and virtualization unit may operate under the control of their respective control weights. The control weights for the upmixer may be related to the upmix weights.

Another aspect relates to a device (eg, an encoder or decoder) that includes a processor coupled to memory that stores instructions for the processor. The processor may be adapted to implement the method according to any of the above aspects and embodiments thereof. A further aspect is a computer program containing instructions, the computer program causing the processor to execute the instructions according to any of the above aspects and embodiments thereof, and the computer program. With respect to each computer readable storage medium that is stored.

Exemplary embodiments of the present disclosure are described below with reference to the accompanying drawings. Here, a similar reference number indicates a similar element or a similar element.
An example of an encoder-decoder system according to an embodiment of the present disclosure is schematically shown. An example of a bitstream to which the embodiments of the present disclosure are applicable is shown schematically. An example of a data field for storing audio content classification information according to an embodiment of the present disclosure is schematically shown. An example of a method of encoding audio content according to an embodiment of the present disclosure is schematically shown in the form of a flowchart. An example of content analysis of audio content according to the embodiment of the present disclosure is schematically shown. Another example of how to encode audio content according to an embodiment of the present disclosure is schematically shown in the form of a flowchart. Another example of how to encode audio content according to an embodiment of the present disclosure is schematically shown in the form of a flowchart. Another example of content analysis of audio content according to an embodiment of the present disclosure is schematically shown. Yet another example of how to encode audio content according to an embodiment of the present disclosure is schematically shown in the form of a flow chart. Yet another example of content analysis of audio content according to an embodiment of the present disclosure is schematically shown. An example of a method of decoding audio content according to an embodiment of the present disclosure is schematically shown in the form of a flowchart. Another example of how to decode audio content according to an embodiment of the present disclosure is schematically shown in the form of a flowchart. An example of control weight calculation according to the embodiment of the present disclosure is schematically shown. Another example of how to decode audio content according to an embodiment of the present disclosure is schematically shown in the form of a flowchart. An example of using control weights in a decoder according to an embodiment of the present disclosure is schematically shown. Yet another example of how to decode audio content according to an embodiment of the present disclosure is schematically shown in the form of a flow chart. Another example of the use of control weights in a decoder according to an embodiment of the present disclosure is schematically shown.

As mentioned above, the same or similar reference numbers in the present disclosure indicate the same or similar elements, and the repeated description thereof may be omitted for the sake of brevity.

Broadly speaking, this disclosure proposes to transfer content analysis from an audio decoder to an audio encoder, thereby creating a dual-ended approach to audio post-processing. That is, at least part of the content analysis module is moved from the decoder to the encoder, and the audio stream (bitstream) is the classification information (eg, confidence value,) generated by (part of) the content analysis module in the encoder. Updated to carry a trust label, or confidence score). The weighting calculation is left in the decoder, which operates on the classification information received with the audio stream.

An example of an encoder-decoder system 100 that implements the above scheme is shown in FIG. 1 in the form of a block diagram. The encoder-decoder system 100 includes an (audio) encoder 105 and an (audio) decoder 115. It is understood that the modules of encoder 105 and decoder 115 described below can be implemented, for example, by their respective processors in their respective computing units.

The encoder 105 has a content analysis module 120 and a multiplexer 130. Therefore, as mentioned above, content analysis is now part of the encoder stage. Encoder 105 receives input audio content 101 to be encoded, possibly in relation to relevant metadata and / or user input. The input audio content 101 is provided to the content analysis module 120 and the multiplexer 130. The content analysis module 120 performs content analysis of audio content 101 (eg, by applying Dolby's media intelligence tools) to derive classification information 125 for audio content. Classification information 125 indicates the content type of the input audio content 101, as estimated by content analysis. As described in more detail below, the classification information 125 has one or more confidence values associated with each content type (eg, "music", "utterance", "background effect" confidence values). Can include. In some embodiments, the confidence value may have a higher particle size. For example, classification information 125 substitutes for or in addition to the confidence value for the content type "music" (for example, the content type "classical music", "rock / pop" for one or more music genres. Confidence value for "music", "acoustic music", "electronic music", etc.) can be included. In some embodiments, the content analysis may further be based on metadata and / or user input for audio content (eg, control input from the content creator).

The multiplexer 130 multiplexes the audio content and classification information 125 into the bitstream 110. Audio content can be encoded according to known audio coding methods, such as encoding according to the AC-4 coding standard. As a result, the audio content 101 and the classification information 125 may be said to be encoded in the bitstream 110, which is said to include the audio content and the relevant classification information about the audio content. You may be broken. The bitstream 110 may then be provided to the decoder 115.

In some implementations, content analysis in the encoder 105 of the encoder-decoder system 100 may be performed for each of a plurality of contiguous windows, each of which comprises a predetermined number of contiguous (audio) frames. ..

Content analysis may be based on one or more calculations of certainty / reliability of each content type, based on determinable features within the audio content.

For example, content analysis may include steps for audio content preprocessing, feature extraction, and confidence value calculation. Preprocessing may be optional and may include downmixing, reframement, calculation of amplitude spectra, and the like. Feature extraction may extract / calculate multiple features (eg, hundreds of features) from audio content. These features may include Mel-Frequency Cepstral Coefficient (MFCC), MFCC flux, zero crossover rate, chroma, autocorrelation, and the like. Calculations that ultimately give confidence values may be performed, for example, by a trained machine learning network.

The calculations performed in the context of content analysis (eg, by machine learning networks) may be variable / adaptive. If the calculations are variable, they can be adjusted to derive classification information according to preferences for certain content types. For example, content analysis (default) has a confidence value of 0.7 for content-type "music", a confidence value of 0.15 for content-type "speech", and a confidence value of 0.15 for content-type "effect" for a given audio content. May be returned (note that the confidence values in this example add up to 1). If the content analysis is adapted to have some preference for the content type "music" (ie, if the calculation is adapted for this purpose), the adapted content analysis / calculation is, for example, the content type "music". A 0.8 confidence value for "music", a 0.1 confidence value for content type "speech", and a 0.1 confidence value for content type "effect" may be given. Further, non-limiting examples to which the calculations are applied will be described later.

In addition, content analysis (eg, machine learning networks) may be adaptive and / or pre-trained with pre-determined audio content. For example, in a dual-ended system such as the encoder-decoder system 100, content analysis can be further developed over time to improve the accuracy of feature labeling. Advances can come from the increased complexity that can be met through increased computing power and / or increased computer processor power on the encode server. Content analysis may be improved over time through manual labeling of specific content types.

Content analysis on the encoder side may use a look-ahead buffer or something similar to reduce the latency of content type determination. This addresses a known limitation in single-ended implementations that require fairly large audio frames to make powerful decisions. For example, a 700ms audio frame may be required to make a decision about the existence of a dialog, at which point the dialog's confidence score is 700ms behind the start of the utterance and the start of the spoken phrase. May be overlooked. Additional or alternative, encoding latency may be introduced to accept the processing time required for content parsing.

In some implementations, content analysis may be performed on multiple passes to improve the accuracy of content typing.

In general, the generation of classification information may also be based on the detection of scene transitions in audio content (or manual instructions for scene transitions). For this purpose, the encoder 105 may have an additional reset detector for detecting such scene transitions / resets in the audio content. Manual labeling or additional reset scene detection may be used to influence the rate of change of confidence values in content analysis. For example, the rate of change of confidence values contained in the classification information may be higher if the scene transition is detected / indicated (ie, greater than in steady state). In other words, when the audio program changes, the confidence values may be allowed to adapt faster than in the steady state of the audio program. This is to ensure that audible transitions between post-processing effects are minimized. According to the scene detection, a scene transition instruction (for example, one or more reset flags (scene transition flags) each indicating each scene transition) is encoded in the bitstream 110 together with the classification information 125 (for example, a confidence value). / May be multiplexed.

The decoder 115 in the encoder-decoder system 100 includes a demultiplexer 160, a weight calculation module 170, and a post-processing module 180. The bitstream 110 received by the decoder 115 is demultiplexed in the demultiplexer 160 and after decoding according to known audio decoding methods such as decoding according to the AC-4 coding standard, classification information 125 and audio. -Content is extracted. As a result, the audio content and classification information 125 may be said to be decoded from the bitstream 110. The decoded audio content is provided to a post-processing module 180 that performs post-processing of the decoded audio content. For this purpose, the decoder 115 selects a post-processing mode for the post-processing module 180 based on the classification information 125 extracted from the bitstream 110. More specifically, the classification information 125 extracted from the bitstream 110 is provided to the weight calculation module 170, which is used for post-processing of decoded audio content based on the classification information 125. Calculate one or more control weights 175. Each control weight may be, for example, a number between 0 and 1 and may determine the strength of each process / module / algorithm for post-processing. The one or more control weights 175 are provided to the post-processing module 180. The post-processing module 180 can select / apply a post-processing mode according to the control weight 175 to post-process the decoded audio content. The choice of post-processing mode may be further based on user input in some embodiments. Post-processing of the decoded audio content by the post-processing module 180 using the selected post-processing mode can result in the output audio signal 102 output by the decoder 115.

The calculated control weight 175 may be a control weight for a post-processing algorithm executed by the post-processing module 180, and may thus be referred to as an algorithm maneuvering weight. Thus, the one or more control weights 175 can provide maneuvering for the post-processing algorithm in the post-processing module 180. In this sense, the control weight 175 may be a control weight for each (sub) module for post-processing of the decoded audio content. For example, the post-processing module 180 may include one or more (intelligent / dynamic) equalizers, (adaptive) virtualization devices, surround processors, dialog improvers, upmixers, and / or crossfaders. Sub) may include modules. The control weight 175 may be a control weight for these (sub) modules, and these (sub) modules may operate under the control of their respective control weights. Thus, the control weight 175 is a control weight for an equalizer (eg, an intelligent equalizer (IEQ)), a control weight for a virtualizer (eg, an adaptive virtualizer), a control weight for a surround processor, and so on. It may include one or more of the control weights for the dialog improver, the control weights for the upmixer, and / or the control weights for the crossfader. Here, it is understood that the intelligent equalizer adjusts a plurality of frequency bands using a target spectrum profile. The gain curve is adapted depending on the audio content to which the intelligent equalizer is applied.

Determining the classification information 125 in the encoder 105 and providing it to the decoder 115 as part of the bitstream 110 can reduce the computational load on the decoder 115. In addition, the higher computational power of the encoder can be used to make content analysis more powerful (eg, more accurate).

FIG. 2 schematically illustrates an AC-4 bitstream as an exemplary implementation of bitstream 110. Bitstream 110 includes multiple frames (AC-4 frames) 205. Each frame 205 contains a sync word, a frame word, a raw frame 210 (AC-4 frame), and a CRC word. Raw frame 210 includes a table of contents (TOC) field and a plurality of substreams as shown in the TOC field. Each substream contains audio data field 211 and metadata field 212. The audio data field 211 may contain encoded audio content and the metadata field 212 may contain classification information 125.

Given such a bitstream structure, classification information 125 may be encoded in a particular data field within a bitstream packet. FIG. 3 schematically shows an example of a data field in (a frame of) a bitstream for carrying classification information 125. This data field may be referred to as an MI data field. The data field may include multiple subfields 310-370. For example, the data field is a b_mi_data_present field 310 that indicates whether classification information (media information or media intelligence) is present in the frame, a music_confidence field 320 that contains a confidence value for the content type "music", and a content type "speech". Speech_confidence field 330 containing confidence values for, effects_confidence field 340 containing confidence values for content type "effects", b_prog_switch field 350, b_more_mi_data_present field 360 indicating whether further classification information (media information) exists, and further classification. It may include any, part or all of the more_mi_data field 370 containing information (eg, confidence values for crowd noise). Classification information (eg, confidence values) is determined by long-term analysis (content analysis) and can change relatively slowly. Therefore, the classification information does not have to be encoded for each packet / frame, and may be encoded in one of N frames, for example, with N ≧ 2.

Alternatively, the classification information 125 (eg, confidence value) may be encoded into a presentation substream of the AC-4 bitstream. In addition, for file-based audio content, the classification information 125 (eg, confidence value) does not have to be encoded for each frame, and the appropriate data in the bitstream so that it is valid for every frame in the file. It may be encoded in a field.

FIG. 4 is a flowchart showing an example of a method 400 for encoding audio content. Method 400 may be performed, for example, by the encoder 105 in the encoder-decoder system 100 of FIG.

In step S410 , content analysis of the audio content is performed.
In step S420 , classification information indicating the content type of the audio content is generated based on (result) of the content analysis.
In step S430 , the audio content and classification information is encoded into a bitstream.
Finally, in step S440 , a bitstream is output.

In particular, the steps of method 400 may be performed in the manner described above for the encoder-decoder system 100.

As mentioned above, the generation of classification information may be further based on the detection of scene transitions (or manual instructions for scene transitions) in audio content. Thus, method 400 (or any of methods 600, 700, or 900 described below) detects scene transitions in audio content (or receives input of manual instructions for scene transitions in audio content) and audio. -It may further include encoding the instruction of the scene transition in the content into the bitstream.

Next, with reference to FIG. 5, the details of the content analysis (for example, the content analysis performed by the content analysis module 120 of the encoder 105 or the content analysis performed in step S410 of the method 400) will be described.

As mentioned above, the content analysis produces classification information 125 indicating the content type of the audio content 101. In some embodiments of the present disclosure, the classification information 125 comprises one or more confidence values (feature confidence values, confidence scores). Each of these confidence values is associated with each content type and gives an indication of the certainty that the audio content is that respective content type. These content types can include one or more of music content, spoken content, and effect (eg, background effect) content. In some implementations, the content type can further include crowd noise content (eg cheers). That is, the classification information 125 indicates a music reliability value indicating the reliability (probability) that the audio content is the content type "music", and the audio content 101 indicates the reliability (probability) that the content type "speech" is. The speech confidence value, and the effect confidence value indicating the reliability (certainty) that the audio content 101 is the content type "effect", and possibly the audio content 101 is the content type "crowd noise". It can include one or more of the crowd noise confidence values that indicate confidence.

In the following, it is assumed that the confidence value is normalized so that it falls in the range of 0 to 1. Here, 0 indicates that the audio content is zero (0%) likely to be of its respective content type, and 1 indicates that the audio content is of its respective potential. Shows certainty (complete certainty, 100%). A value "0" is an unrestricted example of a confidence value indicating zero probability, and a value "1" is an unrestricted example of a confidence value indicating complete certainty. Is understood.

In the example of FIG. 5, the content analysis of the audio content 101 returns a (raw) music confidence value of 125a, a (raw) speech confidence value of 125b, and a (raw) effect confidence value of 125c. In principle, these raw confidence values 125a, 125b, 125c can also be used to be directly encoded into bitstream 110 as (part of) classification information 125. Alternatively, the classification information 125 (ie, raw confidence values 125a, 125b, 125c) can be smoothed (eg, time smoothed) prior to encoding to obtain a substantially continuous confidence value. can. This can be done by the smoothing modules 140a, 140b, 140c, respectively, which output the smoothed confidence values 145a, 145b, 145c, respectively. There, different smoothing modules can apply different smoothings, for example, using different parameters / coefficients for smoothing.

In line with the above, method 400 (or any of methods 600, 700, or 900 described below) may smooth out classification information (eg, confidence values) prior to multiplexing / encoding. It may be further included.

Smoothing of classification information (eg, confidence values) can cause audible distortion under certain circumstances, for example, when smoothing is performed across scene transitions. Therefore, smoothing may be disabled depending on the situation, for example, in a scene transition. In addition, smoothing may be disabled for dynamic (non-static) audio content or according to control input or metadata, as described in more detail below.

The smoothed music confidence value 145a, the smoothed speech confidence value 145b, and the smoothed effect confidence value 145c can be further quantized before encoding in some implementations. This can be done with the quantizers 150a, 150b, 150c, which output the quantized confidence values 155a, 155b, 155c, respectively. There, different quantizers may apply different quantizations, for example using different parameters for quantization.

In line with the above, method 400 (or any of methods 600, 700, or 900 described below) quantizes classification information (eg, confidence values) prior to multiplexing / encoding. It may be further included.

The smoothing of classification information 125 can result in improved continuity and stability of post-processing in the decoder, and thus a listening experience. By quantizing the classification information 125, the bandwidth efficiency of the bitstream 110 can be improved.

As mentioned above, it may be advantageous in terms of computational power to determine the classification information 125 in the encoder 105 and provide it to the decoder 115 as part of the bitstream 110. In addition, doing so may allow some encoder-side control over audio post-processing on the decoder-side by setting the confidence value transmitted in the audio stream to some desired value. For example, by making the classification information (at least partially) dependent on user input on the encoder side, the user on the encoder side (eg, the content creator) gives control over audio post-processing on the decoder side. Can be Next, some exemplary implementations that allow additional control on the encoder side for audio post-processing on the decoder side will be described.

FIG. 6 schematically illustrates in the form of a flow chart an example of a method 600 for encoding audio content that allows such encoder-side control of decoder-side audio post-processing based on user input. Method 600 may be performed, for example, by the encoder 105 in the encoder-decoder system 100 of FIG.

At step S610 , user input is received. The user may be, for example, a content creator. User input can include manual labels for labeling audio content as related to certain content types, or can be related to, for example, manual confidence values.

Step S620 generates classification information indicating the content type of the audio content, at least in part, based on user input. For example, manual labels and / or manual confidence values can be used directly as classification information. If the audio content is manually labeled as of a certain content type, the confidence value for that particular content type can be set to 1 (trust value between 0 and 1). ), Other confidence values can be set to zero. In this case, content analysis will be bypassed. In an alternative implementation, the content analysis output, along with user input, can be used to derive classification information. For example, the final confidence value can be calculated based on the confidence value generated in the content analysis and the manual confidence value. This may be done by averaging these confidence values, or by any other suitable combination.

In step S630 , the audio content and classification information is encoded in the bitstream.

Finally, in step S640 , the bitstream is output.

Additional encoder-side control can be achieved by relying, at least in part, on the metadata associated with the audio content for content classification decisions on the encoder side. Two examples of such encoder-side processing will be described below. A first example will be described with reference to FIGS. 7 and 8. In the first example, the audio content is provided as part of an audio program in a stream of audio content (eg, a linear continuous stream). Metadata for audio content includes at least service-type instructions for audio content (ie, audio programs). Therefore, the service type may be referred to as an audio program type. Examples of service types can include music services (eg, music streaming services or music broadcasts) or news (newscast) services (eg, audio components of news channels). Serviced instructions may be provided on a frame basis or may be the same (uniform / static) for the audio stream. A second example will be described with reference to FIGS. 9 and 10. In the second example, the audio content is provided on a file basis. Each file may contain metadata for its own audio content. The metadata may include the file content type of the file (audio content of). The metadata may further include markers, labels, tags, and the like. Examples of file content types are indications that the file is a music file, indication that the file is a news / newscast file (news clip), and that the file is dynamic (non-static) content (eg, non-static). It may include instructions to include (such as the musical genre of a movie that frequently transitions between spoken scenes and music / song scenes). The file content type may be the same (uniform / static) for the entire file, or it may vary from part to part of the file. The process in the second example may be file-based. File "tagging" with metadata indicating the file content type can be said to assist the encoder in deriving classification information (to provide the encoder with additional control over audio post-processing on the decoder side). father).

Now refer to FIG. FIG. 7 shows, in the form of a flow chart, how to encode audio content provided within a stream of audio content as part of an audio program. This method 700 takes into account audio content metadata when deriving classification information. Method 700 may be performed, for example, by the encoder 105 in the encoder-decoder system 100 of FIG.

At step S710 , the service type instruction is received. As mentioned above, the service type instruction indicates the service type of the audio content.
In step S720 , content analysis of the audio content is performed, at least in part, based on service-type instructions. Non-limiting examples of such content analysis are described below with reference to FIG.
In step S730 , classification information indicating the content type of the audio content is generated based on (result) of the content analysis.
In step S740 , the audio content and classification information is encoded in the bitstream.
Finally, in step S750 , the bitstream is output.

FIG. 8 schematically shows an example of content analysis of audio content in step S720 of method 700. The upper 810 of FIG. 8 relates to an example of a music service, that is, a service-type instruction indicating that the audio content is a service-type “music service”, and in this case, the confidence value for “music” is 1. It may be set and the confidence value for other content types (eg, "speech", "effect", and possibly "crowd noise") is set to 0. That is, the content type "music" may be hard-coded into classification information. As described above, the method 700 may include determining whether or not the service type of the audio content is a music service based on the service type instruction. Then, in response to the determination that the service type of the audio content is a music service, the classification information may be generated to indicate that the content type of the audio content is music content.

The lower 820 of FIG. 8 relates to an example of a news service, that is, a service-type instruction indicating that the audio content is a service-type “news service” (or newscast service, news channel). In this case, the calculations used in the content analysis are adapted to have a clear preference for speech, for example, less preference for music (eg, spoken content given by content analysis (content-type "speech"). ) May be increased, and the confidence value for music content (content type "music") and possibly any remaining content type may be decreased). In other words, the adaptation of the calculation reduces the possibility of false indications of content-type "music". Thus, the method 700 may include determining whether the service type of the audio content is a newscast service based on the service type instruction. Then, in response to the determination that the service type of the audio content is the newscast service, the content analysis in step S720 is adapted to have a higher probability that the audio content is spoken content. Can be done. Further, the content analysis in step S720 may be adapted to have a lower probability of indicating that the audio content is of any other content type.

In some implementations, one or more confidence values for audio content can be provided directly by user input (eg, by the content creator) or as part of the metadata. Whether or not these confidence values are then taken into account may then depend on the service type instruction. For example, the trust value provided by user input or metadata can be used to encode as classification information if the service type of the audio content is of some type (and only then). In some alternative implementations, the trust values provided by user input or metadata can be used as part of the classification information unless the audio content service type is of some type. For example, trust values provided by user input or metadata can be used unless the service type instruction indicates that the service type of the audio content is a music service. If the service type instruction indicates that the service type of the audio content is a music service, the trust value for the music content is set to 1 regardless of the trust value provided by user input or metadata. May be good.

Now refer to FIG. FIG. 9 shows, in the form of a flow chart, a method 900 for encoding audio content provided on a file basis. Thus, Method 900 may be file-based. This method 900 takes into account the file metadata of the audio content when deriving the classification information. Method 900 may be performed, for example, by the encoder 105 in the encoder-decoder system 100 of FIG.

In step S910 , content analysis of the audio content is performed, at least in part, based on the (file) metadata about the audio content. For example, the metadata can include a file content type indication indicating the file content type of the file. Content analysis may then be based, at least in part, on file content type instructions. An unrestricted example of such content analysis based at least in part on the content type of the file is described below with reference to FIG.

In step S920 , classification information indicating the content type of the audio content is generated based on (result) of the content analysis.

In step S930 , the audio content and classification information is encoded in the bitstream.

Finally, in step S940 , the bitstream is output.

FIG. 10 schematically shows an example of content analysis of audio content in step S910 of Method 900. The upper 1010 of FIG. 10 relates to an example of a music file, that is, a file content type instruction indicating that the file content is a file content type “music”, in which case the content type “music” is classified. The information may be hard coded. Further, the classification information may be uniform (static) for the entire file. Thus, Method 900 may further include determining if the file content type of the file is a music file, based on the file content type indication. Then, in response to the determination that the file content type of the file is a music file, the classification information may be generated to indicate that the content type of the audio content is music content.

The middle 1020 of FIG. 10 relates to an example of a news file, that is, a file content type instruction indicating that the file content is the file content type “news”. In this case, Method 900 may further include determining if the file content type of the file is a newscast file, based on the file content type indication. Then, in response to the determination that the file content type of the file is a newscast file, content analysis may be adapted to have a higher probability that the audio content is spoken content. good. This is done by applying one or more calculations (calculation algorithms) of the content analysis to increase the certainty / reliability of the spoken content in the content analysis and / or for content types other than the spoken content. It may be achieved by adapting the one or more calculations above to reduce certainty / confidence. Again, the classification information may be uniform (static) for the entire file.

The lower 1030 of FIG. 10 shows an example of a dynamic (non-static) file (for example, a musical genre of a movie that frequently transitions between a spoken scene and a music / song scene), that is, file content. File content type Regarding the file content type instruction indicating that it is "dynamic". In this case, Method 900 further includes determining if the file content type of the file is dynamic content (ie, dynamic file content) based on the file content type indication of the file. You may. Content analysis then allows higher transition rates between different content types in response to the determination that the file content type of the file is dynamic content (ie, dynamic file content). May be adapted. For example, a content type may be allowed to transition more frequently (ie, more frequently than for steady state) between content types, for example, between music and non-music. Thus, classification information may be allowed to switch between, for example, the music section and the non-music section of the file. Unlike the first two rows 1010 and 1020 in Figure 10, this means that the classification information is not kept uniform (static) throughout the file.

It is also understood that dynamic content (ie, dynamic file content) can have sharp transitions between sections of different content types within a file. For example, there can be sharp transitions between the music section and the non-music section. In such cases, it may not make sense to apply time smoothing to the classification information (eg, confidence values). In some implementations, therefore, classification information smoothing (time smoothing) may be disabled for dynamic content (ie, dynamic file content).

Next, an embodiment and implementation of decoding audio content from a bitstream containing audio content and classification information about the audio content will be described. It is understood that the classification information indicates the content classification (with respect to the content type) of the audio content. It is also understood that the content classification may be based on the content analysis performed on the encoder side.

FIG. 11 shows, in the form of a flow chart, a common method 1100 for decoding audio content from a bitstream. Method 1100 may be performed, for example, by the decoder 115 in the encoder-decoder system 100 of FIG.

In step S1110 , the bitstream is received, for example, by wireless or wired transmission, or via a storage medium that stores the bitstream.
In step S1120 , the audio content and classification information is decoded from the bitstream.
In step S1130 , a post-processing mode for performing (audio) post-processing of the decoded audio content is selected based on the classification information obtained in step S1120. In some implementations, the choice of post-processing mode can be further based on user input.

Further, Method 1100 further comprises performing content analysis of the audio content to determine one or more additional confidence values (eg, for content types not considered by the encoder side). May be good. This content analysis may proceed in the same manner as described above with reference to step S410 in Method 400. The choice of post-processing mode may then be further based on the one or more additional confidence values. For example, if the decoder contains a detector for a content type that was not considered by the (legacy) encoder, the decoder will calculate the confidence value for this content type and select this confidence value to select the post-processing mode. Can be used with any confidence value transmitted in the classification information.

As mentioned above in the context of Figure 1, the post-processing implements, for example, an (intelligent / dynamic) equalizer, an (adaptive) virtualization device, a surround processor, a dialog improver, an upmixer, or a crossfader, respectively. It may be executed using a post-processing algorithm such as the algorithm of. Therefore, selecting the mode for performing post-processing can result in one or more control weights (steering weights, algorithm maneuvering weights, algorithm control weights) for each process / module / algorithm for post-processing. It can be said that it corresponds to making a decision (for example, calculating).

The corresponding method 1200 is shown by the flowchart of FIG. Again, this method 1200 may be performed, for example, by the decoder 115 in the encoder-decoder system 100 of FIG. Step S1210 and step S1220 are identical to step S1110 and step S1120 of method 1100, respectively.

In step S1230 , one or more control weights for post-processing of the decoded audio content are determined (eg, calculated) based on the classification information obtained in step S1220.

Sending confidence values instead of control weights (steering weights), that is, leaving the weighting module in the decoder instead of moving it to the encoder, not only allows the decoder to save computational resources, but also allows the weighting to be calculated. It can also enable a customizable and flexible decoder that can be personalized. For example, weighting can depend on the device type and / or the user's personal preferences. This is in contrast to the traditional approach in which the decoder receives specific instructions from the encoder as to which audio post-processing should be performed on the decoded audio content.

That is, the requirements for audio post-processing may depend on the device type of the device on which the decoded audio content is played. For example, playing decoded audio content on a speaker on a mobile device with only two speakers (for example, a mobile phone) is similar to playing decoded audio content on a soundbar device with five or more speakers. May require different audio post-processing. Thus, in some implementations, the calculation of control weights depends on the device type of the device performing the decoding. In other words, the calculation may be endpoint-specific or personalized. For example, the decoder side may implement a set of endpoint-specific processes / modules / algorithms for post-processing, and the parameters (control weights) for these processes / modules / algorithms are endpoint-specific. It may be determined based on the confidence value in the above manner.

In addition, different users may have different preferences for audio post-processing. For example, utterances are typically not virtualized, but based on user preferences, it can be decided to virtualize utterance-rich audio content (ie, if the user so desires, virtualization). May be applied to utterances). As another example, for audio playback on a personal computer, there is typically no stereo expansion, upmix, and virtualization. However, depending on the user's preferences, stereo expansion, upmix, and / or virtualization may be applied in this case (ie, stereo expansion, upmix, and / or virtualization if the user so desires). May be applied for PC users). Thus, in some implementations, the calculation of control weights is further based on user preference or user input (eg, user input indicating user preference). Thus, user input can override or partially override classification information-based calculations.

If the classification information contains a confidence value (confidence score), each associated with each content type, and, as described above, gives an indicator of the certainty that the audio content is each content type, the control weight is It can be calculated based on these confidence values. Non-limiting examples of such calculations will be described later.

Further, Method 1200 further comprises performing content analysis of the audio content to determine one or more additional confidence values (eg, for content types not considered by the encoder side). You may. This content analysis may proceed in the same manner as described above with reference to step S410 of method 400. The calculation of the control weight mode may then be further based on the one or more additional confidence values. For example, if the decoder has a detector for a content type that was not considered by the (legacy) encoder, the decoder will calculate the confidence value for this content type and this confidence value to calculate the control weights. Can be used with any confidence value transmitted in the classification information.

As mentioned above, the confidence values may be smoothed on the encoder side in a dual-ended encoder-decoder system to accurately and stably reflect the encoded content. Alternatively or additionally, the weight calculation on the decoder side may provide further smoothing in determining the control weight (algorithm maneuvering weight). Thereby, it can be ensured that each post-processing algorithm has an appropriate level of continuity to avoid audible distortion. For example, a virtualizer may want slow changes to avoid unwanted fluctuations in the spatial image, while a dialog improver is responsive to dialog frames but non-dialog frames. You may want fast changes to ensure that any false dialog improvements are minimized. Therefore, the method 1200 may further include a step of smoothing the control weights (time smoothing).

Smoothing may depend on the device type of the device performing the decoding. For example, there may be different smoothing between the virtualization device control weights for mobile devices (eg, mobile phones) and the virtualization device control weights for TV sets or soundbar devices. There, smoothing may differ with respect to a set of smoothing coefficients that determine smoothing, such as the time constant of smoothing.

In addition, smoothing may also depend on the specific control weights to be smoothed. That is, smoothing can differ between at least two control weights. For example, there may be no or little smoothing for dialog improver control weights, and / or stronger smoothing for virtualizer control weights.

Finally, note that smoothing may be disabled in some situations. As mentioned above, smoothing can be counterproductive for audio content flagged as dynamic (non-static), or for scene transitions. Also, smoothing may be disabled according to control inputs and / or metadata.

Another approach to improving the continuity / stability of control weights (and thus audio post-processing) is to apply a non-linear mapping Φ to the control weights. The value of the control weight may be in the range 0 to 1. The nonlinear mapping Φ may be a map Φ: [0,1] → [0,1]. Preferably, the nonlinear mapping Φ places the value of the control value close to the boundary of the value range of the control weight (ie, the domain range such as [0,1]) into the value range of the mapped value (ie, [0]. , 1] Map closer to each boundary of the image range). That is, Φ may be mapped so that the value 0 + ε (ε << 1) approaches 0, that is, Φ (0 + ε) <(0 + ε), and the value 1−ε may be mapped so as to approach 1. Well, that is, Φ (1-ε)> (1-ε). An example of such a nonlinear mapping Φ is a sigmoid function.

FIG. 13 schematically shows an example of a weight calculation module 170 that operates according to the above considerations. It is understood that the weight calculation module 170 described below may be implemented, for example, by the processor of the computing unit.

Without intention of limitation, the weight calculation module 170 of this embodiment determines the control weights for the intelligent / dynamic equalizer and the control weights for the virtualizer. It is understood that other control weights may also be calculated by the weight calculation module 170.

The weighting module 170 receives a confidence value (ie, classification information 125) as input. Based on the confidence value, the control weights for the intelligent / dynamic equalizer are calculated in block 1310. In some implementations, the classification information (eg, confidence) is decoded audio content, as equalization can change the timbre of the utterance, and thus is typically undesirable for the utterance. Intelligent / dynamic, etc. so that equalization is disabled if the content type of is utterance or is likely to be utterance (eg, if the utterance confidence value exceeds a certain threshold). The control weight for the timbre (equalizer control weight) can be calculated. Optionally, at block 1330, the equalizer control weights may be smoothed. The smoothing may depend on the equalizer control weight smoothing factor 1335, which may be specific to the smoothing of the equalizer control weights. Finally, the (smoothed) equalizer control weight 175a is output by the weight calculation module 170.

The confidence value is also used in block 1320 to calculate the control weight for the virtualizer (virtualizer control weight). Virtualization can change the musical timbre, and is therefore typically undesirable for music, so in some implementations the classification information (eg, confidence values) is decoded audio. · Virtualization (speaker virtualization) is disabled if the content type of the content indicates that it is music or is likely to be music (for example, if the music confidence value exceeds a certain threshold). As such, the control weights for the virtualizer may be calculated. Also, the control weights for the virtualizer may be calculated so that the coefficients of the virtualizer scale between pass-through (no processing) and full virtualization. As an example, the control weights of the virtualizer are based on the music confidence value music_confidence, the speech confidence value speech_confidence, and the effect confidence value effect_confidence.
1－music_confidence * {1-max [effects_confidence, speech_confidence] ^ 2} (Equation 1)
Can be calculated by

Optionally, the virtualization device control weights may be smoothed in block 1340. The smoothing may depend on the virtualization device control weight smoothing factor 1345, which may be specific to the smoothing of the virtualization device control weights.

Further, optionally, in block 1350, the (smoothed) virtualization device control weights may be amplified, for example, by a sigmoid function, in order to improve the stability / continuity of the virtualization device control weights. This can reduce audible artifacts in the rendered representation of post-processed audio content. Amplification may proceed according to the nonlinear mapping described above.

Finally, the (smoothed and / or amplified) virtualization control weights 175b are output by the weight calculation module 170.

The confidence value can also be used to calculate the control weights for the dialog improver (dialog improver control weights; not shown). The dialog improver may detect the time-frequency tile containing the dialog in the frequency domain. And these time-frequency tiles can be selectively improved, thereby emphasizing the dialog. The main purpose of the dialog improver is to emphasize the dialog, and applying the dialog improvement to the content without the dialog is often a waste of computational resources, so the classification information is the content of the audio content. If the type is or is likely to be spoken, then the dialog improver control weights are calculated so that the dialog improver's dialog enhancement is enabled (and only then). May be good. This may be the case, for example, if the confidence value for the utterance exceeds a given threshold. Similar to equalizer control weights and virtualizer control weights, dialog improver control weights can also be subject to smoothing and / or amplification.

In addition, confidence values can be used to calculate control weights for surround processors (surround processor control weights; not shown), upmixers, and / or crossfaders.

FIG. 14 shows bitstream to audio content in a special case of two-channel (eg stereo) audio content for playback by a mobile device with two speakers (eg, a mobile phone) according to an embodiment of the present disclosure. How to decode 1400 is shown in the form of a flowchart. It is understood that the bitstream contains classification information or two-channel audio content, and the classification information indicates the content classification (eg, for content type) of the two-channel audio content. Method 1400 may be performed by a decoder on a mobile device with two speakers. The decoder may have the same basic configuration as the decoder 115 in the encoder-decoder system 100 of FIG. 1, and may include, for example, specific implementations of weighting and post-processing.

At step S1410 , the AC-4 bitstream is received.
In step S1420 , the 2-channel audio content and classification information is decoded / multiplexed from the bitstream.
In step S1430 , the 2-channel audio content decoded in step S1420 is upmixed into upmixed 5.1-channel audio content.
In step S1440 , a virtualization device is applied to the upmixed 5.1 channel audio content for 5.1 virtualization for a 2-channel speaker array. The virtualization machine operates under the control of each control weight. Control weights for the virtualizer are calculated based on classification information (eg, confidence values). This can be done, for example, in the manner described above with reference to FIG.
In step S1450 , the crossfader is applied to 2-channel audio content and virtualized upmixed 5.1-channel audio content. The crossfader operates under the control of each control weight. Control weights for crossfaders are calculated based on classification information (eg confidence values).
Finally, in step S1460 , the output of the crossfader is routed to a two-channel speaker array.

FIG. 15 schematically illustrates an example of a decoder 1500 of a two-speaker mobile device 1505 capable of performing method 1400 according to an embodiment of the present disclosure. It is understood that the modules of the decoder 1500 described below may be implemented, for example, by the processor of the computing unit.

The decoder 1500 receives a bitstream 110 (eg, an AC-4 bitstream), which is then decoded / multiplexed by the AC-4 (mobile) decoder module 1510. The AC-4 (mobile) decoder module 1510 outputs decoded 2-channel audio content 1515 and decoded classification information 125. The decoded classification information 125 is provided to the virtualizer crossfade weight calculation module 1570, which calculates the crossfade control weight 1575 based on the classification information 125 (eg, confidence value). The crossfade control weight 1575 may be a parameter that determines the relative weights of the two signals combined by the crossfade module 1540. The decoded 2-channel audio content 1515 is upmixed from 2.0 channels to 5.1 channels by the upmix module 1520, which outputs the upmixed 5.1 channel audio content 1525. The 5.1 virtualization for stereo speakers is then applied to the upmixed 5.1 channel audio content 1525 by the virtualization module (virtualizer) 1530. The virtualization module outputs virtualized upmixed 5.1 channel audio content 1535, which is combined with the original decoded 2-channel audio content by the crossfade module 1540. The crossfade module 1540 operates under the control of the crossfade control weight 1575 and finally outputs the post-processed 2-channel audio content 102 for routing to the speakers of the mobile device 1505.

Although not shown in the figure, the decoder 1500 may also include a module for calculating the virtualization device control weights for the virtualization module 1530 based on classification information 125 (eg, confidence values). good. In addition, the decoder 1500 may include a module for calculating upmix control weights for the upmix module 1520 based on classification information 125 (eg, confidence values).

FIG. 16 is from a bitstream in a special case of two-channel (eg, stereo) audio content for reproduction by an embodiment of the present disclosure, eg, by a 5 (or higher) speaker array of soundbar devices. A method 1600 for decoding audio content is shown in the form of a flowchart. Again, it is understood that the bitstream contains classification information or two-channel audio content, and the classification information indicates the content classification (eg, for content type) of the two-channel audio content. Method 1600 may be performed by a decoder in a device with 5 (or more) speaker arrays, such as a soundbar device. The decoder may have the same basic configuration as the decoder 115 in the encoder-decoder system 100 of FIG. 1 and may include, for example, specific implementations of weighting and post-processing.

At step S1610 , the AC-4 bitstream is received.
In step S1620 , the 2-channel audio content and classification information is decoded / multiplexed from the bitstream.
In step S1630 , an upmixer is applied to the 2-channel audio content to upmix the 2-channel audio content into the upmixed 5.1-channel audio content. The upmixer operates under the control of each control weight. Control weights for the upmixer are calculated based on classification information (eg confidence values). The control weights for the upmixer may be related, for example, to the upmix weights.
In step S1640 , a virtualization device is applied to the upmixed 5.1 channel audio content for 5.1 virtualization for a 5-channel speaker array. The virtualization machine operates under the control of each control weight. The control weights of the virtualizer are calculated based on the classification information (eg, confidence value). This can be done, for example, in the manner described above with reference to FIG.
Finally, in step S1650 , the output of the virtualizer is routed to a 5-channel speaker array.

FIG. 17 schematically illustrates an example of a decoder 1700 of a soundbar device 1705 capable of performing method 1600 according to an embodiment of the present disclosure. It is understood that the modules of the decoder 1700 described below may be implemented, for example, by the processor of the computing unit. The decoder 1700 receives a bitstream 110 (eg, an AC-4 bitstream), which is decoded / multiplexed by the AC-4 (soundbar) decoder module 1710. The AC-4 (soundbar) decoder module 1710 outputs decoded 2-channel audio content 1715 and decoded classification information 125. The decoded classification information 125 is provided to the upmix weight calculation module 1770, which calculates the upmix control weight 1775 based on the classification information 125 (eg, confidence value). The upmix control weight 1775 may be, for example, an upmix weight. The decoded 2-channel audio content 1715 is upmixed from 2.0 channels to 5.1 channels by the upmix module 1720, which outputs the upmixed 5.1 channel audio content. The upmix module 1720 operates under the control of the upmix control weight 1775. For example, different upmixes (with different upmix control weights) may be performed for music and utterances. The virtualization module (virtualizer) 1730 then applies 5.1 virtualization for the 5-channel speaker array to the upmixed 5.1-channel audio content 1725, and the virtualized upmix. Output 5.1-channel audio content. The virtualized upmixed 5.1-channel audio content is finally output as post-processed 5.1-channel audio content 102 for routing to the speakers of soundbar device 1705.

Although not shown in the figure, the decoder 1700 has a virtualization device control for the virtualization module 1730, for example, in the manner described above with reference to FIG. 13, based on classification information 125 (eg, confidence values). It may also include a module for calculating weights. In particular, Methods 1400 and 1600, as well as the corresponding decoders 1500 and 1700, are examples of endpoint-specific audio post-processing.

Various aspects of the invention can be understood from the following bulleted example embodiments (EEEs).
[EEE1]
How to encode audio content:
At the stage of performing content analysis of audio content;
The stage of generating classification information indicating the content type of the audio content based on the content analysis;
The stage of encoding the audio content and the classification information into a bitstream;
Including the step of outputting the bitstream.
Method.
[EEE2]
The method of EEE 1, wherein the content analysis is at least partially based on metadata about the audio content.
[EEE3]
How to encode audio content:
At the stage of receiving user input regarding the content type of the audio content;
The stage of generating classification information indicating the content type of the audio content based on the user input;
The stage of encoding the audio content and the classification information into a bitstream;
Including the step of outputting the bitstream.
Method.
[EEE4]
The user input is:
A label indicating that the audio content is of a given content type; and one or more trust values, each trust value is associated with a respective content type, and the audio content is said to be its respective. The method according to EEE3, which comprises one or more confidence values, which gives an indication of certainty of being content type.
[EEE5]
A method of encoding audio content, wherein the audio content is provided in a stream of audio content as part of an audio program.
At the stage of receiving the service type instruction indicating the service type of the audio content;
At least in part, at the stage of performing content analysis of the audio content based on the service-type instructions;
The stage of generating classification information indicating the content type of the audio content based on the content analysis;
The stage of encoding the audio content and the classification information into a bitstream;
Including the step of outputting the bitstream.
Method.
[EEE6]
Whether or not the service type of the audio content is a music service is determined based on the service type instruction;
Further comprising generating the classification information to indicate that the content type of the audio content is music content in response to the determination that the service type of the audio content is a music service.
The method according to EEE5.
[EEE7]
At the stage of determining whether the service type of the audio content is a newscast service based on the service type instruction;
In response to the determination that the service type of the audio content is a newscast service, the content analysis is adapted to have a higher probability of indicating that the audio content is spoken content. Including stages,
The method according to EEE 5 or 6.
[EEE8]
The method according to any one of EEE 5 to 7, wherein the service type instruction is provided for each frame.
[EEE9]
A method of encoding audio content, wherein the audio content is provided on a file basis, the file contains metadata about each audio content, and the method is:
At least in part with performing a content analysis of the audio content based on the metadata about the audio content;
The stage of generating classification information indicating the content type of the audio content based on the content analysis;
The stage of encoding the audio content and the classification information into a bitstream;
Including the step of outputting the bitstream.
Method.
[EEE10]
The metadata includes a file content type indication indicating the file content type of the file.
The content analysis is at least partially based on the file content type indication.
The method according to EEE9.
[EEE11]
Whether or not the file content type of the file is a music file is determined based on the file content type instruction;
Further comprising generating the classification information to indicate that the content type of the audio content is music content in response to the determination that the file content type of the file is a music file.
The method according to EEE10.
[EEE12]
Whether or not the file content type of the file is a newscast file is determined based on the file content type instruction;
In response to the determination that the file content type of the file is a newscast file, the content analysis is adapted to have a higher probability of indicating that the audio content is spoken content. Including further to let
The method according to EEE 10 or 11.
[EEE13]
Whether or not the file content type of the file is dynamic is determined based on the file content type instruction;
Further comprising adapting the content analysis to allow higher transition rates between different content types in response to the determination that the file content type of the file is dynamic content.
The method according to any one of EEE 10 to 12.
[EEE14]
The classification information comprises one or more confidence values, each confidence value is associated with a respective content type, and gives an indication of the certainty that the audio content is the respective content type.
The method according to any one of EEE 1 to 13.
[EEE15]
The method according to any one of EEE1 to 14, wherein the content type includes one or more of music content, audio content, or effect content.
[EEE16]
The method according to any one of EEE 1 to 15, further comprising encoding a scene transition instruction in the audio content into the bitstream.
[EEE17]
Further including smoothing of the classification information before encoding,
The method according to any one of EEE 1 to 16.
[EEE18]
Further comprising quantizing the classification information prior to encoding,
The method according to any one of EEE 1 to 17.
[EEE19]
Further comprising encoding the classification information into specific data fields in the packet of the bitstream.
The method according to any one of EEE 1 to 18.
[EEE20]
A method of decoding audio content from a bitstream containing audio content and classification information about the audio content, wherein the classification information indicates content classification of the audio content, and the method is:
At the stage of receiving the bitstream;
The stage of decoding the audio content and the classification information;
A step of selecting a post-processing mode for performing post-processing of decoded audio content based on the classification information.
Method.
[EEE21]
The method of EEE20, wherein the post-processing mode selection is further based on user input.
[EEE22]
A method of decoding audio content from a bitstream containing audio content and classification information about the audio content, wherein the classification information indicates content classification of the audio content, and the method is:
At the stage of receiving the bitstream;
The stage of decoding the audio content and the classification information;
A step of calculating one or more control weights for post-processing of decoded audio content based on the classification information.
Method.
[EEE23]
The classification information includes one or more confidence values, each confidence value is associated with each content type and provides an indicator of the certainty that the audio content is the respective content type. ;
The control weight is calculated based on the confidence value.
The method according to EEE22.
[EEE24]
23. The method of EEE 22 or 23, wherein the control weights are control weights for each module for post-processing of the decoded audio content.
[EEE25]
The control weights include one or more of a control weight for an equalizer, a control weight for a virtualizer, a control weight for a surround processor, and a control weight for a dialog improver. The method according to any one of EEE 22 to 24.
[EEE26]
The method according to any one of EEE 22 to 25, wherein the calculation of the control weight depends on the device type of the device that performs the decoding.
[EEE27]
The method according to any one of EEE 22 to 26, wherein the calculation of the control weight is further based on user input.
[EEE28]
The method according to any one of EEE 22 to 27, wherein the calculation of the control weight is further based on the number of channels of the audio content.
[EEE29]
The control weights include control weights for the virtualization device.
The control weight for the virtualization device disables the virtualization device if the classification information indicates that the content type of the audio content is or is likely to be music. Calculated to be,
The method according to any one of EEE 22 to 28.
[EEE30]
The control weights include control weights for the virtualization device.
The control weights for the virtualizer are calculated so that the coefficients of the virtualizer scale between pass-through and full virtualization.
The method according to any one of EEE 22 to 29.
[EEE31]
The control weights include control weights for the dialog improver.
The control weight for the dialog improver is a dialog by the dialog improver if the classification information indicates that the content type of the audio content is or is likely to be an utterance. Calculated to improve the improvement,
The method according to any one of EEE 22 to 30.
[EEE32]
The control weights include control weights for the dynamic equalizer.
The control weight for the dynamic equalizer is the dynamic equalization if the classification information indicates that the content type of the audio content is or is likely to be an utterance. Calculated to invalidate the vessel,
The method according to any one of EEE 22 to 31.
[EEE33]
The method according to any one of EEE 22 to 32, further comprising smoothing the control weights.
[EEE34]
The method of EEE33, wherein the smoothing of the control weights depends on the particular control weights to be smoothed.
[EEE35]
30. The method of EEE 33 or 34, wherein the smoothing of the control weights depends on the device type of the device performing the decoding.
[EEE36]
The method according to any one of EEE 33 to 35, further comprising applying a non-linear mapping function to the control weights to increase the continuity of the control weights.
[EEE37]
A method of decoding audio content from a bitstream containing two-channel audio content and classification information about the two-channel audio content, wherein the classification information is a content classification of the two-channel audio content. The method is:
At the stage of receiving the AC-4 bit stream;
The stage of decoding the two-channel audio content and the classification information;
The stage of upmixing the above-mentioned 2-channel audio content into upmixed 5.1-channel audio content;
With the stage of applying the virtualization device to the upmixed 5.1 channel audio content for 5.1 virtualization for a 2-channel speaker array;
The stage of applying crossfader to the 2-channel audio content and the virtualized upmixed 5.1-channel audio content;
Including the step of routing the output of the crossfader to the two-channel speaker array.
The method further comprises calculating the respective control weights for the virtualizer and the crossfader based on the classification information.
Method.
[EEE38]
A method of decoding audio content from a bitstream containing two-channel audio content and classification information about the two-channel audio content, wherein the classification information is a content classification of the two-channel audio content. The method is:
At the stage of receiving the bitstream;
The stage of decoding the two-channel audio content and the classification information;
At the stage of applying the upmixer to the 2-channel audio content so that the 2-channel audio content is upmixed into the upmixed 5.1-channel audio content;
With the stage of applying the virtualization device to the upmixed 5.1 channel audio content for 5.1 virtualization for a 5-channel speaker array;
Including the step of routing the output of the virtualization device to the 5-channel speaker array.
The method further comprises calculating the respective control weights for the upmixer and the virtualization device based on the classification information.
Method.
[EEE39]
An encoder for encoding audio content, the encoder having a processor, the processor being coupled to a memory storing instructions for the processor, the processor being EEE1-19. An encoder adapted to perform the method described in any one of the sections.
[EEE40]
A decoder for decoding audio content, the decoder having a processor, the processor being coupled to a memory storing instructions for the processor, the processor being EEE 20-38. A decoder adapted to perform the method described in any one of the sections.
[EEE41]
A computer program comprising an instruction, wherein the instruction causes a processor to execute the instruction according to any one of EEE1 to 38.
[EEE42]
A computer-readable storage medium that stores the computer program described in EEE41.

Claims (41)

How to encode audio content:
At the stage of performing content analysis of audio content;
At the stage of generating classification information indicating the content type of the audio content based on the content analysis, the classification information includes one or more trust values, and each trust value is associated with each content type. And giving an indication of the certainty that the audio content is the respective content type;
The stage of encoding the audio content and the classification information into a bitstream;
Including the step of outputting the bitstream.
Method. The method of claim 1, wherein the content analysis is at least partially based on metadata about the audio content. Further including the step of receiving user input regarding the content type of the audio content.
The step of generating information is based on the user input.
The method according to claim 1 or 2. The user input is:
The method of claim 3, comprising a label indicating that the audio content is of a given content type. The audio content is provided in a stream of audio content as part of an audio program, further:
At the stage of receiving the service type instruction indicating the service type of the audio content;
At least in part, including performing a content analysis of the audio content based on the service-type instruction.
The step of generating classification information indicating the content type of the audio content is based on the content analysis.
The method according to any one of claims 1 to 4. Whether or not the service type of the audio content is a music service is determined based on the service type instruction;
Further comprising generating the classification information to indicate that the content type of the audio content is music content in response to the determination that the service type of the audio content is a music service.
The method according to claim 5. At the stage of determining whether the service type of the audio content is a newscast service based on the service type instruction;
The said so as to have a certainty value higher than a predetermined threshold indicating that the audio content is spoken content in response to the determination that the service type of the audio content is a newscast service. Including the stage of adapting content analysis,
The method according to claim 5 or 6. The method according to any one of claims 5 to 7, wherein the service type instruction is provided for each frame. The method according to any one of claims 1 to 7, wherein the audio content is provided on a file basis, and the file contains metadata for each audio content, as long as it is dependent on claim 2. .. 9. The method of claim 9, wherein the metadata includes a file content type indication indicating the file content type of the file, and the content analysis is at least partially based on the file content type instruction. Whether or not the file content type of the file is a music file is determined based on the file content type instruction;
Further comprising generating the classification information to indicate that the content type of the audio content is music content in response to the determination that the file content type of the file is a music file.
The method according to claim 10. Whether or not the file content type of the file is a newscast file is determined based on the file content type instruction;
The said so as to have a certainty value higher than a predetermined threshold indicating that the audio content is spoken content in response to the determination that the file content type of the file is a newscast file. Further including adapting content analysis,
The method according to claim 10 or 11. Whether or not the file content type of the file is dynamic is determined based on the file content type instruction;
Further comprising adapting the content analysis to allow higher transition rates between different content types in response to the determination that the file content type of the file is dynamic content.
The method according to any one of claims 10 to 12. The method of any one of claims 1-13, wherein the content type is: music content, audio content, effect content, and one or more of the content selected from the group of crowd noise. The method of any one of claims 1-14, further comprising encoding a scene transition instruction in the audio content into the bitstream. Further including smoothing of the classification information before encoding,
The method according to any one of claims 1 to 15. Further comprising quantizing the classification information prior to encoding,
The method according to any one of claims 1 to 16. Further comprising encoding the classification information into specific data fields in the packet of the bitstream.
The method according to any one of claims 1 to 17. An encoder for encoding audio content, wherein the processor has a processor, the processor is coupled to a memory storing instructions for the processor, and the processor is claimed 1. Or an encoder adapted to perform the method according to any one of 18. A method of decoding audio content from a bitstream that includes audio content and classification information about the audio content, wherein the classification information indicates the content type of the audio content, and the classification information is one. It comprises one or more confidence values, each confidence value is associated with each content type and provides an indicator of the certainty that the audio content is the respective content type.
At the stage of receiving the bitstream;
The stage of decoding the audio content and the classification information;
A step of selecting a post-processing mode for performing post-processing of decoded audio content based on the classification information;
At the stage of calculating one or more control weights for the post-processing of the decoded audio content based on the classification information, the control weights are calculated based on the confidence value. Including stages,
Method. The bitstream contains channel-based audio content and the post-processing is:
The stage of upmixing the channel-based audio content into upmixed channel-based audio content;
To obtain the virtualized upmixed channel-based audio content for virtualization for a speaker array of the desired number of channels, to the upmixed channel-based audio content. Including the stage of applying the virtualizer,
The method according to claim 20. The method of claim 20 or 21, wherein the post-processing mode selection is further based on user input. At the stage of routing the output of the virtualization device to the speaker array;
Further comprising calculating the respective control weights for the upmixer and the virtualization device based on the classification information.
The method of claim 21 or 22. After applying the virtualization device, the method further:
The stage of applying a crossfader to the channel-based audio content and the virtualized upmixed audio content;
At the stage of routing the output of the crossfader to the speaker array;
Further comprising calculating the respective control weights for the upmixer and the crossfader based on the classification information.
The method of claim 21 or 22. The method according to any one of claims 20 to 24, wherein the control weight is a control weight for each module for post-processing of the decoded audio content. The control weights include one or more of a control weight for an equalizer, a control weight for a virtualizer, a control weight for a surround processor, and a control weight for a dialog improver. The method according to any one of claims 20 to 25. The method according to any one of claims 20 to 26, wherein the calculation of the control weight depends on the device type of the device that performs the decoding. The method according to any one of claims 20 to 27, wherein the calculation of the control weight is further based on user input. The method according to any one of claims 20 to 28, wherein the calculation of the control weight is further based on the number of channels of the audio content. The control weights include control weights for the virtualization device.
The control weight for the virtualization device disables the virtualization device if the classification information indicates that the content type of the audio content is or is likely to be music. Calculated to be,
The method according to any one of claims 20 to 29. The control weights include control weights for the virtualization device.
The control weights for the virtualizer are calculated so that the coefficients of the virtualizer scale between pass-through and full virtualization.
The method according to any one of claims 20 to 30. The control weights include control weights for the dialog improver.
The control weight for the dialog improver is a dialog by the dialog improver if the classification information indicates that the content type of the audio content is or is likely to be an utterance. Calculated to enhance improvement,
The method according to any one of claims 20 to 31. The control weights include control weights for the dynamic equalizer.
The control weight for the dynamic equalizer is the dynamic equalization if the classification information indicates that the content type of the audio content is or is likely to be an utterance. Calculated to invalidate the vessel,
The method according to any one of claims 20 to 32. The method according to any one of claims 20 to 33, further comprising smoothing the control weights. 34. The method of claim 34, wherein the smoothing of the control weights depends on the particular control weights to be smoothed. 34. The method of claim 34 or 35, wherein the smoothing of the control weights depends on the device type of the device performing the decoding. The method of any one of claims 33-36, further comprising applying a non-linear mapping function to the control weights to increase the continuity of the control weights. The bitstream is an AC-4 bitstream, and the method is:
The stage of decoding 2-channel audio content and the classification information;
The stage of upmixing the above-mentioned 2-channel audio content into upmixed 5.1-channel audio content;
With the stage of applying the virtualization device to the upmixed 5.1 channel audio content for 5.1 virtualization for a 2-channel speaker array;
The stage of applying crossfader to the 2-channel audio content and the virtualized upmixed 5.1-channel audio content;
Including the step of routing the output of the crossfader to the two-channel speaker array.
The method further comprises calculating the respective control weights for the virtualizer and the crossfader based on the classification information.
The method according to any one of claims 21 to 37. The bitstream contains classification information about the two-channel audio content and the two-channel audio content, the classification information indicating the content classification of the two-channel audio content, the method. :
The stage of decoding the two-channel audio content and the classification information;
At the stage of applying the upmixer to the 2-channel audio content so that the 2-channel audio content is upmixed into the upmixed 5.1-channel audio content;
With the stage of applying the virtualization device to the upmixed 5.1 channel audio content for 5.1 virtualization for a 5-channel speaker array;
Including the step of routing the output of the virtualization device to the 5-channel speaker array.
The method further comprises calculating the respective control weights for the upmixer and the virtualization device based on the classification information.
The method according to any one of claims 21 to 38. A decoder for decoding audio content, wherein the decoder has a processor, the processor is coupled to a memory storing instructions for the processor, and the processor is claimed 20. A decoder adapted to perform the method according to any one of 39 to 39. A computer program comprising an instruction, wherein the instruction causes a processor to execute the instruction according to any one of claims 1 to 39.

Images