WO2007133901A1 - Prioritization of audio streams for platform adaptive audio decoding - Google Patents
Prioritization of audio streams for platform adaptive audio decoding Download PDFInfo
- Publication number
- WO2007133901A1 WO2007133901A1 PCT/US2007/067234 US2007067234W WO2007133901A1 WO 2007133901 A1 WO2007133901 A1 WO 2007133901A1 US 2007067234 W US2007067234 W US 2007067234W WO 2007133901 A1 WO2007133901 A1 WO 2007133901A1
- Authority
- WO
- WIPO (PCT)
- Prior art keywords
- audio
- application
- audio streams
- streams
- prioritized
- Prior art date
Links
- 238000012913 prioritisation Methods 0.000 title abstract description 8
- 230000003044 adaptive effect Effects 0.000 title description 5
- 238000000034 method Methods 0.000 claims abstract description 49
- 238000012545 processing Methods 0.000 claims description 23
- 238000004891 communication Methods 0.000 claims description 4
- 230000008569 process Effects 0.000 claims description 3
- 238000004148 unit process Methods 0.000 claims 1
- 238000012805 post-processing Methods 0.000 description 22
- 238000010586 diagram Methods 0.000 description 13
- 230000009471 action Effects 0.000 description 5
- 230000000694 effects Effects 0.000 description 5
- 230000015572 biosynthetic process Effects 0.000 description 4
- 230000006837 decompression Effects 0.000 description 4
- 230000001413 cellular effect Effects 0.000 description 3
- 230000007717 exclusion Effects 0.000 description 3
- 238000004880 explosion Methods 0.000 description 3
- 230000006870 function Effects 0.000 description 3
- 230000005236 sound signal Effects 0.000 description 3
- 230000008901 benefit Effects 0.000 description 2
- 230000006835 compression Effects 0.000 description 2
- 238000007906 compression Methods 0.000 description 2
- 230000006378 damage Effects 0.000 description 2
- 230000002452 interceptive effect Effects 0.000 description 2
- 230000009191 jumping Effects 0.000 description 2
- 238000004519 manufacturing process Methods 0.000 description 2
- 238000003491 array Methods 0.000 description 1
- 230000008859 change Effects 0.000 description 1
- 230000007423 decrease Effects 0.000 description 1
- 230000004044 response Effects 0.000 description 1
- 230000001360 synchronised effect Effects 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F15/00—Digital computers in general; Data processing equipment in general
- G06F15/16—Combinations of two or more digital computers each having at least an arithmetic unit, a program unit and a register, e.g. for a simultaneous processing of several programs
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L65/00—Network arrangements, protocols or services for supporting real-time applications in data packet communication
- H04L65/1066—Session management
- H04L65/1101—Session protocols
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L65/00—Network arrangements, protocols or services for supporting real-time applications in data packet communication
- H04L65/60—Network streaming of media packets
- H04L65/75—Media network packet handling
- H04L65/764—Media network packet handling at the destination
Definitions
- This disclosure relates to audio coding techniques and, more particularly, audio decoding techniques for the audio streams of an application that can be executed on different platforms.
- Audio coding is used in many applications and environments such as satellite radio, digital radio, internet streaming (web radio), digital music players, wireless mobile devices, and a variety of mobile multimedia applications.
- audio coding standards such as standards according to the motion pictures expert group (MPEG), windows media audio (WMA), and standards by Dolby Laboratories, Inc.
- MPEG motion pictures expert group
- WMA windows media audio
- AAC advanced audio coding
- Audio coding standards generally seek to achieve low bitrate, high quality audio coding using compression.
- Audio coding is "loss-less,” meaning that the coding does not degrade the audio signal, while other audio coding may introduce some loss in order to achieve additional compression.
- audio coding is used along with video coding in order to provide multi-media content for applications such as video telephony (VT) or streaming video. Audio coding is also commonly used in mobile devices that support multi-media applications, such as video games, and the like. Indeed, mobile devices increasingly incorporate a wide variety of functionality and content that provides audio output to a user, and therefore makes use of audio coding techniques. Audio and video information may be segmented into frames, which comprise blocks of audio and video data. A stream of audio output can be defined by a sequence of audio frames, following decode of the frames.
- this disclosure describes audio coding techniques that exploit a prioritization scheme of audio streams defined for an application, such as a video game, that executes on a mobile device. Since mobile device platforms can vary in different devices, the amount of audio decoding that is supported by a given platform may be unknown to the content creators that design such applications to run on the mobile devices. In accordance with this disclosure, the content creators are given an ability to define the priority of different audio streams of the applications that they create. Then, upon execution in a mobile device, a subset of the prioritized audio streams can be combined according to priority to form a common audio stream, which defines the audio output for the application.
- applications can become more platform independent, and may be come more compatible with multiple mobile device platforms.
- Lower quality platforms may use fewer audio streams in the creation of the common audio stream, but generally use the highest priority streams to ensure that the application is still functional.
- Higher quality platforms in contrast, can use more of the audio streams in the creation of the common audio stream to provide an enhanced audio experience to the user.
- the same application executes on different platforms, but the audio output can differ, since the different platforms may have different amounts of memory and processing power.
- the content creators can define the audio prioritization of different audio streams of an application to ensure that the most important audio streams will be included upon execution of the application in any platform, while one or more less important audio streams may be excluded upon execution of the application in lower quality platforms.
- one or more additional audio streams unassociated with the application are prioritized as well.
- an audio alert (such as an alert notification of an incoming telephone call) can be prioritized in the scheme, typically by the device manufacture.
- the creation of the common audio stream may comprise combining a subset of the prioritized audio streams associated with the application and the additional prioritized audio stream according to priority.
- Each audio stream may have a unique priority, and typically the audio alert is given highest priority in the scheme, although this disclosure is not necessarily limited in this respect.
- this disclosure provides a method comprising receiving a plurality of prioritized audio streams of an application executing on a mobile device, receiving an additional prioritized audio stream not associated with the application, and combining a subset of the prioritized audio streams associated with the application and the additional prioritized audio streams according to priority to form a common audio stream.
- this disclosure provides a device comprising a processor that executes an application, and an audio decoding unit that receives a plurality of prioritized audio streams of an application executing on a mobile device, receives an additional prioritized audio stream not associated with the application, and combines a subset of the prioritized audio streams associated with the application and the additional prioritized audio streams according to priority to form a common audio stream.
- the techniques of this disclosure may be implemented in software that is stored on a computer readable medium.
- this disclosure provides a computer readable medium comprising executable instructions that upon execution in a mobile device cause the mobile device to receive a plurality of prioritized audio streams of an application executing on a mobile device, receive an additional prioritized audio stream not associated with the application, and combine a subset of the prioritized audio streams associated with the application and the additional prioritized audio stream according to priority to form a common audio stream.
- this disclosure provides a method that is executed by the content creators and device manufacturers.
- the method comprises defining priorities of audio streams of an application, defining priority of an audio alert not associated with the application, supplying the application to a first mobile device that defines a first platform, wherein the first mobile device uses a first subset of the audio streams of the application to define combined audio output of the application and the alert, and supplying the application to a second mobile device that defines a second platform, wherein the second mobile device uses a second subset of the audio streams of the application to define combined audio output of the application and the alert, wherein the first subset is different than the second subset.
- the priority of the audio streams of the application are typically defined by the content creators, while the priority of the additional audio stream not associated with the application may be assigned by the device manufacturer.
- FIG. 1 is a block diagram of an exemplary mobile device that can implement the techniques of this disclosure.
- FIG. 2 is a flow diagram illustrating a technique that may be performed by the device of FIG. 1 or other similar devices.
- FIG. 3 is a block diagram illustrating a technique in which multiple audio streams are processed and then combined based on priority of the audio streams.
- FIG. 4 is another block diagram of an exemplary device that can implement the techniques of this disclosure.
- FIG. 5 is a block diagram illustrating storage of various audio information in a shared memory location.
- FIG. 6 is a flow diagram illustrating a technique that may be performed by the device of FIG. 4 or other similar devices.
- FIG. 7 is a flow diagram illustrating a technique that may executed by a content creator of an application to define audio priority within the application, and then distribute the application to different platforms.
- This disclosure describes audio coding techniques that exploit a prioritization scheme for audio streams of a software application, which can be defined by a content creator of the application.
- a content creator of an application such as a video game that executes on a mobile device
- one or more additional audio streams unassociated with the application are prioritized as well, typically by the device manufacturer.
- an audio alert (such as an alert notification of an incoming telephone call) can be prioritized in the scheme.
- a common audio stream can be created by combining a subset of the prioritized audio streams associated with the application and the additional prioritized audio stream according to priority.
- Each audio stream may have a unique priority, and typically the audio alert is given highest priority in the scheme, although this disclosure is not necessarily limited in this respect.
- Lower quality platforms may use fewer audio streams in the creation of the common audio stream, but generally use the highest priority streams so that the application is still functional.
- Higher quality platforms can use more of the audio streams in the creation of the common audio stream to provide an enhanced audio experience to the user.
- the same application executes on different platforms, but the audio output may differ, since the different platforms may have different amounts of memory and processing power.
- the content creators can prioritize the different audio streams of an application to ensure that the most important audio streams will be included in the output upon execution of the application in any platform, while less important audio streams may be excluded from the output upon execution of the application in lower quality platforms.
- audio streams unassociated with the application can be prioritized in the scheme, typically by the device manufacturer.
- the audio of an application can be platform adaptive, while still supporting the ability to interject additional audio over that of the application, e.g., in the case of an incoming telephone call or another event that may require an alert to a user.
- FIG. 1 is a block diagram of an exemplary mobile device that can implement the techniques of this disclosure.
- Mobile device 10 may comprise any of a wide variety of mobile devices that can include audio decoding capabilities. Examples include digital music players such as iPods devices (commercially available from Apple Computer, Inc.) and similar digital music devices, mobile digital television devices, wireless communication devices, personal digital assistants (PDAs), laptop computers, handheld computers, digital cameras, digital recording devices, cellular or satellite radio telephones, direct two-way communication devices (sometimes referred to as "walkie- talkies"), hand-held video game devices such as Game Boy devices (commercially available from Nintendo of America, Inc.) and similar gaming devices, and the like.
- digital music players such as iPods devices (commercially available from Apple Computer, Inc.) and similar digital music devices
- mobile digital television devices wireless communication devices
- PDAs personal digital assistants
- laptop computers handheld computers
- digital cameras digital recording devices
- cellular or satellite radio telephones direct two-way communication devices
- hand-held video game devices such as Game Boy devices (commercially
- Exemplary mobile device 10 includes a processor 12 coupled to a memory 14, a user interface 13 (such as a keyboard, mouse, trackball, joystick, or the like), and a display 17.
- Device 10 may include other components, not illustrated in FIG. 1, such as an antenna and a modem to facilitate wireless communication of information.
- Processor 12 is coupled to an audio decoding unit 16, which may be implemented as a digital signal processor (DSP). Audio decoding unit 16 is also coupled to memory 14 and may access hardware 18 in order to accelerate aspects of the audio decoding process. In some cases, processor 12 may also be coupled to hardware 18.
- processor 12 is coupled to a drive circuit 15, which receives audio control signals from processor and generates drive signals in order to drive speakers 19A and 19B and thereby generate audible sounds.
- Hardware 18, for example may include a digital to analog converter (DAC), which may be coupled directly to drive circuit 15 to deliver analog signals that can be amplified to drive speakers 19A and 19B.
- DAC digital to analog converter
- Processor 12 may comprise a microprocessor, and may be a general purpose processor, or a specialized processor designed specifically for mobile devices. In any case, according to this disclosure, processor 12 generally executes software applications. A user may interact with the software applications via user interface 13 and multi-media output can be delivered to the user via display 17 and speakers 19A and 19B. Audio decoding unit 16 is invoked by processor 12 to handle intensive audio decoding tasks, such as the decoding/decompression of coded audio streams. [0026] In accordance with this disclosure, the application executing on processor 12 defines prioritized audio streams, the priority of which are defined by the creator of the application. For example, the application may comprise a video game that a user can execute on mobile device 10 for entertainment purposes.
- Such video game applications typically include several audio streams, such as one or more types of background music, and a number of interactive real-time sound effects (such as explosions or gunfire).
- the audio streams may correspond to different audio channels of the game.
- Some video games provide many sound effects associated the real-time action of the characters of the video game, e.g., sounds for jumping, running, sliding, driving, turning, changing direction, and the like.
- sounds may be associated with background characters as well, or the appearance or destruction of such characters. Indeed, the number of sound effect channels in a video game is generally unlimited and typically defined by the game designer (also referred to more generally as the content provider of the application). [0027]
- audio instructions may be delivered to the user, from the game, in response to action by the video game characters. Such instructions may be needed for the user to understand the game.
- Other sounds may be less important, such as gunfire or explosions, and may enhance the user experience, but may be unnecessary for the user to understand the game. Still other sounds may be even less important, such as background music or other sounds.
- audio streams are encoded separately in the application that executes on processor 12. Moreover, the encoded audio streams are prioritized by the content creator of the application. Upon execution of the application in processor 12, the audio streams are decoded by unit 12 according to the priority assigned by the content creator. In addition, one or more audio streams unassociated with the application (such as alerts) can be prioritized as well. A subset of the prioritized audio streams associated with the application and the additional prioritized audio stream can be combined according to priority to form a common audio stream.
- Audio decoding unit 16 generally decodes and combines the audio streams to provide a common audio stream, which can then be delivered to drive circuit 15 to generate drive signals for speakers 19A and 19B.
- the "higher priority" audio streams are used in the formation of the common audio stream, while one or more "lower priority” audio streams are excluded from the common audio stream due to limitations of device 10.
- mobile device 10 can execute the application, and can generate audio output in a manner that ensures that low priority audio streams are never given precedence over high priority audio steams. In other words, high priority audio streams are never excluded from the output at the expense of lower priority audio streams.
- the higher quality device may generate enhanced audio relative to the lower quality device. In this case, both devices would execute the same application. However, since one of the devices has a more capable computing platform, the higher quality device may use more of the "lower priority" audio streams in the creation of the audio output in order to enhance the user's experience.
- the same application may result in different levels of audio output on different devices, due to the different platforms of the different devices that execute the same application.
- Content providers can design one application, with prioritized audio streams, and that same application can result in different amounts of audio output, depending on the platform that executes the application.
- the audio streams are prioritized by the content provider, the most important audio (as defined by the content provider and device manufacturer) is always included in the output even when the application is executed on low quality platforms. Again, this means that the content provider may generate a single application that can execute at different quality levels, depending on the quality of the platform. Moreover, the content provider may define audio priority of a video game in a manner that ensures that even the lowest quality platform, defined to execute the application at a minimalist level, can still allow the user to effectively use an application, such as a video game. The same application, when executed on a higher quality platform, may generate enhanced audio relative to the same application executing on a lower quality platform.
- Processor 12 may execute the application that includes the prioritized audio, but may also execute other applications. These other applications may also generate audio streams that, in some cases, can supersede any audio in the application.
- the application with prioritized audio may comprise a video game
- one or more audio streams not associated with the application may comprise an audio alert not associated with the video game (such as the ringing of a cellular telephone in the case where mobile device 10 is a cellular telephone).
- the audio alert not associated with the video game may be given precedence or priority and may be included in the common audio stream, or may supersede any audio from the game.
- Processor 12 may be interrupt driven, in which case an incoming phone call could generate an interrupt that causes an audible alert to supersede any audio of the video game, or be combined with the highest priority audio of the video game and included in the audio output.
- the alert may be given a priority just like the different audio streams of the application, even though the alert is not associated with the application.
- Each audio stream may have a unique priority, and typically the audio alert is given highest priority in the scheme, although this disclosure is not necessarily limited in this respect.
- the audio stream not associated with the application may be defined by the device manufacture. In any case, each audio stream is given a unique priority.
- FIG. 2 is a flow diagram illustrating a technique that may be performed by the device of FIG. 1 or other similar devices.
- audio decoding unit 16 receives prioritized audio steams associated with an application executing on device 10, e.g., in processor 12 (21).
- audio decoding unit 16 receives an additional prioritized audio stream no associate with the application (22).
- Audio decoding unit 16 combines a subset of the audio streams of the application with the additional audio stream according to priority to form a common audio stream (23). Audio decoding unit 16 can then generate output signals (24), which may be used by drive circuit 15 to generate audible signals via speakers 19A and 19B.
- output signals 24
- the prioritization of the audio streams and the formation of the common audio stream from a subset of the prioritized audio steams and the additional audio stream allows the application to operate differently yet efficiently on different platforms.
- FIG. 3 is a block diagram illustrating a technique that may be implemented by audio decoding unit 16 (FIG. 1).
- audio decoding unit 16 FIG. 1
- multiple audio streams are processed and then combined based on priority of the audio streams.
- audio streams 31A-31E are processed to define processed audio streams 32A-32E (collectively processed audio streams 32).
- the processing of audio streams 31 may include decompression, in which case audio streams 32 may be decompressed versions of audio streams 31 respectively. Additional processing may also be performed on audio streams 31 , if desired.
- Control logic 35 combines a subset of processed audio streams 32 to generate a common audio stream 36.
- all of the processed audio streams 32A-32E shown in the illustration are combined by logic 35 to generate common audio stream 36.
- greater or fewer numbers of audio streams may be included in the common audio stream 36.
- one or more lower priority audio streams may be excluded from common audio stream 36.
- Logic 35 may control which ones of processed audio streams 32 are included in the common audio stream 36. Logic 35 makes such determinations based on priority and the computational resources available for audio signal generation.
- Logic 35 could be implemented in hardware, in which case logic 35 may maintain a metric of the computational resources that have been used and the available resources.
- processed audio stream 0 32A
- processed audio stream 1 32B
- stream 2 32C
- stream 3 32D
- stream 4 32E
- post processing may be performed to create processed common audio stream 38.
- the post processing may adjust the intensity (volume) of the signal, or may include audio enhancement of the common audio stream 36. According to this disclosure, however, post processing may be excluded in some cases, particularly if processing power is limited.
- FIG. 4 is another block diagram of an exemplary device that can implement the techniques of this disclosure.
- the device of FIG. 4 includes a processor 42 an audio decoding unit 46 and hardware 48. These respective components shown in FIG. 4 may correspond to elements 12, 16 and 18 of mobile device 10 (FIG. 1), or may be implemented in other devices that have architectures different from that of FIG. 1.
- Processor 42 executes an application that includes the prioritized audio streams, as described herein. Again, a content provider of the application selects the priority of the audio streams of the application.
- the application comprises a video game, although the techniques could work for a variety of other applications that include many audio streams.
- audio streams may include background music, interactive real-time sound effects (such as explosions or gunfire), and a wide variety of sound effects associated the action of game, e.g., sounds for jumping, running, sliding, driving, turning, changing direction and the like. Also, sounds may be associated with background characters, or the appearance or destruction of such characters.
- the different audio streams may correspond to different audio channels of the video game, and the number of channels is generally unlimited and defined by the game itself.
- audio tasks 41A-41E generally represent the tasks performed on the respective prioritized audio steams.
- audio tasks 41A-41E represent tasks performed on audio streams of an application executing on processor 42 and at least one addition audio stream (such as an alert) not associated with the application.
- Processor 42 forwards each of the audio streams to audio decoding unit 46 for processing (e.g., decoding) of the respective audio streams.
- Audio tasks 41A-41E may be event driven tasks in that audio tasks 41A-41E send request signals to processor 42 as more data is needed. Audio streams can be forwarded from processor 42 to the respective audio tasks 41A-41E in audio frames that comprise blocks of data. As more audio frames are needed by audio tasks 41A-41E, control signals are sent to processor 42, and processor 42 forwards addition data for processing by the respective audio task that requests additional data.
- Audio task 0 may be associated with a stream having a highest priority, followed by audio task 1 (41B), then audio task 2 (41C), then audio task 3 (41D), then audio task 4 (41E), and so forth.
- the audio tasks 41A-41E may comprise audio decoding or decompression tasks performed on the respective audio streams.
- audio tasks 41A-41E may receive compressed audio streams from a memory 44 (which may correspond to memory 14 of FIG. 1), and following decode can store decompressed audio streams back into memory 44. Audio tasks 41A-41E may be performed in parallel or overlapping manner such that several decoding tasks execute at the same time.
- decompressed audio streams are received by an audio task, however, then decompression is not needed, and the same information received is stored back into memory 44 as the decompressed audio stream. In this case, the data is still sent to the audio task since it may not be known whether such data is compressed or decompressed.
- post processing (PP) task 43 generally refers to the decoding task that combines a subset of prioritized and decompressed audio streams to form a combined audio stream. PP task 43 also performs any post processing (such as volume adjustments or audio enhancements) to the combined audio stream. PP task 43 may also be an event driven task that sends control signals to audio tasks 41A-4E as data is needed. The data can be forwarded or identified by an address location in memory. PP task 43 can perform its processing in a prioritized manner that ensures that any data associated with audio task 41 A is given highest priority, followed by data from audio task 4 IB, then data from audio task 41C, then data from audio task 4 ID, then data from audio task 4 IE, and so forth.
- the same memory 44 may be used to stored the compressed and decompressed data associated with audio tasks 41A-41E, and may also be used to store the combined and post processed audio stream associated with audio PP task 43.
- the use of a shared memory 44 can promote efficiency by allowing PP task 43 to directly access the decompressed audio streams of audio tasks 41A-41E without needing to reload such data individually prior to the post processing of the combined audio stream.
- the audio streams may be divided into audio frames that comprise sequential blocks of audio data.
- audio decoding unit 46 can maintain address records of memory 44 to facilitate the use of shared memory 44 and avoid the need for unnecessary reloads of data.
- audio PP task 43 accesses the decoded results of audio tasks 41A-41E to generate a combined audio stream, which can be processed further, i.e., post processed.
- the exemplary memory layout of FIG. 5 is shown for clarity. However, under other circumstances, it may be possible that memory areas for one or more task audio tasks may overlap in the memory.
- audio PP task 43 selects the decompressed audio streams of audio tasks 41A-41E based on priority. To this end, one or more of the lower priority audio streams may be excluded from the combined audio stream that is created by audio PP task 43 if platform constraints (such as limited memory 44 or limited processing power of decoding unit 46) exist.
- audio PP task 43 generates the combined audio stream from a subset of the decompressed audio streams, and makes this combination based on priority.
- Each respective audio stream is included only if sufficient memory and processing power are available after inclusion of the higher priority audio streams.
- the system may operate in an event driven manner, and the audio streams may be broken into audio frames. In this case, the memory and processing speed of the system can be exploited in an adaptive fashion. If the amount of audio data associated with a given "high priority" audio stream increases, this can bear on whether a given "low priority" audio stream is excluded from the common audio stream in a given instance.
- the formation of a common audio stream from the subset of prioritized audio streams can adapt over time to more fully utilize the system resources of the platform. Indeed, the number of prioritized audio streams that are included in the common audio stream may change as an application executes on processor 42 in real-time.
- the combined and post processed audio stream includes the decompressed audio streams associated with audio tasks 41A-41D, but excludes that audio stream associated with audio task 4 (41E).
- the decompressed audio stream associated with the lowest priority audio task i.e., audio task 4 (41E) is not accessed by audio PP task 43, and is therefore not included in the combined audio stream.
- the decoding tasks may be avoided for any audio stream that is likely to be excluded from the combined audio stream.
- compressed and decompressed sizes of audio streams may be known "a priori.” The priorities may be used in two ways. First, when an audio stream starts, the respective task may determine whether enough memory exists to handle the audio stream.
- audio front end (AFE) task 45 may use the combined and post processed audio stream to generate and output signal.
- Hardware 48 may include a digital to analog converter (DAC) 49 to generate an analog signal from samples of the combined audio stream.
- AFE 45 may access DAC 49 in order to generate analog output, which can then be delivered to a drive circuit, amplified, and used to drive speakers to provide audio output to a user.
- DAC digital to analog converter
- FIG. 6 is a flow diagram illustrating a technique that may be performed by the device of FIG. 4 or other similar devices.
- audio decoding unit 46 receives prioritized audio streams associated with an application executing on processor
- audio decoding unit 46 receives an additional prioritized audio stream not associated with the application (52).
- Audio tasks 41A-41E decode the prioritized audio streams (including that unassociated with the application) and store decoded audio streams in memory 44 (FIG. 5) (53).
- PP task 43 then combines a subset of the decoded audio streams of the application with the additional audio stream according to the assigned priority of the streams to form a common audio stream (54). Again, this typically involves the inclusion of one or more high priority streams and the exclusion of one or more low priority streams from the common audio stream.
- Audio decoding unit 46 may be implemented as a digital signal processor (DSP), in which case, audio tasks 41A-41E, PP task 43 and AFE task 45 may be software modules that execute on the DSP. More generally, however, the techniques described herein may be implemented in hardware, software, firmware, or any combination thereof.
- DSP digital signal processor
- this disclosure may be directed to a computer readable medium comprising instructions, that when executed in a device that decodes audio streams, performs one or more of the audio decoding techniques described herein.
- the computer readable medium may comprise random access memory (RAM) such as synchronous dynamic random access memory (SDRAM), read-only memory (ROM), non-volatile random access memory (NVRAM), electrically erasable programmable read-only memory (EEPROM), FLASH memory, and the like.
- RAM random access memory
- SDRAM synchronous dynamic random access memory
- ROM read-only memory
- NVRAM non-volatile random access memory
- EEPROM electrically erasable programmable read-only memory
- FLASH memory and the like.
- the instructions may be computer-readable instructions, and a DSP can execute instructions stored in memory in order to carry out one or more of the audio decoding techniques.
- the techniques may be executed by a DSP that invokes various hardware components to accelerate the coding process.
- the units or modules described herein may be implemented as a microprocessor, one or more application specific integrated circuits (ASICs), one or more field programmable gate arrays (FPGAs), or some other hardware-software combination.
- FIG. 7 is a flow diagram illustrating aspects of this disclosure from the perspective of a content creator.
- the technique of FIG. 7 may executed by a content creator of an application and a device manufacturer in order to define audio priority within the application and audio unassociated with the application respectively.
- the application can then be distributed to different platforms.
- the content creator (such as a video game designer), defines priority of audio streams of an application (such as a video game) to operate on mobile devices (61).
- the device manufacturer defines priority of an audio alert not associated with the application (62).
- the content creator supplies the application to a first mobile device and the first mobile uses a first subset of the audio streams of the application to define combined audio output for the application and the alert (63).
- the content creator then supplies the application to a second mobile device, which has a different platform than the first mobile device (64).
- the second mobile device uses a second subset of the audio streams of the application to define its combined audio output for the application and the alert.
- the second subset may be different than the first subset due to the fact that the first and second devices have different platforms.
- the first subset may be smaller than the second subset due to the fact that memory associated with the first mobile device is smaller than memory associated with the second mobile device.
- the content creator is able to distribute identical applications to both devices, and let the respective devices determine the extent of audio stream inclusion (and the extent of audio steam exclusion) in the audio output signal of the application as the application executes on the respective devices.
- the content creator Since the content creator has control over the priority scheme of the audio streams included in an application, the content creator can ensure that any sounds needed by the user to effectively play the game (or otherwise navigate a given application) are given highest priority.
- higher priority audio streams can include audio streams that are needed for a user to effectively play the video game
- lower priority audio streams may include audio streams that are not needed for a user to effectively play the video game.
- a low quality platform may still have sufficient memory and processor speed to ensure the inclusion of the higher priority audio streams needed for the game to be functional.
- a high quality platform may have additional memory and processor speed (relative to the low quality platform) that allows both the inclusion of the higher priority audio streams needed for the game to be functional and the inclusion of the lower priority audio streams, which enhance the audio output of the game.
- the techniques described herein allow handing of audio streams not associated with the application.
- the device may combine the audio stream not associated with the application (e.g., the alert) with the subset of the prioritized audio streams of the application in order to form the common audio stream.
- the device may include the audio alert not associated with the video game as part of the audio output of the game by combining not only the prioritized audio of the video game, but also the alert not associated with the video game.
- the alert like the audio streams of the application, is defined by a priority, which may be unique relative to all of the audio streams of the application.
- An incoming telephone call may be assigned a highest possible priority, and may supersede the action of the video game, and e.g., an incoming telephone call may cause an alert to supersede any audio of the video game in order to notify the user of the incoming call.
- the priority of audio streams (and the alert) can be defined by adjustable variables in the application. In this case, a content creator can easily adjust or re-design the priorities of different audio streams defined in the application. Each audio stream may be assigned a priority via a variable, which can be adjusted by the content creator. Such flexibility may be desirable, as it can allow the designer or device manufacturer to re-prioritize audio very easily in order to better accommodate a specific platform.
- the audio stream not associated with the application could be other things, rather than an alert.
- the "shutter sound" associated with image capture could be given a higher priority than any of the game content.
- the common audio stream would include the shutter sound, which would be combined with some or all of the audio or the game.
- Other types of audio streams unassociated with the application could also be prioritized, according to this disclosure, relative to those of the application.
- a content provider does not assign priorities to the content of an application
- the application might still function on the device by defining priority based on timing of the streams.
- the priority of the audio streams might simply be assigned on a f ⁇ rst-in basis.
- the first-in audio could be given highest priority or lowest priority. While these alternatives might not result in the best possible audio output, the game might still be somewhat functional in the event that audio steam prioritization was not defined by the game designer.
- Another way of dealing with audio streams with unassigned priority may include dynamically assigning the priority based on the content type. In these ways, an application designed to have assigned priorities of audio sounds, may still function even if such priorities are left unassigned.
- the techniques described herein may also apply during an arbitration.
- the selection or exclusion of audio streams may be avoided unless and until limitations in processing power or memory occur.
- the techniques of this disclosure may be applied as an arbitration to include high priority audio and exclude low priority audio to the extent necessary, given the processing or memory constraints.
Landscapes
- Engineering & Computer Science (AREA)
- Multimedia (AREA)
- Computer Networks & Wireless Communication (AREA)
- Signal Processing (AREA)
- Computer Hardware Design (AREA)
- Theoretical Computer Science (AREA)
- General Business, Economics & Management (AREA)
- Business, Economics & Management (AREA)
- Software Systems (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Telephone Function (AREA)
Abstract
Description
Claims
Priority Applications (2)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
JP2009509945A JP2009536752A (en) | 2006-05-10 | 2007-04-23 | Prioritizing audio streams for audio decoding adapted to the audio platform |
EP07761137A EP2016740A1 (en) | 2006-05-10 | 2007-04-23 | Prioritization of audio streams for platform adaptive audio decoding |
Applications Claiming Priority (2)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
US11/431,322 | 2006-05-10 | ||
US11/431,322 US8498723B2 (en) | 2006-05-10 | 2006-05-10 | Prioritization of audio streams for platform adaptive audio decoding |
Publications (1)
Publication Number | Publication Date |
---|---|
WO2007133901A1 true WO2007133901A1 (en) | 2007-11-22 |
Family
ID=38455098
Family Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
PCT/US2007/067234 WO2007133901A1 (en) | 2006-05-10 | 2007-04-23 | Prioritization of audio streams for platform adaptive audio decoding |
Country Status (6)
Country | Link |
---|---|
US (1) | US8498723B2 (en) |
EP (1) | EP2016740A1 (en) |
JP (1) | JP2009536752A (en) |
KR (1) | KR20090008474A (en) |
CN (1) | CN101438562A (en) |
WO (1) | WO2007133901A1 (en) |
Families Citing this family (14)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US9185445B2 (en) | 2009-09-24 | 2015-11-10 | At&T Intellectual Property I, L.P. | Transmitting a prioritized audio stream along with multimedia content |
US9264835B2 (en) * | 2011-03-21 | 2016-02-16 | Microsoft Technology Licensing, Llc | Exposing off-host audio processing capabilities |
US20130067050A1 (en) * | 2011-09-11 | 2013-03-14 | Microsoft Corporation | Playback manager |
US8930601B2 (en) * | 2012-02-27 | 2015-01-06 | Arm Limited | Transaction routing device and method for routing transactions in an integrated circuit |
TWI517691B (en) * | 2012-07-24 | 2016-01-11 | 美商微晶片科技公司 | A system to deliver prioritized game audio wirelessly with a minimal latency |
US9523585B2 (en) | 2013-03-15 | 2016-12-20 | Google Inc. | Systems and methods for handling application notifications |
JP6492736B2 (en) | 2015-02-17 | 2019-04-03 | 東京エレクトロン株式会社 | Substrate processing apparatus, substrate processing method, and storage medium |
US10776072B2 (en) * | 2016-03-29 | 2020-09-15 | Intel Corporation | Technologies for framework-level audio device virtualization |
US10325610B2 (en) | 2016-03-30 | 2019-06-18 | Microsoft Technology Licensing, Llc | Adaptive audio rendering |
US10056086B2 (en) | 2016-12-16 | 2018-08-21 | Microsoft Technology Licensing, Llc | Spatial audio resource management utilizing minimum resource working sets |
US10117083B1 (en) | 2017-04-28 | 2018-10-30 | Motorola Solutions, Inc. | Method and apparatus for audio prioritization |
US10885921B2 (en) * | 2017-07-07 | 2021-01-05 | Qualcomm Incorporated | Multi-stream audio coding |
KR20220004825A (en) | 2019-06-03 | 2022-01-11 | 인텔렉추얼디스커버리 주식회사 | Method, apparatus, computer program and recording medium for controlling audio data in a wireless communication system |
US11601757B2 (en) | 2020-08-28 | 2023-03-07 | Micron Technology, Inc. | Audio input prioritization |
Citations (2)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
WO2002005466A2 (en) * | 2000-07-07 | 2002-01-17 | Qualcomm Incorporated | Method and apparatus for proportionately multiplexing data streams onto one data stream |
FR2859862A1 (en) * | 2003-09-11 | 2005-03-18 | France Telecom | Service quality differentiating method for mobile communication network, involves determining total priority level defined by combination of service quality parameters related to service type and priority level of subscriber |
Family Cites Families (14)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
JPS6437600A (en) | 1987-08-04 | 1989-02-08 | Mitsubishi Electric Corp | Voice file apparatus |
US5285792A (en) | 1992-01-10 | 1994-02-15 | Physio-Control Corporation | System for producing prioritized alarm messages in a medical instrument |
US5703794A (en) | 1995-06-20 | 1997-12-30 | Microsoft Corporation | Method and system for mixing audio streams in a computing system |
JPH10307653A (en) * | 1997-05-09 | 1998-11-17 | Toshiba Corp | Computer system and sound controller, and power-down control method applied to the same |
JP3444806B2 (en) | 1999-02-10 | 2003-09-08 | 株式会社スクウェア・エニックス | GAME DEVICE, MUSIC REPRODUCTION METHOD, AND RECORDING MEDIUM |
JP2001333153A (en) | 2000-05-23 | 2001-11-30 | Yamaha Corp | Method of processing incoming call in mobile phone with game function |
JP3615143B2 (en) | 2000-11-16 | 2005-01-26 | 株式会社フェイス | Pronunciation control device for game machines |
JP3992513B2 (en) | 2002-02-28 | 2007-10-17 | ソフトバンクモバイル株式会社 | Information communication terminal capable of executing application program and control method thereof |
JP3667714B2 (en) | 2002-04-23 | 2005-07-06 | コナミ株式会社 | GAME DEVICE AND PROGRAM |
JP2003319020A (en) | 2002-04-24 | 2003-11-07 | Kyocera Corp | Portable terminal |
US6935959B2 (en) * | 2002-05-16 | 2005-08-30 | Microsoft Corporation | Use of multiple player real-time voice communications on a gaming device |
JP3873013B2 (en) | 2002-08-01 | 2007-01-24 | ソフトバンクモバイル株式会社 | Information communication terminal and control method thereof |
KR20040046483A (en) | 2002-11-27 | 2004-06-05 | 삼성전자주식회사 | Computer system and control method thereof |
US20060023900A1 (en) * | 2004-07-28 | 2006-02-02 | Erhart George W | Method and apparatus for priority based audio mixing |
-
2006
- 2006-05-10 US US11/431,322 patent/US8498723B2/en not_active Expired - Fee Related
-
2007
- 2007-04-23 EP EP07761137A patent/EP2016740A1/en not_active Withdrawn
- 2007-04-23 JP JP2009509945A patent/JP2009536752A/en active Pending
- 2007-04-23 CN CNA2007800164688A patent/CN101438562A/en active Pending
- 2007-04-23 WO PCT/US2007/067234 patent/WO2007133901A1/en active Application Filing
- 2007-04-23 KR KR1020087030140A patent/KR20090008474A/en active IP Right Grant
Patent Citations (2)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
WO2002005466A2 (en) * | 2000-07-07 | 2002-01-17 | Qualcomm Incorporated | Method and apparatus for proportionately multiplexing data streams onto one data stream |
FR2859862A1 (en) * | 2003-09-11 | 2005-03-18 | France Telecom | Service quality differentiating method for mobile communication network, involves determining total priority level defined by combination of service quality parameters related to service type and priority level of subscriber |
Also Published As
Publication number | Publication date |
---|---|
CN101438562A (en) | 2009-05-20 |
US8498723B2 (en) | 2013-07-30 |
JP2009536752A (en) | 2009-10-15 |
EP2016740A1 (en) | 2009-01-21 |
US20070266168A1 (en) | 2007-11-15 |
KR20090008474A (en) | 2009-01-21 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
US8498723B2 (en) | Prioritization of audio streams for platform adaptive audio decoding | |
US7715481B2 (en) | System and method for allocation of resources for processing video | |
CN109076303B (en) | Spatial audio resource management and mixing for applications | |
US20100063825A1 (en) | Systems and Methods for Memory Management and Crossfading in an Electronic Device | |
US20120183040A1 (en) | Dynamic Video Switching | |
US10432543B2 (en) | Dual jitter buffers | |
US10416954B2 (en) | Streaming of augmented/virtual reality spatial audio/video | |
US10535355B2 (en) | Frame coding for spatial audio data | |
US9819429B2 (en) | Efficient load sharing and accelerating of audio post-processing | |
US20150100324A1 (en) | Audio encoder performance for miracast | |
US20140324199A1 (en) | Audio pipeline for audio distribution on system on a chip platforms | |
US8064608B2 (en) | Audio decoding techniques for mid-side stereo | |
CN105657540B (en) | A kind of video encoding/decoding method and its device being adapted to Android system | |
WO2018200176A1 (en) | Progressive streaming of spatial audio | |
US20050262256A1 (en) | Method and device for multimedia processing | |
US20230118415A1 (en) | Systems and methods of using a media bit rate curve for streaming bit rate adaption | |
US20090154570A1 (en) | Method and system to stream and render video data on processing units of mobile devices that have limited threading capabilities | |
Chakraborty et al. | A perception-aware low-power software audio decoder for portable devices | |
JP2008532405A (en) | Adaptive fallback system for MPEG decoder | |
CN116389851A (en) | Processing method, processing device, processing equipment and storage medium | |
CN117376509A (en) | Video decoding method, device, equipment and storage medium | |
KR100715202B1 (en) | Apparatus and method for using storage device in portable terminal | |
JP2009020565A (en) | Information processing device | |
Wang et al. | Implementation and optimization of multimedia framework on heterogeneous multi-core platform | |
KR20160084214A (en) | System for cloud streaming service, method of image cloud streaming service based on selective streaming pipeline method and apparatus for the same |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
121 | Ep: the epo has been informed by wipo that ep was designated in this application |
Ref document number: 07761137 Country of ref document: EP Kind code of ref document: A1 |
|
WWE | Wipo information: entry into national phase |
Ref document number: 2248/MUMNP/2008 Country of ref document: IN |
|
WWE | Wipo information: entry into national phase |
Ref document number: 2007761137 Country of ref document: EP |
|
WWE | Wipo information: entry into national phase |
Ref document number: 200780016468.8 Country of ref document: CN |
|
WWE | Wipo information: entry into national phase |
Ref document number: 2009509945 Country of ref document: JP |
|
NENP | Non-entry into the national phase |
Ref country code: DE |
|
WWE | Wipo information: entry into national phase |
Ref document number: 1020087030140 Country of ref document: KR |