JP2012503426A - Upstream signal processing for client devices in small cell radio networks - Google Patents

Abstract

Methods, apparatuses and program logic in storage media to process media data for quality enhancement. The media data is for rendering at a wireless device. Environmental quantities are accepted from one or more sensors located remote from the wireless device but sufficiently close to be indicative of similar quantities in the device's environment. The sensors at in or connected to a network node. At the network node, the method includes processing the media data using the environmental quantities to generate processed data, and wirelessly sending the processed output to the resource constrained device for rendering, such that the processed output is usable by the resource constrained device to render or to process and render the media data.

Description

<Cross-reference to related applications>
This application claims priority to US Provisional Application No. 61 / 098,566, filed Sep. 19, 2008, which is hereby incorporated by reference in its entirety.

<Field of Invention>
The present disclosure relates to media data signal processing, such as streaming audio data signal processing and video data signal processing.

  As wireless handheld devices become more popular, their functionality continues to grow. Such devices that are gaining popularity include media players such as Apple iPods, mobile phones, Helio devices, wireless IP-based phones such as Linksys, Microsoft Zune, Apple iPhone, etc. Examples include media devices, handheld game machines such as Sony PSP, Nokia N Gage, and many other devices that can be used to transmit, receive and / or render communications and / or multimedia data. Such a device typically includes a processing system, such as a digital signal processor (DSP) or microcontroller, and memory containing software instructions. While such portable devices continue to improve processing power and functionality and / or extend battery life, these devices are non-portable such as computers, network adapters and routers, and / or fixed core network devices. Compared to processing systems, signal processing capabilities and / or other resources are still limited. Typically, portable devices are preferably as small as possible and as cheap as possible, but have limited processing power, limited memory resources, and / or limited battery power.

  Small cell wireless networks such as wireless personal area networks (WPANs) and picocell mobile phone networks are well known. In general, it has a base station that communicates with one or more client devices in a cell, called a base station in the case of a cellular telephone network and an access point in the case of a wireless network. They generally also have relatively small cells on the order of a few meters or less. It is important that such systems have small client devices that have a relatively long battery life and are relatively inexpensive.

  In stark contrast to client devices, fixed processing systems, such as server computer systems, network adapters, network routers, wireless base stations / access points and / or some fixed core network devices with which wireless client devices communicate, are better than client devices. It has a significantly large signal processing capability and can use power relatively without restriction. Thus, fixed systems are typically characterized by relatively fast processing capabilities, much larger memory capabilities than portable devices, and virtually unlimited power usage.

  In this specification, a device such as a portable device that has limited resources compared to a fixed processing system is referred to as a “restricted resource device”. As used herein, a base station refers to an access point, cellular base station, or similar wireless transceiver that wirelessly transmits media data to a resource restriction device.

It is a simplified block diagram of an example of a radio | wireless mechanism provided with the radio | wireless resource restriction | limiting device which concerns on embodiment of this invention. FIG. 6 is a simplified block diagram of another example of a radio mechanism comprising a radio resource limiting device according to an embodiment of the present invention. 1B is a simplified block diagram of an apparatus embodiment that includes upstream processing of media data and that describes, for example, the embodiment shown in FIGS. 1A and 1B. FIG. FIG. 6 is a simplified diagram of several rows of seats in a passenger aircraft having one passage according to an application example of an embodiment of the present invention. FIG. 6 is a diagram illustrating an example picocell in which each seat is an example of a picocell of six seats including a voice input sensor, and an embodiment of the present invention can operate using the picocell. FIG. 4 shows a simplified flowchart of a method embodiment of the present invention. 1 is an embodiment of a simplified block diagram of the present invention for streaming video data wirelessly via a signal processing device to a portable device with a high dynamic range (HDR) video display. FIG. 1 is a simplified block diagram of an example of an apparatus for performing upstream noise compensation processing of audio content using metadata and environmental noise information generated at a location remote from a client device according to an embodiment of the present invention. FIG. 4 illustrates one embodiment of quality-enhanced signal processing that uses a feedforward mechanism to modify an audio signal to generate a modified audio signal such that a partial specific loudness (specific loudness in the presence of interference noise) approaches a target specific loudness. It is a functional block diagram. FIG. 4 illustrates one embodiment of quality-enhanced signal processing that uses a feedforward mechanism to modify an audio signal to generate a modified audio signal such that a partial specific loudness (specific loudness in the presence of interference noise) approaches a target specific loudness. It is a functional block diagram. FIG. 4 illustrates one embodiment of quality-enhanced signal processing that uses a feedforward mechanism to modify an audio signal to generate a modified audio signal such that a partial specific loudness (specific loudness in the presence of interference noise) approaches a target specific loudness. It is a functional block diagram. A function that represents an embodiment of quality-enhanced signal processing that uses a feedforward mechanism to generate a modified audio signal by changing the audio signal to bring the partial specific loudness (specific loudness in the presence of interference noise) closer to the target specific loudness It is a block diagram. It is a figure which shows the embodiment of the feedforward mechanism for noise compensation which isolate | separates input audio into a frequency band group in a pre-processing.

<Outline>
Embodiments of the invention include a method, apparatus, and program logic encoded in one or more computer-readable tangible media to perform the method. The method of the present invention is for performing quality enhancement signal processing on media data using one or more environmental quantities. In the present invention, the environmental quantity is far from the resource-constrained device, but it is provided in a place sufficiently close to the resource-constrained device, and the environmental quantity by the sensor in the vicinity of the resource-constrained device becomes an index equivalent to each environmental quantity. collect. The signal processing of the present invention is for generating a processed output that is used in a resource constrained device to render media data.

  Particular embodiments of the present invention include a method for processing media data using processing hardware for quality improvement. Media data is rendered by a resource constrained wireless device. The method of the present invention includes receiving one or more sensing environment quantities determined from one or more sensors. The sensor is far from the resource constraining device, but the environmental quantity is close enough to the resource constraining device to be the same indicator of the respective environmental quantity as the sensor in the vicinity of the resource constraining device. The method of the present invention further comprises the step of processing media data and generating processed data using environmental quantities at a network node located at or coupled to the resource constrained device, comprising: A sensor is in or connected to the network node; and wirelessly sending the processed output to a resource constrained device for rendering, whereby the processed output renders media data To be used by a resource constrained device for processing or rendering.

  In an embodiment of the method of the present invention, the network node includes a base station of a wireless network.

  In an embodiment of the inventive method, the processed output includes processed media data for rendering by a resource constrained device.

  Further, in the method embodiment of the present invention, some media data processing is performed at the resource constrained device and the processed output is an auxiliary to be used by the resource constrained device for media data processing at the resource constrained device. Contains data.

  In an embodiment of the method of the invention, referred to as a “streaming embodiment”, the media data includes both 1) media data streamed to a resource constrained device and / or 2) via a wireless network, the resource constrained device. One or more of the media data that is interactively streamed to the resource constrained device as part of the directed communication is included.

  In the streaming embodiment, the media data includes audio data, the one or more environmental quantities include at least one quantity indicative of an acoustic profile of noise in the environment, and the quality enhancement process includes noise compensation. . In a variation, noise compensation includes generating change parameters from the audio data using one or more loudness level parameters and one or more parameters of the acoustic noise profile. Change parameters are generated by performing operations on information in the perceptual loudness region. Noise compensation further includes changing the audio data based on the change parameter to generate processed audio data. The one or more loudness level parameters include one or more of audio noise compensation on / off, a reference level for a resource constrained device, a desired reproduction level, and / or an amount of noise compensation.

  In an embodiment of the noise compensation method, the media data quality enhancement process includes one or more of automatic gain control, dynamic range compression and / or equalization applied to the audio data.

  In the streaming embodiment, the media data includes video data, the one or more environmental quantities include at least one parameter that indicates lighting in the environment, and the quality enhancement process includes one of the parameters. This includes changing the contrast and / or brightness of the video data according to the above.

  In the streaming embodiment, the media data includes video data, the one or more environmental quantities include at least one parameter indicative of lighting in the environment, and the resource constrained device includes a resource along with the video data. A flat panel display device is included having position dependent backlight elements that are each adjusted according to image dependent adjustment data sent to the constraining device. The quality improvement process includes changing the contrast and / or brightness of the video data. Data processing at the network node includes generating image dependent adjustment data according to at least one of the one or more parameters.

  Particular embodiments include program logic that, when executed by at least one processor of the processing system, causes any one of the method embodiments described in this overview section to be executed. Such program logic is provided, for example, in a computer-readable storage medium.

  Certain embodiments include a computer-readable medium containing program logic that, when executed by at least one processor of a processing system, causes any one of the method embodiments described in this overview section to be executed. It is.

  Particular embodiments include an apparatus that performs at least a portion of a media data quality enhancement process. The apparatus comprises a network node configured to wirelessly connect to a resource constrained device, as well as one or more sensors connected to or within the network node. Although this sensor is far from the resource constraining device, it is in a location sufficiently close to the resource constraining device, and the environmental amount detected by the sensor in the vicinity of the resource constraining device is an index of each environmental amount of 1 or more. The apparatus is connected to or within a network node, receives one or more environmental quantities, and performs data processing of media data using at least a portion of the received environmental quantities to achieve quality improvement And processing hardware configured to generate processed output. Further, the network node is configured to wirelessly send the processed output to the resource constrained device, so that the processed output can be used by the resource constrained device to render or process and render media data. . The apparatus of the present invention is configured to perform any one of the method embodiments described in this overview section.

  Particular embodiments include an apparatus that performs at least a portion of a media data quality enhancement process. The apparatus includes a processing system including at least one processor and a storage device. The storage device is provided with program logic, and when the program logic is executed, the apparatus executes any one of the method embodiments.

  Particular embodiments may provide all of these aspects, features or advantages, some may provide, or none at all. Certain embodiments may provide one or more other aspects, features or advantages, one or more of which will be readily apparent to those skilled in the art from the figures, descriptions, and claims herein. Let's go.

<Example Embodiment>
<Typical Architecture of Embodiment>
1A and 2B show simplified block diagrams of two different examples of a radio mechanism including a radio resource restriction device according to an embodiment of the present invention.

  FIG. 1C is a simplified block diagram illustrating an embodiment of the present invention. The particular embodiment shown in each of FIGS. 1A and 1B can be described by the mechanism of FIG. 1C, respectively.

  FIG. 1A illustrates a first example mechanism in which media data is streamed to a client device for rendering at the client device. A network 101 coupled to the base station 103 is also included. Note that in this description, the term “base station” is used synonymously and interchangeably. “Base station” is a term commonly used in describing cellular communication networks, while “access point” is a term commonly used in describing infrastructure-type wireless local area networks. Base station 103 comprises a radio transceiver 107 coupled to an antenna subsystem 105 that includes at least one antenna, and a processing subsystem that includes a processor 111 and a storage subsystem 113. The storage subsystem 113 includes one or more memory elements, and possibly one or more other storage elements such as a magnetic disk. In the present application, the subsystem 113 is collectively referred to as a storage device. In some embodiments, the storage device comprises program logic, eg, program logic such as instructions that, when executed, cause a base station to perform the steps of the method according to embodiments of the present invention.

  The base station is configured to communicate wirelessly with the resource constrained client device 121. The base station is designed to stream media data, such as audio data, or audio and video data, for rendering at the (resource constrained) client device 121. The resource constrained client device renders video with antenna subsystem 123 for receiving wireless transmissions, such as from base station 103, and loudspeakers / earphones for rendering audio with other components, and / or other components. One or more transducers, such as a display screen.

  In one embodiment of the invention, media data such as audio data or audio data and video data is streamed.

  In FIG. 1A, the source of media data is illustrated in the form of one or more media servers (one media server example 131 is shown) coupled to the network 101. As is common to such servers, the server 101 includes a processing system including a processor 133, memory, and optionally one or more other storage elements such as optical and / or magnetic media systems. Subsystem 135 is included. Computer readable storage subsystem 135 includes instructions 137 that, when executed by processor (s) 133, cause the server to supply media data over network 101.

  In a particular example, the client device 121 is a personal audio playback device, and the output transducer includes an in-plane headset for listening to audio programs streamed over the network 101. As another specific example, client device 121 is a personal video playback device, and the output transducer includes a display screen in the plane for viewing video data streamed over network 101. In either case, the media data is streamed for rendering at the resource constrained client device 121, and the resource constrained client device 121 renders the media data for listening and / or viewing via one or more output transducers 125. .

  One aspect of the present invention includes changing media data for rendering by the client device 121 by performing signal processing that uses one or more sensing environment quantities indicative of the environment of the client device 121. One or more sensors 109 are configured to detect one or more environmental quantities. Examples of environmental quantities include noise characteristics for audio and background illumination for video. However, it is not limited to them. In one aspect of the present invention, the one or more sensors are located away from the client device 121, but the detected environmental quantity is an index equivalent to one or more environmental quantities in the vicinity of the resource constrained device. Is sufficiently close to the resource constrained device. In the embodiment of FIG. 1A, sensor 109 is at base station 103, i.e., is directly coupled to base station 103. This suppresses the use of power in the resource limit device and also reduces the amount of signal processing in the resource limit client device 121.

  One aspect of the present invention is a method for modifying media data for rendering by a resource constraining device 121 that is wirelessly coupled to a wireless network including a base station 103. A processing system, for example, a processing system including processor 111 receives one or more detected environmental quantities from sensor 109. The processing system processes the media data using the received sensing environment quantity and is used in the mobile device for processing processed output, eg, processed media data, and / or media data in the mobile device. Configured to generate auxiliary data. In one embodiment, the data processing by the processing system includes generating a change parameter using the received detected environment quantity and processing the media data based on the change parameter to generate processed media data. . The processed media data and / or auxiliary data is sent wirelessly to the resource constraint device 121 for rendering via the output transducer or for processing and rendering. Therefore, a signal processing method that uses an environment quantity in the environment of the rendering device 121 but requires a resource that is restricted in the rendering device 121 has no resource restriction as in the device 121, and is a processor separated from the device 121. Executed by.

  Several such signal processing methods that can advantageously use the mechanism shown are described in more detail later herein.

  In some embodiments, the source of media data is sent from upstream to the client device 121, but in other embodiments of the invention the source of media data is not on a separate device upstream of the network, but on the client Located in device 121. In such an embodiment, the media data is sent upstream to a processing system that includes a processor 111. The processing system receives media data and one or more detected environmental quantities.

  FIG. 1B represents a second example, representing an example of a wireless client device 121, such as a mobile phone, used for two-way conversation via a base station 103 coupled to the network 101. FIG. Note that the same reference numerals are used herein for reference elements having functions similar to those in FIG. 1A. Even if the same reference signs are used, the details of any particular element may vary from embodiment to embodiment, including the means used by the element to accomplish the function. For this example of FIG. 1B, the base station 103 has minimal components, including a transceiver 107, whose functionality is controlled by a controller 141 that is also coupled to the network 101, so-called “lightweight”. It operates differently than that of FIG. 1A in that it is a base station. The controller 141 includes a processing subsystem including at least one processor 143 and a storage subsystem 145. The storage subsystem 145 comprises one or more memory elements and possibly one or more other storage elements such as a magnetic disk. In the present specification, the subsystem 145 is collectively referred to as a storage device. Control data and instructions are communicated over the network by a secure control protocol, whereby the base station operates in a manner similar to the richer base station than that shown in FIG. 1A in combination with the controller. As described herein, in some embodiments, the storage device comprises instructions, for example as a computer program 147, that cause the base station 103 to perform the steps of the method embodiment of the present invention.

  The base station 103 is configured to wirelessly communicate with the resource constrained client device 121, for example, by a program in the storage subsystem 145. The base station is designed to receive media data as (resource-constrained) audio data for rendering on the client device 121 and receive media data such as audio data collected by a sensor such as a microphone in the client device 121. Designed to do. The resource constraining device is for example an antenna subsystem 123 for receiving radio transmissions from the base station 103, and one or more input / output transducers 125, eg for receiving loudspeakers / earphones and audio for rendering. A microphone is provided.

  In one embodiment of the invention, media data, such as audio data, is interactively streamed to a resource constraining device 121 as part of a two-way communication that includes the resource constraining device 121 through a wireless network including the base station 103 and the resource. Design to do. A typical example of such an application is a mobile phone.

  As shown in FIG. 1B, in addition to the controller 131, a media server 131 similar to the media server in FIG. 1A is provided as a media source. The media server 131 can stream media data, such as entertainment or other messages, to the resource limiting device 121 or some other resource limiting device.

  In one aspect of the present invention, media data for rendering by the client device 121 is changed by executing signal processing using one or more detection environment quantities indicating the environment of the client device 121. One or more sensors 109 are configured to detect one or more environmental quantities. Examples of environmental quantities include noise characteristics for audio and / or background lighting for video. However, it is not limited to them. In one aspect of the present invention, the one or more sensors are located away from the client device 121, but the detected environmental quantity is an index equivalent to one or more environmental quantities in the vicinity of the resource constrained device. It is close enough to a resource constrained device. In the embodiment of FIG. 1B, sensor 109 is at base station 103, i.e., is directly coupled to base station 103. This suppresses the use of power in the resource limit device and also reduces the amount of signal processing in the resource limit client device 121.

  A processing system, such as a processing system including a processor 143 in the controller 141, receives one or more detected environmental quantities from the sensor 109 via the base station 103 and the network 101. The processing system processes the media data using the received sensing environment quantity and processes the processed output, eg, media data, and / or auxiliary data to be used to further process the media data at the resource constraint device 121. Configured to generate. In one embodiment, such data processing by the processing system includes generating a change parameter using the received detected environmental quantity and processing the media data based on the change parameter to generate processed media data. Is included. The controller is configured to send processed media data and / or auxiliary data wirelessly to the resource constraint device 121 for rendering via the output transducer or for use in processing and rendering. Accordingly, a signal processing method that uses resources in the environment of the rendering device 121 but requires resources that are constrained in the rendering device 121 is a processor that is not constrained by resources like the device 121 and that is remote from the device 121. Can be executed by.

  In some embodiments, the source of media data is sent from upstream to the client device 121, but in other embodiments of the invention the source of media data is not on a separate device upstream of the network, Located in the client device 121. Therefore, as shown in FIGS. 1A and 1B, the client device 121 includes a storage area capable of storing media data. In such an embodiment, the media data is sent to an upstream processing system that includes the processor 111. The processing system receives media data and one or more detected environmental quantities.

  Several quality enhancement signal processing methods that can advantageously use the mechanism shown are described in more detail later herein.

<Application example>
In the example of FIGS. 1A and 2B, the distance between the sensor and the client device 121 is relatively short. One example is a picocell network. For example, consider an airplane seat. Assume that the client device is a pair of wireless headphones for listening to material that is streamed to a picocell client via a base station. A picocell includes a subset of seats in a set of seats in an airplane. In another application, for example, again considering an airplane seat, the client device is a mobile phone for two-way conversations, including listening to material that is streamed to the picocell client device via the base station. To do. A picocell includes a subset of seats in a set of seats in an airplane. As a second example, assume that the wireless client device 121 is a music player designed to be wirelessly connected to, for example, a picocell in the vicinity of an aircraft seat, and the aircraft is configured to store music data stored in the wireless client device 121. To provide quality improvement services.

  FIG. 2A shows a simplified diagram of several rows of seats 201 in a passenger aircraft having one passage in an application example of the embodiment of the present invention. In this example, two rows and two aisle seats form a picocell 203, and therefore four such picocells are shown, each picocell 203 including a base station 209 and up to six seats. It is.

  FIG. 2B shows an example of one picocell 203 having six seats. Each seat 205 is near the head of any passenger, such as a voice input sensor, and includes a microphone 207 in the back of the seat. The front leftmost seat is shown with the passengers there. The passenger cell phone 211 is shown in a simplified and enlarged form. Each microphone 207 is directly coupled to the base station. Wireless headphones 213 (also shown in simplified, exaggerated and enlarged form) are also available as alternative client devices for each passenger.

  The cell phone handset 211 and headphones are extremely inexpensive so that they do not require a lot of processing and do not require frequent recharging and / or battery replacement and do not require much power. It is advantageous to do so. Nevertheless, there remains a desire to improve the passenger experience, such as audio transmitted to the passenger, by performing signal processing that takes into account the environment, eg, environmental noise in the vicinity of the passenger.

  In another scenario, it is advantageous for the aircraft to provide quality enhancement services for passengers connecting their audio playback devices via their picocell wireless network.

  Each of client devices 211 and 213 has the general structure shown in FIGS. 1A and 1B. The base station may be a complete base station having the general architecture shown in FIG. 1A, or a lightweight base station having the architecture of FIG. 1B, ie a base station using some other architecture.

  Note that in another variation of the application, fewer microphones 207 are used within the picocell 203. For example, the amount of background noise for the entire picocell region 203 can be obtained using one microphone.

  Although FIGS. 2A and 2B illustrate audio related applications, a similar application is applied to viewing video or still images on a portable video device that is a client device and has constrained resources. can do.

<Method Embodiment>
FIG. 3 shows a simplified flowchart of a method embodiment of the present invention, as implemented, for example, by the apparatus shown in FIG. 1C and the specific example embodiment shown in each of FIGS. 1A and 1B. . FIG. 10 illustrates a method 300 for changing media data, eg, a method 300 for improving quality for rendering by a resource constrained device wirelessly coupled to a wireless network. The method includes receiving one or more detected environmental quantities from one or more sensors 109 in step 301. The sensor 109 is located away from the resource constraint device 121, but is sufficient for the detected environment quantity to be an index equivalent to one or more environment quantities in the vicinity of the resource constraint device 121. It is close.

  The method further includes, at step 303, media data using the sensed environment quantity received at the network node 161 at or coupled to the location of the sensor 109, i.e., away from the resource constraint device 121. Data processing to generate processed output, such as processed media data, or auxiliary data for use in processing at a resource constrained device, or both processed media data and auxiliary data.

  In this method, in step 305, the processed output (processed media data and / or auxiliary data) is sent wirelessly to the resource constraining device 121 for rendering at the resource constraining device 121 or for processing and rendering. Is included.

  In an embodiment, the media data includes audio data, for example, audio data for quality improvement processing. In other embodiments, the media data includes video data for quality enhancement processing. In one embodiment, the media data includes both audio data for quality improvement processing and video data for quality improvement processing.

  In some embodiments, network node 161 includes a base station of a wireless network. The term “base station of a wireless network” includes a base station 131 that itself has processing capability to perform processing, or a base coupled to an element that has processing capability to perform processing via a network. Any of the stations 131 is included. Therefore, the term “base station” covers the scenario of FIG. 1A and likewise FIG. 1B.

  In some embodiments, the media data includes media data that is streamed to a resource constrained device. In some embodiments, the media data includes media data that is interactively streamed over a wireless network to a resource constrained device as part of an interactive communication that includes the resource constrained device. In still other embodiments, the media data includes media data that is streamed to a resource constrained device, as well as media data that is interactively streamed over a wireless network to a resource constrained device as part of a two-way communication that includes the resource constrained device. Both are included.

  In some embodiments, the media data is created at the wireless client and then sent back to the wireless client after processing.

<Type of processing using the detected environmental quantity>
The present invention is not limited to the type of processing that uses the detected environment quantity, and many such types of signal processing operations are known that can advantageously utilize the present invention. A quality-enhanced signal processing technique for audio data that can utilize an estimate of the acoustic noise profile in the vicinity of the environment of the mobile device is performed in the perceptual domain, also called the perceived loudness domain, or simply the loudness domain. Processing is included. In one embodiment, such processing includes noise compensation. In a noise compensation embodiment, the processing further includes one or more of automatic gain control, dynamic range compression, and / or dynamic equalization.

  Video data quality-enhancing signal processing techniques that can take advantage of ambient lighting estimates away from, but close to, the wireless device include saturation adjustment, brightness adjustment, contrast adjustment, etc. Generating image-dependent signals for adjusting backlight elements for flat panel display devices using a number of individually adjustable light elements, such as backlighting LEDs for so-called high dynamic range (HDR) displays Is included.

<Exemplary Embodiment for HDR Displays>
HDR displays and the technology behind them are marketed as Dolby Contrast, Dolby HDR and Dolby Vision by Dolby Laboratories, Inc., related to the assignee of the present invention. Currently constructed HDR displays use a regulated light source such as a light emitting diode (LED) for the backlight. Such a backlight is sometimes referred to as an IMLED (Individually Modulated Array of LED) backlight. In one variation, the brightness of each LED is controlled by an 8-bit signal, so each LED has 256 brightness steps. Rather than having a single light source behind the LCD screen, multiple sub-regions are backlit in a manner that is adjusted according to local brightness and contrast in the scene shown.

  The adjustment signal is obtained by performing processing on the video signal and generates a signal that further adjusts the backlighting LED. For further details see, for example, Helge Seezen, Wolfgang Heidrich, Wolfgang Stuerslinger, Greg Ward, Greg Ward, Lone Matthew Trentacoste, Abhijeet Ghosh, Andreis Vorozkovs: "High Dynamic Range Display System", ACM Transactions on Graphics (ACM TransonGacts) s (TOG)), Vol. 23, No. 3, special issue: see the minutes of the 2004 SIGGRAPH Conference (August 2004). See also US Patent Application No. 6,891,672.

  The signal processing for obtaining the adjustment signal from the video signal is not easy. Therefore, a viewing device with resource constraints may not have such processing capability. However, such signal processing can be advantageously performed upstream based on one or more parameters provided, determined or measured in the vicinity of the environment of the resource constrained viewing device.

  Therefore, it is known to change the contrast and brightness according to local viewing conditions, for example, the brightness of the viewing environment. In one embodiment, an environmental sensor connected to the processing element but remote from the wireless client determines a brightness indicator close to the viewing environment. Environmental brightness is provided to upstream processes. In one embodiment, one or more other parameters are sent, such as a parameter indicating the amount of contrast enhancement, a parameter indicating the brightness setting desired by the viewer, and the like. The upstream processor receives such parameters from the resource limited viewing device and performs signal processing to determine the level for the backlighting of the LED device. Typically, backlighting is monochromatic and has a much coarser resolution than the main video. The lower resolution backlight data is sent along with the video signal to the resource limit device, adjusted according to the received parameters, and combined and rendered by the resource limit viewing device.

  FIG. 4 shows a simplified block diagram of an embodiment of the present invention. In this embodiment, media data is streamed to the portable device via the signal processing device 403. In this embodiment, the portable device is a wireless device 511 comprising an LCD panel 415 and an HDR video display 413 composed of a number of individually adjusted light emitting diode devices 417 for backlighting that is adjusted depending on the environment. is there. As a matter of course, FIG. 4 shows the display 413 in the most simplified two-dimensional form. A device 421 coupled to the signal processing device 403 includes an optical sensor 325 configured to measure an indication of ambient light near the wireless portable device 411. In the illustrated embodiment, a microphone 423 for obtaining an audio noise environment quantity is also included as a user interface (UI) for providing metadata. The ambient light environment quantity is combined with one or more other items of metadata relating to audio and / or video quality enhancement processing and / or other environment quantities at a certain frequency, for example, once a second. To the upstream signal processing device 403. The upstream signal processing device is configured to receive the ambient light environment quantity and the video media data and process the video data to generate a signal for the individually adjusting LED device 417 in the HDR display 413. The HDR signal processor 405 is further configured to generate an adjustment signal usable by Processing is not only based on the luminance in the video signal, but also enhances the contrast in the video based on the ambient illumination of the field where the video is being viewed on the portable device 409. The processing device 403 further includes another audio and / or video quality enhancement process due to other items of metadata, and another audio and / or video quality enhancement process due to other environmental quantities such as background noise, for example. May be.

  In some embodiments, signal processing block 403 includes a processor and a storage device comprising program logic comprised of instructions that perform method steps according to some embodiments of the present invention.

<Example Embodiment of Noise Compensation>
FIG. 5 shows that at least one environmental quantity measured by at least one sensor 523 of the device 521 in the vicinity of the wireless portable device 511 includes audio data in the media data, and the acoustic noise profile in the vicinity of the environment of the wireless device 511. FIG. 3 shows a simplified block diagram of an embodiment of an apparatus used as an indicator. The quality improvement processing is executed in the upstream processing block 503, and the quality improvement processing includes noise compensation using the metadata generated by the device 521 and the environmental noise amount. Details of the noise compensation will be described later. In some embodiments, noise compensation includes generating change parameters from audio data using one or more loudness level parameters and one or more environmental quantities indicative of an acoustic noise profile. The change parameter is generated by performing an operation on the information in what is called a “perception region”. Noise compensation includes changing the audio data based on the change parameter and generating processed audio data that is sent wirelessly to the portable device 511. By way of example, the loudness level parameter includes one or more of: whether the audio noise compensation switch is on, a reference level for a resource constrained device, a desired reproduction level, and / or an amount of noise compensation. In some variations, the processing of the audio data further includes one or more of AGC, dynamic range compression and / or dynamic equalization applied to the audio data. FIG. 5 shows some typical values of metadata and environmental quantities that may be sent. The environmental quantity includes an estimated value of noise power. For example, the magnitude of noise in a combination of several bands is the environmental quantity. These environmental quantities are determined by a processor included in the device 521 that receives input from the included microphone 523. In one embodiment, 20 noise spectral intensity values are determined and used. In another embodiment, 40 noise spectral intensity values are used. Those skilled in the art with experience in equalization and noise compensation have found that performing equalization and noise compensation within the 20 to 40 frequency bands will give good results based on current processing capabilities. Yes. Of course, as technology advances, more spectrum bands may be used, and in some situations where even some upstream processing is somewhat constrained, fewer frequency bands, and thus spectrum noise values, are used. There is a possibility that it will only become necessary. This metadata and environmental quantity is received by an upstream signal processing system 503 having the capacity and ability to perform audio noise compensation, eg, a system 503 comprising a noise compensation processing block 505.

  Typically, the environmental quantity is measured and provided at a frequency much lower than the frequency range of the audio data, for example, approximately one set every second.

  In some embodiments, signal processing block 503 includes a processor and a storage device comprising program logic comprised of instructions that perform the method steps according to some embodiments of the invention.

In one embodiment, in loudness-based noise compensation, information required to process audio data that is streamed to a client device such as a mobile device in upstream signal processing, including noise compensation processing, includes: Provided by metadata parameters and environmental information. Illustrate the units used in certain embodiments, and some typical values:
Parameter 1: Noise compensation on / off (0 or 1)
Parameter 2: portable device reference reproduction level (75 dB)
Parameter 3: Target reproduction level (−20 dB)
Parameter 4: Noise spectrum estimate, for example, 20 to 40 noise spectrum intensity values sent approximately once per second.
Parameter 5: Amount of noise compensation (1 to 10)

  4 and 5 show just a few examples of possible audio data quality enhancement signal processing and / or video data quality enhancement signal processing that can be performed by various embodiments having the general structure of FIG. 1C. It will be apparent to those skilled in the art.

<Further details of processing in the perceptual domain>
The present invention is not limited to quality enhancement processing of any particular format of audio media data. However, to clearly demonstrate the advantageous use of embodiments of the present invention, noise compensation—audio quality enhancement signal processing performed in what is referred to herein as a perceptual region, also referred to as a perceptual loudness region or simply referred to as a loudness region. An example of a method is described in this section. In quality-enhanced signal processing, it is known to obtain and use a perceived loudness index in the perceptual region. For example, a method, apparatus and computer program for calculating and adjusting perceived loudness of an audio signal (METHOD, APPARATUS, AND COMPUTER PROGRAM FOR CALCULATINGING and ADJUSTING THE PERCEIVED LOUDNESS OF AN AUDIO 94, published in 1994). International Patent Application No. PCT / US2004 / 016964 published as and the calculation and adjustment of perceived loudness and / or perceived spectral balance of audio signals (CALCULATING AND ADJUSTING THE PERCEIVED LOUDNESS AND / OR THE PERCEIVED SPECTRAL BAL NCE OF AN AUDIO SIGNAL) and entitled, see International Patent Application PCT / US2005 / 038579 published as WO 2006047600. Calculation and adjustment of perceived loudness and / or perceived spectral balance of audio signals (CALCULATING AND ADJUSTING THE PERCEVED LOUDNESS AND / OR THE PERCEIVED SPECTRAL BALANCE OF AN AUDIO SIGNAL) See also published international patent application PCT / US2000075 / 007946. Each of these applications is designated the United States. The contents of each of the above publications, WO20041111994, WO2006047600 and WO2007120453 are incorporated herein by reference. Alain Seefeld: "Loudness Domain Signal Processing", Paper 7180, Minutes, Audio Engineering Society 123rd Annual Meeting, New York, New York, USA, 2007, 5 See also -8. Some details of the audio quality enhancement signal processing methods described herein are in the above-mentioned published patent applications and published papers.

  The quality-enhanced signal processing method includes obtaining change parameters from calculations performed in the perceptual loudness region and changing the audio media data according to the change parameters. If a change parameter is obtained in the perceptual loudness region, it may be possible to achieve a control with a better balance between the perceived loudness and the perceived spectrum than when such a change parameter is derived in the electrical signal region. In addition, when performing calculations in the loudness region, using a psychoacoustic filter bank that simulates the basement membrane or equivalent, the perceived spectrum rather than a mechanism that derives the modified parameters in the electrical signal region. May be able to be controlled in detail.

  Often, audio media data is expected to be reproduced at a specified reference level. However, in many cases, media data is played at a reduced level. It is known that there is a change in audio perception according to the reproduction level. Such changes are related to psychoacoustics and the isoloudness curve and the minimum audible value in silence. Changing the playback level can make dramatic differences in audio timbre and spatial perception when compared to the same media data played at the reference level. Quality-enhancing signal processing in some embodiments of the invention includes determining and adjusting perceived loudness in the audio signal in an improved manner. A psychoacoustic model is used to calculate the loudness index of the audio signal in perceptual units. Such an indicator of perceived loudness is called specific loudness and is an indicator of perceived loudness as a function of frequency and time. As one example, a volume control method using parameters determined in the perceptual region includes calculating a wideband multiplication gain using a signal processing method, and the wideband multiplication gain, when applied to audio, is a reference loudness. Causes the loudness of the gain-changed audio to be substantially the same. The gain adjustment method includes a signal processing method in which the audio is analyzed and changed according to the reproduction level and restored to a state perceived at the reference reproduction level. This has been found to provide improved imaging, clarity and audio media data audibility. Further details will be presented later.

<Noise compensation as sound volume equalization (optionally using dynamic range control and / or automatic gain control)>
Noise compensation is an example of volume equalization in the presence of noise (and taking noise into account). Volume equalization, also called loudness equalization and loudness compensation equalization, is used to control the specific loudness of the audio signal, especially by changing the audio signal to reduce the difference between the specific loudness and the target specific loudness. Deriving available information is also included. In practical implementations, the specific loudness of the modified audio signal may be made to approximate the target specific loudness. The approximation may be affected not only by normal signal processing considerations, but also by time and / or frequency smoothing that may be used in the change. The method includes determining a perceived loudness of the audio signal in the form of a specific loudness of the audio signal and determining a multi-band gain to be applied to multiple bands of the audio signal to modify the audio signal. In some embodiments, signal changes dynamically apply multiband gain changes to the audio so that the perceived loudness consistency of the audio media data is maintained. When these are used in conjunction with audio system volume control, the volume controller is transformed and no longer emulates an electrical resistor that controls the level of the audio signal being sent to the amplifier. Instead, the volume controller provides an input to the homogenization method and indicates the user's desired perceived loudness reproduction level.

<Noise compensation-homogenization in the presence of noise interference>
In many audio playback environments, there is background noise that interferes with the audio that the listener wants to hear. For example, a listener in a moving car may be playing music through an on-board stereo system, and noise from the engine and road may significantly change the perception of the music. In particular, the perceived loudness of the music is reduced in the part of the spectrum where the noise energy is considerably larger than the music energy. If the noise level is high enough, the music will be completely masked. Quality-enhanced signal processing in some embodiments of the present invention includes a method of compensating for background noise that interferes in an audio playback environment. Audio partial loudness is defined as the perceived loudness of audio in the presence of secondary interfering speech signals such as noise. Signal processing in some embodiments seeks information that can be used to control the partial specific loudness of the audio signal by modifying the audio signal to reduce the difference between the partial specific loudness and the target specific loudness. It is included. By doing so, the effect of noise is reduced in a perceptually accurate manner.

<Noise compensation using automatic gain control or dynamic range compression>
Uniformation in the presence of noise can be used to determine change information used to change the perceived loudness of the reproduced audio to match the user's desired loudness level. This can be used to achieve automatic gain control and / or dynamic range compression. Details of equalization to achieve automatic gain control and dynamic range compression will be described later.

<Dynamic equalization (DEQ)>
Unlike simple homogenization in the presence of noise, instead of changing the audio to match the user's desired perceived loudness level, dynamic equalization uses a preset or user-defined equalization or spectral balance profile. To match. Since specific loudness is a measure of the perceived loudness of an audio signal as a function of frequency and time, changes are made to the audio signal as a function of frequency in order to reduce the difference between the specific loudness of the audio signal and the target specific loudness. You can change as follows. In some cases, the target specific loudness may be time-invariant and the audio signal itself may be a steady-state time-invariant signal, but the change usually changes the audio signal as a function of time. May be. For a time and frequency varying scale factor, the specific loudness may be scaled by the ratio of the desired spectral shape index to the spectral shape index of the audio signal. Using such scaling, the perceived spectrum of the audio signal may be converted from a time-varying perceived spectrum to a substantially time-invariant perceived spectrum. If the specific loudness is scaled by the ratio of the desired spectral shape index to the spectral shape index of the audio signal, such scaling may be used as a dynamic equalizer.

  In the case of perceptual region processing including equalization, such as noise compensation, the amount of detected environment received includes one or more parameters indicative of the acoustic noise profile of the environment close to the environment of the resource constrained device. The metadata includes one or more loudness equalization parameters. The processing of the media data includes noise compensation that optionally includes dynamic range compression and / or equalization applied to the audio data. For noise compensation, one or more of the loudness level parameters and one or more parameters of the acoustic noise profile are used to generate change parameters from the audio data, and the audio data is changed based on the change parameters and processed audio data. Is included. Change parameters are generated by performing operations on information in the perceptual loudness region. The one or more loudness level parameters include one or more of a reference reproduction level, a desired reproduction level, and / or an amount of homogenization.

  In certain embodiments, the sensed environmental quantity received includes one or more parameters indicative of an acoustic noise profile within the environment of the resource constrained device. Noise compensation applied to the audio data includes: (a) one or more of the received sensing environment quantities that include at least one parameter indicative of an acoustic noise profile within the environment of the resource constrained device; and (b) audio. Based on one or more parameters such as whether the noise compensation switch is on, a reference level for the resource constrained device, and / or one or more processing parameters such as a desired reproduction level and amount of noise compensation.

<A more detailed overview of perceptual domain-based quality improvement processing>
Throughout the following description, terms such as “filter” or “filter bank” include essential elements such as infinite impulse response (IIR) filters or transforms, and finite impulse response (FIR) filters. Any form of recursive and non-recursive filtering is included. “Filtered” information means the result of applying such a filter or group of filters. In the embodiment described later, a filter bank executed by conversion is used.

  As described above, audio quality enhancement signal processing operations in the perceptual loudness region that can be included in embodiments of the present invention include noise compensation, such as equalization in the presence of noise interference. Such noise compensation could be advantageously combined with dynamic range control or automatic gain control, and / or dynamic equalization (DEQ).

  Such embodiments include determining the specific loudness of the audio signal and the specific loudness of the noise based on the amount of environment sent from a location close to the location of the resource constrained device. For quality-enhanced signal processing, a noise indicator is received from one or more sensors that are located remotely from the resource limiting device, but are close enough to be an indicator of the noise in the environment of the resource limiting device. Further included is controlling the partial specific loudness of the audio signal by modifying the audio signal to reduce the difference between the partial specific loudness and the target specific loudness. Quality-enhancing signal processing includes processing the audio signal by processing the audio signal or its indication according to one or more processes and one or more process control parameters to create a signal having a target specific loudness. It may be.

  Target specific loudness may or may not be a function of the audio signal. In the latter case, the target specific loudness is the stored target specific loudness, or the target specific loudness received as a parameter, or may be determined from the received parameter. In such a case, the modification or derivation may be performed by explicitly or implicitly calculating the specific loudness or the partial specific loudness. An example of an implicit calculation is by a look-up table or by calculating a mathematical formula in which specific loudness and / or partial specific loudness is essentially determined.

<Feed forward mechanism>
6A-6D, functional blocks representing several embodiments of quality-enhanced signal processing that alters an audio signal to produce modified audio to bring the partial specific loudness closer to the target specific loudness using a feedforward mechanism. The figure is shown. In particular, FIG. 6A shows that the audio signal 611 has two paths: a signal modification path 601 having one or more processes or devices configured to modify the audio signal 611 in response to one or more modification parameters 619. And a feedforward topology applied to a parameter generation control path having a parameter generation control path 602 configured to generate a modified parameter 619. The signal change path 601 in the feedforward topology example of FIG. 6A changes the audio signal, eg, its amplitude, in a manner that changes in frequency and / or time according to a change parameter 619 received from the parameter generation control path 602. Any device or process that can be used. In one embodiment, at least a portion of the parameter generation control path 602 operates in the perceptual loudness region. On the other hand, the signal change path 601 operates in the electrical signal domain and creates a changed audio signal 615.

  The signal change path 601 and the parameter generation control path 602 are configured to change the audio signal to reduce the difference between the specific loudness and the target specific loudness 623.

  In one embodiment, each of the signal modification path 601 and the parameter generation control path 602 processes a signal that has been pre-processed by a pre-processing operation or device. For this reason, FIG. 6A illustrates a preprocessing function block 603 that creates a preprocessed audio 613.

  In the feedforward example of FIG. 6A, the parameter generation control path 602 may comprise several processes and / or devices: In the example shown in FIG. 6A, the parameter generation control path 602 includes an audio signal 611, Or comprising one or more processes and / or devices configured to calculate a specific loudness 617 of the audio signal in response to the indication of the audio signal and in response to the preprocessed audio signal 613, the specific loudness A block 605 to calculate is included. The parameter generation control path 602 includes a block 607 for calculating a change parameter. The change parameter calculation block 607 calculates the change parameter in response to excitation or specific loudness due to secondary interference audio signal 621 such as specific loudness or excitation 617, target specific loudness 623, noise, and the like. Accordingly, the change parameter calculation block 607 receives such a secondary interference audio signal indicator as an input, or receives the secondary interference signal itself as one of its inputs. As described in more detail later in this specification and in WO2006047600 and WO2007120453, the indicator of the secondary interference signal may be its excitation. A change parameter 619 that takes into account the interference signal to achieve noise compensation by an appropriately configured process or device by applying an indicator of the interference signal or the signal itself to the block 607 for calculating the change parameter shown in FIG. 6A. Can be calculated.

  In the feedforward example of FIG. 6A, the partial specific loudness is not calculated explicitly. The change parameter calculation block 607 of FIG. 6A calculates the appropriate change parameter to approximate the target specific loudness 623 to the partial specific loudness of the modified audio. For feedback and hybrid configurations, partial specific loudness may also be calculated.

  In one embodiment, as shown in FIG. 6B, the target specific loudness of block 607 for calculating the change parameter of the control path 602 is typically in the audio signal or its indicator, as well as in the illustrated embodiment the preprocessed audio signal 613. Determined by a target specific loudness block 631 comprising one or more processes or devices configured to calculate target specific loudness 623 in response. This target specific loudness calculation block 631 implements one or more functions “F” each having a function parameter. For example, block 631 may calculate a specific loudness of the audio signal and then apply one or more functions F to it to provide a target specific loudness 623. To this end, FIG. 6A schematically illustrates a “function and / or function parameter” input 633 to a block 631 for calculating target specific loudness.

  In some embodiments, as shown in FIG. 6C, the target specific loudness 623 is provided from the storage element 635 schematically shown and included in or associated with the parameter generation control path 602.

  Further, as shown in FIG. 6D, in one embodiment, target specific loudness 623 is provided by a source external to the process or the entire device.

  Accordingly, the change parameter 619 is based at least in part on calculations in the perceptual (acoustic psychological) loudness region.

  The calculations performed by the processes or devices 605 and 607 in the example of FIG. 6A and 631 in FIG. 6B may be performed explicitly and / or implicitly. Examples of implicit implementations include: (1) a lookup table in which all or part of the items are based on specific loudness and / or target specific loudness 623 and / or calculation of modified parameters; and (2) essentially all or one. And closed mathematical formulas whose parts are based on specific loudness and / or target specific loudness 623 and / or modified parameters.

  Although the calculation blocks 605, 607, and 631 in the example of FIGS. 6A and 6B are schematically illustrated and described as separate, this is for illustrative purposes only. It will be appreciated that some or all of these processes or devices may be combined within a single process or device, or various combinations within multiple processes or devices.

  The target specific loudness may be a scaling of an index of the audio signal such as the specific loudness of the audio signal. For example, as described in detail in WO2006047600 and WO2007120453, the scaling may be one or a combination of the following scalings for a specific loudness. Here, b represents a frequency index, for example, the number of bands when the preprocessing 603 separates the input signal into a plurality of frequency bands, t represents a time index,

は目標特定ラウドネス６２３を示し、Ｎ［ｂ，ｔ］はオーディオ信号６１１の特定ラウドネス６１７を示す。

Indicates the target specific loudness 623, and N [b, t] indicates the specific loudness 617 of the audio signal 611.

  (A) Scaling a specific loudness as a function of time and frequency change scale factor Ξ [b, t]:

  (B) Scaling of a specific loudness with the following relationship by time-varying, frequency-invariant scale factor Φ [t]:

  (C) Time-invariant, frequency change scale factor Θ [b] with a specific loudness scale as:

  (D) Scaling the specific loudness of the audio signal with a time-invariant, frequency-invariant scale factor α, such that:

  The target specific loudness 623, denoted as, may thus be expressed as one or more functions denoted by F in the audio signal or a combination of audio signal indicators, and the specific loudness N [b, t] Possible indicators are as follows:

If the function or function group F is reversible, the specific loudness N [b, t] of the unchanged audio signal 611 is the inverse function of the target specific loudness 623 or the function group F ⁻¹ (N [b, t]). May be calculated.

Although FIG. 6A shows a feedforward mechanism, it is also known to use a feedback and hybrid feedforward / feedback mechanism in which an inverse function or function group F ⁻¹ () is calculated. See, for example, WO2006047600 and WO2007120453. However, for the sake of brevity, only the feedforward configuration is described herein.

  Whether using a look-up table, a closed formula, or some other technique, the operation of the parameter generation control path 602 is calculated even though the specific loudness and target specific loudness 623 may not be calculated explicitly. Is based in the perceptual (acoustic psychological) loudness domain. There is either explicit specific loudness or conceptual or implicit specific loudness. Similarly, there is either an explicit target specific loudness 623 or a conceptual, implicit target specific loudness 623. In any case, the calculation of the change parameter seeks to generate a change parameter that changes the audio signal to reduce the difference between the specific loudness and the target specific loudness 623.

  The change parameter 619 when applied to the audio signal (or preprocessed audio signal) by the signal change path 601 reduces the difference between the resulting changed audio partial specific loudness and the target specific loudness 623. Ideally, the partial specific loudness of the modified audio signal 615 closely approximates or is the same as the target specific loudness 623.

  In some embodiments, the preprocessing separates the audio signal into multiple frequency bands using, for example, a filter bank. In such an embodiment, the change parameter 619 may take the form of a time-varying gain factor applied to the frequency band derived from the filter bank within 603, as in the example described in more detail later. In an alternative embodiment, the change parameter 619 is applied to the coefficients of the time varying filter. Accordingly, in the example of FIG. 6A, the signal modification path 601 is implemented as, for example, a plurality of amplitude scalers each operating within a certain frequency band, or a time-varying filter, such as a multi-tap FIR filter or a multipole IIR filter. That's fine.

  While not critical or indispensable in determining specific loudness or partial specific loudness, in one embodiment, the technique described in WO 2004/111964 described above is used in block 605 to calculate specific loudness. The calculation selects one or a combination of two or more specific loudness model functions from a group of two or more specific loudness model functions, the selection being controlled by an indicator of the characteristics of the input audio signal. Some use techniques.

  According to further aspects of the invention, the unmodified audio signal 611 and (1) the modified parameter 619 or (2) the target specific loudness 623 or a representation of the target specific loudness 623, eg, the target specific loudness 623 may be explicit or implicit. Any of the scale factors available in calculating can be stored or transmitted, for example, for use in temporally and / or spatially separated devices or processes. The change parameter, the target specific loudness 623, or the expression of the target specific loudness 623 may be obtained by any appropriate method. In practice, a feedforward mechanism such as that in the example of FIG. 6A is the least complex and the fastest. This is because it avoids calculations based on the modified audio signal 615.

  FIG. 7 shows details of an example embodiment of an aspect of the present invention that includes a feedforward mechanism that separates input audio into frequency band groups during preprocessing. In an actual embodiment, the audio processing is performed in the digital domain, and thus analog signal sampling and digitization is performed. Such details are omitted from this description but will be apparent to those skilled in the art.

  The audio 611 is first passed through an analysis filter bank function or device 703 that divides the audio signal into a plurality of frequency bands. This is the preprocessing 603 in this embodiment. The thick lines in FIG. 7 indicate a plurality of signals, and thus each is a plurality of outputs that are frequency bands from the analysis filter bank 703. Each of these frequency bands is subjected to the various processing steps shown, leading to a synthesis filter bank 723, which sums the bands into a combined wideband signal to produce a modified audio signal 615.

  The filter response associated with each frequency band in the analysis filter bank 703 is designed to mimic the response at a specific location of the basement membrane in the human inner ear. In one embodiment, the analysis filter bank 703 includes a set of linear filters that have a constant bandwidth and spacing on the frequency scale of the equivalent rectangular bandwidth (ERB).

  The analysis filter bank 703 may be efficiently executed by using short-time discrete Fourier transform (short-time DFT, STDFT) or modified discrete cosine transform (modified DCT, MDCT). STDFT or MDCT may be used in the same way for the execution of the synthesis filter bank 723.

  The output of each filter in the analysis filter bank 703 then proceeds into a conduction filter function or device 705 that is designed to simulate the filtering effect of audio conduction through the human outer and middle ears.

  In order to calculate the loudness of the input audio signal, an indication of the short-time energy of the audio signal after application of the conduction filter 705 within each filter of the analysis filter bank 703 is obtained. This time and frequency change index is called excitation and is denoted as E [b, t]. Here, b indicates a frequency band, and t indicates time. In order to obtain excitation, the output of the conduction filter 705 then proceeds into the excitation function or device 707. The excitation function or output of device 707 is designed to simulate the distribution of energy along the basement membrane of the human ear. Depending on the desired effect, the excitation energy value may be smoothed over time by a smoothing function or device 709 that is designed to have a time constant set according to the desired effect requirements of the process. The output of the excitation function 707 is a frequency domain representation of the energy, denoted E, per time, denoted t, within the respective ERB band denoted b.

  A specific loudness function or device 711 converts the smoothed excitation signal to a specific loudness (SL). The specific loudness may be expressed by, for example, a thorn unit per unit frequency, for example, a thorn per ERB. Note that from a specific loudness, the total or total loudness is the sum of the specific loudness over the entire band b. The design of the specific loudness function 711 includes determining gains for narrowband and wideband estimates that are chosen to be consistent with experimental data regarding loudness increases for music and noise. Furthermore, the specific loudness function 711 is such that when the excitation is at the minimum audible value, the specific loudness is some small value instead of zero, and as the excitation decreases to zero, the specific loudness decreases monotonically to zero. Designed as such. The transformation of the excitation E [b, t] into a specific loudness, denoted N [b, t], is due to the function denoted by Ψ {·} herein, and therefore the specific loudness is N [b , T] = Ψ {E [b, t]}.

  Depending on the particular desired effect or group of effects of the process, the specific loudness component associated with the frequency band is sent to a specific loudness change function or device 713 that produces the target specific loudness. Referring to FIG. 6B, as described above, in one embodiment, the target specific loudness is a function of the specific loudness of the input audio according to the desired effect of the processing. The target specific loudness can be calculated using, for example, a scale factor for volume control. For automatic gain control (AGC) or dynamic range control (DRC), the target specific loudness can be calculated using the ratio of the desired output loudness to the input loudness. . In one embodiment, instead of performing DRC for all bands, it is represented by N [b, t] so that the amount of DRC applied from one band to the next does not change abruptly. A specific loudness is applied and smoothed over the band.

  In the case of dynamic equalization (DEQ), the target specific loudness is calculated using a relationship that takes into account the spectrum of the flowing audio. Specifically, the spectrum of the signal is measured, and then the signal is dynamically changed, and the measured spectrum is referred to as EQ [b], which is basically a static desired Convert to shape. The spectral shape of the audio signal is represented by L [b, t] and, in one embodiment, is determined as a smoothing over time of a specific loudness denoted N [b, t]. As with multiband DRC, the DEQ change may not want to change rapidly from one band to the next, so a band smoothing function is applied to generate a band smoothed spectrum. In order to preserve the original dynamic range of the audio, the desired spectrum EQ [b] must be normalized to have the same overall loudness as the measured spectral shape given by L [b, t]. In one embodiment, a parameter is specified that varies from 0 to 1 representing the amount of DEQ applied, for example, a value of 0 indicating no DEQ. Accordingly, the SL change 713 may operate independently of each band, or there may be interdependence between bands or within a band group.

  This embodiment also includes measurement of noise excitation by analysis filter bank 733, conduction filter 735, excitation 737 and smoothing 739 in a manner corresponding to the operation of blocks 703, 705, 707 and 709. The noise excitation is sent to the gain solver 631 along with the audio excitation from the smoothing 709 and the target specific loudness from the SL change 713.

  A gain solver function or device 715 that takes as input the smoothed excitation frequency band component from the smoother 709, the smoothed excitation frequency band component from the smoother 739, and the target specific loudness 623 from the SL change 713. In order to convert the determined partial specific loudness to the target specific loudness 623, it is configured to determine a gain that needs to be applied to each band. The required gains are typically frequency and time-varying gains, and when these gains are applied to the original excitation of the audio input and noise, they produce a partial specific loudness that is ideally equal to the desired target specific loudness. In practice, this results in changing the audio signal to reduce the difference between the partial specific loudness and the target specific loudness. The gain solver 515 may be executed by various methods. If a closed calculation is possible, it is applied. If a table index is possible, such a table index may be used. In one embodiment, the gain solver may include an iterative process in which the partial specific loudness is evaluated using the current estimate of gain at each iteration. The resulting partial specific loudness is compared to the desired target, and the gain is iteratively updated based on the error. Such an iterative method is disclosed in the aforementioned international patent application published as WO2004111964. Other methods may be devised to calculate the modified parameters through either explicit or implicit calculation of specific loudness and target specific loudness, and the present invention is intended to cover all such methods in scope. .

  The gain per band generated by the gain solver 715 is further smoothed over time by any smoothing function or device 719 to minimize perceptual artifacts. Alternatively, it may be advantageous that temporal smoothing is applied elsewhere in the process or the entire device.

  Finally, the gain determined by the gain solver 715 is applied to each band through each multiplying function or combiner 721. Multiply combining function or combiner 721 applies the gain to the delayed output from the analysis filter bank. The output from the analysis filter bank is delayed by a device 725 configured to compensate for any latency associated with the gain calculation or an appropriate delay function.

  The modified audio 615 is synthesized from the synthesized filter bank function or the gain modified band in the device 723. As described above, the analysis filter bank 703 may be efficiently executed by using the short-time DFT or the modified DCT, and the STDFT or MDCT may be similarly used for the execution of the synthesis filter bank 723. The synthesis filter for each band is determined from the filter used in the analysis filter bank 703 and the delay of delay 725.

  Alternatively, instead of calculating the gain used to change the gain in the frequency band, the gain solver 715 may calculate a filter coefficient for controlling a time-varying filter such as a multi-tap FIR filter or a multipole IIR filter. Please keep in mind. For the sake of brevity, the case of using a gain factor mainly applied to the frequency band is described in the aspect of the present invention. However, in the actual embodiment, a filter coefficient and a time-varying filter may be used. It will be understood that it is good.

For noise compensation, the gain from the gain solver denoted G [b, t] is such that the specific loudness of the processed audio in the presence of interference noise is equal to or close to the target specific loudness. In order to obtain this effect, the partial loudness concept may be used. E _N [b, t] represents excitation from noise and E _A [b, t] represents excitation from noiseless audio. The combined specific loudness of audio and noise is given by:
N _Tot [b, t] = Ψ {E _A [b, t] + E _N [b, t]}
Here, again, Ψ {·} indicates a conversion from excitation to a specific loudness. It may be assumed that the listener's hearing partitions the combined specific loudness between the audio specific loudness and the noise partial specific loudness in a manner that preserves the combined specific loudness. Here, the part specific loudness of the audio, denoted as N _A [b, t], is the value that we want to control and therefore must be solved for this value. WO2006047600 and WO2007120453 disclose E _N [b, t], N _Tot [b, t], masked threshold in the presence of noise, and the minimum audible range value in band b in silence. Describes how the noise specific loudness should be approximated, and when the audio excitation is equal to the noise mask threshold, the audio partial loudness is the signal loudness at the silent threshold. Equally, when the audio excitation is much greater than that of noisy, the audio specific loudness has the property that it is roughly equal to that in the absence of noise, with a partial audio loudness N _A [b, t]. Can reach the formula for. In other words, when the audio is much larger than the noise, the noise is masked by the audio. The formula includes an exponent value that can be chosen empirically to fit the data about the loudness of the musical tone in noise as a function of the signal-to-noise ratio. The noise masked threshold may be approximated as a function of the noise excitation itself.

  In the case of noise compensation, a gain G [b, t] is calculated using a modified gain solver so that the partial specific loudness of the processed audio in the presence of noise is equal to or close to the target specific loudness.

  In the most basic mode of operation, the SL change 713 in FIG. 7 simply sets the target specific loudness equal to the original specific loudness of the audio N [b, t]. In other words, changing the SL provides a specific loudness scaling of the audio signal by a frequency invariant scale factor. Using the mechanism of FIG. 7 and the like, the gain is calculated so that the perceived loudness spectrum of the processed audio in the presence of noise is equal to the loudness spectrum of the audio in the absence of noise. Further, any one or combination of the above techniques that calculate target specific loudness including volume control, AGC, DRC and DEQ as a function of the original may be used in conjunction with a noise compensated loudness modification system. Good.

  In an actual embodiment, noise measurements may be obtained from a microphone provided in or near the environment where audio is played. In one aspect of the invention, noise measurement is performed by a sensor coupled to a network element in the system where signal processing is performed, rather than a playback resource constrained device.

  Comparing FIGS. 6A-4B with FIG. 7, preprocessing block 603 is performed by analysis filter bank 703 and audio modification is performed by a combination of delay 725, gain multiplication 721 and synthesis filter bank. The block 605 for calculating the specific loudness is executed by a combination of the conduction filter 705, the excitation 707, the smoothing 709 and the specific loudness function 711. The change parameter is calculated by calculating the gain G (b, t), and in the absence of noise compensation, the gain solver 715 optionally executes in combination with the smoothing 719. The gain solver 715 also performs analysis filter bank 733, conduction It is performed in combination with filter 735, excitation 737, smoothing 739 and specific loudness function 611, and optionally in combination with smoothing 719. In various applications, the target specific loudness calculation 631 shown in FIG. 6B is performed by a specific loudness change block 713.

  Noise compensation has been described in some detail herein, possibly with one or more of volume control, AGC, dynamic range control, and / or dynamic equalization, which is signal processing to which the present invention is limited. It is never meant to limit the form of. The present invention is related to the environment of a resource limiting device but is obtained remotely from the resource limiting device to generate processed output that can be used by the resource constraining device to render or process and render media data. Applicable to signal processing operations on media data that can be advantageously performed upstream in a network element where one or more resources sufficient for processing with information are available.

  It is noted that the above description, as well as WO2004111994 and WO2006047600, describe several methods for determining specific loudness, but other methods for determining specific loudness are also known. I want. See, for example, WO2007120453.

  In one embodiment, a computer-readable medium is configured with program logic, eg, a set of instructions that, when executed by at least one processor, causes a set of method steps of the methods described herein to be performed. Is done.

  In accordance with common industry terminology, the terms “base station”, “access point”, and “AP” are used to describe an electronic device that can communicate with a plurality of other electronic devices wirelessly and substantially simultaneously. While the terms are used interchangeably in the specification, the terms “client”, “mobile device”, “portable device”, and “resource constrained device” refer to any of a plurality of other electronic devices that have the ability to render media data. Used interchangeably to represent. However, the scope of the present invention is not limited to the devices represented by these terms.

  In the context of this document, the term “wireless” and its derivatives refer to circuits, devices, systems, methods, techniques, communication channels, etc. that may communicate data using modulated electromagnetic radiation over a non-solid medium. May be used to describe. The term does not imply that the associated device does not contain any wire. However, it may not be included at all depending on the embodiment.

  Unless otherwise specified, as will be apparent from the following description, explanations using terms such as “process,” “calculate,” “calculate,” “seek,” or the like throughout the specification, Refers to actions and / or processes that process and / or convert data expressed as physical quantities, such as electronic quantities, into other data that is also expressed as physical quantities in a computer or computing system or similar electronic computing device Will be understood.

  Similarly, the term “processor” processes electronic data from, for example, registers and / or memories, and converts the electronic data into other electronic data that may be stored in, for example, registers and / or memory. Any device or part of a device may be used. The “computer” or “computer” or “computation platform” may include at least one processor.

  It should be noted that when a method is described that includes several elements, eg, several steps, the ordering of such elements, eg, the ordering of steps, is not implied unless specifically stated otherwise.

  The techniques described herein may be implemented, in one embodiment, by one or more processors that receive computer-executable (also called machine-executable) program logic embodied on one or more computer-readable media. It can be implemented. The program logic includes a set of instructions that, when executed by one or more of the processors, perform at least one of the methods described herein. Any processor capable of executing a set of instructions (sequential or other form) specifying the action to be taken is included. Thus, one example is a typical processing system that includes one processor or more. Each processor may include one or more of a CPU, a graphics processing unit, and a programmable DSP unit. The processing system may further include a storage subsystem including a main RAM and / or static RAM, and / or a memory subsystem including a ROM. The storage subsystem may further include one or more other storage devices. A bus subsystem may be included for communication between components. The processing system may further be a distributed processing system comprising processors coupled by a network. If the processing system requires a display, such a display may be included, for example, a liquid crystal display (LCD) or a cathode ray tube (CRT) display. If manual data entry is required, the processing system also includes an input device such as one or more of an alphanumeric input unit such as a keyboard, a pointing control device such as a mouse, and the like. The term units such as storage devices, storage subsystems, etc. as used herein includes storage devices such as disk drive units, unless otherwise apparent from the context and unless explicitly stated otherwise. The processing system in some configurations may include an audio output device and a network interface device. Accordingly, a storage subsystem is a computer that retains program logic (eg, software) that includes a set of instructions that, when executed by one or more processors, cause one or more of the methods described herein to be performed. Includes readable media. The program logic may be in the hard disk, or may be wholly or at least partially in RAM and / or in the processor during its execution by the processing system. Thus, the memory and processor are also components of a computer readable medium in which program logic is encoded, for example in the form of instructions.

  Further, a computer readable medium may form or be included in a computer program product.

  In alternative embodiments, one or more processors may operate as a single device or may be connected to other processor (s), eg, networked within a network deployment. These processors may operate within the capacity of a server or client machine in a server-client network environment or as a peer machine in a peer-to-peer or distributed network environment. One or more processors are: a personal computer (PC), a tablet PC, a set-top box (STB), a personal digital assistant (Personal Digital Assistant, PDA), a mobile phone, a web device, a network router , A switch or bridge, or any machine capable of executing a set of instructions (sequential or other form) that specify the action to be taken by the machine.

  Although some figures (s) only show a single processor and a single memory holding the logic containing the instructions, those skilled in the art will appreciate that many of the components described above are included, Note that it will be understood that it has not been explicitly shown or described in order not to obscure. For example, although only a single machine is illustrated, the term “machine” refers to a set (or sets) of instructions that perform any one or more of the techniques described herein. It must also be considered to include any set of machines that perform together or jointly.

  Accordingly, one embodiment of each of the methods described herein is a computer program for execution on one or more processors, such as one or more processors that are part of a signal processing apparatus. Or the like. Thus, as will be appreciated by those skilled in the art, embodiments of the present invention may be embodied as a method, a device such as a dedicated device, a device such as a data processing system, or a computer readable medium, such as a computer program product. . The computer-readable medium retains logic that includes a set of instructions that, when executed on one or more processors, cause the method steps to be performed. Thus, aspects of the invention may take the form of a method, an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Furthermore, the present invention may take the form of a computer readable medium comprised of program logic, such as a computer program on a computer readable storage medium, or computer readable program code, such as a computer program product.

  Although a computer readable medium is shown as being a single medium in the example embodiments, the term “medium” refers to a single medium or multiple media (eg, centralized or multiple) that store one or more sets of instructions. Distributed database and / or associated cache and server). The term “computer-readable medium” is for execution by one or more of the processors and stores a set of instructions that cause execution of any one or more of the techniques of the present invention; It must also be considered to include any computer readable medium that is capable of encoding or otherwise composed thereof. Computer-readable media can take many forms, including but not limited to, non-volatile media and volatile media. Non-volatile media includes, for example, optical, magnetic disks, and magneto-optical disks. The volatile medium includes a dynamic memory such as a main memory.

  It is understood that the method steps described are performed, in one embodiment, by a suitable processor (or group of processors) of a processing system (eg, a computer system) that executes instructions stored in storage. Like. The embodiments of the present invention are not limited to any particular implementation or programming technique, and the present invention employs any suitable technique for performing the functionality described herein. It will also be understood that it may be implemented using. In addition, embodiments are not limited to any particular programming language or operating system.

  Throughout this specification, reference to “one embodiment” or “one embodiment” refers to a particular feature, structure, or characteristic described in connection with the embodiment within at least one embodiment of the invention. Means included. Thus, the appearances of the phrases “in one embodiment” or “in one embodiment” in various places throughout this specification are not necessarily all referring to the same embodiment. . Furthermore, the particular features, structures or characteristics may be combined in any suitable manner within one or more embodiments, as will be apparent to those skilled in the art from this disclosure.

  Similarly, in the above description of example embodiments of the invention, for purposes of simplifying the disclosure and assisting in understanding one or more of the various aspects of the invention, It should be understood that they may be grouped together in a single embodiment, figure, or description thereof. This method of disclosure, however, should not be interpreted as reflecting an intention that the claimed invention requires more features than are expressly recited in each claim. Rather, as the appended claims reflect, aspects of the invention reside in less than all features of a single embodiment disclosed above. Thus, the claims following the detailed description are hereby expressly incorporated into this Detailed Description, with each claim standing on its own as an independent embodiment of the invention. Exists as.

  Further, some embodiments described herein are intended to include some of the features included in other embodiments but not other features, but will be understood by those skilled in the art. As such, combinations of features of different embodiments are intended to be within the scope of this invention and form different embodiments. For example, in the appended claims, any of the claimed embodiments can be used in any combination.

  Furthermore, some of the embodiments are described herein as a method or combination of elements of a method that can be performed by a processor of a computer system or by other means of performing that function. . Accordingly, a processor with the necessary instructions to carry out such a method or method element forms the means for carrying out the method or method element. Furthermore, elements of apparatus embodiments described herein are examples of means for performing the functions performed by the elements for the purpose of carrying out the invention.

  In the description provided herein, numerous specific details are set forth. However, it will be understood that embodiments of the invention may be practiced without these specific details. In other instances, well-known methods, structures and techniques have not been shown in detail in order not to obscure the understanding of this description.

  As used herein, unless otherwise specified, the use of the adjectives “first”, “second”, “third”, etc. to indicate a common thing is similar for describing common things. It is merely an indication that different examples of the object are mentioned, and the object so described is either temporally, spatially, in rank, or in any other manner Is not intended to imply that it must be in a given order.

  Any discussion of prior art in this specification is in no way an acknowledgment that such prior art is well known, known or forms part of the general knowledge in the field. Should not be considered.

  In the appended claims and the description herein, the terms “comprising”, “comprised of”, or “comprising of” each include at least the following element / feature. Is an open term meaning that it does not exclude others. Therefore, the term “comprising”, as used in the claims, should not be interpreted as being restricted to the means or elements or steps listed thereafter. For example, the scope of the expression device comprising A and B should not be limited to devices consisting only of elements A and B. As used herein, the terms including, including, or including, including, that at least includes the elements / features that follow the term. It is a non-limiting word that also means that it includes but does not exclude others. Accordingly, including is synonymous with and meaning “comprising”.

  Similarly, the term coupled, as used in the claims, should not be construed as limited to direct connections only. The terms “coupled” and “connected” and their derivatives may be used. It should be understood that these terms are not intended as synonyms for each other. Accordingly, the scope of the expression device A coupled to device B should not be limited to devices or systems where the output of device A is directly connected to the input of device B. That means that there is a path between the output of A and the input of B that may be a path that includes other devices or means. “Coupled” means that two or more elements are in direct physical or electrical contact, or two or more elements are not in direct contact with each other However, it only needs to mean that they cooperate or interact with each other.

  Thus, while what has been considered as a preferred embodiment of the present invention has been described, those skilled in the art will recognize that other and further modifications may be made thereto, and such changes without departing from the spirit of the present invention. It will be understood that all changes and modifications are intended to be claimed as falling within the scope of the invention. For example, any of the above equations are only representative of procedures that may be used. Functionality may be added or deleted from the block diagram, and operations may be switched within the functional block group. Steps may be added or deleted from the described methods within the scope of the present invention.

In the present specification, a device such as a portable device that has limited resources compared to a fixed processing system is referred to as a “restricted resource device”. As used herein, a base station refers to an access point, cellular base station, or similar wireless transceiver that wirelessly transmits media data to a resource restriction device.
US Pat. No. 5,802,467 describes a wireless communication, command, control and detection system for voice and data transmission and reception. WO 02/27985 describes a system for automating volume control for a radio device based on measured parameters.

Claims (15)

A method of processing media data by processing hardware to improve quality, wherein the media data is media data for rendering by a resource constrained wireless device, the method comprising:
Receiving one or more environmental quantities detected by one or more sensors, wherein the sensors are located far from the resource constraining device but close enough to the resource constraining device, Making the environmental quantity by the sensor an index equivalent to one or more environmental quantities;
At a network node located at or remote from the resource constrained device:
Processing the media data using the environmental quantity to generate processed data, wherein the sensor is in or connected to the network node;
Wirelessly sending the processed output to the resource constrained device for rendering;
A method for rendering or processing and rendering the media data such that the processed output is made available to the resource constrained device. The method of claim 1, wherein the network node includes a base station of a wireless network. The method of claim 1, wherein the processed output includes processed media data for rendering by the resource constrained device. The media data processing is performed in the resource constrained device, and the processed output includes auxiliary data used by the resource constrained device during media data processing in the resource constrained device. The method described. For the media data:
Media data streamed to the resource constrained device or media data interactively streamed to the resource constrained device as part of a two-way communication involving the resource constrained device via the wireless network;
5. A method according to any one of claims 1 to 4, wherein one or more of: are included. Audio data is included in the media data,
The one or more environmental quantities include at least one quantity indicative of an acoustic profile of noise in the environment;
The quality enhancement process includes noise compensation,
The method of claim 5. Audio data is included in the media data,
The one or more environmental quantities include at least one quantity indicative of an acoustic profile of noise in the environment;
The quality enhancement process of the media data includes noise compensation, and the noise compensation is:
Generating modified parameters from the audio data using one or more loudness level parameters and one or more parameters of an acoustic noise profile, wherein the modified parameters are manipulated on information in a perceived loudness domain. Generating by executing, and
Modifying the audio data based on the modification parameter to generate processed audio data;
With
The one or more loudness level parameters include one or more of: whether an audio noise compensation switch is on, a reference level of the resource constrained device, a desired reproduction level, and / or an amount of noise compensation. The method of claim 5. The method of claim 7, wherein the quality enhancement process of the media data includes one or more of automatic gain control, dynamic range compression and / or equalization applied to the audio data. The media data includes media data streamed to the resource constrained device;
The media data includes video data,
The one or more environmental quantities include at least one parameter indicative of illumination in the environment;
The quality improvement process includes a step of changing the contrast and / or brightness of the video data according to one or more parameters serving as indicators of illumination in the environment.
The method of claim 5. The media data includes media data streamed to the resource constrained device;
The media data includes video data,
The one or more environmental quantities include at least one parameter indicative of illumination in the environment;
The resource constraining device comprises a flat panel display device having position dependent backlight elements that are each adjusted according to image dependent adjustment data sent to the resource constraining device along with the video data;
The quality improvement process includes a step of changing a contrast and / or brightness of the video data,
The data processing at the network node includes generating the image dependent adjustment data with at least one of one or more parameters indicative of illumination in the environment.
The method of claim 5. 11. Program logic that, when executed by at least one processor of a processing system, causes the method of any one of claims 1 to 10 to be executed. A computer readable medium containing program logic that, when executed by at least one processor of a processing system, causes the method of any one of claims 1 to 10 to be performed. An apparatus for performing at least a part of a media data quality improvement process, wherein the apparatus:
A network node configured to wirelessly connect to a resource constrained device;
One or more sensors connected to or within the network node, the sensor being provided at a location remote from the resource constraining device but sufficiently close to the resource constraining device, wherein the sensor provides one or more environments A sensor that detects an amount, and causes the environmental amount by the sensor in the vicinity of the resource-constrained device to be an index equivalent to one or more environmental amounts;
Processing hardware connected to or within the network node, comprising:
A process configured to receive the one or more environmental quantities and perform data processing of the media data by generating a processed output using at least a portion of the received environmental quantities to achieve improved quality With hardware,
The network node is further configured to wirelessly send the processed output to the resource constrained device;
An apparatus for rendering or processing and rendering the media data such that the processed output is made available to the resource constrained device. 14. An apparatus according to claim 13, configured to perform the method according to any one of claims 1 to 10. An apparatus comprising a processing system including at least one processor and a storage device,
11. An apparatus comprising program logic, wherein the storage device causes the apparatus to perform the method of any one of claims 1-10.

Images