JP2023519596A - Systems and methods for enhancing audio in changing environments - Google Patents

Abstract

Novel methods and systems for generating and using user profiles for speech boosting and audio equalizer adjustments to compensate for various environmental audio conditions. When not ambient audio, synthesized/pre-recorded ambient noise can be mixed with media to generate profiles to simulate noise conditions.

Description

[Cross reference to related applications]
This application is filed April 2, 2020, International Patent Application No. PCT/CN2020/083083, U.S. Provisional Patent Application No. 63/014,502, filed April 23, 2020, and U.S. Provisional Patent Application No. No. 62/125,132, filed Dec. 14, 2020, is claimed. These applications are incorporated herein by reference in their entireties.

[Technical field]
The present disclosure relates to improvements for audio playback of media content. In particular, the present disclosure relates to setting and applying suitable noise compensation for audio of media content played in various environments, especially on mobile devices.

Audio playback of media with dialogue (movies, television shows, etc.) is typically designed to be enjoyed in relatively quiet environments such as homes and theaters. However, it is becoming more common for people to consume such content on the go with their mobile devices. This can affect what an actor says when there is too much environmental noise (vehicle noise, crowds, etc.), or due to the audio quality limitations of mobile hardware or the type of audio playback device used (headphones, etc.). This is a problem because it makes it difficult to understand what is happening.

A common solution is to use noise canceling headphones/earbuds. However, this is an expensive solution and has the drawback of blocking out environmental noises that the user may want to hear (car horns, sirens, yell warnings, etc.).

Various audio processing systems and methods are disclosed herein.

According to a first aspect, for a user of a mobile device, a method of configuring said mobile device for use with environmental noise, comprising:
receiving from the user a case identification of the environmental noise;
receiving from the user a noise level of the environmental noise;
receiving from the user a speech boost level of the environmental noise at the noise level;
receiving from the user a graphic equalizer setting for the ambient noise at the noise level;
playing sample audio of the user from the mobile device while the user sets the dialogue boost level and the graphic equalizer settings;
storing the speech boost level and the graphic equalizer settings of the case identification at noise levels in a profile on the mobile device, wherein the device activates the speech boost when the profile is selected by the user; configured to play audio media using levels and the graphic equalizer settings;
A method is described comprising:

According to a second aspect, a method of adjusting mobile device audio for a user, comprising:
receiving a profile selection from the user, the profile selection relating to at least environmental noise conditions;
receiving from the user a noise level for the environmental noise condition;
reading speech boost levels and graphic equalizer settings from memory at the mobile device;
adjusting the level of the audio using the dialog boost level and the graphic equalizer settings;
A method is described comprising:

Some or all of the methods described herein may be performed by one or more devices according to instructions (eg, software) stored on one or more non-transitory media. Such non-transitory media may include memory devices such as those described herein, including but not limited to random access memory (RAM), read only memory (ROM), and the like. Accordingly, various innovative aspects of the subject matter described in this disclosure can be embodied in non-transitory media containing software. Software may be executable by one or more components of a control system, for example, as disclosed herein. Software may, for example, include instructions for performing one or more of the methods disclosed herein.

At least some aspects of the disclosure can be implemented via a device or devices. For example, one or more devices may be configured to at least partially perform the methods disclosed herein. In some implementations, the device may include an interface system and a control system. The interface system may include one or more network interfaces, one or more interfaces between the control system and the memory system, one or more interfaces between the control system and another device, and/or one or more external It may also include a device interface.

Details of one or more implementations of the subject matter described in this specification are set forth in the accompanying drawings and the description below. Other features, aspects, and advantages will become apparent from the description, drawings, and claims. Relative dimensions in the following drawings may not be drawn to scale. Like reference numbers and designations in different drawings, reference numbers and designations in the various drawings generally indicate similar elements, but different reference numbers do not necessarily indicate different elements between the different drawings. .

4 shows an exemplary flowchart for setting case-specific audio settings.

4 shows an exemplary flow chart for utilizing case-specific audio settings.

4 shows an exemplary flow chart for setting case-specific audio settings including synthesizing environmental noise.

4 shows an exemplary comparison of unconditioned and conditioned audio obtained through subjective test experience.

FIG. 4 is an exemplary frequency response curve of speech boost applied to a codec of a type showing the response curve of an output speech segment; FIG. FIG. 10 is an exemplary frequency response curve of speech boost applied to a codec of one type, showing the response curve of a non-speech segment of the output; FIG.

FIG. 5C is an exemplary frequency response curve of speech boost applied to a codec of a different type than FIGS. 5A and 5B, showing the response curve of the output speech segment; FIG. 5B is an exemplary frequency response curve of speech boost applied to a codec of a different type than FIGS.

Figure 3 shows an exemplary graphical user interface for the method of the present application;

1 illustrates an exemplary hardware/software configuration for the method of the present application;

A solution to the problem of providing clear speech during media playback (audio or audio/visual) in noisy environments (environmental noise) is to reduce noise within a particular user's profile at a particular noise level and type (environment type) Generating and using speech boost and equalizer settings is described herein.

As used herein, the term "mobile device" refers to a user's device that is capable of playing audio and that can be carried and used in multiple locations. Examples include mobile phones, laptop computers, tablet computers, mobile gaming systems, wearable devices, small media players, and the like.

The terms "environmental conditions" or "case" or "case identification" as used herein refer to noisy locations/environments that may or may not interfere with the enjoyment of listening to audio media on a mobile device. represents the category of Examples include homes (eg, "default"), residential areas (eg, walking), public transportation, noisy indoor environments (eg, airports), and the like.

The term "speech boost" refers to the application of general speech amplification of speech components of audio with negligible amplification of non-speech components. For example, speech boost can be implemented as an algorithm that continuously monitors the audio being played, detects the presence of speech, and dynamically applies processing that enhances the clarity of spoken portions of the audio content. . In some embodiments, speech boost analyzes features from the audio signal and applies a pattern recognition system to detect the presence of speech moment by moment. When speech is detected, the speech spectrum is modified as needed to emphasize speech content so that the listener can hear the speech more concisely.

The term "equalization" or "graphic equalizer" or "GED" refers to frequency-based amplitude adjustment of audio. In a true GED, the amplitude setting is set by a slider and the position of the slider corresponds to the frequency range it controls. However, here GED also represents a specific setting that a graphic equalizer can have that gives a specific frequency response curve.

As used herein, the terms "media" or "content" refer to anything that has audio content. This can be music, movies, videos, video games, phone conversations, alarms, and the like. In particular, the systems and methods described herein are most useful for media having a combination of conversational and non-conversational components, but the systems and methods are applicable to any media.

FIG. 1 shows an exemplary flow chart for generating profiles for different environmental conditions (cases). The user selects (110) to initiate setup from the device user interface (UI) and an exemplary playback sample is played (140) for the user. This sample can be selected by the system or by the user. The user selects (141) what volume to play. The user can then automatically or manually (selected by the user) use different environmental noise cases (default, walking, public transport, airport, etc.) (142). The system can utilize a selected subset containing all cases or one selected case. If the selected case is not the default case, the user can set the estimated ambient noise level (125), the speech boost level (130) for their current situation, and in combination give the user the optimal listening experience in their subjective opinion. Enter the graphic equalizer (GEQ) settings (135). These settings 125, 130 can be done multiple times in any order (not necessarily the order shown) and are set based on a sample playback 140 of audio with dialogue components. Once the speech boost and GEQ settings 130, 135 are set according to the user's preferences, they are stored in the profile database/memory 145 for future use. The system can then determine whether all applicable cases have been set (115). If so, the setup ends (150). If not, then the system proceeds to the next case 142 and repeats the setup process for that case. In some embodiments, the saved setting profiles are also indexed (125) based on the injected noise level.

In some embodiments, the speech boost level and/or GEQ settings are each one value from a short list of possible settings. For example, "3" from the range 0-5. In some embodiments, the setting is a real value associated with the setting, eg +10 dB (eg, at a particular frequency range).

FIG. 2 shows an exemplary flow chart for using profiles generated according to the methods described herein. The user initiates their media (205) and selects a case profile that best describes their current situation (210). If the profile is indexed based on ambient noise level, that can also be selected. Next, the system retrieves from database/memory 25 a profile that matches the selected case (and noise level, if applicable) (210). The system then determines (220) whether playback is in a mobility situation (ie, a situation requiring speech boost and GEQ adjustment). This determination can be from user input, device identification, location data, or other means. If the system determines that there is no mobility situation, normal playback/mixing from the media 245 being played occurs (240) when environmental noise 250 is present for the user. This continues until the case profile is changed (255), at which point a new profile is read (210) and processing begins again. In some embodiments, a new mobility state check is performed by or before case switch 255 and the process is simply repeated if mobility conditions exist. If mobility conditions are found 220 to exist, speech boost 230 and GEQ adjustments 235 are applied to mixing 240 to adjust media playback 245 to provide clear speech despite ambient noise 250 .

FIG. 3 shows a profile including the use of synthetic ambient noise for virtual mixing of media and ambient noise (as opposed to the actual mixing of media and actual ambient noise as provided in FIGS. 1 and 2). 4 shows an exemplary flow chart for generating; The system is similar to that of FIG. 1, but to see if the user is generating a profile in a noisy location or presetting cases from a relatively quiet environment (e.g., home). is checked (310). This check can be determined by querying the user or by location services determining that the mobile device is "at home". In some embodiments, the system always assumes that the user is in a relatively quiet environment. If the user is not at the location, the case (environmental noise conditions) is selected 320 by the user or the system. The ambient noise for that case is synthesized (330). In some embodiments, this can be or be based on pre-recorded noise stored in database/memory 340 . This noise is added to the reproduced sample 350 and the noise level 360 can be set by the user for the level the user expects to experience. The simulated noise is thereby adjusted (330). The speech boost level 370 and GEQ level 380 can be set in the same way that at-location setting is performed. The settings are then saved in database/memory 390 for future use. In some embodiments, recorded ambient noise is taken from an ambisonic source and rendered into a binaural format.

FIG. 4 shows examples of perceptual differences that the system can generate and how comparisons made to a reference condition can be used to assess performance. As can be seen, the speech boost is better than the no-boost baseline, improving speech intelligibility and making adjustments beneficial to the user for media where understanding speech is important. For example, FIG. 4 shows that level 2 dialogue enhancement (DE) 415 indicates a high level of user preference 405 and subjective clarity 410 and may therefore be the preferred setting for many users.

Figures 5A and 5B and Figures 6A and 6B show exemplary graphs of speech boost. Figures 5A and 5B show graphs of different dialogue boost settings for the dialogue component of the media. As shown, different settings show different curves. In contrast, Figures 5B and 6B show graphs of the non-speech component of the media, but with the same different speech boost settings. There is negligible difference between curves for different settings (ie speech boost does not boost non-speech components). Figures 5A and 5B represent speech boost levels with smaller level intervals than Figures 6A and 6B. Different curves can be used depending on how noisy the environment is. The louder the environmental noise, the more active the curve in terms of having more dialogue boost throughout the reproduced content. FIG. 5A shows a response curve showing a strong boost in the speech boost level at lower frequencies 505 than at higher frequencies 510 for the speech component of the audio. In contrast, FIG. 6A shows a strong boost at high frequencies 610 compared to low frequencies 605 . In both cases, FIGS. 5B and 6B show that the non-speech components have negligible boost across all frequencies.

FIG. 7 shows an exemplary UI (specifically, in this case a graphical user interface (GUI)) for setting a profile. In mobile device 700, inputs for settings can be presented in a simplified form for ease of use. The noise level control 710 can be presented (in real dB or perceptual steps) as a finite number (eg, 0 to 5) of noise levels, eg, starting at 0 with no noise and increasing evenly as the noise level increases. A speech boost setting 720 can be presented as a graphical slider from no boost to maximum boost. Similarly, the GEQ settings 730 are simplified to a single range of one value for selecting preset GEQ settings (e.g., "bright", "shallow", "deep", etc. tones). (shown here as a slider). Case 740 can be shown as an icon (with or without text). For example, 'default' could be indicated as a house, 'walking' could be indicated with people, 'public transportation' could be indicated as a train or bus, and 'indoor location' could be indicated (to indicate an airport). ) can be shown with the plane. Other cases and icons can be used, and the icon provides the user with a quick reference to the case it represents.

FIG. 8 illustrates an exemplary mobile device architecture implementing features and processes described herein, according to an embodiment. Architecture 800 can be any electronic device including, but not limited to, desktop computers, consumer audio/visual (AV) devices, radio broadcast devices, mobile devices (e.g., smart phones, tablet computers, laptop computers, wearable devices). can be implemented in In the exemplary embodiment shown, architecture 800 is for a smart phone and includes processor 801, peripheral interface 802, audio subsystem 803, speaker 804, microphone 805, sensors 806 (e.g., accelerometer, gyro, barometer, magnetometer, camera), position processor 807 (e.g., GNSS receiver), wireless communication subsystem 808 (e.g., Wi-Fi, Bluetooth, cellular), and touch controller 810 and other input controllers 811. /O subsystem 809 , touch surface 812 , and other input/control devices 813 . Memory interface 814 is coupled to processor 801, peripherals interface 802, and memory 815 (eg, flash, RAM, ROM). Memory 815 stores, without limitation, operating system instructions 816, communication instructions 817, GUI instructions 818, sensor processing instructions 819, telephony instructions 820, electronic messaging instructions 821, web browsing instructions 822, audio processing instructions 823, GNSS/navigation instructions. It stores computer program instructions and data, including 824 and applications/data 825 . Audio processing instructions 823 include instructions for performing the audio processing described herein. Other architectures with more or fewer components can also be used to implement the disclosed embodiments.

The system can be provided as service driven from a remote server, as a standalone program on the device, integrated into a media player application, or included as part of the operating system, such as the audio settings part of the operating system.

A number of embodiments of the disclosure have been described. However, it will be understood that various modifications can be made without departing from the spirit and scope of this disclosure. Accordingly, other aspects are within the scope of the claims.

As described herein, embodiments of the present invention may therefore relate to one or more of the exemplary embodiments listed below. Accordingly, the present invention is described herein including, but not limited to, the following enumerated exemplary embodiments (EEE) that describe the structure, features, and functions of some portions of the invention. It may be embodied in any of the forms.

EEE1: For users of mobile devices, a method of configuring a mobile device for use with environmental noise, comprising:
receiving from the user a case identification of the environmental noise;
receiving from the user a noise level of the environmental noise;
receiving from the user a speech boost level of the environmental noise at the noise level;
receiving from the user a graphic equalizer setting for the ambient noise at the noise level;
playing sample audio of the user from the mobile device while the user sets the dialogue boost level and the graphic equalizer settings;
storing the speech boost level and the graphic equalizer settings of the case identification at noise levels in a profile on the mobile device, wherein the device activates the speech boost when the profile is selected by the user; configured to play audio media using levels and the graphic equalizer settings;
method including.

EEE2: simulating the environmental noise at the noise level;
mixing the simulated environmental noise with the sample audio prior to playing the sample audio;
The method of EEE1 further comprising:

EEE3: The method of EEE2, wherein the simulating step includes reading stored pre-recorded environmental noise from memory.

EEE4: The method of clause 3, wherein said stored pre-recorded environmental noise is in binaural format.

EEE5: Any of EEE1-EEE4, further comprising presenting, on the mobile device, graphical user interface controls for setting the case identification, the noise level, the speech boost level, and the graphic equalizer settings. described method.

EEE6:
The method of any of EEE1-EEE5, wherein the profile corresponds to both the case identification and the noise level.

EEE7: A method of adjusting mobile device audio for a user, comprising:
receiving a profile selection from the user, the profile selection relating to at least environmental noise conditions;
receiving from the user a noise level for the environmental noise condition;
reading speech boost levels and graphic equalizer settings from memory at the mobile device;
adjusting the level of the audio using the dialog boost level and the graphic equalizer settings;
method including.

EEE8: The method of EEE7, further comprising presenting on the mobile device a graphical user interface control for selecting a profile corresponding to environmental noise conditions.

EEE9: A device configured to perform in software or firmware at least one of the methods described in EEE1-EEE8.

EEE10: A non-transitory computer-readable medium that, when read by a computer, instructs said computer to perform at least one of the methods recited in EEE1-EEE8.

EEE11: The device according to EEE9, wherein said device is a telephone.

EEE12: The device of EEE9, wherein the device is at least one of a cellular phone, a laptop computer, a tablet computer, a mobile gaming system, a wearable device, and a small media player.

EEE13: A device according to any of EEE9, EEE11 or EEE12, wherein said software or firmware is part of said device's operating system.

EEE14: A device according to any of EEE9, EEE11, or EEE12, wherein said software or firmware runs a standalone program on said device.

EEE15: The apparatus according to any of EEE1-EEE8, wherein the method is executed by an operating system of the mobile device.

The present disclosure is directed to specific implementations for the purpose of describing some of the innovative aspects described herein and examples of contexts in which these innovative aspects can be implemented. However, the teachings herein can be applied in a variety of different ways. Moreover, the described embodiments may be implemented in various hardware, software, firmware, and the like. For example, aspects of the present application may be implemented, at least in part, in apparatus, systems including more than one device, methods, computer program products, and the like. Accordingly, aspects of the present application may take the form of hardware embodiments, software embodiments (including firmware, resident software, microcode, etc.), and/or embodiments combining both software and hardware aspects. you can Such embodiments may be referred to herein as "circuits," "modules," "devices," "instruments," or "engines." Some aspects of the present application may take the form of a computer program product embodied on one or more non-transitory media having computer-readable program code embodied therein. Such non-transitory media include, for example, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), portable compact disc read-only memory (CD-ROM). ROM), optical storage, magnetic storage, or any suitable combination thereof. Accordingly, the teachings of the present disclosure are not limited to the implementations illustrated and/or described herein, but rather have broad applicability.

Claims (15)

A method, for a user of a mobile device, of configuring the mobile device for use with environmental noise, comprising:
receiving from the user a case identification of the environmental noise;
receiving from the user a noise level of the environmental noise;
receiving from the user a speech boost level of the environmental noise at the noise level;
receiving from the user a graphic equalizer setting for the ambient noise at the noise level;
playing sample audio of the user from the mobile device while the user sets the dialogue boost level and the graphic equalizer settings;
storing the speech boost level and graphic equalizer settings of the case identification at noise levels in a profile on the mobile device, the device displaying the speech boost level when the profile is selected by the user; and playing the audio media using the graphic equalizer settings;
method including. simulating the environmental noise at the noise level;
mixing the simulated environmental noise with the sample audio prior to playing the sample audio;
2. The method of claim 1, further comprising: 3. The method of claim 2, wherein the simulating step comprises reading stored pre-recorded environmental noise from memory. 4. The method of claim 3, wherein the stored prerecorded environmental noise is in binaural format. 5. Any of claims 1-4, further comprising presenting, on the mobile device, graphical user interface controls for setting the case identification, the noise level, the speech boost level, and the graphic equalizer settings. described method. A method according to any preceding claim, wherein said profile corresponds to both said case identification and said noise level. A method of adjusting mobile device audio for a user, comprising:
receiving profile selections from the user, wherein the profile selections relate to at least environmental noise conditions;
receiving from the user a noise level for the environmental noise condition;
reading speech boost levels and graphic equalizer settings from memory at the mobile device;
adjusting the level of the audio using the dialog boost level and the graphic equalizer settings;
method including. 8. The method of claim 7, further comprising presenting on the mobile device a graphical user interface control for selecting a profile corresponding to environmental noise conditions. Apparatus configured to perform in software or firmware at least one of the methods of claims 1-8. A non-transitory computer readable medium that, when read by a computer, instructs the computer to perform at least one of the methods of claims 1-8. 10. The device of Claim 9, wherein the device is a telephone. 10. The device of claim 9, wherein the device is at least one of a cellular phone, laptop computer, tablet computer, mobile gaming system, wearable device, and miniature media player. 13. A device according to any of claims 9, 11 or 12, wherein said software or firmware is part of an operating system of said device. 13. A device according to any of claims 9, 11 or 12, wherein said software or firmware runs a standalone program on said device. A method according to any preceding claim, wherein the method is executed by an operating system of a mobile device.

Images