TW202028929A - Spatial repositioning of multiple audio streams - Google Patents

Abstract

An audio rendering system includes a processor that combines audio input signals with personalized spatial audio transfer functions preferably including room responses. The personalized spatial audio transfer functions are selected from a database having a plurality of candidate transfer function datasets derived from in-ear microphone measurements for a plurality of individuals. Alternatively, the personalized transfer function datasets are derived from actual in-ear measurements of the listener. Foreground and background positions are designated and matched with transfer function pairs from the selected dataset for the foreground and background direction and distance. Two channels of input audio such as voice and music are processed. When a voice communication such as a phone call is accepted the music being rendered is moved from a foreground to a background channel corresponding to a background spatial audio position using the personalized transfer functions. The voice call is simultaneously transferred to the foreground channel.

Description

Spatial relocation of multi-audio streaming

The present invention relates to a method and system for generating audio for presentation through a headset. More particularly, the present invention relates to a data set using a personalized spatial audio transfer function, and uses the personalized spatial audio transfer function to generate spatial audio positions, so as to create more realistic audio rendering through headphones, The spatial audio transfer function has location impulse response information associated with the audio stream and the location of the spatial audio.

Cross-reference of related applications

This application is a reference to incorporate the overall disclosure content from the following previous case: a US patent application filed on January 7, 2018 and titled "Method of Generating Customized Spatial Audio by Head Tracking" Case No. 62/614,482; International Application No. PCT/SG2016/050621 filed on December 28, 2016 and titled "Method for Generating Customized/Personalized Head Related Transfer Functions", Its claim comes from the priority benefit of Singapore Patent Application No. 10201510822Y filed on December 31, 2015 and titled "Method for Generating Customized/Personalized Head Related Transfer Functions". The overall content is Incorporated by reference for all purposes. This application is a reference to further incorporate the overall disclosure content from the following previous cases: US patent application filed on May 2, 2018 and titled "System and Processing Method for Customized Audio Experience" Serial No. 15/969,767; and U.S. Patent Application Serial No. 16/136,211 filed on September 19, 2018 and titled "Method of Generating Customized Spatial Audio by Head Tracking".

Often, users who are listening to music on their mobile phones may want the music to continue uninterrupted when the call comes. Unfortunately, most cell phones are equipped to mute the music when answering the phone. What is needed is an improved system that allows music or other audio to continue uninterrupted when answering the phone, and allows the user to distinguish between two different audio sources.

In order to achieve the foregoing, the present invention proposes in various embodiments a processor and system equipped to provide binaural signals to a headset. The system includes a system for placing audio in the first input audio channel A mechanism in a first position (such as a foreground position), and a mechanism for placing audio in a second input audio channel in a second position (such as a background position).

In some embodiments of the present invention, the system includes a data set of a personalized spatial audio transfer function, and the data set of the personalized spatial audio transfer function has a location pulse associated with the spatial audio location along with at least two audio streams. Response information (such as: HRTF or BRIR). The personalized BRIR for at least two locations is used together with two input audio streams to create a foreground space audio source and a background space audio source to provide a listener with an immersive experience through a headset.

The preferred embodiments of the present invention will be discussed in detail. Examples of preferred embodiments are illustrated in the accompanying drawings. Although the present invention will be described in relation to these preferred embodiments, it will be understood that there is no intention to limit the present invention to the above-described preferred embodiments. On the contrary, it is intended to cover alternatives, modifications, and equivalents that may be included in the spirit and scope of the invention as defined by the scope of the appended application. In the following description, many specific details are set forth in order to provide a thorough understanding of the present invention. The present invention can be implemented without some or all of these specific details. In other cases, well-known mechanisms are not described in detail in order to avoid unnecessarily obscuring the present invention.

It should be pointed out here that the same reference numerals in various drawings refer to the same parts. The various diagrams illustrated and described herein are used to illustrate various features of the present invention. To a certain extent, specific features are described in one schema but not in another, unless otherwise specified or the structure itself prohibits the inclusion of the features. It should be understood that those features can be adapted It may be included in the embodiments represented by the other figures as if it were fully described in those figures. Unless otherwise specified, the drawings need not be to scale. Any scale provided in the drawings is not intended to limit the scope of the present invention but is merely illustrative.

Binaural technology, generally refers to the technology related to or using two ears, which enables users to perceive audio in a three-dimensional field. In some embodiments, this is achieved through the determination and use of the Binaural Room Impulse Response (BRIR) and its associated Binaural Room Transfer Function (BRTF). BRIR simulates the interaction of sound waves from speakers and the listener's ears, head and torso, walls and other objects in the venue. Alternatively, a head related transfer function (HRTF, Head Related Transfer Function) is used in some embodiments. HRTF is a transfer function in the frequency domain, which corresponds to an impulse response representing an interaction in a silent environment. That is, the impulse response here represents the interaction with the sound of the listener's ears, head, and torso.

According to known methods for determining HRTF or BRTF, real or virtual head and binaural microphones are used to record the stereo impulse response (IR, impulse response) for each of several speaker positions in the real place. That is, a pair of impulse responses in which each ear is one is generated for each position, and this pair is called BRIR. Music tracks or other audio streams can then be convolved (filtered) with these BRIRs, and the results are mixed together and played through headphones. If the correct equalization is applied, the music channel will sound like it is played at the speaker position in the location where the BRIR was recorded.

Frequently, a user who is listening to music on a mobile phone may want the music to continue uninterrupted when the call comes. Instead of using the mute function, two separate audio signals (ie, telephone conversation and music) can be fed to the same channel. But in general, it is difficult for people to distinguish sound sources from the same direction. To solve this problem, and according to one embodiment, when an incoming call comes in, the music is directed from the first location to the speaker or channel in the second location (such as the background location), that is, the music and voice communication are placed in different locations. position. Unfortunately, although these methods of locating and displaying audio streams allow separation of sources when using a multi-speaker setup, most voice communications today are made through mobile phones, which are usually not connected to a multi-channel speaker setup. Moreover, even if the above method of multi-channel setting is used, when the audio source is at a position designated by pan, the position is not completely aligned with the actual position of the speaker. The above method sometimes provides lower than optimal results. This is partly due to the difficulty of the listener to accurately locate the spatial audio position when the position is roughly estimated by the traditional translation method to move the perceived audio position to a place between the positions of the multi-channel speaker.

The present invention solves these problems of voice communication through headsets by automatically locating voice calls and music at different spatial audio positions. By using the positions virtualized by using transfer functions (such as by using HRTF), The transfer function at least simulates the audio effects from at least the individual's head, torso, and ears. More preferably, the location effect on audio is considered by BRIR to process audio streams. However, the non-individualized commercial BRIR data set gives most users a poor sense of direction and a worse sense of distance from the sound source. This may cause difficulty in distinguishing the sound source.

To solve these additional problems, in some embodiments, the present invention uses individualized BRIR. In one embodiment, the individualized HRTF or BRIR data set is generated by inserting a microphone into the listener's ear and recording the impulse response during the recording. This is a time-consuming process and may be inconvenient to include in the sale of mobile phones or other audio units. In a further embodiment, the individualized BRIR (or related BRTF) derived from the capture based on the image properties for each individual listener is used, and the sound source of voice and music is located in the first separate (for example: foreground) With the second (eg, background) location, the property is used to determine a suitable individualized BRIR from a database of candidate libraries with individualized spatial audio transfer functions for a plurality of measured individuals. The individualized BRIR corresponding to each of the at least two separate spatial audio locations is preferably used to direct the first and second audio streams to two different spatial audio locations.

Furthermore, since it is known that people can better distinguish between the two sound sources when one is determined by the listener to be closer and the other is determined to be farther away, in some embodiments, the use of extraction is based on In the individualized BRIR derived from the nature of the image, the music is automatically placed at a certain distance in the background space and the voice is placed at a closer distance.

In yet another embodiment, the captured image-based nature is generated by a mobile phone. In another embodiment, when it is determined that the voice call is of lower priority, when receiving a control signal from the listener, such as by actuating a switch, the voice call is directed from the foreground to the background and the music is directed To the foreground. In another embodiment, when it is determined that the voice call is of lower priority and when the control signal from the listener is received, the individualized BRIR corresponding to different distances in the same direction is used, and the apparent distance of the voice call is reduced by Increase and the apparent distance of music is reduced.

Although it should be understood that most of the embodiments herein describe personalized BRIR for headsets, the technology for positioning media streams combined with voice communications can also be implemented according to the steps described in relation to FIG. 3 Extend to any suitable transfer function customized for the user.

It should be understood that the scope of the present invention is intended to cover placing separate first audio sources and voice communications anywhere around the user. Furthermore, the use of the text of the foreground and the background is not intended to be limited to the area in front of the listener or behind the listener, respectively. To be precise, the foreground is interpreted in the most general sense as referring to the more significant or important of the two separated locations, while the background refers to the less significant of the separated locations. Moreover, it should be pointed out that the scope of the present invention exists in a very general sense and uses HRTF or BRIR to direct the first audio stream to the first position and direct the second audio stream to the first position according to the technology described herein. 2. Space audio location. It should be further pointed out that some embodiments of the present invention can apply signal attenuation at the same time, instead of designating a relatively short distance as a foreground position and a relatively distant position as a background position, and extend to areas selected around the user. The position in any direction is either the foreground or the background position. In its simplest form, a filter circuit system applying two pairs of BRIRs to represent the foreground and background positions will be initially shown according to the embodiment of the present invention.

FIG. 1 is a schematic diagram illustrating spatial audio positions for audio processed according to some embodiments of the present invention. Initially, the listener 105 can listen to the first audio signal such as music through the headset 103. Using the BRIR applied to the first audio stream, the listener feels that the first audio stream comes from the first audio location 102. In some embodiments, this is the foreground position. In one embodiment, a technique places this foreground position at zero degrees relative to the listener 105. When a triggering event occurs, such as receiving a phone call in one embodiment, the second stream (for example: voice communication or telephone call) is routed to the first location (102) and the first audio signal is routed to the second location 104. In the illustrated example embodiment, this second position is placed at a 200 degree position, which in some embodiments is described as a less prominent or background position. The 200 degree position was chosen only as a non-limiting example. Placing the audio stream in the second position is preferably achieved by using the BRIR (or BRTF) corresponding to the azimuth, elevation, and distance of the second position for the relevant listener.

In one embodiment, the transition from the first audio stream to the second location (for example, the background) occurs suddenly, without providing the awareness that the first audio stream is moving through the intermediate spatial location. This is graphically depicted by the path 110, which shows no intermediate spatial position. In another embodiment, the audio is positioned at the intermediate points 112 and 114 during a short transition time to provide a sense of direct movement from the foreground position 102 to the background position 104 or an alternative arc-shaped movement. In the preferred embodiment, the BRIR for intermediate positions 112 and 114 is used to spatially locate the audio stream. In an alternative embodiment, the sense of movement is achieved by using BRIR for foreground and background positions, and by panning between those virtual speakers corresponding to those foreground and background positions. In some embodiments, the user can confirm that the voice communication (e.g., phone call) should not be given priority status, and choose to hand over the phone call to a second location (e.g., background location) or even a third location selected by the user , And return the music to the first (for example: foreground) position. In one embodiment, this is performed by sending the audio stream corresponding to the music back to the foreground (first) location 102 and sending the voice communication to the background location 104. In another embodiment, the reordering of the priority order is performed by making the voice call farther away and the music closer to the listener's head 105. This is preferably done by specifying a new HRTF or BRTF for the listener that is captured at different distances, calculated from captured measurements, or interpolated to represent the new distance. For example, in order to increase the priority of music from the background position 104, the apparent distance can be reduced to the spatial audio position 118 or 116. It is better to use the new HRTF or BRTF to handle the reduced distance achieved by the music audio stream, which increases the volume of the music relative to the voice communication signal. In some embodiments, also for the choice of capturing HRTF/BRTF values or interpolation, the voice signal can increase the distance from the listener's head 105 at the same time. Interpolation/calculation can be done using more than 2 points. For example, in order to find a point where two lines (AB and CD) intersect, points A, B, C, and D may be needed for interpolation/calculation.

Alternatively, the spatial audio position where the voice communication is generated can be maintained at a fixed position or increased during the reordering step. In some embodiments, two separate audio streams share equal significance.

In still other embodiments, the user can select at least one of the audio locations in the user interface space for streaming, and more preferably, a single or multiple locations for all streams.

2 is a schematic diagram illustrating a system for simulating audio sources and voice communication at different spatial audio locations according to some embodiments of the present invention. Figure 2 depicts a summary of two different streams (202 and 204) entering the spatial audio positioning system, by using separate pairs of filters for the first spatial audio location (ie: filters 207, 208) and Filters 209 and 210 at the second spatial audio position. After the signals respectively used for the left headset cup are added to the adder 214 and the filtered results of the right headset cup for the headset 216 are similarly added before the adder 215, the gain 222-225 can be applied to all filtered streams. Although this combined hardware module shows the basic principles involved, other embodiments use BRRI or HRTF stored in memory, such as the memory 732 of the audio display module 730 shown in FIG. 3 (such as a mobile phone). In some embodiments, the listener assists in distinguishing the first and second spatial audio positions based on the fact that those spatial audio positions are generated by personally selecting transfer functions that have a place response in addition to the HRTF. In a preferred embodiment, the first and second positions are determined using BRIR customized for the listener.

When the HRTF or BRTF is individualized for the listener, the system and method for display through the headset are optimized, whether the individualization is measured by a direct in-ear microphone, or by combining Individualized BRIR/HRIR data set when no in-ear microphone is used for measurement. According to a preferred embodiment of the present invention, a customized method for generating BRIR is used, which involves extracting from the user based on the image properties, and determining a suitable BRIR from the BRIR candidate library, as outlined in Figure 3 . In more detail, FIG. 3 illustrates a system according to an embodiment of the present invention, which is used to generate an HRTF for customized purposes, obtain listener properties for customization, and select customized listeners for use. HRTF provides a rotating filter suitable for working with the movement of the user’s head, and is used to display audio corrected by BRIR. The capture device 702 is a device equipped to identify and capture the audio-related physical properties of the listener. Although the block 702 can be configured to directly measure those properties (for example, ear height), in a preferred embodiment, the relevant measurement is captured from the user's acquired image to include at least the user's ears or both ears. The processing necessary to capture those properties preferably occurs in the capture device 702, but can also be located elsewhere. For a non-limiting example, the image from the image sensor 704 may be captured by the processor of the remote server 710 after the image is received.

In a preferred embodiment, the image sensor 704 obtains the image of the user's ear, and the processor 706 is configured to capture the relevant properties for the user and send it to the remote server 710. For example, in one embodiment, the Active Shape Model can be used to identify landmarks in the auricle image, and use those landmarks and their geometric relationships and linear distances to identify user-related The property is related to generating a customized BRIR from a set of stored BRIR data sets (ie, from a candidate library of the BRIR data set). In other embodiments, a regression tree model (RGT, Regression Tree Model) is used to capture properties. In still other embodiments, machine learning such as neural networks and other forms of artificial intelligence (AI) are used to capture properties. An example of a neural network is a convolutional neural network. A complete discussion of several methods for identifying the unique physical properties of new listeners is described on December 28, 2016 and titled "Methods for Generating Customized Personalized Head Related Transfer Functions" The application No. PCT/SG2016/050621, the disclosure of which is incorporated herein by reference in its entirety.

The remote server 710 is preferably accessible through a network such as the Internet. The remote server preferably includes a selection processor 710 to access the memory 714 to use the physical properties or other image-related properties captured by the capture device 702 to determine the best matching BRIR data set. The selection processor 712 preferably accesses the memory 714 having a plurality of BRIR data sets. That is, each data set in the candidate library will have a BRIR pair that is better for each point in the azimuth and elevation angle and perhaps the appropriate angle of the head tilt. For example, the measurement can be performed every 3 degrees of the azimuth and elevation angles to generate a BRIR data set for the sampled individuals that constitute the BRIR candidate library.

As discussed earlier, these are preferably derived from in-ear microphone measurements on a moderately sized group (ie: greater than 100 individuals), but can be operated with smaller groups of individuals, and are associated with each Similar images of the BRIR collection are stored with related properties. These can be a spherical grid of BRIR pairs produced partly by direct measurement and partly by interpolation. Even if there is a partially measured/partially interpolated grid, once the appropriate azimuth and elevation values are used to identify the appropriate BRIR pair for a point from the BRIR data set, other points that do not fall on the grid line can be interpolated . For example, any suitable interpolation method can be used, including but not limited to adjacent linear interpolation, bilinear interpolation, and spherical trigonometric interpolation, preferably in the frequency domain.

In one embodiment, each BRIR data set stored in the memory 714 includes at least one complete spherical grid for the listener. In the above situation, any angle in the azimuth (at the horizontal plane surrounding the user, ie at the ear height) or the elevation angle can be selected for placing the sound source. In other embodiments, the BRIR data set is more limited. In one example, it is limited to produce speakers in a venue that conform to conventional stereo settings (ie: +30 degrees and -30 degrees relative to the zero position of the forward straight forward) Place the required BRIR pairs, or in another subset of the full spherical grid, there are no restrictions on the placement of speakers for multi-channel settings, such as 5.1 systems or 7.1 systems.

HRIR is head-related impulse response. It fully describes the sound transmission from source to receiver in the time domain under silent conditions. Most of its information is about the physiology and anthropometry of the person being measured. HRTF is a head-related transfer function. It is equivalent to HRIR, except that it is described in the frequency domain. BRIR is the binaural room impulse response. It is equivalent to HRIR, except that it is measured at the location, and therefore additionally includes the location response for the specific configuration captured in it. BRTF is the frequency domain version of BRIR. It should be understood that in this specification, since BRIR and BRTF are easily interchangeable, and HRIR and HRTF are similarly interchangeable, the embodiments of the present invention are intended to cover those steps that are easy to interchange, even if they are not This is clearly described. So, for example, when the description refers to accessing another BRIR data set, it should be understood that it covers accessing another BRTF.

Figure 3 further depicts the logical relationship of the samples with respect to the data stored in the memory. The memory is shown as including BRIR data sets for several individuals in row 716 (for example: HRTF DS1A, HRTF DS2A, etc.). These are indexed and accessed based on the properties associated with each BRIR data set (preferably image-related properties). The properties of the association shown in row 715 enable the new listener properties to be matched with the properties of the associated BRIR. These properties associated with the BRIT are measured and stored in rows 716, 717, and 718. That is, it functions as an index to the candidate library of the BRIR data set displayed in those rows. Line 717 relates to the storage of BRIR at the reference position zero, which is associated with the rest of the BRIR data set, and when the listener's head rotation is detected and follows the listener's head rotation, it can be combined with the rotation filter for efficient storage and deal with. The further explanation of this option is detailed in the joint trial application No. 16/136,211, which was filed on September 19, 2018 and titled "Methods to Generate Customized Spatial Audio by Head Tracking". It is incorporated herein in its entirety by reference.

In summary, one purpose of accessing the candidate database of BRIR (or HRTF) data sets is to generate customized audio response characteristics (such as BRIR data sets) for individuals. In some embodiments, these are used to process input audio signals such as voice communications and media streams in order to locate the input audio signals for accurate perception of spatial audio associated with the first location and the second location as described above. In some embodiments, generating the customized audio response characteristics such as personalized BRIR includes capturing image-related properties of the individual such as biometric measurement data. For example, the biometric measurement data may include information about the pinna of the ear, the entire ear of the individual, the head, and/or the shoulder. In a further embodiment, such as (1) multiple matching, (2) multiple recognizer types, and (3) cluster-based processing strategies are used to generate intermediate data sets, which are later combined (caused by multiple hits) Situation) to generate a customized BRIR data set for individuals. These can be combined by using a weighted sum between other methods. In the case of a single match, there is no need to combine intermediate results. In one embodiment, the intermediate data set is based at least in part on the proximity of the extracted BRIR data set (from the candidate database) with respect to the matching of the retrieval properties. In other embodiments, multiple identifier matching steps are used, whereby the processor extracts one or more data sets based on a plurality of training parameters corresponding to the biometric measurement data. In still other embodiments, a cluster-based processing strategy is used, whereby the potential data set is clustered based on retrieved data (for example, biometric measurement data). The cluster includes a plurality of data sets having a relationship, where they are clustered or clustered together to form a model of the corresponding BRIR data set that matches the captured data (for example, biometric measurement) from the image.

In some embodiments of the invention, 2 or more distance balls are stored. This refers to spherical grids produced at 2 different distances from the user. In one embodiment, a reference position BRIR is stored and associated with 2 or more different spherical grid distance balls. In other embodiments, each spherical grid will have its own reference BRIR for use in applicable rotating filters. The selection processor 712 is used to match the properties in the memory 714 with the capture properties that are received from the capture device 702 for new listeners. Various methods are used to match the nature of the association so that the correct BRIR data set can be derived. As mentioned above, these methods include multi-match-based processing strategies, multi-identifier processing strategies, cluster-based processing strategies, and as proposed on May 2, 2018 and titled "Used for Customized Audio Experience System and Processing Method" US Patent Application No. 15/969,767 is another method for comparing biometric measurement data. The disclosure of the above US patent application is incorporated herein by reference in its entirety. Row 718 is a set of BRIR data for the group of measured individuals at the second distance. That is, this row announces the BRIR data set recorded by the measuring individual at the second distance. As an example, the first BRIR data set in row 716 can be obtained at a distance of 1.0 m to 1.5 m from the listener, and the BRIR data set in row 718 can refer to those data measured at a distance of 5 m from the listener set. Ideally, the BRIR data set forms a complete spherical grid, but the embodiments of the present invention apply to any and all subsets of the complete spherical grid, including but not limited to subsets of BRIR pairs containing conventional stereo sets, 5.1 multi-channels Settings, 7.1 multi-channel settings, and all other variations and subsets of spherical grids, including BRIR pairs that are every 3 degrees or less in both azimuth and elevation, and those spherical grids whose density is irregular. For example, this might include a spherical grid, where the density of grid points at the front position is much greater than the density of grid points behind the listener. Furthermore, the content configuration in lines 716 and 718 is not only applied to the stored BRIR pairs as derived by the measurement and interpolation method, but also to the BRIR data set by generating a conversion reflecting the former to a BRIR containing a rotating filter The further improvement.

After one or more matching or calculation BRIR data sets are determined, the data sets are sent to the audio display device 730 for storage of the entire BRIR data set determined by matching or other techniques for new listeners as described above Or, in some embodiments, a subset corresponding to the selected spatialized audio position. In one embodiment, the audio display device then selects the BRIR pair for the desired azimuth or elevation angle position and applies it to the input audio signal to provide spatialized audio to the headset 735. In other embodiments, the selected BRIR data set is stored in a separate module coupled to the audio display device 730 and/or the headset 735. In other embodiments, in the case where only limited storage is available for the display device, the display device only stores the identification of the associated property data that best matches the listener or the identification of the best match BRIR data set, and as required The remote server 710 downloads the desired BRIR pair (for the selected azimuth and elevation angle) in real time. As discussed earlier, these BRIR pairs are preferably derived from in-ear microphone measurements on a moderately sized population (ie: greater than 100 individuals) and stored along with similar image-related properties associated with each BRIR data set . Not to obtain all 7200 points, these can be partly generated by direct measurement and partly by interpolation to form a spherical grid of BRIR pairs. Even if there is a partially measured/partially interpolated grid, other points that do not fall on the grid line can be interpolated once the appropriate azimuth and elevation values are used to identify the appropriate BRIR pair for a point from the BRIR data set.

Once the HRTF or BRIR data set selected by the customer is selected for the individual, these individualized transfer functions are used to enable the user or the system to provide at least the first and second spatial audio locations for locating separate media streams and Voice communication. In other words, a pair of transfer functions are used for each of the first and second spatial audio locations to virtually place those streams, and due to their separate spatial audio locations, the listener can focus on his preferred audio stream ( For example: telephone conversation or media streaming). The scope of the present invention is intended to cover all media streams including, but not limited to, audio and music related to video.

Although the foregoing invention has been described in certain details for clear understanding, it will be apparent that certain changes and modifications can be implemented within the scope of the appended patent application. Therefore, the present embodiment will be regarded as illustrative rather than restrictive, and the present invention is not limited to the details given herein, but can be modified within the scope of the appended patent application and equivalents.

102: The first audio position (foreground position) 103: Headphones 104: second position (background position) 105: listener (head) 110: Path 112, 114: middle point (middle position) 116, 118: Spatial audio position 202, 204: Streaming 207, 208: filter 209, 210: filter 214, 215: adder 216: Headphones 222, 223, 224, 225: gain 700: System 702: Capture Device 704: Image Sensor 706: processor 710: remote server 712: select processor 714: memory 715, 716, 717, 718: OK 720: BRIR generation module 730: Audio display module 732: memory 735: Headphones

[Fig. 1] is a schematic diagram illustrating the spatial audio position of audio processed according to some embodiments of the present invention.

[FIG. 2] is a schematic diagram illustrating a system for audio sources (such as from any of several different types of media) and voice communication presented in different spatial audio locations according to some embodiments of the present invention.

[FIG. 3] It is an illustration of an embodiment of the present invention for generating BRIR for customization, obtaining listener properties for customization, selecting a customized BRIR for listeners, and displaying Schematic diagram of the audio system modified by BRIR.

700: System

702: Capture Device

704: Image Sensor

706: processor

710: remote server

712: select processor

714: memory

715, 716, 717, 718: OK

720: BRIR generation module

730: Audio display module

732: memory

735: Headphones

Claims (20)

An audio processing device for processing events by using a spatial audio position transfer function data set, the device comprising: The audio display module is assembled to locate the first and second audio signals respectively including at least one voice communication stream and a media stream at the selected ones of at least the first spatial audio position and the second spatial audio position, said Each of the first and second spatial audio positions is revealed by using respective first and second transfer functions from the spatial audio position transfer function data set; The monitoring module is used to monitor the start of a voice communication event, the event includes receiving a phone call, and at the start of the phone call, by positioning the voice communication to the first spatial audio position And locate the media stream to the second spatial position to process the first and second audio signals; and The output module is equipped to display the synthesized audio signal to the coupled pair of headphones through two output channels. The audio processing device according to claim 1, wherein the spatial audio position transfer function data set is an individualized head related impulse response (HRIR, Head Related Impulse Response) data set for the data set customized by the individual Or one of the individualized Binaural Room Impulse Response (BRIR, Binaural Room Impulse Response) data sets. The audio processing device according to claim 2, which further includes a second processor, which is configured to retrieve the image-based properties for the individual from the input image and is used to transfer the image-based properties to the selection process The selection processor is configured to determine the individualized HRIR or BRIR data set from the memory having a plurality of candidate HRIR or BRIR data sets provided for a group of individuals, the HRIR or BRIR The data sets are each associated with their corresponding image-based properties. The audio processing device according to claim 3, wherein the selection processor accesses the candidate library, compares the captured image-based properties for the individual and the candidate library The properties of the capture determine the individualized BRIR data set, identify one or more BRIR data sets based on the proximity measurement, and the processing strategies used are multi-matching, multi-identifier type, and cluster-based One. The audio processing device according to claim 2, wherein the first and second spatial audio positions from the determined individualized BRIR data set are obtained from a captured data set in the memory by interpolation Method or other calculation methods, and wherein the first and second spatial audio positions include foreground and background positions, respectively. The audio processing device according to claim 5, wherein when the individual listener determines that the voice call is of lower priority and generates a corresponding control signal, the voice call is directed to the background position and the The music is directed to the foreground position. The audio processing device according to claim 2, wherein when the individual listener determines that the voice call is of lower priority and generates the corresponding control signal, the individualized BRIR corresponding to different distances in the same direction is used , The apparent distance of the voice call is increased and the apparent distance of the music is reduced. The audio processing device according to claim 2, wherein from the respective initial positions of the voice communication and the media stream, the positioning of the voice communication to the first spatial audio position, and to the The positioning of the media stream of the second spatial audio position is performed in a burst mode. The audio processing device according to claim 2, which further includes a portable image capturing device equipped to obtain the input image, and wherein the audio processing device obtains the image and captures the image-based One of a mobile phone, a communication device, or a tablet computer. The audio processing device according to claim 1, wherein the audio processing device is configured to relocate the media stream to the first spatial audio position when the voice communication stream is terminated. The audio processing device according to claim 1, wherein the media stream includes music. The audio processing device according to claim 1, wherein respective first and second spatial audio position sound transfer functions from individualized BRIRs corresponding to different distances in the same direction are used, and the apparent distance of the voice call Is increased and the apparent distance of the music is reduced. The audio processing device according to claim 1, wherein the output module is coupled to the headset via one of a wireless connection and a wired connection. The audio processing device of claim 1, wherein the output module includes a digital-to-analog converter, and the coupling to the headset is through an analog port. The audio processing device according to claim 1, wherein the output module is equipped to pass digital signals to the headset, and the headset includes a digital-to-analog converter. The audio processing device according to claim 1, further comprising a user interface configured to select a position for at least one of the first spatial audio position and the second spatial audio position. A method for processing audio streaming to a set of headsets, the method comprising: Locating the first and second audio signals of at least one voice communication stream and a media stream in the selected ones of at least a first spatial audio position and a second spatial audio position, respectively, the first and second spatial audio positions Each of is revealed by using the respective first and second transfer functions from the spatial audio position transfer function data set; Monitor the start of a voice communication event, the event includes receiving a phone call, and at the start of the phone call, by positioning the voice communication to the first spatial audio position and linking the media string The stream is positioned to the second spatial audio location to process the first and second audio signals, wherein at least one associated place impulse response exists for the second spatial audio location; and The synthesized audio is displayed to the headset of the coupling pair through two output channels. The method according to claim 17, wherein the spatial audio position transfer function data set is one of the HRIR data set or the BRIR data set customized for the individual. The method of claim 18, wherein the customization includes extracting image-based properties for the individual from the input image, and transmitting the image-based properties to a selection processor, the selection processor Is assembled to determine an individualized HRIR or BRIR data set from a memory with a plurality of candidate HRIR or BRIR data sets that have been provided for a group of individuals, each of which is associated with its corresponding Image nature. The method according to claim 19, wherein it is determined that the individualized BRIR data set is included in the candidate library to perform interpolation between existing BRIR data sets.

Images