CN116456247A

CN116456247A - Stereo playback method, apparatus, microphone device, sound box device, and medium

Info

Publication number: CN116456247A
Application number: CN202310200988.5A
Authority: CN
Inventors: 刘益帆; 徐银海; 赵明洲; 丁丹
Original assignee: Anson Chongqing Electronic Technology Co ltd
Current assignee: Anson Chongqing Electronic Technology Co ltd
Priority date: 2023-03-03
Filing date: 2023-03-03
Publication date: 2023-07-18
Also published as: CN117835121A; CN117835121A8

Abstract

Embodiments of the present disclosure disclose stereo playback methods, apparatus, microphone devices, speaker devices, and media. One embodiment of the method comprises the following steps: acquiring a main audio signal group of each channel of corresponding target equipment; determining a target transfer function set, wherein each target transfer function in the target transfer function set corresponds to target sound transmission path information, and the target sound transmission path information represents a sound transmission path between a sound channel and a listening position; determining a secondary audio signal group corresponding to each sound channel according to the primary audio signal group and the target transfer function group; the respective channels are driven to replay the set of superimposed signals corresponding to the set of primary audio signals and the set of secondary audio signals to minimize the target energy received at each of the respective listening positions. The embodiment can improve the degree of distinction between stereo effects heard by listening objects in the same space and at different positions for an acoustic scene with an open ear canal of a user.

Description

Stereo playback method, apparatus, microphone device, sound box device, and medium

Technical Field

Embodiments of the present disclosure relate to the field of audio technology, and in particular, to a stereo playback method, apparatus, microphone device, speaker device, and medium.

Background

Stereo is a sound having a spatially distributed characteristic such as azimuth level. Currently, when implementing stereo playback, the following methods are generally adopted: the method comprises the steps of carrying out distributed layout in a three-dimensional space with a considerable volume through a large number of loudspeakers, or carrying out binaural differential rendering on an original audio stream, and replaying left and right ears of a user through wearable equipment such as Bluetooth headphones, virtual reality head-mounted display equipment and the like to form stereophonic effect.

However, the inventors found that when stereo sound is played back in the above manner, there are often the following technical problems: firstly, realizing independent stereo effect of hearing of listening objects (such as left ear and right ear) at different positions by means of binaural difference rendering of an original audio stream, wherein an auditory canal of a user is required to be closed by wearable equipment, and once the auditory canal and the wearable equipment are in an open state, the stereo effect is seriously weakened, and audio components heard by the left ear and the right ear of the user tend to be consistent; secondly, when a large-scale scene such as a cinema is faced by adopting a mode of carrying out distributed layout in a three-dimensional space with a considerable volume through a large number of loudspeakers, the auditory canal of each user is open, and the audio components received by the listening objects (for example, the users) at different positions are the same, so that the stereo effect distinction degree of hearing of the listening objects at the same space and different positions is poor for the acoustic scene with the auditory canal of the user in the various stereo playback modes.

The above information disclosed in this background section is only for enhancement of understanding of the background of the inventive concept and, therefore, may contain information that does not form the prior art that is already known to those of ordinary skill in the art in this country.

Disclosure of Invention

The disclosure is in part intended to introduce concepts in a simplified form that are further described below in the detailed description. The disclosure is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to be used to limit the scope of the claimed subject matter.

Some embodiments of the present disclosure propose a stereo playback method, apparatus, microphone device, speaker device, and computer readable medium to solve one or more of the technical problems mentioned in the background section above.

In a first aspect, some embodiments of the present disclosure provide a stereo playback method, the method comprising: acquiring a main audio signal group of each channel of corresponding target equipment; determining a target transfer function set, wherein each target transfer function in the target transfer function set corresponds to target sound transmission path information, and the target sound transmission path information represents a sound transmission path between a sound channel and a listening position; determining a secondary audio signal group corresponding to each channel according to the primary audio signal group and the target transfer function group; and driving the audio channels to play back the superposition signal groups corresponding to the main audio signal groups and the auxiliary audio signal groups so as to minimize the target energy received by each listening position in the listening positions, wherein the target energy is the energy of the main audio signal of the channel except for the target channel corresponding to the listening position.

Optionally, before the acquiring the main audio signal group corresponding to each channel of the target device, the method further includes: and acquiring a channel selection information set corresponding to each user, wherein the channel selection information in the channel selection information set comprises channel identifiers, and each channel identifier included in the channel selection information set corresponds to each channel.

Optionally, the target channel corresponds to channel selection information corresponding to a target user; and the determining the target transfer function set includes: acquiring a user interaction voice information set corresponding to each user; generating a user listening position information set according to the user interaction voice information set; and determining a target transfer function set according to the channel selection information set and the user listening position information set.

Optionally, each channel includes a left channel and a right channel, and the target channel is a channel corresponding to the channel and having the same direction as the direction of the listening position; and the determining the target transfer function set includes: acquiring equipment pose information and user voice information of the target equipment; generating user mouth position information according to the user voice information; and determining a target transfer function set according to the equipment pose information and the user mouth position information.

Optionally, the determining the target transfer function set according to the equipment pose information and the user mouth position information includes: generating a channel pose information group corresponding to each channel according to the equipment pose information; generating user binaural position information according to the user mouth position information; and determining a target transfer function set according to the channel pose information set and the user double-ear position information.

Optionally, the determining a target transfer function set according to the channel pose information set and the user binaural position information includes: and selecting a preset transfer function set meeting preset position conditions from a preset transfer function set according to the sound channel pose information set and the user double-ear position information as a target transfer function set.

In a second aspect, some embodiments of the present disclosure provide a stereo playback apparatus, the apparatus comprising: an acquisition unit configured to acquire a main audio signal group of each channel of a corresponding target device; a first determining unit configured to determine a set of target transfer functions, wherein each target transfer function in the set of target transfer functions corresponds to target sound transmission path information, the target sound transmission path information characterizing a sound transmission path between a sound channel and a listening position; a second determining unit configured to determine a sub-audio signal group corresponding to each channel based on the main audio signal group and the target transfer function group; and a playback unit configured to drive the respective channels to play back the superimposed signal group corresponding to the main audio signal group and the sub audio signal group so as to minimize target energy received by each of the respective listening positions, wherein the target energy is energy of a main audio signal of a channel other than a target channel of the corresponding listening position.

Optionally, the stereo playback apparatus may further include: and a channel selection information acquisition unit. Wherein the channel selection information obtaining unit may be configured to obtain a channel selection information set corresponding to each user, wherein channel selection information in the channel selection information set includes channel identifiers, and each channel identifier included in the channel selection information set corresponds to each channel.

Optionally, the target channel corresponds to channel selection information corresponding to a target user. The first determining unit may be further configured to obtain a user interaction voice information set corresponding to the respective users; generating a user listening position information set according to the user interaction voice information set; and determining a target transfer function set according to the channel selection information set and the user listening position information set.

Optionally, the above-mentioned respective channels include a left channel and a right channel. The target channel is a channel corresponding to the channel and having the same direction as the listening position. The first determining unit may be further configured to obtain device pose information and user voice information of the target device; generating user mouth position information according to the user voice information; and determining a target transfer function set according to the equipment pose information and the user mouth position information.

Optionally, the first determining unit may be further configured to generate a channel pose information group corresponding to each channel according to the device pose information; generating user binaural position information according to the user mouth position information; and determining a target transfer function set according to the channel pose information set and the user double-ear position information.

Optionally, the first determining unit may be further configured to select, as the target transfer function set, a set of preset transfer functions satisfying a preset position condition from a set of preset transfer function sets according to the above-mentioned channel pose information set and the above-mentioned user binaural position information.

In a third aspect, some embodiments of the present disclosure provide a microphone apparatus comprising: one or more processors; a storage device having one or more programs stored thereon, a microphone body for a user to hold; at least two speakers provided in the microphone body and corresponding to at least two channels of the microphone device; the microphone assembly is arranged at the upper end of the microphone main body and comprises a microphone array and a shell, and is used for collecting the sound of a user; the sound source port is used for transmitting main audio signal groups corresponding to the at least two channels to the at least two loudspeakers; the one or more programs, when executed by the one or more processors, cause the one or more processors to implement the method described in any of the implementations of the first aspect described above.

In a fourth aspect, some embodiments of the present disclosure provide a sound box apparatus, including: one or more processors; a storage device having one or more programs stored thereon, a housing; at least two speakers, corresponding to at least two sound channels of the sound box equipment, arranged in the box body; the sound source port is used for transmitting main audio signal groups corresponding to the at least two channels to the at least two loudspeakers; the one or more programs, when executed by the one or more processors, cause the one or more processors to implement the method described in any of the implementations of the first aspect described above.

In a fifth aspect, some embodiments of the present disclosure provide a computer readable medium having a computer program stored thereon, wherein the program, when executed by a processor, implements the method described in any of the implementations of the first aspect above.

The above embodiments of the present disclosure have the following advantageous effects: by the stereo playback method of some embodiments of the present disclosure, the degree of distinction between stereo effects heard by listening objects in the same space and at different positions for an acoustic scene in which the user's ear canal is open is improved. For an acoustic scene in which the auditory canal of a user is open, a sound transmission medium (such as air) existing between listening objects in the same space and at different positions is a connectivity medium in the whole space and is not physically isolated, so that the stereo effect heard by each listening object is poor in distinction degree, and remarkable hearing independence is difficult to realize. Specifically: firstly, realizing independent stereo effect of hearing of listening objects (such as left ear and right ear) at different positions by means of binaural difference rendering of an original audio stream, wherein an auditory canal of a user is required to be closed by wearable equipment, and once the auditory canal and the wearable equipment are in an open state, the stereo effect is seriously weakened, and audio components heard by the left ear and the right ear of the user tend to be consistent; secondly, when a large-scale scene such as a cinema is faced by adopting a mode of carrying out distributed layout in a three-dimensional space with a considerable volume through a large number of loudspeakers, the auditory canal of each user is open at the moment, and the audio components received by the listening objects (for example, the users) at different positions are the same, so that the stereo effect distinction degree of hearing of the listening objects at the same space and different positions is poor for the acoustic scene with the auditory canal of the user in the various stereo playback modes. Based on this, the stereo playback method of some embodiments of the present disclosure first acquires a main audio signal group corresponding to each channel of the target device. Thereby, all audio signals that the individual listening object needs to hear can be obtained, and can be used for generating the sub audio signal group. Next, a set of target transfer functions is determined. Wherein, each objective transfer function in the objective transfer function set corresponds to objective sound transmission path information. The target sound transmission path information characterizes a sound transmission path between the sound channel and the listening position. Thus, the respective transfer functions of the respective channels for the respective listening positions can be obtained, and thus can be used to determine the audio signals of all channels heard by each listening position. Then, a set of secondary audio signals corresponding to the respective channels is determined based on the set of primary audio signals and the set of target transfer functions. Finally, the respective channel playback is driven to correspond to the superimposed signal group of the main audio signal group and the sub audio signal group so as to minimize the target energy received by each of the respective listening positions. The target energy is the energy of the main audio signal of the channel other than the target channel corresponding to the listening position. Thus, the main audio component of the target channel corresponding to each listening position in the audio energy received by each listening position can be maximized, and the main audio component of the non-target channel can be minimized, so that the degree of distinction between stereo effects heard by listening objects in the same space and at different positions can be improved for an acoustic scene with an open ear canal of a user. Whether in a miniature scene facing a wearable device or the like, for listening objects in different positions (for example, the left ear and the right ear of a user) or in a large scene facing a cinema or the like, for listening objects in different positions (for example, each user), the audio components received by the open ear canal are different from each other, and remarkable individual hearing independence is realized in a connectivity sound transmission medium.

Drawings

The above and other features, advantages, and aspects of embodiments of the present disclosure will become more apparent by reference to the following detailed description when taken in conjunction with the accompanying drawings. The same or similar reference numbers will be used throughout the drawings to refer to the same or like elements. It should be understood that the figures are schematic and that elements and components are not necessarily drawn to scale.

Fig. 1 is a schematic diagram of one application scenario of a stereo playback method according to some embodiments of the present disclosure;

FIG. 2 is a flow chart of some embodiments of a stereo playback method according to the present disclosure;

FIG. 3 is a flow chart of other embodiments of a stereo playback method according to the present disclosure;

FIG. 4 is a flow chart of still further embodiments of a stereo playback method according to the present disclosure;

fig. 5 is a schematic structural view of some embodiments of a stereo playback device according to the present disclosure;

fig. 6 is a schematic structural diagram of a microphone apparatus suitable for use in implementing some embodiments of the present disclosure;

fig. 7 is a schematic structural diagram of a sound box apparatus suitable for use in implementing some embodiments of the present disclosure.

Detailed Description

Embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. While certain embodiments of the present disclosure are shown in the drawings, it should be understood that the present disclosure may be embodied in various forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided so that this disclosure will be thorough and complete. It should be understood that the drawings and embodiments of the present disclosure are for illustration purposes only and are not intended to limit the scope of the present disclosure.

It should be noted that, for convenience of description, only the portions related to the present invention are shown in the drawings. Embodiments of the present disclosure and features of embodiments may be combined with each other without conflict.

It should be noted that the terms "first," "second," and the like in this disclosure are merely used to distinguish between different devices, modules, or units and are not used to define an order or interdependence of functions performed by the devices, modules, or units.

It should be noted that references to "one", "a plurality" and "a plurality" in this disclosure are intended to be illustrative rather than limiting, and those of ordinary skill in the art will appreciate that "one or more" is intended to be understood as "one or more" unless the context clearly indicates otherwise.

The names of messages or information interacted between the various devices in the embodiments of the present disclosure are for illustrative purposes only and are not intended to limit the scope of such messages or information.

The present disclosure will be described in detail below with reference to the accompanying drawings in conjunction with embodiments.

Fig. 1 is a schematic diagram of one application scenario of a stereo playback method according to some embodiments of the present disclosure.

In the application scenario of fig. 1, first, the computing device 101 may acquire a main audio signal group 102 corresponding to each channel of the target device. Second, the computing device 101 may determine the set of target transfer functions 103. Wherein, each objective transfer function in the objective transfer function set 103 corresponds to objective sound transmission path information. The target sound transmission path information characterizes a sound transmission path between the sound channel and the listening position. The computing device 101 may then determine the set of secondary audio signals 104 corresponding to the respective channels from the set of primary audio signals 102 and the set of target transfer functions 103. Finally, the computing device 101 may drive the respective channel playback of the set of superimposed signals corresponding to the set of primary audio signals 102 and the set of secondary audio signals 104 to minimize the target energy received by each of the respective listening positions. The target energy is the energy of the main audio signal of the channel other than the target channel corresponding to the listening position.

The computing device 101 may be hardware or software. When the computing device is hardware, the computing device may be implemented as a distributed cluster formed by a plurality of servers or terminal devices, or may be implemented as a single server or a single terminal device. When the computing device is embodied as software, it may be installed in the hardware devices listed above. It may be implemented as a plurality of software or software modules, for example, for providing distributed services, or as a single software or software module. The present invention is not particularly limited herein.

It should be understood that the number of computing devices in fig. 1 is merely illustrative. There may be any number of computing devices, as desired for an implementation.

With continued reference to fig. 2, a flow 200 of some embodiments of a stereo playback method according to the present disclosure is shown. The stereo playback method includes the steps of:

step 201, acquiring a main audio signal group corresponding to each channel of a target device.

In some embodiments, an execution subject of the stereo playback method (e.g., the computing device shown in fig. 1) may acquire the main audio signal group corresponding to each channel of the target device from the terminal through a wired connection or a wireless connection. The terminal can be a mobile phone or a computer. The target device may be a speaker device or a microphone device. Each of the channels may include at least two channels. Each of the main audio signals included in the main audio signal group may correspond to one channel. Each of the channels includes a listening object. The above-mentioned listening object may be an object of an audio signal played back by the listening target apparatus. The listening object may be any ear of the user, may be a sound collecting device, or may be the user. The listening object may correspond to a listening position. The channels corresponding to the main audio signals included in the main audio signal group may have a mapping relation with the listening object (listening position) or may be selected by the listening object as a target channel corresponding to the listening position. It should be noted that the wireless connection may include, but is not limited to, 3G/4G connections, wiFi connections, bluetooth connections, wiMAX connections, zigbee connections, UWB (ultra wideband) connections, and other now known or later developed wireless connection means.

Step 202, a set of target transfer functions is determined.

In some embodiments, the execution body may determine the set of target transfer functions. Wherein, each objective transfer function in the objective transfer function set may correspond to objective sound transmission path information. The target transfer function in the target transfer function set may be a transfer function having an audio signal as an input and a sound signal as an output. The target sound transmission path information may characterize a sound transmission path between the sound channel and the listening position. The above-mentioned listening position may include position coordinates of the listening object. The position coordinates may be coordinates in a spatial coordinate system established with the target device as an origin. The sound transmission path may be a path through which sound propagates in the air. The above-mentioned sound transmission path can be represented by a sequence of coordinates. The target sound transmission path information may include, but is not limited to, a channel identification, a listening object identification, and a listening position. The channel identifier may uniquely identify the channel. For example, the channel identifiers may be a first channel and a second channel, or may be a left channel and a right channel. The listening object identification may be a unique identification of the listening object. For example, the listening object identifier may be a left ear, a right ear, a first sound collecting device, a second sound collecting device, or a user identifier. The user identification may be unique to the user. For example, the user identification may be a seat number in which the user is located. In practice, the execution body may determine the preset transfer function set as the target transfer function set. The preset transfer function in the preset transfer function set may be a preset transfer function corresponding to the target sound transmission path information.

In step 203, a set of secondary audio signals corresponding to each channel is determined according to the set of primary audio signals and the set of target transfer functions.

In some embodiments, the executing body may determine a secondary audio signal group corresponding to each channel according to the primary audio signal group and the target transfer function group. The secondary audio signal included in the secondary audio signal group may be an audio signal linearly related to a primary audio signal of at least one channel other than the corresponding channel. In practice, first, the execution body may execute the following steps for each listening object included in the respective listening objects corresponding to the respective channels:

first, a channel having a predetermined mapping relation with the listening object (listening position) or a channel selected by the listening object in real time is determined as a target channel.

And a second step of determining a listening position corresponding to the listening object as a target listening position.

And thirdly, selecting at least one target transfer function meeting the preset function condition from the target transfer function group as a signal transfer function. The preset function condition may be that a listening position identifier included in the target sound transmission path information corresponding to the target transfer function corresponds to the target listening position.

And step four, generating a signal objective function according to the main audio signal group and the selected signal transfer functions. Wherein, the signal objective function can be represented by the following formula:

X _i ＝x _i +Δx _i

where Δx may represent a secondary audio signal. k may represent the target listening position described above. Δx _k The secondary audio signal of the target channel corresponding to the target listening position may be represented. m may represent a target channel. H _mk A signal transfer function corresponding to the first sound transmission path information may be represented. The first sound transmission path information may be target sound transmission path information characterizing a sound transmission path between the target channel and the target listening position. i may representAn i-th channel other than the target channel. X is X _i The superimposed signal corresponding to the i-th channel may be represented. X is X _i The main audio signal of the i-th channel included in the signal group may be represented. Δx _i The secondary audio signal of the i-th channel included in the above-described signal group may be represented. H _ik A signal transfer function corresponding to the second sound transmission path information may be represented. The second sound transmission path information may be target sound transmission path information characterizing a sound transmission path between the i-th channel and the target listening position.

Then, traversing the k value, and solving each generated signal objective function to obtain a secondary audio signal corresponding to the sound channel. In practice, the execution main body can solve each generated signal objective function through an unconstrained optimization algorithm to obtain a secondary audio signal group corresponding to each channel. Wherein, the above-mentioned unconstrained optimization algorithm can be, but is not limited to, one of the following: gradient descent method, least square method.

Step 204, driving each channel to replay the superimposed signal group of the corresponding primary audio signal group and secondary audio signal group to minimize the target energy received by each of the respective listening positions.

In some embodiments, the execution body may drive the superimposed signal group of the main audio signal group and the sub audio signal group corresponding to the respective channel playback so as to minimize the target energy received by each of the respective listening positions. The target energy is the energy of the main audio signal of the channel other than the target channel corresponding to the listening position. The target energy may be represented by a value of the signal objective function.

The above embodiments of the present disclosure have the following advantageous effects: by the stereo playback method of some embodiments of the present disclosure, the degree of distinction between stereo effects heard by listening objects in the same space and at different positions for an acoustic scene in which the user's ear canal is open is improved. For an acoustic scene in which the auditory canal of a user is open, a sound transmission medium (such as air) existing between listening objects in the same space and at different positions is a connectivity medium in the whole space and is not physically isolated, so that the stereo effect heard by each listening object is poor in distinction degree, and remarkable hearing independence is difficult to realize. Specifically: firstly, realizing independent stereo effect of hearing of listening objects (such as left ear and right ear) at different positions by means of binaural difference rendering of an original audio stream, wherein an auditory canal of a user is required to be closed by wearable equipment, and once the auditory canal and the wearable equipment are in an open state, the stereo effect is seriously weakened, and audio components heard by the left ear and the right ear of the user tend to be consistent; secondly, when a large-scale scene such as a cinema is faced by adopting a mode of carrying out distributed layout in a three-dimensional space with a considerable volume through a large number of loudspeakers, the auditory canal of each user is open at the moment, and the audio components received by the listening objects (for example, the users) at different positions are the same, so that the stereo effect distinction degree of hearing of the listening objects at the same space and different positions is poor for the acoustic scene with the auditory canal of the user in the various stereo playback modes. Based on this, the stereo playback method of some embodiments of the present disclosure first acquires a main audio signal group corresponding to each channel of the target device. Thereby, all audio signals that the individual listening object needs to hear can be obtained, and can be used for generating the sub audio signal group. Next, a set of target transfer functions is determined. Wherein, each objective transfer function in the objective transfer function set corresponds to objective sound transmission path information. The target sound transmission path information characterizes a sound transmission path between the sound channel and the listening position. Thus, the respective transfer functions of the respective channels for the respective listening positions can be obtained, and thus can be used to determine the audio signals of all channels heard by each listening position. Then, a set of secondary audio signals corresponding to the respective channels is determined based on the set of primary audio signals and the set of target transfer functions. Finally, the respective channel playback is driven to correspond to the superimposed signal group of the main audio signal group and the sub audio signal group so as to minimize the target energy received by each of the respective listening positions. The target energy is the energy of the main audio signal of the channel other than the target channel corresponding to the listening position. Thus, the main audio component of the target channel corresponding to each listening position in the audio energy received by each listening position can be maximized, and the main audio component of the non-target channel can be minimized, so that the degree of distinction between stereo effects heard by listening objects in the same space and at different positions can be improved for an acoustic scene with an open ear canal of a user. Whether in a miniature scene facing a wearable device or the like, for listening objects in different positions (for example, the left ear and the right ear of a user) or in a large scene facing a cinema or the like, for listening objects in different positions (for example, each user), the audio components received by the open ear canal of the listening objects are different from each other, and remarkable individual hearing independence is realized in a connectivity sound transmission medium.

With further reference to fig. 3, a flow 300 of further embodiments of a stereo playback method is shown. The process 300 of the stereo playback method comprises the steps of:

step 301, a channel selection information set corresponding to each user is acquired.

In some embodiments, the executing entity (e.g., the computing device shown in fig. 1) may obtain the channel selection information set corresponding to each user. The channel selection information in the channel selection information set may be information of a channel selected by a user. The channel selection information in the channel selection information set may include, but is not limited to: channel identification, channel location coordinates. The channel position coordinates may be coordinates in a coordinate system established with the center of the target device as an origin. The users among the users may be in one-to-one correspondence with the channel selection information in the channel selection information set. Each channel identifier included in the channel selection information set corresponds to each channel. The channel identifiers in the respective channel identifiers may be in one-to-one correspondence with the channels in the respective channels. In practice, the above-described execution subject may acquire the channel selection information sets corresponding to the respective users from the terminals at the respective listening positions by means of wired connection or wireless connection.

Step 302, a main audio signal group corresponding to each channel of the target device is acquired.

In some embodiments, the specific implementation of step 302 and the technical effects thereof may refer to step 201 in those embodiments corresponding to fig. 2, which are not described herein.

Step 303, obtaining user interaction voice information sets corresponding to the users.

In some embodiments, the executing entity may obtain a set of user interaction voice information corresponding to the respective users. The correspondence between the user included in each user and the user interaction voice information included in the user interaction voice information set may be one-to-one correspondence. The user interaction voice information included in the user interaction voice information set may be a sound made by the user to the target device. In practice, the executing body may acquire the user interaction voice information sets corresponding to the respective users from the target device through a wired connection or a wireless connection. The target device may perform echo cancellation on the collected acoustic signals, and filter out the audio signals from the target device, so as to obtain the user interaction voice information set.

Step 304, a user listening position information set is generated according to the user interactive voice information set.

In some embodiments, the executing entity may generate the user listening position information set according to the user interactive voice information set. Wherein the user listening position information in the user listening position information set may characterize a listening position of the user. The user listening position information may include, but is not limited to, user listening position coordinates. The user listening position coordinates may be coordinates in a coordinate system established with the target device as an origin. In practice, the executing body may generate the user listening position information according to a sound source localization algorithm for each user interactive voice information included in the user interactive voice information set. The sound source localization algorithm may include, but is not limited to: MVDR (Minimum Variance Distortionless Response, least mean square undistorted response) beamforming algorithm, TDOA (Time Difference Of Arrival ) algorithm.

Step 305, determining a target transfer function set according to the channel selection information set and the user listening position information set.

In some embodiments, the executing body may determine the target transfer function set according to the channel selection information set and the user listening position information set. In practice, for each of the individual users, the execution body may execute the following steps:

And determining channel selection information corresponding to the user in the channel selection information set as target channel selection information.

And a second step of determining the user listening position information corresponding to the user in the user listening position information set as target user listening position information.

And thirdly, selecting a preset transfer function set meeting preset matching conditions from a preset transfer function set to serve as a target transfer function according to the target sound channel selection information and the target user listening position information. The preset matching condition may be that a channel corresponding to a channel identifier included in the target channel selection information is the same as a channel represented by target sound transmission path information corresponding to a preset transfer function, and a distance between a user listening position represented by target user listening position information and a listening position represented by target sound transmission path information corresponding to the preset transfer function is within a preset distance range. The preset distance range may be a preset distance range.

Step 306, determining a secondary audio signal group corresponding to each channel according to the primary audio signal group and the target transfer function group.

Step 307 drives the respective channels to play back the superimposed signal groups corresponding to the primary audio signal groups and the secondary audio signal groups to minimize the target energy received by each of the respective listening positions.

In some embodiments, the specific implementation and the technical effects of steps 306-307 may refer to steps 203-204 in those embodiments corresponding to fig. 2, which are not described herein.

Optionally, the target channel corresponds to channel selection information corresponding to a target user. The channel identifier corresponding to the target channel is the same as the channel identifier included in the channel selection information corresponding to the target user. The target user may listen to the superimposed signal group played back for each channel of the target device for the target user.

As can be seen in fig. 3, the flow 300 of the stereo playback method in some embodiments corresponding to fig. 3 embodies the step of expanding the set of determined target transfer functions as compared to the description of some embodiments corresponding to fig. 2. Therefore, the schemes described in the embodiments can realize that when a plurality of users are in a large three-dimensional sound field environment (e.g. cinema), stereo components heard by the users at different positions are different from each other through respectively performing sound field active control on a plurality of communicated local spaces in the three-dimensional space, so that obvious individual hearing independence is achieved, and user experience is improved.

With further reference to fig. 4, a flow 400 of still further embodiments of a stereo playback method is shown. The process 400 of the stereo playback method comprises the steps of:

step 401, acquiring a main audio signal group corresponding to each channel of the target device.

In some embodiments, the specific implementation of step 401 and the technical effects thereof may refer to step 201 in those embodiments corresponding to fig. 2, which are not described herein.

Alternatively, the respective channels may include a left channel and a right channel.

Step 402, acquiring equipment pose information and user voice information of a target equipment.

In some embodiments, the executing entity (e.g., the computing device shown in fig. 1) may obtain the device pose information and the user voice information of the target device. The device pose information may be pose data of a microphone array in the target device output by the pose sensor. The user voice information may be a user voice obtained by echo cancellation of an acoustic signal collected by the microphone array.

Step 403, generating user mouth position information according to the user voice information.

In some embodiments, the executing body may generate user mouth position information according to the user voice information. Wherein the user mouth position information may characterize the position of the user's mouth. The user mouth position information may be coordinates of the user's mouth in a target three-dimensional coordinate system. The target three-dimensional coordinate system may be a three-dimensional coordinate system with an equivalent center point of the microphone array as an origin, a plane parallel to the face of the user as an x-y plane, and a horizontal plane as an x-z plane. In practice, the executing body may generate the user mouth position information according to the sound source positioning algorithm and the user voice information.

Step 404, determining a target transfer function set according to the equipment pose information and the user mouth position information.

In some embodiments, the executing body may determine the target transfer function set according to the device pose information and the user mouth position information. In practice, first, the executing body may generate the first distance value according to the pose information of the device, the position information of the mouth of the user, and the euclidean distance algorithm. As an example, the execution subject may determine the euclidean distance between the device pose information and the user mouth position information as the first distance value.

Then, a first preset transfer function set satisfying a preset distance condition is selected from a preset first transfer function set as a target transfer function set. The first preset transfer function set included in the first preset transfer function set corresponds to a preset distance value. The preset distance value may be a preset distance value between the microphone array and the mouth of the user. The first predetermined transfer function set may be a predetermined transfer function set. The preset distance condition may be that a difference between the first distance value and a preset distance value corresponding to the first preset transfer function set is smaller than a preset distance threshold. The preset distance threshold may be a preset threshold.

In some optional implementations of some embodiments, the executing entity may further determine the target transfer function set according to the device pose information and the user mouth position information by:

first, generating a channel pose information group corresponding to each channel according to the equipment pose information. The correspondence between the channels included in each channel and the channel pose information included in the channel pose information set may be one-to-one correspondence. The channel pose information in the channel pose information group may represent a position of the channel in the target three-dimensional coordinates. The channel pose information in the channel pose information group may include channel coordinates. In practice, the execution body may input the device pose information into a preset device structure model to obtain a channel pose information group corresponding to each channel. The preset device structure model may be a preset function using device pose information as input and using a channel pose information set corresponding to each channel as output. For example, the function may be a linear function or a nonlinear function. The preset device structure model may be obtained by fitting according to the relative positional relationship between each channel in a large number of target devices and the microphone array.

And a second step of generating user binaural position information according to the user mouth position information. Wherein the user binaural position information may characterize the position of the user ears. The user binaural position information may include left ear position coordinates and right ear position coordinates. The left ear position coordinates may be coordinates of the left ear of the user in the target three-dimensional coordinate system. The right ear position coordinates may be coordinates of the user's right ear in the target three-dimensional coordinate system. In practice, the executing body may input the user mouth position information into a preset ear position information model to obtain user binaural position information. The preset ear position information model may be a preset function using user mouth position information as input and user binaural position information as output. The preset ear position information model may be obtained based on fitting a large number of face samples.

And thirdly, determining a target transfer function set according to the channel pose information set and the user double-ear position information. In practice, the execution body may input the channel pose information set and the user binaural position information into a pre-trained transfer function set generation model, to obtain a transfer function set as a target transfer function set. The transfer function set generating model may be a neural network that takes the channel pose information set and the user binaural position information as inputs and takes the transfer function set as an output. The neural network may be a convolutional neural network or a cyclic neural network.

In some optional implementations of some embodiments, the executing body may select, as the target transfer function set, a set of preset transfer functions that satisfies a preset position condition from a set of preset transfer function sets according to the set of channel pose information and the user binaural position information. The preset transfer function set included in the preset transfer function set may be a preset transfer function set. Each preset transfer function group included in the preset transfer function group set corresponds to a preset distance average value and a preset angle average value. The preset distance average value may be a preset distance average value. The preset angle average value may be a preset angle average value. The preset position condition may be that the first target difference value is less than or equal to a first preset threshold value and the second target difference value is less than or equal to a second preset threshold value. The first target difference may be a difference between a preset distance average value and a distance average value corresponding to a preset transfer function set. The distance average value may be an average value of each distance value of each channel corresponding to the user's ears. The second target difference may be a difference between a preset angle average value and an angle average value corresponding to a preset transfer function set. The average value of angles may be an average value of azimuth angles of the channels corresponding to the ears of the user. The preset angle average value may be a preset angle average value. The first preset threshold and the second preset threshold may be preset thresholds. In practice, first, the above-described execution body may execute the following steps for each channel pose information in the channel pose information group:

And a first step of generating a left ear position distance value and a left ear position azimuth angle according to the Euclidean metric method, the channel coordinates included in the channel pose information and the left ear position coordinates included in the user double ear position information. As an example, the execution body may input the channel coordinates included in the channel pose information and the left ear position coordinates included in the user binaural position information into an euclidean metric formula, and the obtained euclidean distance is used as a left ear position distance value, and the obtained spatial angle is used as a left ear position azimuth angle.

And a second step of generating a right ear position distance value and a right ear position azimuth angle according to the Euclidean metric method, the channel coordinates included in the channel pose information and the right ear position coordinates included in the user double ear position information. As an example, the execution body may input the channel coordinates included in the channel pose information and the right ear position coordinates included in the user binaural position information into an euclidean metric formula, and the obtained euclidean distance is used as a right ear position distance value, and the obtained space angle is used as a right ear position azimuth angle.

Next, the sum of the generated respective left ear position distance values and the generated respective right ear position distance values is determined as a total distance value, and the sum of the generated respective left ear position azimuth angles and the generated respective right ear position azimuth angles is determined as a total angle value. Then, the number of the channel pose information included in the channel pose information group is multiplied by two to obtain a target number value. Then, a ratio of the total distance value to the target number value is determined as a distance average value, and a ratio of the total angle value to the target number value is determined as an angle average value. And finally, selecting a preset transfer function set meeting the preset position condition from the preset transfer function set as a target transfer function set.

Therefore, the target transfer function can be determined according to the real-time distance and the real-time direction between the user and the target equipment, so that the transfer function which is more matched with the actual scene can be obtained, and the distortion of the replayed superimposed signal in the process of being transmitted to the user can be reduced.

Step 405, determining a secondary audio signal group corresponding to each channel according to the primary audio signal group and the target transfer function group.

Step 406 drives the respective channels to play back the superimposed signal sets corresponding to the primary audio signal set and the secondary audio signal set to minimize the target energy received by each of the respective listening positions.

In some embodiments, the specific implementation of steps 405-406 and the technical effects thereof may refer to steps 203-204 in those embodiments corresponding to fig. 2, which are not described herein.

Optionally, the target channel is a channel corresponding to the direction same as the direction of the listening position.

As an example, if the direction corresponding to the channel is the left channel and the direction of the listening position is the left side of the face, the target channel is the left channel. The direction corresponding to the sound channel is the right sound channel, the direction of the listening position is the right side of the face, and the target sound channel is the right sound channel.

As can be seen in fig. 4, the flow 400 of the stereo playback method in some embodiments corresponding to fig. 4 embodies the step of expanding the set of determined target transfer functions as compared to the description of some embodiments corresponding to fig. 2. Therefore, the schemes described in the embodiments can enable the left and right ears of the user to hear different audio components through performing asymmetric sound field active control on the open spaces on the two sides of the face, and enable the binaural hearing of the user to have remarkable independence, so that the user generates sound source space sense based on the binaural effect, and the stereophonic effect and the user experience sense heard by the user can be improved.

With further reference to fig. 5, as an implementation of the method shown in the above figures, the present disclosure provides some embodiments of a stereo playback apparatus, which apparatus embodiments correspond to those shown in fig. 2, which apparatus is particularly applicable in various electronic devices.

As shown in fig. 5, a stereo playback apparatus 500 of some embodiments includes: an acquisition unit 501, a first determination unit 502, a second determination unit 503, and a playback unit 504. Wherein the acquisition unit 501 is configured to acquire a main audio signal group of each channel of the corresponding target device; the first determining unit 502 is configured to determine a set of target transfer functions, wherein each target transfer function in the set of target transfer functions corresponds to target sound transmission path information, which characterizes a sound transmission path between a sound channel and a listening position; the second determining unit 503 is configured to determine a set of sub audio signals corresponding to the respective channels based on the set of main audio signals and the set of target transfer functions; the playback unit 504 is configured to drive the respective channels to play back the superimposed signal group corresponding to the main audio signal group and the sub audio signal group so as to minimize the target energy received by each of the respective listening positions, wherein the target energy is the energy of the main audio signal of the channel other than the target channel of the corresponding listening position.

Optionally, the stereo playback apparatus 500 may further include: a channel selection information acquisition unit (not shown in the figure). Wherein the channel selection information obtaining unit may be configured to obtain a channel selection information set corresponding to each user, wherein channel selection information in the channel selection information set includes channel identifiers, and each channel identifier included in the channel selection information set corresponds to each channel.

Optionally, the target channel corresponds to channel selection information corresponding to a target user. The first determining unit 502 may be further configured to obtain a set of user interaction speech information corresponding to the above-mentioned respective users; generating a user listening position information set according to the user interaction voice information set; and determining a target transfer function set according to the channel selection information set and the user listening position information set.

Optionally, the above-mentioned respective channels include a left channel and a right channel. The target channel is a channel corresponding to the channel and having the same direction as the listening position. The first determining unit 502 may be further configured to obtain the device pose information and the user voice information of the target device; generating user mouth position information according to the user voice information; and determining a target transfer function set according to the equipment pose information and the user mouth position information.

Optionally, the first determining unit 502 may be further configured to generate a channel pose information group corresponding to each channel according to the device pose information; generating user binaural position information according to the user mouth position information; and determining a target transfer function set according to the channel pose information set and the user double-ear position information.

Alternatively, the first determining unit 502 may be further configured to select, as the target transfer function set, a set of preset transfer functions satisfying a preset position condition from a set of preset transfer function sets according to the above-described channel pose information set and the above-described user binaural position information.

It will be appreciated that the elements described in the stereo playback device 500 correspond to the various steps in the method described with reference to fig. 2. Thus, the operations, features and resulting benefits described above with respect to the method are equally applicable to the apparatus 500 and the units contained therein, and are not described in detail herein.

Referring now to fig. 6, a schematic diagram of a microphone apparatus 600 suitable for use in implementing some embodiments of the present disclosure is shown. Microphone devices in some embodiments of the present disclosure may include, but are not limited to, singing microphones. The microphone device shown in fig. 6 is only one example and should not impose any limitation on the functionality and scope of use of the embodiments of the present disclosure.

As shown in fig. 6, the microphone apparatus 600 may include a microphone main body (not shown in the drawing), a processing device 601 (e.g., a central processing unit, a graphic processor, etc.), a memory 602, an input unit 603, and an output unit 604. Wherein, the microphone body can be used for holding by a user. The processing device 601, the memory 602, the input unit 603, and the output unit 604 are connected to each other via a bus 605. Here, a method according to an embodiment of the present disclosure may be implemented as a computer program and stored in the memory 602. The processing means 601 in the microphone device embody the stereo playback method of the present disclosure by calling the above-described computer program stored in the memory 602. In some implementations, the input unit 603 may include a microphone assembly and a source port. The microphone assembly may be disposed at an upper end of the microphone body. The audio port may be an audio interface, and may be configured to transmit a main audio signal group corresponding to the at least two channels to the at least two speakers. The microphone assembly may include a microphone array and a housing for capturing sound of a user. Optionally, the input unit 603 may further include a pose sensor. The pose sensor may be disposed inside a housing included in the microphone assembly, for detecting a pose of the microphone assembly. The output unit 604 may include at least two speakers. The at least two speakers may be disposed in the microphone body and correspond to at least two channels of the microphone device.

Referring now to fig. 7, a schematic diagram of a sound box apparatus 700 suitable for use in implementing some embodiments of the present disclosure is shown. Microphone devices in some embodiments of the present disclosure may include, but are not limited to, portable speakers, car speakers. The sound box apparatus shown in fig. 7 is only one example and should not impose any limitation on the functionality and scope of use of the embodiments of the present disclosure.

As shown in fig. 7, the speaker apparatus 700 may include a cabinet (not shown), a processing device 701 (e.g., a central processing unit, a graphics processor, etc.), a memory 702, an input unit 703, and an output unit 704. The processing device 701, the memory 702, the input unit 703 and the output unit 704 are connected to each other via a bus 705. Here, a method according to an embodiment of the present disclosure may be implemented as a computer program and stored in the memory 702. The processing means 701 in the loudspeaker device embody the stereo playback method of the present disclosure by calling the above-mentioned computer program stored in the memory 702. In some implementations, the input unit 703 may include an audio source port. The audio port may be an audio interface, and may be configured to transmit a main audio signal group corresponding to the at least two channels to the at least two speakers. Optionally, the input unit 703 may further comprise a microphone assembly. The microphone assembly may include a microphone array and a housing for capturing sound of a user. The output unit 704 may include at least two speakers. The at least two speakers may be disposed in the case and correspond to at least two channels of the speaker apparatus.

It should be noted that, the computer readable medium described in some embodiments of the present disclosure may be a computer readable signal medium or a computer readable storage medium, or any combination of the two. The computer readable storage medium can be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or a combination of any of the foregoing. More specific examples of the computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer diskette, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. In some embodiments of the present disclosure, a computer readable storage medium may be any tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device. In some embodiments of the present disclosure, however, the computer-readable signal medium may comprise a data signal propagated in baseband or as part of a carrier wave, with the computer-readable program code embodied therein. Such a propagated data signal may take any of a variety of forms, including, but not limited to, electro-magnetic, optical, or any suitable combination of the foregoing. A computer readable signal medium may also be any computer readable medium that is not a computer readable storage medium and that can communicate, propagate, or transport a program for use by or in connection with an instruction execution system, apparatus, or device. Program code embodied on a computer readable medium may be transmitted using any appropriate medium, including but not limited to: electrical wires, fiber optic cables, RF (radio frequency), and the like, or any suitable combination of the foregoing.

In some implementations, the clients, servers may communicate using any currently known or future developed network protocol, such as HTTP (Hyper Text Transfer Protocol ), and may be interconnected with any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network ("LAN"), a wide area network ("WAN"), the internet (e.g., the internet), and peer-to-peer networks (e.g., ad hoc peer-to-peer networks), as well as any currently known or future developed networks.

The computer readable medium may be contained in the microphone device and the audio box device; or may exist alone without being incorporated into the electronic device. The computer readable medium carries one or more programs which, when executed by the electronic device, cause the electronic device to: acquiring a main audio signal group of each channel of corresponding target equipment; determining a target transfer function set, wherein each target transfer function in the target transfer function set corresponds to target sound transmission path information, and the target sound transmission path information represents a sound transmission path between a sound channel and a listening position; determining a secondary audio signal group corresponding to each channel according to the primary audio signal group and the target transfer function group; and driving the audio channels to play back the superposition signal groups corresponding to the main audio signal groups and the auxiliary audio signal groups so as to minimize the target energy received by each listening position in the listening positions, wherein the target energy is the energy of the main audio signal of the channel except for the target channel corresponding to the listening position.

Computer program code for carrying out operations for some embodiments of the present disclosure may be written in one or more programming languages, including an object oriented programming language such as Java, smalltalk, C ++ and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The program code may execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any kind of network, including a Local Area Network (LAN) or a Wide Area Network (WAN), or may be connected to an external computer (for example, through the Internet using an Internet service provider).

The units described in some embodiments of the present disclosure may be implemented by means of software, or may be implemented by means of hardware. The described units may also be provided in a processor, for example, described as: a processor includes an acquisition unit, a first determination unit, a second determination unit, and a playback unit. The names of these units do not constitute limitations on the unit itself in some cases, and the acquisition unit may also be described as "a unit that acquires a main audio signal group of each channel of the corresponding target device", for example.

The functions described above herein may be performed, at least in part, by one or more hardware logic components. For example, without limitation, exemplary types of hardware logic components that may be used include: a Field Programmable Gate Array (FPGA), an Application Specific Integrated Circuit (ASIC), an Application Specific Standard Product (ASSP), a system on a chip (SOC), a Complex Programmable Logic Device (CPLD), and the like.

The foregoing description is only of the preferred embodiments of the present disclosure and description of the principles of the technology being employed. It will be appreciated by those skilled in the art that the scope of the invention in the embodiments of the present disclosure is not limited to the specific combination of the above technical features, but encompasses other technical features formed by any combination of the above technical features or their equivalents without departing from the spirit of the invention. Such as the above-described features, are mutually substituted with (but not limited to) the features having similar functions disclosed in the embodiments of the present disclosure.

Claims

1. A stereo playback method comprising:

acquiring a main audio signal group of each channel of corresponding target equipment;

determining a target transfer function set, wherein each target transfer function in the target transfer function set corresponds to target sound transmission path information, and the target sound transmission path information represents a sound transmission path between a sound channel and a listening position;

Determining a secondary audio signal group corresponding to each sound channel according to the primary audio signal group and the target transfer function group;

and driving the channels to replay the superposition signal groups corresponding to the main audio signal group and the auxiliary audio signal group so as to minimize the target energy received by each listening position in each listening position, wherein the target energy is the energy of the main audio signal of the channel except for the target channel of the corresponding listening position.

2. The method of claim 1, wherein prior to the acquiring the main set of audio signals corresponding to the respective channels of the target device, the method further comprises:

and acquiring a channel selection information set corresponding to each user, wherein the channel selection information in the channel selection information set comprises channel identifiers, and each channel identifier included in the channel selection information set corresponds to each channel.

3. The method of claim 2, wherein the target channel corresponds to channel selection information corresponding to a target user; and

the determining the target transfer function set includes:

acquiring user interaction voice information sets corresponding to the users;

generating a user listening position information set according to the user interaction voice information set;

And determining a target transfer function set according to the sound channel selection information set and the user listening position information set.

4. The method of claim 1, wherein the respective channels include a left channel and a right channel, and the target channel is a channel corresponding to the channel in the same direction as the listening position; and

the determining the target transfer function set includes:

acquiring equipment pose information and user voice information of the target equipment;

generating user mouth position information according to the user voice information;

and determining a target transfer function set according to the equipment pose information and the user mouth position information.

5. The method of claim 4, wherein the determining a set of target transfer functions from the device pose information and the user mouth position information comprises:

generating a channel pose information group corresponding to each channel according to the equipment pose information;

generating user binaural position information according to the user mouth position information;

and determining a target transfer function set according to the channel pose information set and the user double-ear position information.

6. The method of claim 5, wherein the determining a set of target transfer functions from the set of channel pose information and the user binaural position information comprises:

And selecting a preset transfer function set meeting preset position conditions from a preset transfer function set according to the sound channel pose information set and the user double-ear position information as a target transfer function set.

7. A stereo playback apparatus comprising:

an acquisition unit configured to acquire a main audio signal group of each channel of a corresponding target device;

a first determining unit configured to determine a set of target transfer functions, wherein each target transfer function in the set of target transfer functions corresponds to target sound transmission path information, the target sound transmission path information characterizing a sound transmission path between a sound channel and a listening position;

a second determining unit configured to determine a set of sub audio signals corresponding to the respective channels based on the set of main audio signals and the set of target transfer functions;

and a playback unit configured to drive the respective channels to play back the superimposed signal group corresponding to the main audio signal group and the sub audio signal group so as to minimize target energy received by each of the respective listening positions, wherein the target energy is energy of a main audio signal of a channel other than a target channel of the corresponding listening position.

8. A microphone apparatus, comprising:

one or more processors;

a storage device having one or more programs stored thereon;

a microphone body for a user to hold;

at least two speakers disposed within the microphone body, corresponding to at least two channels of the microphone device;

the microphone assembly is arranged at the upper end of the microphone main body and comprises a microphone array and a shell, and is used for collecting the sound of a user;

a sound source port for transmitting a main audio signal group corresponding to the at least two channels to the at least two speakers;

when executed by the one or more processors, causes the one or more processors to implement the method of any of claims 1-6.

9. A loudspeaker apparatus comprising:

one or more processors;

a storage device having one or more programs stored thereon;

a case;

at least two loudspeakers corresponding to at least two sound channels of the sound box equipment and arranged in the box body;

10. A computer readable medium having stored thereon a computer program, wherein the computer program, when executed by a processor, implements the method of any of claims 1-6.