WO2024027315A1

WO2024027315A1 - Audio processing method and apparatus, electronic device, storage medium, and program product

Info

Publication number: WO2024027315A1
Application number: PCT/CN2023/097184
Authority: WO
Inventors: 秦宇; 谢仁礼
Original assignee: 深圳Tcl数字技术有限公司
Priority date: 2022-08-05
Filing date: 2023-05-30
Publication date: 2024-02-08
Also published as: CN117135557A

Abstract

The embodiments of the present invention disclose an audio processing method and apparatus, an electronic device, a storage medium, and a program product; the method comprises: obtaining from each of at least two external loudspeakers an audio signal to be played, and a target source source angle corresponding to each audio signal to be played; determining a signal transmission angle between each external loudspeaker and the head of a user, and based on the signal transmission angle and the target sound source angle, calculating an anti-crosstalk function corresponding to the signal transmission angle, the anti-crosstalk function being used for eliminating crosstalk generated by the at least two external loudspeakers when playing audio externally; based on the target sound source angle, the signal transmission angle, and the anti-crosstalk function, performing signal transformation on the audio signals to be processed of each external loudspeaker, to obtain target playback audio signals corresponding to audio signals to be played. The invention allows for generating an audio signal capable of representing the orientation of a sound source, and allows for eliminating crosstalk generated when external loudspeakers are used for playing audio, allowing a user to enjoy audio having a better playback effect.

Description

Audio processing methods, devices, electronic equipment, storage media and program products

This disclosure requires the priority of the Chinese patent application with the application number 202210940126.1 and the invention name is "Audio processing method, device, electronic equipment, storage medium and program product" submitted to the China Patent Office on August 5, 2022, and its entire content is approved by This reference is incorporated into this disclosure.

Technical field

The present invention relates to the technical field of audio processing, and specifically to audio processing methods, devices, electronic equipment, storage media and program products.

Background technique

With the current rapid development of economy and technology, people have begun to pursue surround sound listening effects in their daily lives. If you use methods such as arranging additional audio equipment to achieve surround sound effects, it will take a certain amount of time and economic resources.

At present, modern electroacoustic technology can also be used to achieve a stereo effect without adding additional audio equipment. However, with this solution, the audio is generally only processed into two-channel audio data for playback.

technical problem

It is impossible to fully realize the true surround sound effect, and if this solution is used in an external playback scenario, the already limited stereo playback effect will be further weakened due to factors such as crosstalk.

Technical solutions

Embodiments of the present invention provide audio processing methods, devices, electronic equipment, storage media and program products, which can generate audio signals that can express the direction of sound sources without adding audio playback equipment, and eliminate the problem of using external speakers to play audio. The cross-talk generated during playback allows users to enjoy audio with better playback effects.

An embodiment of the present invention provides an audio processing method, including:

Obtain the audio signal to be played from each of the at least two external speakers and the target sound source angle corresponding to each of the audio signals to be played;

Determine the signal transmission angle between each of the external speakers and the user's head;

Based on the signal transmission angle and the target sound source angle, an anti-crosstalk function corresponding to the signal transmission angle is calculated. The anti-crosstalk function is used to eliminate the crossover generated by the at least two external speakers when the audio is externally played. Crosstalk;

Based on the anti-crosstalk function, signal transformation is performed on the audio signals to be processed from each of the external speakers to obtain a target playback audio signal corresponding to the audio signal to be played.

Correspondingly, an embodiment of the present invention provides an audio processing device, including:

A signal acquisition unit configured to acquire the audio signal to be played from each of the at least two external speakers and the target sound source angle corresponding to each of the audio signals to be played;

An angle determination unit, used to determine the signal transmission angle between each of the external speakers and the user's head;

A function calculation unit configured to calculate an anti-crosstalk function corresponding to the signal transmission angle based on the signal transmission angle and the target sound source angle. The anti-crosstalk function is used to eliminate the problem caused by the at least two external loudspeakers being externally placed. When playing audio The resulting crosstalk;

A signal conversion unit, configured to perform signal conversion on the audio signals to be processed from each of the external speakers based on the anti-crosstalk function, to obtain a target playback audio signal corresponding to the audio signal to be played.

Optionally, the function calculation unit is configured to determine, based on the signal transmission angle, the speaker head-related transfer function corresponding to the signal transmission angle;

Based on the target sound source angle, determine the sound source related transfer function corresponding to the target sound source angle;

According to the speaker head related transfer function and the sound source related transfer function, the anti-crosstalk function corresponding to the signal transmission angle is calculated.

Optionally, the at least two external speakers include a left speaker and a right speaker, and the angle determination unit is used to determine the left signal transmission angle between the left speaker and the user's head;

Determine a right signal transmission angle between the right speaker and the user's head;

The function calculation unit is configured to determine, based on the left signal transmission angle, a first left ear head-related transfer function between the left speaker and the user's left ear, and a first left ear head-related transfer function between the left speaker and the user's right ear. First right ear head related transfer function;

Based on the right signal transmission angle, a second left ear head-related transfer function between the right speaker and the user's left ear, and a second right ear head-related transfer function between the right speaker and the user's right ear are determined ;

The first left ear tip related transfer function, the first right ear tip related transfer function, the second left ear tip related transfer function and the second right ear tip related transfer function are used as speaker head related transfer functions.

Optionally, the at least two external speakers include a left speaker and a right speaker, and the angle determination unit is used to determine the positional relationship between the left speaker, the right speaker and the user's head;

If the left speaker and the right speaker have a left-right symmetrical relationship with respect to the user's head, the angle between any external speaker and the user's head is used as the signal transmission angle;

The function calculation unit is configured to determine a first head-related transfer function between the left speaker and the user's left ear and between the right speaker and the user's right ear based on the signal transmission angle, and the left speaker a second head-related transfer function between the user's right ear and between the right speaker and the user's left ear;

The first head-related transfer function and the second head-related transfer function are regarded as speaker head-related transfer functions.

Optionally, the function calculation unit is configured to perform matrix merging processing according to the speaker head related transfer function to obtain the speaker crosstalk matrix corresponding to the audio signal to be processed;

Perform matrix cancellation on the speaker crosstalk matrix to calculate the crosstalk cancellation matrix of the speaker crosstalk matrix;

Based on the crosstalk cancellation matrix and the sound source related transfer function, an anti-crosstalk function corresponding to the signal transmission angle is calculated.

Optionally, the audio processing device provided by the embodiment of the present invention also includes a function processing unit for obtaining a preset discrete head related transfer function;

Perform function approximation processing on the discrete head related transfer function to obtain the target head related transfer function;

The function calculation unit is configured to determine the speaker head-related transfer function corresponding to the signal transmission angle based on the signal transmission angle and the target head-related transfer function;

Based on the target sound source angle and the target head related transfer function, a sound source related transfer function corresponding to the target sound source angle is determined.

Optionally, the signal acquisition unit is used to acquire the audio signal to be played by each of the at least two external speakers;

Position the sound source of the audio signals to be played, and determine the target sound source angle corresponding to each of the audio signals to be played.

Optionally, the signal acquisition unit is configured to acquire the audio signal to be played from each of the at least two external speakers and the video frame to be played corresponding to the audio signal to be played;

Determine the voicing position information of the voicing object from the video frame to be played;

Based on the sounding position information of the sounding object, the target sound source angle corresponding to each of the audio signals to be played is calculated.

Optionally, the video frame to be played includes at least one candidate voicing object. The audio processing device provided by the embodiment of the present invention further includes a voicing object determination unit, used to determine the voicing object corresponding to the audio signal to be played, and obtain the voicing object. Describes the object identification information of the vocal object;

The signal acquisition unit is configured to perform information matching based on the object identification information from the candidate voicing objects included in the video frame to be played, and determine the voicing object;

Obtain the target display area of the utterance object in the video frame to be played, and use the position information of the target display area as the utterance position information of the utterance object.

Optionally, the video frame to be played includes a display area of at least one candidate voicing object, and the signal acquisition unit is configured to perform voicing action detection for each display area in the video frame to be played, respectively. If it is detected that a candidate voicing object in the display area has performed a voicing action, use the candidate voicing object as the voicing object;

Optionally, the audio processing device provided by the embodiment of the present invention further includes a first position determination unit, configured to determine the corresponding sound receiving position information of the user in the video frame to be played in response to the user's position selection operation;

The signal acquisition unit is configured to determine the signal transmission direction between the sounding position and the sound receiving position based on the sounding position information of the sounding object and the sound receiving position information;

According to the signal transmission direction, the target sound source angle corresponding to each audio signal to be played is calculated.

Optionally, the audio processing device provided by the embodiment of the present invention further includes a second position determination unit, configured to determine the reference object corresponding to the user in the video frame to be played in response to the user's reference object selection operation;

Obtain the reference position information of the reference object in the video frame to be played;

The signal acquisition unit is configured to determine a signal transmission angle between the sounding object and the reference object based on the sounding position information of the sounding object and the reference position information, as each audio signal to be played Corresponding purpose Mark the sound source angle.

Optionally, the signal acquisition unit is configured to receive a data packet to be played sent by the signal transmitting end. The data packet to be played is based on the audio signal to be played from each external speaker and the target corresponding to each of the audio signals to be played. The sound source angle is encoded;

The data packet to be played is decoded to obtain the audio signal to be played from each of the at least two external speakers and the target sound source angle corresponding to each of the audio signals to be played.

Optionally, the signal acquisition unit is used to receive the data packet to be played sent by the cloud. The data packet to be played is sent to the cloud by the signal sending end. The data packet to be played is sent by the signal sending end based on The audio signals to be played by each external speaker and the target sound source angle corresponding to each of the audio signals to be played are obtained by encoding;

Optionally, the signal conversion unit is configured to calculate an anti-crosstalk resonance function based on the anti-crosstalk function. The anti-crosstalk resonance function is used to eliminate cross-talk generated by the at least two external speakers when the audio is externally played. The influence of sound and resonance of the external auditory canal of the human ear;

Based on the anti-crosstalk resonance function, signal transformation is performed on the audio signals to be processed from each of the external speakers to obtain a target playback audio signal corresponding to the audio signal to be played.

Optionally, the audio processing device provided by the embodiment of the present invention also includes an adjustment parameter calculation unit for obtaining the playback setting parameters of each of the external speakers and the audio playback preference parameters corresponding to the target user;

Perform audio parameter extraction on the audio signal to be processed to obtain the parameters to be played corresponding to the audio signal to be processed;

Calculate the audio adjustment function corresponding to the audio signal to be processed according to the playback setting parameters, the audio playback preference parameters and the to-be-played parameters;

The signal transformation unit is configured to perform signal transformation on the audio signal to be processed based on the audio adjustment function, the target sound source angle, the signal transmission angle and the anti-crosstalk function to obtain the audio signal to be processed. The target corresponding to the audio signal plays the audio signal.

Optionally, the audio processing device provided by the embodiment of the present invention further includes an audio playback unit, configured to send the target playback audio signal corresponding to each of the audio signals to be played to the target playback audio signal corresponding to each of the audio signals to be played. An external speaker triggers the external speaker to play the corresponding target audio signal.

Correspondingly, embodiments of the present invention also provide an electronic device, including a memory and a processor; the memory stores application programs, and the processor is used to run the application programs in the memory to execute the tasks provided by the embodiments of the present invention. steps in any audio processing method.

Correspondingly, embodiments of the present invention also provide a computer-readable storage medium that stores a plurality of instructions, and the instructions are suitable for loading by the processor to execute any of the instructions provided by the embodiments of the present invention. Steps in an audio processing method.

In addition, embodiments of the present invention also provide a computer program product, including a computer program or instructions, and the computer When the program or instructions are executed by the processor, the steps in any of the audio processing methods provided by the embodiments of the present invention are implemented.

beneficial effects

Using the solution of the embodiment of the present invention, the audio signal to be played by each of the at least two external speakers and the target sound source angle corresponding to each of the audio signals to be played can be obtained, and the angle between each external speaker and the audio signal to be played can be determined. The signal transmission angle between the user's heads. Based on the signal transmission angle and the target sound source angle, the anti-crosstalk function corresponding to the signal transmission angle is calculated. The anti-crosstalk function is used to eliminate the problem caused by the at least two external speakers. The cross-talk generated during audio processing is based on the target sound source angle, the signal transmission angle and the anti-crosstalk function. The audio signal to be processed of each external speaker is signal transformed to obtain the target corresponding to the audio signal to be played. Play the audio signal; because in the embodiment of the present invention, the anti-crosstalk function that can eliminate the cross-talk generated in the open sound field is calculated, and the audio signal to be played is also transformed based on the target sound source angle and the anti-crosstalk function, so that the target Playing audio signals can express sound source position information and offset crosstalk generated during playback. Therefore, it is possible to generate audio signals that can express the sound source position without adding additional audio playback equipment, and eliminate the problem when using external speakers. The cross-talk generated when playing audio allows users to enjoy audio with better playback effects. .

Description of the drawings

In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings needed to be used in the description of the embodiments will be briefly introduced below. Obviously, the drawings in the following description are only some embodiments of the present invention. For those skilled in the art, other drawings can also be obtained based on these drawings without exerting creative efforts.

Figure 1a is a schematic scene diagram of an audio processing method provided by an embodiment of the present invention;

Figure 1b is a schematic diagram of another scenario of the audio processing method provided by an embodiment of the present invention;

Figure 2 is a flow chart of an audio processing method provided by an embodiment of the present invention;

Figure 3 is a schematic diagram of a video frame to be played according to an embodiment of the present invention;

Figure 4 is another schematic diagram of a video frame to be played according to an embodiment of the present invention;

Figure 5 is a schematic diagram of the ideal state and actual state of audio signal transmission provided by the embodiment of the present invention;

Figure 6 is a schematic diagram of the technical implementation of virtual 5.1 surround sound provided by an embodiment of the present invention;

Figure 7 is a schematic structural diagram of an audio processing device provided by an embodiment of the present invention;

Figure 8 is another structural schematic diagram of an audio processing device provided by an embodiment of the present invention;

Figure 9 is a schematic structural diagram of an electronic device provided by an embodiment of the present invention.

Embodiments of the invention

The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only some of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without making creative efforts fall within the scope of protection of the present invention.

Embodiments of the present invention provide an audio processing method, device, electronic equipment and computer-readable storage medium. Specifically, embodiments of the present invention provide an audio processing method suitable for an audio processing device, and the audio processing device can be integrated in an electronic device.

The electronic device may be a terminal or other equipment, including but not limited to mobile terminals and fixed terminals. For example, mobile terminals include but are not limited to smartphones, smart watches, tablets, laptops, smart vehicles, etc., wherein fixed terminals include but are not limited to Desktop computers, smart TVs, etc.

The electronic device can also be a server or other equipment. The server can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers. It can also provide cloud services, cloud databases, cloud computing, and cloud functions. , cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN (Content Delivery Network, content distribution network), and cloud servers for basic cloud computing services such as big data and artificial intelligence platforms, but not Limited to this.

The audio processing method in the embodiment of the present invention can be implemented by the server, or can be implemented by the terminal and the server together.

The method will be described below by taking the audio processing method jointly implemented by the terminal and the server as an example.

As shown in Figure 1a, the audio processing system provided by the embodiment of the present invention may include a signal sending end 10 and a signal receiving end 20; the signal sending end 10 and the signal receiving end 20 are connected through a network, for example, through a wired or wireless network connection, etc., wherein the signal sending end 10 can exist as an electronic device that sends the audio signal to be played and the target sound source angle to the signal receiving end 20 .

In some examples, the signal sending end 10 can be used to receive the user's voice input, generate an audio signal to be played, perform sound source analysis on the audio signal to be played to obtain the target sound source angle, and send the audio signal to be played and the audio signal to the signal receiving end 20. Target sound source angle.

The signal receiving end 20 can be used to obtain the audio signal to be played from each of the at least two external speakers and the target sound source angle corresponding to each of the audio signals to be played, and determine the angle of each external speaker. The signal transmission angle between the speaker and the user's head, based on the signal transmission angle and the target sound source angle, calculates the anti-crosstalk function corresponding to the signal transmission angle, the anti-crosstalk function is used to eliminate the at least The crosstalk generated by the two external speakers when playing audio externally is based on the anti-crosstalk function. The audio signal to be processed of each external speaker is signal transformed to obtain the target corresponding to the audio signal to be played. Play audio signal.

It can be understood that in some embodiments, the audio processing steps performed by the signal receiving end 20 can also be performed by the signal transmitting end 10, and the signal transmitting end 10 can directly send the signal-transformed target playback audio signal to the signal receiving end. 20, etc., the embodiment of the present invention does not limit this.

As shown in Figure 1b, the audio processing system provided by the embodiment of the present invention may include a signal sending end 10, a signal receiving end 20, a cloud 30, etc.; the signal sending end 10, the signal receiving end 20 and the cloud 30 are connected through a network, such as , through wired or wireless network connection, etc., wherein the signal sending end 10 can be used to send the audio signal to be played and the target sound to the cloud 30 Source angle of electronic equipment exists.

In some examples, the signal sending end 10 can be used to receive the user's voice input, generate an audio signal to be played, perform sound source analysis on the audio signal to be played to obtain the target sound source angle, and send the audio signal to be played and the target sound to the cloud 30 source angle.

The cloud 30 can be used to forward the audio signal to be played and the target sound source angle to the signal receiving end 20 .

The signal receiving end 20 can be used to obtain the audio signal to be played by each of the at least two external speakers and the target sound source angle corresponding to each of the audio signals to be played, and determine the audio signal of each external speaker. The signal transmission angle between the speaker and the user's head, based on the signal transmission angle and the target sound source angle, calculates the anti-crosstalk function corresponding to the signal transmission angle, the anti-crosstalk function is used to eliminate the at least The crosstalk generated by the two external speakers when playing audio externally is based on the anti-crosstalk function. The audio signal to be processed of each external speaker is signal transformed to obtain the target corresponding to the audio signal to be played. Play audio signal.

It can be understood that in some embodiments, the audio processing steps performed by the signal receiving end 20 can also be performed by the signal transmitting end 10 or the cloud 30. For example, the signal transmitting end 10 or the cloud 30 can directly play the target after signal transformation. The audio signal is sent to the signal receiving end 20 and so on, which is not limited in the embodiment of the present invention.

Each is explained in detail below. It should be noted that the order of description of the following embodiments does not limit the preferred order of the embodiments.

The embodiments of the present invention will be described from the perspective of an audio processing device. The audio processing device may be integrated in a server or a terminal.

As shown in Figure 2, the specific process of the audio processing method in this embodiment can be as follows:

201. Obtain the audio signal to be played from each of the at least two external speakers and the target sound source angle corresponding to each of the audio signals to be played.

In the embodiment of the present invention, the external speaker is a speaker used for audio playback in an open sound field. Among them, the number of external speakers is at least two.

For example, the open sound field can be used in scenarios such as using TVs, smart speakers and other home appliances to play audio and video, and using car speakers to play audio and video in cars. The external speakers may be speakers installed in electronic devices such as televisions, mobile phones, and car terminals, etc.

The audio signal to be played is the original audio signal obtained by the audio processing device. For example, in a video conferencing scenario, the audio signal to be played may be the voice signal of the speaking user collected by the video conferencing client. Or, in the audio and video playback scenario, the audio signal to be played can be the audio signal obtained from the audio file that needs to be played, etc. wait.

It can be understood that the audio signals to be played by each of the external speakers may be the same or different. For example, an audio file that needs to be played can naturally include audio signals to be played in the left channel and the right channel respectively. Then each external speaker can correspond to the audio signal of the left channel or the audio of the right channel respectively. Signal.

Specifically, the target sound source angle is the angle of the sound source expected to appear to the user during audio playback. For example, during a video call, if the angle between the position of the speaking user in the video screen and the position of the user of the video call application in the video screen is 30°, then the target sound source angle can be 30°.

The target sound source angle corresponding to each audio signal to be played may be the same or different. For example, if the audio signals to be played by each external speaker are the same, the target sound source angle is the same at this time, or, in a scenario where multiple users speak at the same time, different audio signals to be played can be based on the voice input of different users. The signal is obtained. At this time, the target sound source angles corresponding to different audio signals to be played may be different.

Even if the audio signals to be played by the external speakers are different, the target sound source angles corresponding to the different audio signals to be played can be the same. For example, the speech of the same user is divided into audio signals to be played in the left channel and the right channel respectively. At this time, each external speaker can correspond to the audio signal of the left channel or the audio signal of the right channel, but the left channel The audio signal of the channel or the audio signal of the right channel corresponds to the same speaking user, and the corresponding target sound source angle is also the same.

In some optional embodiments, the target sound source angle may need to be obtained by positioning the audio signal to be played. Step 201 may specifically include:

Obtain the audio signal to be played from each of the at least two external speakers;

Among them, sound source location positioning can be achieved using Direction of Arrival (DOA), microphone array system positioning and other technologies. Specifically, the sound source position positioning can be performed at the signal sending end that generates the audio signal to be played. For example, during a video conference, the target sound source angle can be obtained through MIC (microphone) array positioning by the terminal of the user who is speaking.

Alternatively, the sound source position positioning can be performed on the cloud or signal receiving end that obtains the audio signal to be played. For example, the signal receiving end can obtain the target sound source angle based on information such as the time delay between the speaking time of the speaking user and the time when the audio signal to be played is received.

In other optional embodiments, the audio signal to be played may correspond to a video frame, and the target sound source angle may be obtained based on the position of the sounding object in the video frame. Step "Obtain at least two external speakers for each of the "The audio signal to be played by the speaker and the target sound source angle corresponding to each of the audio signals to be played", which may specifically include:

Obtain the audio signal to be played from each of the at least two external speakers and the video frame to be played corresponding to the audio signal to be played;

Wherein, the video frame to be played corresponding to the audio signal to be played can be a video frame synchronized with the audio signal to be played, or the video frame to be played can be a synchronized video frame synchronized with the audio signal to be played and the sequence is located before and/or the synchronized video frame. or the next N video frames.

Specifically, the number of N can be set by technicians according to actual conditions, and this is not limited in the embodiment of the present invention.

For example, the video frame to be played may include a synchronized video frame synchronized with the audio signal to be played and 10 video frames sequentially located after the synchronized video frame. The cloud or the signal receiving end can detect the vocal objects in advance based on the 10 video frames that are sequentially located after the synchronized video frame to improve the speed of audio processing.

The sound position information may be the position of the display area of the sound object in the video frame to be played. Alternatively, the utterance position information may be the virtual position information of the utterance object in the virtual space corresponding to the video frame to be played, and so on.

In some examples, when the audio signal to be played is obtained, it can be determined which object in the video frame to be played corresponds to the audio signal to be played. After determining the corresponding sound-producing object, the sound-producing object can be placed in the video frame to be played according to the sound-producing object. The position of the target sound source angle is calculated. That is, the video frame to be played may include at least one candidate voicing object. Before the step of "determining the voicing position information of the voicing object from the video frame to be played", the audio processing method provided by the embodiment of the present invention may also include :

Determine the sound-emitting object corresponding to the audio signal to be played, and obtain the object identification information of the sound-emitting object.

The object identification information may be identification information that can identify the speaking object, such as the account nickname of the speaking object, a unique identity ID, etc. The embodiment of the present invention does not limit the content and form of the object identification information.

Correspondingly, the step "determine the voicing position information of the voicing object from the video frame to be played" may specifically include:

From the candidate voicing objects included in the video frame to be played, perform information matching based on the object identification information to determine the voicing object;

For example, as shown in Figure 3, three different candidate utterance objects, User 1, User 2 and User 3, are displayed on the smart TV. When user 1 speaks in the video conference, based on the application data of the video conference application, it can be determined that the speaking object corresponding to the audio signal to be played is user 1, and its object identification information can be "user 1".

The smart TV can determine the voicing object corresponding to "User 1" and the target display area of the voicing object in the video frame to be played from each candidate voicing object based on the object identification information of "User 1".

It can be understood that the display area of each candidate sounding object may be fixed. After determining the sounding object corresponding to the audio signal to be played, the position information of the target display area of the sounding object can be obtained. Alternatively, the user can also set the display area of the candidate utterance object by himself. In this case, the position information of the target display area needs to be determined based on the object identification information of the utterance object and the user's personalized setting information.

In other examples, it is possible to determine which object is speaking by detecting whether each object has mouth movements, etc., based on the area where each object in the video frame to be played is located. That is to say, the video frame to be played may include the display area of at least one candidate voicing object, and the step "determining the voicing position information of the voicing object from the video frame to be played" includes:

Perform voice action detection for each display area in the video frame to be played respectively. If it is detected that a candidate voice object in the display area has performed a voice action, use the candidate voice object as the voice object;

Specifically, the utterance action detection may be to only detect whether the mouth of each candidate utterance object moves. Or, in order to avoid misrecognition caused by the mouth movements of the candidate vocal target when they are not speaking, such as pursed lips, coughing, etc., facial muscle recognition can be added on the basis of mouth movement recognition to improve the accuracy of vocal movement detection.

For example, as shown in Figure 3, user 1's mouth moves. At this time, user 1, the candidate utterance target, is the utterance target. At this time, the target display area is the local area where user 1 is located.

In other optional examples, the user can manually select his or her own position. Before the step of "calculating the target sound source angle corresponding to each of the audio signals to be played based on the sound-emitting position information of the sound-emitting object", the present invention implements The audio processing methods provided by the example can also include:

In response to the user's position selection operation, the corresponding sound receiving position information of the user in the video frame to be played is determined.

For example, during a video conference, users can select their position in the video frame to be played, such as the lower left corner of the screen, the center of the screen, etc. The user's position selection operation can be the user's drag operation of the corresponding screen display frame, Click/double-click and other trigger operations in the video conference screen, etc.

Correspondingly, the step "calculate the target sound source angle corresponding to each of the audio signals to be played based on the sounding position information of the sounding object" may specifically include:

Based on the sound-emitting position information of the sound-emitting object and the sound-receiving position information, determine the signal transmission direction between the sound-emitting position and the sound-receiving position;

Among them, the signal transmission direction refers to the direction between the sounding position of the sounding object and the sound receiving position selected by the user.

It can be understood that the position indicated by the sound receiving position information may not be in the video frame to be played. For example, in an online concert scenario, users can pre-select their virtual location in the virtual concert venue. At this time, the virtual position is the sound receiving position corresponding to the video frame to be played in the user's online concert. If the virtual location selected by the user can be displayed in the video screen of the concert, it may or may not be displayed in the video screen of the concert.

Correspondingly, the vocal object is the performer in the concert, and the vocal location information is the performer's location information in the virtual venue.

Optionally, in order to enhance the user's viewing experience when watching audio and video, the user can be allowed to select an object in a certain video to enhance the user's immersive experience. That is, before the step "calculating the target sound source angle corresponding to each of the audio signals to be played based on the sounding position information of the sounding object", the audio processing method provided by the embodiment of the present invention may also include:

In response to the user's reference object selection operation, determine the reference object corresponding to the user in the video frame to be played;

Obtain the reference position information of the reference object in the video frame to be played.

For example, as shown in Figure 4, there are two objects in the video screen displayed by the terminal, and the user can select one of them as a reference object.

Based on the sounding position information of the sounding object and the reference position information, the signal transmission angle between the sounding object and the reference object is determined as the target sound source angle corresponding to each of the audio signals to be played.

It is understood that the position of the reference object and the position of the sounding object may vary. For example, if a drama is played on the terminal, the reference object selected by the user can be the protagonist of the drama, and the speaking object can change at any time. The position of the sounding object and the position of the reference object can be moved.

202. Determine the signal transmission angle between each of the external speakers and the user's head;

Among them, the signal transmission angle is the angle between the external speaker and the user's head. Specifically, when there is only one person in the human body detection range of the external speaker or the terminal where the external speaker is located, the head position of this person can be the user's head position by default.

If there are at least two people in the human body detection range of the external speaker or the terminal where the external speaker is located, the nearest person can be used as the user; or, the face of the logged-in account on the terminal can be compared with the faces of each person. Matching, the successfully matched person will be regarded as the user, etc.

In some embodiments, when the signal sending end and the signal receiving end cannot communicate directly, the data packet to be played can be forwarded to the signal receiving end through the cloud. That is, the step "obtaining the audio signal to be played from each of the at least two external speakers and the target sound source angle corresponding to each of the audio signals to be played" may include:

Receive the data packet to be played sent by the cloud. The data packet to be played is sent to the cloud by the signal sending end. The data packet to be played is sent by the signal sending end based on the audio signal to be played by each external speaker and the audio signal of each external speaker. The target sound source angle corresponding to the audio signal to be played is obtained by encoding;

Among them, the signal sending end can directly perform sound source positioning based on the audio signal to be played to obtain the target sound source angle, and then encode the audio signal to be played and the target sound source angle to obtain the data packet to be played.

Alternatively, the signal sending end can only encode the audio signal to be played, obtain the data packet to be played, and send the data packet to be played to the cloud. After the cloud forwards the data packet to be played to the signal receiving end, the signal receiving end can obtain the data to be played. The package is decoded, and the target sound source angle is determined based on the obtained audio signal to be played.

It is understandable that due to the limited processing capabilities of the signal sending end and the signal receiving end and the fact that the signal sending end and the signal receiving end are too far apart to communicate directly, the audio signal to be played can be obtained in the cloud for processing.

In order to reduce the transmission pressure, the signal sending end can perform encoding processing on the audio signal to be played. The step "obtains the audio signal to be played by each of the external speakers in at least two external speakers and the audio signal corresponding to each of the audio signals to be played. "Target sound source angle", which can include:

Receive the data packet to be played sent by the signal transmitting end, the data packet to be played is encoded based on the audio signal to be played from each external speaker and the target sound source angle corresponding to each of the audio signals to be played;

Decode the data packet to be played to obtain each of the at least two external speakers. The audio signals to be played and the target sound source angle corresponding to each of the audio signals to be played.

Alternatively, the signal sending end can only encode the audio signal to be played, obtain the data packet to be played, send the data packet to be played to the cloud, trigger the cloud to decode the data packet to be played, and determine the target sound source based on the obtained audio signal to be played. angle.

Furthermore, after the cloud obtains the signal transmission angle, it can also directly perform step 203 on the cloud.

203. Based on the signal transmission angle and the target sound source angle, calculate an anti-crosstalk function corresponding to the signal transmission angle. The anti-crosstalk function is used to eliminate the noise generated by the at least two external speakers when playing audio externally. of crosstalk.

As shown in Figure 5, in an ideal state, the sound source E that is virtually at the θ angle (i.e., the target sound source angle) is processed by the left ear HRTF (Head Related Transfer Function) at the θ angle, that is, α0, and is given to the left ear. Ear playback; after processing the HRTF of the right ear at the θ angle, that is, β0, and playing it to the right ear, the user can hear the virtual sound source at the θ angle. In other words, under ideal conditions, the physical pathways of the user's left and right ears are basically isolated, and it is basically impossible for the sound from the left ear to be heard by the right ear, and vice versa.

That is, the ideal virtual azimuth listening signal is:
L＝α ₀ (θ,f)E(f)
R＝β ₀ (θ,f)E(f)

However, in an open sound field scenario, sounds that are only sent to the user's left ear can actually be received by the user's right ear, and sounds that are only sent to the user's right ear can actually be received by the user's left ear. The situation is cross-talk.

That is, the actual listening signal is the result of crosstalk and interference from the position of the external speakers. Taking the external speakers including the left speaker and the right speaker as an example, the unprocessed audio signal heard by the user can be expressed as:
L＝α ₁ (θ ₁ ,f)α ₀ (θ,f)E(f)+β ₂ (θ ₂ ,f)β ₀ (θ,f)E(f)
R＝α ₂ (θ ₂ ,f)β ₀ (θ,f)E(f)+β ₁ (θ ₁ ,f)a ₀ (θ,f)E(f)

Therefore, in order to achieve better virtual sound source effects, it is necessary to eliminate the crosstalk generated during the spatial propagation of signals. The step "calculate the anti-crosstalk function corresponding to the signal transmission angle based on the signal transmission angle and the target sound source angle" may include:

Based on the signal transmission angle, determine the speaker head related transfer function corresponding to the signal transmission angle;

In the actual application process, each external speaker has a spatial position. If the signal that has been processed according to the target sound source angle is directly played by the external speaker, the actual played sound must be multiplied by the distance between the external speaker and the user's head. The HRTF corresponding to the signal transmission angle is the sound that the user actually hears.

Therefore, before calculating the anti-crosstalk function, it is necessary to determine the speaker head-related transfer function corresponding to the external speaker based on the signal transmission angle.

Optionally, taking the at least two external speakers including a left speaker and a right speaker as an example, the step "determining the signal transmission angle between each of the external speakers and the user's head" specifically includes:

Determine the left signal transmission angle between the left speaker and the user's head;

The step "based on the signal transmission angle, determine the speaker head-related transfer function corresponding to the signal transmission angle" may include:

Based on the left signal transmission angle, a first left ear head-related transfer function between the left speaker and the user's left ear, and a first right ear head-related transfer function between the left speaker and the user's right ear are determined ;

It should be noted that the left speaker and the right speaker do not limit the positional relationship between the external speaker and the user. Generally speaking, the left speaker refers to the left speaker among the two speakers, and the right speaker refers to the right speaker among the two speakers.

Specifically, when determining the first left ear head related transfer function, the first right ear head related transfer function, the second left ear head related transfer function and the second right ear head related transfer function, the pre-established head related transfer function can be obtained When applying the library, after determining the head-related transfer function library, according to the left signal transmission angle and the right signal transmission angle, the first left ear head-related transfer function and the first head-related transfer function corresponding to each angle are obtained from the head-related transfer function library. a right ear tip related transfer function, a second left ear tip related transfer function and a second right ear tip related transfer function.

Among them, the left signal transmission angle can have only one angle, that is, the angle between the left speaker and a certain position on the user's head. When determining the first left ear head related transfer function and the first right ear head related transfer function, it can only be based on This perspective is taken from the head-related transfer function library.

Alternatively, in order to improve audio processing accuracy, the left signal transmission angle may include two angles, namely the angle between the left speaker and the user's left ear and the angle between the left speaker and the user's right ear. Correspondingly, when determining the first left ear head related transfer function, it can be obtained from the head related transfer function library according to the angle between the left speaker and the user's left ear; when determining the first right ear head related transfer function, it can be obtained according to the left ear head related transfer function. The angle between the speaker and the user's right ear is obtained from a library of head-related transfer functions.

The signal transmission angle on the right side is similar to the signal transmission angle on the left side, and will not be described again in the embodiment of the present invention.

In some optional embodiments, if the left speaker and the right speaker are left and right symmetrical with respect to the direction of the user's head, at this time, the transfer function related to the speaker head can be further simplified. That is, taking the at least two external speakers including a left speaker and a right speaker as an example, the step "determining the signal transmission angle between each of the external speakers and the user's head" may specifically include:

Determine the positional relationship between the left speaker, the right speaker and the user's head;

If the left speaker and the right speaker have a left-right symmetrical relationship with respect to the user's head, the angle between any external speaker and the user's head is used as the signal transmission angle.

Correspondingly, the step "based on the signal transmission angle, determine the speaker head-related transfer function corresponding to the signal transmission angle" includes:

Based on the signal transmission angle, a first head-related transfer function between the left speaker and the user's left ear and between the right speaker and the user's right ear is determined, and a first head-related transfer function between the left speaker and the user's right ear and the Describe the second head-related transfer function between the right speaker and the user's left ear;

In other words, the HRTF at this time is symmetrical on the same side. The head-related transfer function from the left speaker to the left ear is equal to the head-related transfer function from the right speaker to the right ear. The same is true for the opposite side, that is, α ₁ = α ₂
β ₁ = β ₂

Among them, as shown in Figure 5, α ₁ and β ₁ are the transfer functions from the left speaker with an angle θ1 to the user’s left and right ears, α ₂ and β ₂ are the transfer functions from the right speaker with an angle θ2 to the user’s right and left ears. Head-related transfer function of the ear.

Optionally, an anti-crosstalk function that can eliminate crosstalk can be determined based on the generation rules of crosstalk. The step "calculate the anti-crosstalk function corresponding to the signal transmission angle based on the related transfer function of the speaker head and the related transfer function of the sound source" may specifically include:

Perform matrix merging processing according to the speaker head related transfer function to obtain the speaker crosstalk matrix corresponding to the audio signal to be processed;

Taking the external speakers including the left speaker and the right speaker as an example, the unprocessed audio signals heard by the user are processed by matrix merging according to the relevant transfer function of the speaker head to obtain the matrix representation of the audio signal:

Among them, the speaker crosstalk matrix is:

Furthermore, a crosstalk cancellation matrix can be designed for the speaker crosstalk matrix so that the processed audio signal can eliminate the speaker crosstalk matrix during spatial propagation.

Specifically, the crosstalk cancellation matrix A can be expressed as follows:

At this time, based on the original audio signal to be played, the result of pre-cancellation is performed through matrix A, and then the effect of A is offset during the spatial propagation process, and finally the user can hear the appropriate sound.

The audio signals x and y sent to the left speaker and the right speaker respectively after being processed by the crosstalk cancellation matrix A can be expressed by the following formula:

Right now:

Therefore, the anti-crosstalk function corresponding to the signal transmission angle can be calculated based on the crosstalk cancellation matrix and the sound source related transfer function.

The anti-crosstalk function can be expressed by the following formula:

When the left and right speakers are symmetrical with respect to the user's head, the anti-crosstalk function can be simplified to the following form:

It should be noted that although the foregoing embodiment uses dual speakers as an example for description, the audio processing method of the embodiment of the present invention is also applicable to a multi-speaker scenario. The core problem in a multi-speaker scenario is similar to that of dual speakers, which is the crosstalk sum between external speakers and the additional transfer function of each external speaker. It can be understood that the more external speakers, the greater the interaction between the external speakers. The more severe the crosstalk is, the more complex the problem becomes.

However, in a scenario with multiple external speakers, the crosstalk cancellation matrix A can be calculated based on the principle of superposition of audio signals in the user's left and right ears. Multiply the audio signal to be played by a crosstalk cancellation matrix A to calculate the anti-crosstalk function. Specifically, in a multi-speaker scenario, A is the size of n*m, n is the number of speakers, and m is the number of virtual sound sources.

It is understandable that in practical applications, it may not be possible to obtain accurate angle-corresponding HRTFs using only the HRTF library obtained from actual measurements. Therefore, estimation processing can be performed based on the existing angles and corresponding HRTFs. Before the step "calculate the anti-crosstalk function corresponding to the signal transmission angle according to the related transfer function of the speaker head and the related transfer function of the sound source", the audio processing method provided by the embodiment of the present invention may also include:

Get the preset discrete head related transfer function;

Perform function approximation processing on the discrete head-related transfer function to obtain a target head-related transfer function.

Specifically, function approximation processing can include interpolation methods or curve fitting methods, etc. Technical personnel can choose an appropriate function approximation processing method according to actual application requirements.

Correspondingly, the step "based on the signal transmission angle, determine the speaker head-related transfer function corresponding to the signal transmission angle" may specifically include:

Based on the signal transmission angle and the target head-related transfer function, determine the speaker head-related transfer function corresponding to the signal transmission angle;

Correspondingly, the step "based on the target sound source angle, determine the sound source related transfer function corresponding to the target sound source angle" may specifically include:

Optionally, the data in the preset header-related transmission function library can be obtained from an existing database. For example, the data in the header-related transmission function library in this embodiment can be obtained from the CIPIC database. Furthermore, embodiments of the present invention can perform interpolation processing according to the CIPIC database to obtain a target head-related transfer function that includes HRTFs corresponding to more precise angles.

It is understandable that due to differences in head shapes of different users, the same head-related transfer function library may not be suitable for each user. Therefore, in this embodiment, head-related transmission function libraries of different users can be established in advance. When applying, first determine the target user who wants to receive the target playback audio signal, which can be determined through the information of the account logged in in the terminal or application, the facial feature information of the target user, etc. After determining the target user's head-related transfer function library, the speaker head-related transfer function and the sound source-related transfer function are obtained from the target user's head-related transfer function library.

204. Based on the anti-crosstalk function, perform signal transformation on the to-be-processed audio signals of each of the external speakers to obtain a target playback audio signal corresponding to the to-be-played audio signal.

Specifically, the target playback audio signals x and y sent to the left speaker and the right speaker respectively can be expressed by the following formula:
x＝G _L (θ ₀ ,f)E(f)
y＝G _R (θ ₀ ,f)E(f)

In an optional embodiment, the audio processing method provided by the embodiment of the present invention can be applied to achieve the virtual 5.1 surround sound effect using dual external speakers. As shown in Figure 6, the bass path is not considered, and it is assumed that the dual external speakers and In a 5.1-channel scene, the front left and right speakers have the same angle, that is, 30° and 330° positions. The technology of using dual external speakers to virtualize 5.1 surround sound can be expressed by the following formula:
L′＝L+0.707C+G _L (θ _LS ,f)LS+G _L (θ _RS ,f)RS
R′＝R+0.707C+G _R (θ _LS ,f)LS+G _R (θ _RS ,f)RS

in,

α is the HRTF from the front left speaker to the same side, β is the HRTF from the front left speaker to the opposite side. Ideally, the front left speaker and the front right speaker are symmetrical, so α is also the HRTF from the front right speaker to the same side, and β is also the HRTF from the front right speaker to the opposite side. HRTF.

HL and HR respectively represent the HRTF from the left rear and right rear speakers to both ears at f frequency at different angles.

In the above formula, the two angle parameters required for HL and HR are known in the simulated 5.1 scenario, that is, the positions of the left and right rear speakers, 120° and 240°. The parameters to be determined are 8 transfer functions, That is, 8 functions from four directions to both ears.

That is to say, the target audio signal to be played by the left speaker can include the audio signal to be played that simulates the left front channel, the audio signal to be played that simulates the front channel, and after being processed by the anti-crosstalk function, it can offset the audio signal transmitted from the left speaker to the right. The audio signal of the crosstalk effect from the right ear to the left ear and the audio signal processed by the anti-crosstalk function can offset the crosstalk effect from the right speaker to the left ear.

The content of the target audio signal to be played by the right speaker is similar to the above, and will not be described again here.

In addition, it is assumed that the general human head is symmetrical, so the head-related transfer function is also symmetrical. For example, the head-related transfer functions from the 330° and 30° directions to the left and right ears are symmetrical, so the head-related transfer function can be simplified into 4 , equivalent to:
H _R (θ _RS ,f)=H _L (θ _LS ,f)=α ₂
H _R (θ _LS ,f)=H _L (θ _RS ,f)=β ₂

at this time,

After sorting out Figure 6, it can be expressed as:

In another optional embodiment, the audio processing method provided by the embodiment of the present invention can be applied to achieve the virtual sound field broadening effect of using dual external speakers. The sound field of dual external speakers is widened, which is equivalent to using the front left and front right channels to virtual the rear left and right channels in a 5-channel situation. At the same time, there are no front left, right, and center channels. Therefore, it can be expressed by the following formula :

In practical applications, HRTF is generally collected through sound pickup in the inner auditory canal. That is to say, the HRTF measured by this acquisition method not only includes the effect of air channel transmission outside the ear canal, but also includes the resonance generated by the human ear canal. Impact. If this HRTF is directly applied to the signal, although the user can hear an obvious sense of space, the intermediate frequency will be distorted because the signal actually heard by the user is the result of two human ear resonance effects.

Therefore, an anti-crosstalk resonance function can be designed to eliminate the effects of crosstalk and ear canal resonance. Step "Based on the anti-crosstalk function, perform signal transformation on the audio signal to be processed from each of the external speakers to obtain the target audio signal to be played corresponding to the audio signal to be played", which may specifically include:

Calculate an anti-crosstalk resonance function based on the anti-crosstalk function, the anti-crosstalk resonance function is used to eliminate the impact of cross-talk and resonance of the external auditory canal of the human ear due to the at least two external speakers when the audio is externally played;

Specifically, for the same frequency, the energy ratio and phase relationship of the left and right channels remain unchanged, which will not cause direction error. The anti-crosstalk resonance function can be expressed by the following formula:

Since the denominator of the anti-crosstalk resonance function is equivalent to the average frequency energy spectrum of the left and right channels, it does not affect the phase between frequency points, and the same frequency point of the left and right channels is divided by the same number, which does not affect the energy of the left and right channels at the same frequency point. relationship, and the energy of the transmitted signals of the left and right channels remains unchanged after passing through the anti-crosstalk resonance function. Therefore, through the processing of the anti-crosstalk resonance function, the signal remains stable, and the poles at each frequency point are also eliminated.

Optionally, when performing audio processing, the audio signal to be played can be processed accordingly according to the user's volume setting, etc. Before the step of "conducting signal transformation on the audio signal to be processed based on the anti-crosstalk function", the audio processing method provided by the embodiment of the present invention also includes:

Obtain the playback setting parameters of each of the external speakers and the audio playback preference parameters corresponding to the target user;

According to the playback setting parameters, the audio playback preference parameters and the to-be-played parameters, the audio adjustment function corresponding to the to-be-processed audio signal is calculated.

In this embodiment of the present invention, the target user is a user logged in to the audio playback device or a user currently using the audio playback device.

Among them, the audio playback preference parameters are generated based on the playback effect adjustment operation performed by the target user before determining the audio adjustment parameters. Specifically, the audio playback preference parameters may include but are not limited to audio frequency (pitch) adjustment parameters, audio loudness adjustment parameters, and so on.

Extracting audio parameters from the audio signal to be processed may include audio analysis of the audio signal to be processed to obtain specific parameters such as frequency, number of sampling bits, number of channels, loudness, bit rate, etc. corresponding to the audio signal to be processed; or , or feature extraction of the audio signal to be processed is performed to obtain a feature vector corresponding to the audio signal to be processed, which feature vector can represent the parameter characteristics of the audio signal to be processed, and so on.

Correspondingly, the step "based on the anti-crosstalk function, perform signal transformation on the audio signal to be processed to obtain the target playback audio signal corresponding to the audio signal to be processed" includes:

Based on the audio adjustment function, the target sound source angle, the signal transmission angle and the anti-crosstalk function, the audio signal to be processed is signal transformed to obtain the target playback audio signal corresponding to the audio signal to be processed. .

The process of calculating the audio adjustment function corresponding to the audio signal to be processed may be to perform operations such as amplifying or reducing the parameters to be played, so that the target playback audio signal processed by the audio adjustment function is played based on the playback setting parameters. , the audio playback preference parameters can be met. Alternatively, the parameters to be played in the form of matrices or vectors can also be convolved or multiplied, etc.

In some optional embodiments, the audio processing method provided by the embodiment of the present invention may also include:

The target playback audio signal corresponding to each of the audio signals to be played is sent to the external speaker corresponding to each of the audio signals to be played, and the external speaker is triggered to play the corresponding target playback audio signal.

That is to say, after the target playback audio signal is obtained through processing in the cloud or the signal receiving end, the target playback audio signal can be sent to the corresponding external speaker for playback.

It can be seen from the above that the embodiment of the present invention can obtain the audio signal to be played from each of the at least two external speakers and the target sound source angle corresponding to each audio signal to be played, and determine the distance between each external speaker and the user's head. Based on the signal transmission angle and the target sound source angle, the anti-crosstalk function corresponding to the signal transmission angle is calculated. The anti-crosstalk function is used to eliminate the crosstalk generated by at least two external speakers when the audio is played externally. Based on the target sound source angle, signal transmission angle and anti-crosstalk function, signal transformation is performed on the audio signal to be processed of each external speaker to obtain the target playback audio signal corresponding to the audio signal to be played; because in the embodiment of the present invention, the The anti-crosstalk function can eliminate the cross-talk generated in the open sound field. It also transforms the audio signal to be played based on the target sound source angle and the anti-crosstalk function, so that the target audio signal to be played can express the sound source position information and offset the sound source position information during playback. Therefore, it is possible to generate an audio signal that can express the direction of the sound source without adding additional audio playback equipment, and eliminate the cross-talk generated when using external speakers to play audio, so that users can enjoy Play better audio.

In order to better implement the above method, correspondingly, an embodiment of the present invention also provides an audio processing device.

Referring to Figure 7, the device includes:

The signal acquisition unit 701 is used to acquire the audio signal to be played by each of the at least two external speakers and the target sound source angle corresponding to each of the audio signals to be played;

Angle determination unit 702, used to determine the signal transmission angle between each of the external speakers and the user's head;

Function calculation unit 703, configured to calculate an anti-crosstalk function corresponding to the signal transmission angle based on the signal transmission angle and the target sound source angle. The anti-crosstalk function is used to eliminate the interference caused by the at least two external speakers. Crosstalk generated when playing audio out;

The signal conversion unit 704 is configured to perform signal conversion on the audio signals to be processed from each of the external speakers based on the anti-crosstalk function to obtain a target playback audio signal corresponding to the audio signal to be played.

Optionally, the function calculation unit 703 is configured to determine, based on the signal transmission angle, the speaker head-related transfer function corresponding to the signal transmission angle;

Optionally, the at least two external speakers include a left speaker and a right speaker, and the angle determination unit 702 is used to determine the left signal transmission angle between the left speaker and the user's head;

Optionally, the at least two external speakers include a left speaker and a right speaker, and the angle determination unit 702 is used to determine the positional relationship between the left speaker, the right speaker and the user's head;

The function calculation unit 703 is configured to determine a first head-related transfer function between the left speaker and the user's left ear and between the right speaker and the user's right ear based on the signal transmission angle, and the left a second head-related transfer function between the speaker and the user's right ear and between the right speaker and the user's left ear;

Optionally, the function calculation unit 703 is configured to perform matrix merging processing according to the speaker head related transfer function to obtain the speaker crosstalk matrix corresponding to the audio signal to be processed;

Based on the crosstalk cancellation matrix and the sound source related transfer function, the immunity corresponding to the signal transmission angle is calculated. crosstalk function.

Optionally, as shown in Figure 8, the audio processing device provided by the embodiment of the present invention also includes a function processing unit 705, used to obtain a preset discrete head related transfer function;

Optionally, the signal acquisition unit 701 is used to acquire the audio signal to be played by each of the at least two external speakers;

Optionally, the signal acquisition unit 701 is configured to acquire the audio signal to be played from each of the at least two external speakers and the video frame to be played corresponding to the audio signal to be played;

Optionally, the video frame to be played includes at least one candidate voicing object. The audio processing device provided by the embodiment of the present invention also includes a voicing object determination unit 706, used to determine the voicing object corresponding to the audio signal to be played, and obtain The object identification information of the vocal object;

The signal acquisition unit 701 is configured to perform information matching based on the object identification information from the candidate voicing objects included in the video frame to be played, and determine the voicing object;

Optionally, the video frame to be played includes a display area of at least one candidate utterance object, and the signal acquisition unit 701 is configured to perform utterance action detection for each display area in the video frame to be played. , if it is detected that a candidate voicing object in the display area has performed a voicing action, use the candidate voicing object as the voicing object;

Optionally, the audio processing device provided by the embodiment of the present invention also includes a first position determination unit 707, configured to determine the corresponding sound receiving position information of the user in the video frame to be played in response to the user's position selection operation. ;

The signal acquisition unit 701 is configured to determine the signal transmission direction between the sounding position and the sound receiving position based on the sounding position information of the sounding object and the sound receiving position information;

The signal acquisition unit 701 is configured to determine the signal transmission angle between the sounding object and the reference object based on the sounding position information of the sounding object and the reference position information, as each of the audio to be played The target sound source angle corresponding to the signal.

Optionally, the signal acquisition unit 701 is configured to receive a data packet to be played sent by the signal sending end. The data packet to be played is based on the audio signal to be played from each external speaker and the audio signal corresponding to each audio signal to be played. The target sound source angle is encoded;

Optionally, the signal acquisition unit 701 is used to receive the data packet to be played sent by the cloud. The data packet to be played is sent to the cloud by the signal sending end. The data packet to be played is sent by the signal sending end. Obtained by encoding based on the audio signals to be played from each external speaker and the target sound source angle corresponding to each of the audio signals to be played;

Optionally, the signal transformation unit 704 is configured to calculate an anti-crosstalk resonance function based on the anti-crosstalk function. The anti-crosstalk resonance function is used to eliminate crossover caused by the at least two external speakers when playing audio externally. The impact of cross-sound and resonance of the external auditory canal of the human ear;

Perform audio parameter extraction on the audio signal to be processed to obtain the parameters to be played corresponding to the audio signal to be processed. number;

The signal transformation unit 704 is configured to perform signal transformation on the audio signal to be processed based on the audio adjustment function, the target sound source angle, the signal transmission angle and the anti-crosstalk function to obtain the audio signal to be processed. Process the audio signal corresponding to the target to play the audio signal.

It can be seen from the above that through the audio processing device, it is possible to generate audio signals that can express the direction of the sound source without adding additional audio playback equipment, and eliminate the cross-talk generated when using external speakers to play audio, so that users can enjoy to play better audio.

In addition, an embodiment of the present invention also provides an electronic device, which can be a terminal or a server, etc., as shown in Figure 9, which shows a schematic structural diagram of the electronic device involved in the embodiment of the present invention. Specifically, :

The electronic device may include a radio frequency (RF, Radio Frequency) circuit 901, a memory 902 including one or more computer-readable storage media, an input unit 903, a display unit 904, a sensor 905, an audio circuit 906, a wireless fidelity (WiFi) , Wireless Fidelity) module 907, including a processor 908 with one or more processing cores, a power supply 909 and other components. Those skilled in the art can understand that the structure of the electronic device shown in FIG. 9 does not constitute a limitation on the electronic device, and may include more or fewer components than shown in the figure, or combine certain components, or arrange different components. in:

The RF circuit 901 can be used to receive and send information or signals during a call. In particular, after receiving the downlink information of the base station, it is handed over to one or more processors 908 for processing; in addition, the uplink data is sent to the base station. . Typically, the RF circuit 901 includes, but is not limited to, an antenna, at least one amplifier, a tuner, one or more oscillators, a Subscriber Identity Module (SIM) card, a transceiver, a coupler, a low noise amplifier (LNA, Low Noise Amplifier), duplexer, etc. In addition, RF circuit 901 can communicate with networks and other devices through wireless communications. Wireless communication can use any communication standard or protocol, including but not limited to Global System of Mobile communication (GSM, Global System of Mobile communication), General Packet Radio Service (GPRS, General Packet Radio Service), Code Division Multiple Access (CDMA, Code Division Multiple Access), Wideband Code Division Multiple Access (WCDMA, Wideband Code Division Multiple Access), Long Term Evolution (LTE, Long Term Evolution), email, Short Message Service (SMS, Short Messaging Service), etc.

The memory 902 can be used to store software programs and modules, and the processor 908 executes various functional applications and data processing by running the software programs and modules stored in the memory 902 . The memory 902 may mainly include a storage program area and a storage data area, wherein the storage program area may store an operating system, an application program required for at least one function (such as a sound playback function, an image playback function, etc.), etc.; the storage data area may store a program based on Data created by the use of electronic devices (such as audio data, phone books, etc.), etc. In addition, memory 902 may include high-speed random access memory, and may also include non-volatile memory, such as at least one magnetic disk storage device, flash memory device, or other volatile solid-state storage device. Accordingly, the memory 902 may also include a memory controller to provide the processor 908 and the input unit 903 with access to the memory 902 .

The input unit 903 may be used to receive input numeric or character information and generate keyboard, mouse, joystick, optical or trackball signal input related to user settings and function control. Specifically, in a specific embodiment, the input unit 903 may include a touch-sensitive surface as well as other input devices. A touch-sensitive surface, also known as a touch display or trackpad, can collect the user's touch operations on or near it (such as the user using a finger, stylus, or any suitable object or accessory on or near the touch-sensitive surface). operations near the surface), and drive the corresponding connection device according to the preset program. Optionally, the touch-sensitive surface may include two parts: a touch detection device and a touch controller. Among them, the touch detection device detects the user's touch orientation, detects the signal brought by the touch operation, and transmits the signal to the touch controller; the touch controller receives the touch information from the touch detection device, converts it into contact point coordinates, and then sends it to the touch controller. to the processor 908, and can receive commands sent by the processor 908 and execute them. In addition, touch-sensitive surfaces can be implemented using a variety of types including resistive, capacitive, infrared, and surface acoustic waves. In addition to touch-sensitive surfaces, the input unit 903 may also include other input devices. Specifically, other input devices may include, but are not limited to, one or more of a physical keyboard, function keys (such as volume control keys, switch keys, etc.), trackball, mouse, joystick, etc.

The display unit 904 may be used to display information input by the user or information provided to the user as well as various graphical user interfaces of the electronic device. These graphical user interfaces may be composed of graphics, text, icons, videos, and any combination thereof. The display unit 904 may include a display panel, which may optionally be configured in the form of a liquid crystal display (LCD), an organic light-emitting diode (OLED), or the like. Further, the touch-sensitive surface can cover the display panel. When the touch-sensitive surface detects a touch operation on or near it, it is sent to the processor 908 to determine the type of the touch event. The processor 908 then displays the display panel according to the type of the touch event. Corresponding visual output is provided on the panel. Although in FIG. 9 , the touch-sensitive surface and the display panel are used as two independent components to implement the input and input functions, in some embodiments, the touch-sensitive surface and the display panel can be integrated to implement the input and output functions.

The electronic device may also include at least one sensor 905, such as a light sensor, a motion sensor, and other sensors. Specifically, the light sensor may include an ambient light sensor and a proximity sensor. The ambient light sensor may adjust the brightness of the display panel according to the brightness of the ambient light. The proximity sensor may close the display panel and/or when the electronic device moves to the ear. Backlight. As a kind of motion sensor, the gravity acceleration sensor can detect the magnitude of acceleration in various directions (usually three axes). It can detect the magnitude and direction of gravity when stationary. It can be used to identify applications of mobile phone posture (such as horizontal and vertical screen switching, related Games, magnetometer attitude calibration), vibration recognition related functions (such as pedometer, knock), etc.; as for other sensors such as gyroscopes, barometers, hygrometers, thermometers, infrared sensors, etc. that can also be configured in electronic devices, they are not mentioned here. Again.

The audio circuit 906, speaker, and microphone can provide an audio interface between the user and the electronic device. The audio circuit 906 can transmit the electrical signal converted from the received audio data to the speaker, which converts it into a sound signal and outputs it; on the other hand, the microphone converts the collected sound signal into an electrical signal, which is received and converted by the audio circuit 906 The audio data is processed by the audio data output processor 908 and then sent to, for example, another electronic device through the RF circuit 901, or the audio data is output to the memory 902 for further processing. Audio circuitry 906 may also include an earphone jack to provide communication between peripheral earphones and electronic devices.

WiFi is a short-distance wireless transmission technology. Electronic devices can help users send and receive emails, browse web pages, and access streaming media through the WiFi module 907. It provides users with wireless broadband Internet access. Although FIG. 9 shows the WiFi module 907, it can be understood that it is not a necessary component of the electronic device and can be omitted as needed without changing the essence of the invention.

The processor 908 is the control center of the electronic device, using various interfaces and lines to connect various parts of the entire mobile phone, by running or executing software programs and/or modules stored in the memory 902, and calling data stored in the memory 902, Perform various functions of electronic devices and process data. Optionally, the processor 908 may include one or more processing cores; preferably, the processor 908 may integrate an application processor and a modem processor, where the application processor mainly processes operating systems, user interfaces, application programs, etc. , the modem processor mainly handles wireless communications. It can be understood that the above-mentioned modem processor may not be integrated into the processor 908.

The electronic device also includes a power supply 909 (such as a battery) that supplies power to various components. Preferably, the power supply can be logically connected to the processor 908 through a power management system, so that functions such as charging, discharging, and power consumption management can be implemented through the power management system. Power supply 909 may also include one or more DC or AC power supplies, recharging systems, power failure detection circuits, power converters or inverters, power status indicators, and other arbitrary components.

Although not shown, the electronic device may also include a camera, a Bluetooth module, etc., which will not be described again here. Concrete in reality In this embodiment, the processor 908 in the electronic device will load the executable files corresponding to the processes of one or more application programs into the memory 902 according to the following instructions, and the processor 908 will run the executable files stored in the memory 902 application to achieve various functions, as follows:

Those of ordinary skill in the art can understand that all or part of the steps in the various methods of the above embodiments can be completed by instructions, or by controlling relevant hardware through instructions. The instructions can be stored in a computer-readable storage medium, and loaded and executed by the processor.

To this end, embodiments of the present invention provide a computer-readable storage medium in which a plurality of instructions are stored, and the instructions can be loaded by a processor to execute the steps in any audio processing method provided by embodiments of the present invention. . For example, this command can perform the following steps:

For the specific implementation of each of the above operations, please refer to the previous embodiments and will not be described again here.

Among them, the computer-readable storage medium may include: read-only memory (ROM, Read Only Memory), random access memory (RAM, Random Access Memory), magnetic disk or optical disk, etc.

Since the instructions stored in the computer-readable storage medium can execute the steps in any audio processing method provided by the embodiment of the present invention, therefore, the steps of any audio processing method provided by the embodiment of the present invention can be realized. The beneficial effects that can be achieved are detailed in the previous embodiments and will not be described again here.

According to one aspect of the present application, a computer program product or computer program is also provided. The computer program product or computer program includes computer instructions, and the computer instructions are stored in a computer-readable storage medium. The processor of the electronic device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the electronic device performs the methods provided in various optional implementations in the above embodiments.

The audio processing methods, devices, electronic equipment, storage media and program products provided by the embodiments of the present invention have been introduced in detail. This article uses specific examples to illustrate the principles and implementation methods of the present invention. The description of the above embodiments It is only used to help understand the method and its core idea of the present invention; at the same time, for those skilled in the art, there will be changes in the specific implementation and application scope according to the idea of the present invention. In summary, this The content of the description should not be construed as limiting the invention.

Claims

An audio processing method, characterized by including:

Obtain the audio signal to be played from each of the at least two external speakers and the target sound source angle corresponding to each of the audio signals to be played;

Determine the signal transmission angle between each of the external speakers and the user's head;

Based on the signal transmission angle and the target sound source angle, an anti-crosstalk function corresponding to the signal transmission angle is calculated. The anti-crosstalk function is used to eliminate the crossover generated by the at least two external speakers when the audio is externally played. Crosstalk;

Based on the anti-crosstalk function, signal transformation is performed on the audio signals to be processed from each of the external speakers to obtain a target playback audio signal corresponding to the audio signal to be played.
The audio processing method according to claim 1, characterized in that, based on the signal transmission angle and the target sound source angle, calculating the anti-crosstalk function corresponding to the signal transmission angle includes:

Based on the signal transmission angle, determine the speaker head related transfer function corresponding to the signal transmission angle;

Based on the target sound source angle, determine the sound source related transfer function corresponding to the target sound source angle;

According to the speaker head related transfer function and the sound source related transfer function, the anti-crosstalk function corresponding to the signal transmission angle is calculated.
The audio processing method according to claim 2, wherein the at least two external speakers include a left speaker and a right speaker, and determining the signal transmission angle between each of the external speakers and the user's head, include:

Determine the left signal transmission angle between the left speaker and the user's head;

Determine a right signal transmission angle between the right speaker and the user's head;

Determining the speaker head-related transfer function corresponding to the signal transmission angle based on the signal transmission angle includes:

Based on the left signal transmission angle, a first left ear head-related transfer function between the left speaker and the user's left ear, and a first right ear head-related transfer function between the left speaker and the user's right ear are determined ;

Based on the right signal transmission angle, a second left ear head-related transfer function between the right speaker and the user's left ear, and a second right ear head-related transfer function between the right speaker and the user's right ear are determined ;

The first left ear tip related transfer function, the first right ear tip related transfer function, the second left ear tip related transfer function and the second right ear tip related transfer function are used as speaker head related transfer functions.
The audio processing method according to claim 2, characterized in that the at least two external speakers include For the left speaker and the right speaker, determining the signal transmission angle between each of the external speakers and the user's head includes:

Determine the positional relationship between the left speaker, the right speaker and the user's head;

If the left speaker and the right speaker have a left-right symmetrical relationship with respect to the user's head, the angle between any external speaker and the user's head is used as the signal transmission angle;

Determining the speaker head-related transfer function corresponding to the signal transmission angle based on the signal transmission angle includes:

Based on the signal transmission angle, a first head-related transfer function between the left speaker and the user's left ear and between the right speaker and the user's right ear is determined, and a first head-related transfer function between the left speaker and the user's right ear and the Describe the second head-related transfer function between the right speaker and the user's left ear;

The first head-related transfer function and the second head-related transfer function are regarded as speaker head-related transfer functions.
The audio processing method according to claim 2, characterized in that, based on the speaker head related transfer function and the sound source related transfer function, calculating the anti-crosstalk function corresponding to the signal transmission angle includes:

Perform matrix merging processing according to the speaker head related transfer function to obtain the speaker crosstalk matrix corresponding to the audio signal to be processed;

Perform matrix cancellation on the speaker crosstalk matrix to calculate the crosstalk cancellation matrix of the speaker crosstalk matrix;

Based on the crosstalk cancellation matrix and the sound source related transfer function, an anti-crosstalk function corresponding to the signal transmission angle is calculated.
The audio processing method according to claim 2, characterized in that, before calculating the anti-crosstalk function corresponding to the signal transmission angle according to the speaker head related transfer function and the sound source related transfer function, the method Also includes:

Get the preset discrete head related transfer function;

Perform function approximation processing on the discrete head related transfer function to obtain the target head related transfer function;

Determining the speaker head-related transfer function corresponding to the signal transmission angle based on the signal transmission angle includes:

Based on the signal transmission angle and the target head-related transfer function, determine the speaker head-related transfer function corresponding to the signal transmission angle;

Determining the sound source related transfer function corresponding to the target sound source angle based on the target sound source angle includes:

Based on the target sound source angle and the target head related transfer function, determine the sound corresponding to the target sound source angle. Source dependent transfer function.
The audio processing method according to claim 1, characterized in that: obtaining the audio signal to be played by each of the at least two external speakers and the target sound corresponding to each of the audio signals to be played. Source angles, including:

Obtain the audio signal to be played from each of the at least two external speakers;

Position the sound source of the audio signals to be played, and determine the target sound source angle corresponding to each of the audio signals to be played.
The audio processing method according to claim 1, characterized in that: obtaining the audio signal to be played by each of the at least two external speakers and the target sound corresponding to each of the audio signals to be played. Source angles, including:

Obtain the audio signal to be played from each of the at least two external speakers and the video frame to be played corresponding to the audio signal to be played;

Determine the voicing position information of the voicing object from the video frame to be played;

Based on the sounding position information of the sounding object, the target sound source angle corresponding to each of the audio signals to be played is calculated.
The audio processing method according to claim 8, characterized in that the video frame to be played includes at least one candidate voicing object, and the voicing position information of the voicing object is determined from the video frame to be played. The above methods also include:

Determine the sound object corresponding to the audio signal to be played, and obtain the object identification information of the sound object;

Determining the voicing position information of the voicing object from the video frame to be played includes:

From the candidate voicing objects included in the video frame to be played, perform information matching based on the object identification information to determine the voicing object;

Obtain the target display area of the utterance object in the video frame to be played, and use the position information of the target display area as the utterance position information of the utterance object.
The audio processing method according to claim 8, characterized in that the video frame to be played includes a display area of at least one candidate voicing object, and the voicing position information of the voicing object is determined from the video frame to be played. ,include:

Perform voice action detection for each display area in the video frame to be played respectively. If it is detected that a candidate voice object in the display area has performed a voice action, use the candidate voice object as the voice object;

Obtain the target display area of the utterance object in the video frame to be played, and use the position information of the target display area as the utterance position information of the utterance object.
The audio processing method according to claim 8, characterized in that, before calculating the target sound source angle corresponding to each of the audio signals to be played based on the voicing position information of the voicing object, the method further includes:

In response to the user's position selection operation, determine the corresponding sound receiving position information of the user in the video frame to be played;

Calculating the target sound source angle corresponding to each of the audio signals to be played based on the sounding position information of the sounding object includes:

Based on the sound-emitting position information of the sound-emitting object and the sound-receiving position information, determine the signal transmission direction between the sound-emitting position and the sound-receiving position;

According to the signal transmission direction, the target sound source angle corresponding to each audio signal to be played is calculated.
The audio processing method according to claim 8, characterized in that, before calculating the target sound source angle corresponding to each of the audio signals to be played based on the voicing position information of the voicing object, the method further includes:

In response to the user's reference object selection operation, determine the reference object corresponding to the user in the video frame to be played;

Obtain the reference position information of the reference object in the video frame to be played;

Calculating the target sound source angle corresponding to each audio signal to be played based on the sounding position information of the sounding object includes:

Based on the sounding position information of the sounding object and the reference position information, the signal transmission angle between the sounding object and the reference object is determined as the target sound source angle corresponding to each of the audio signals to be played.
The audio processing method according to claim 1, characterized in that: obtaining the audio signal to be played by each of the at least two external speakers and the target sound corresponding to each of the audio signals to be played. Source angles, including:

Receive the data packet to be played sent by the signal transmitting end, the data packet to be played is encoded based on the audio signal to be played from each external speaker and the target sound source angle corresponding to each of the audio signals to be played;

The data packet to be played is decoded to obtain the audio signal to be played from each of the at least two external speakers and the target sound source angle corresponding to each of the audio signals to be played.
The audio processing method according to claim 1, characterized in that: obtaining the audio signal to be played by each of the at least two external speakers and the target sound corresponding to each of the audio signals to be played. source angle degrees, including:

Receive the data packet to be played sent by the cloud. The data packet to be played is sent to the cloud by the signal sending end. The data packet to be played is sent by the signal sending end based on the audio signal to be played by each external speaker and the audio signal of each external speaker. The target sound source angle corresponding to the audio signal to be played is obtained by encoding;

The data packet to be played is decoded to obtain the audio signal to be played from each of the at least two external speakers and the target sound source angle corresponding to each of the audio signals to be played.
The audio processing method according to claim 1, characterized in that, based on the anti-crosstalk function, the audio signal to be processed of each of the external speakers is signal transformed to obtain the corresponding audio signal to be played. The target plays audio signals including:

Calculate an anti-crosstalk resonance function based on the anti-crosstalk function, and the anti-crosstalk resonance function is used to eliminate the influence of cross-talk and resonance of the external auditory canal of the human ear caused by the at least two external speakers when the audio is played out;

Based on the anti-crosstalk resonance function, signal transformation is performed on the audio signals to be processed from each of the external speakers to obtain a target playback audio signal corresponding to the audio signal to be played.
The audio processing method according to claim 1, characterized in that, before performing signal transformation on the audio signal to be processed based on the anti-crosstalk function, the method further includes:

Obtain the playback setting parameters of each of the external speakers and the audio playback preference parameters corresponding to the target user;

Perform audio parameter extraction on the audio signal to be processed to obtain the parameters to be played corresponding to the audio signal to be processed;

Calculate the audio adjustment function corresponding to the audio signal to be processed according to the playback setting parameters, the audio playback preference parameters and the to-be-played parameters;

The step of performing signal transformation on the audio signal to be processed based on the anti-crosstalk function to obtain the target playback audio signal corresponding to the audio signal to be processed includes:

Based on the audio adjustment function, the target sound source angle, the signal transmission angle and the anti-crosstalk function, the audio signal to be processed is signal transformed to obtain the target playback audio signal corresponding to the audio signal to be processed. .
The audio processing method according to any one of claims 1-16, characterized in that the method further includes:

The target playback audio signal corresponding to each of the audio signals to be played is sent to the external speaker corresponding to each of the audio signals to be played, and the external speaker is triggered to play the corresponding target playback audio signal.
An audio processing device, characterized by including:

A signal acquisition unit, configured to acquire the audio signal to be played by each of the at least two external speakers. number and the target sound source angle corresponding to each audio signal to be played;

An angle determination unit, used to determine the signal transmission angle between each of the external speakers and the user's head;

A function calculation unit configured to calculate an anti-crosstalk function corresponding to the signal transmission angle based on the signal transmission angle and the target sound source angle. The anti-crosstalk function is used to eliminate the problem caused by the at least two external loudspeakers being externally placed. Crosstalk generated when playing audio;

A signal conversion unit, configured to perform signal conversion on the audio signals to be processed from each of the external speakers based on the anti-crosstalk function, to obtain a target playback audio signal corresponding to the audio signal to be played.
An electronic device, characterized in that it includes a memory and a processor; the memory stores an application program, and the processor is used to run the application program in the memory to execute the method described in any one of claims 1 to 17 Steps in audio processing methods.
A computer-readable storage medium, characterized in that the computer-readable storage medium stores a plurality of instructions, and the instructions are suitable for loading by a processor to perform the audio processing described in any one of claims 1 to 17 steps in the method.
A computer program product, including a computer program or instructions, characterized in that when the computer program or instructions are executed by a processor, the steps of the audio processing method according to any one of claims 1 to 17 are implemented.