WO2023173285A1

WO2023173285A1 - Audio processing method and apparatus, electronic device, and computer-readable storage medium

Info

Publication number: WO2023173285A1
Application number: PCT/CN2022/080925
Authority: WO
Inventors: 莫品西; 边云锋; 高建正
Original assignee: 深圳市大疆创新科技有限公司
Priority date: 2022-03-15
Filing date: 2022-03-15
Publication date: 2023-09-21

Abstract

Provided in one or more embodiments of the present application are an audio processing method and apparatus, an electronic device and a computer-readable storage medium, the audio processing method being applied to the electronic device and the electronic device comprising at least two players and a motion sensor. The method comprises: acquiring, in respect of an audio to be played, azimuth information of a virtual sound source in a virtual sound field, the virtual sound field being established on the basis of a position relationship between the at least two players and the left and right auditory organs of a user; acquiring motion information of the user by means of the motion sensor, and, on the basis of the motion information, generating a sound source azimuth adjustment instruction; and, on the basis of the sound source position adjustment instruction, the azimuth information of the virtual sound source and said audio, determining an audio signal corresponding to each of the at least two players, and playing the audio signals by means of the corresponding players.

Description

Audio processing method, device, electronic equipment and computer-readable storage medium

Technical field

The present application relates to the field of audio processing technology, specifically, to an audio processing method, device, electronic equipment and computer-readable storage medium.

Background technique

With the continuous advancement of audio processing technology, people's hearing requirements for audio are getting higher and higher; existing binaural audio playback solutions generally output the original audio signal through the player, and all sounds heard by the user are It's the same, these solutions can no longer meet the personalized needs of users.

Contents of the invention

In view of this, the present application provides an audio processing method, device, electronic equipment and computer-readable storage medium to solve the technical problem in related technologies that audio processing cannot meet the personalized needs of users.

In a first aspect, an audio processing method is provided. The method is applied to an electronic device. The electronic device includes at least two players and a motion sensor. The method includes:

Obtaining the orientation information of the virtual sound source in the audio to be played in the virtual sound field, the virtual sound field is established based on the positional relationship between the at least two players and the user's left and right hearing organs;

Using the motion sensor to obtain the user's motion information, and generating a sound source orientation adjustment instruction based on the motion information;

Based on the sound source position adjustment instruction, the orientation information of the virtual sound source and the audio to be played, an audio signal corresponding to each of the at least two players is determined, and passed through the corresponding The player plays the audio signal.

In a second aspect, an audio processing device is provided. The device includes a processor, a memory, and a computer program stored on the memory and executable by the processor. When the processor executes the computer program, the aforementioned first step is implemented. Method embodiments described in one aspect.

In a third aspect, an electronic device is provided. The electronic device includes at least two players and a motion sensor. The electronic device further includes a processor, a memory, and a computer stored on the memory and executable by the processor. program;

When the processor executes the computer program, the method embodiment described in the first aspect is implemented.

In a fourth aspect, a computer-readable storage medium is provided. Several computer instructions are stored on the computer-readable storage medium. When the computer instructions are executed, the method embodiment described in the first aspect is implemented.

Applying the solution provided by this application, a virtual sound field can be established based on the positional relationship between the at least two players and the user's left and right hearing organs, thereby obtaining the orientation information of the virtual sound source in the audio to be played in the virtual sound field; and, all the information can be used The motion sensor obtains the user's motion information, and generates a sound source orientation adjustment instruction based on the motion information, so that the sound source orientation adjustment instruction can be used to adjust the orientation of the virtual sound source, determine a new audio signal, and pass the at least Two players play; this allows the user to hear the sound source with changing position when moving, and allows the user to feel that the position of the sound source in the sound changes with the user's movement. Therefore, the audio processing solution of this embodiment realizes the generation of audio that conforms to the corresponding motion sound effects based on the user's movements, achieving a realistic spatial sound effect experience.

Description of the drawings

In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings needed to be used in the description of the embodiments will be briefly introduced below. Obviously, the drawings in the following description are only some embodiments of the present application. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without exerting any creative effort.

Figure 1 is a schematic diagram of a real concert scene according to an embodiment of the present application.

Figure 2 is a schematic flowchart of an audio processing method according to an embodiment of the present application.

FIG. 3A is a schematic flowchart of an audio processing method according to another embodiment of the present application.

Figure 3B is a schematic flowchart of step 303 according to an embodiment of the present application.

Figure 3C is a schematic diagram of the orientation of each discrete frequency in a coordinate system according to an embodiment of the present application.

Figure 3D is a schematic diagram showing user action changes in a coordinate system according to an embodiment of the present application.

Figure 4 is a hardware structure diagram of an audio processing device according to an embodiment of the present application.

Figure 5 is a hardware structure diagram of an electronic device according to an embodiment of the present application.

Detailed ways

The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present application. Obviously, the described embodiments are only some of the embodiments of the present application, rather than all of the embodiments.

Ordinary binaural audio playback solutions do not process the original audio signal, that is, play the original audio signal, so the user can only feel the static effect of the audio signal. For example, if the user puts on headphones to listen to music, then any user will The music you hear at any time is the same. With the rise of personalized needs and VR (Virtual Reality, virtual reality technology) equipment, it is increasingly difficult for static sound effects to meet the needs and experience of users.

As shown in Figure 1, taking a real concert scene as an example, the user faces the concert stage. To the left of the user is a performer playing gongs, and to the right of the user is a performer playing drums. The sounds of gongs and drums are respectively from The user's left front and right front are transmitted to the user's ears, and the user can feel the sound of gongs and drums in their left front and right front respectively. When the user turns right, the concert stage is on the left side of the user, the drums are in front of the user's left, and the gongs are behind the user's left side. When the user turns sideways, the positions of his left and right ears relative to the gongs and drums change, that is, From the user's perspective, gongs and drums have a new position, and the human auditory system can feel the changes in the gongs and drums and their own orientation.

However, when the user wears headphones and listens to the music played in the headphones, the existing audio processing solution cannot express the above changes. This is because the audio signals played are pre-recorded. When recording audio, the position of the radio equipment is fixed. For example, if the gongs and drums in the audio signal are fixed to the front left and right of the radio equipment, the user will always only be able to feel the gongs and drums in the audio signal coming from the front left and right when listening to music with headphones. In the front, when the user turns sideways, he cannot feel the sound of gongs and drums and the changes in his own position as in a real concert scene. If the position of the radio equipment is not fixed, for example, the radio equipment has a changing position during the recording process, the gongs and drums as the sound source in the recorded audio signal change relative to the position of the radio equipment, or it can also be The position of the radio equipment is fixed, but the position of the gongs and drums as the sound source changes; the audio recorded in this way can make people feel the position changes of the gongs and drums; although dynamic effects of sound changes can be achieved, However, this method needs to be realized by changing the position of the radio equipment or the position of the sound source during recording. After the recording is completed, the change in the position of the sound source in the audio is fixed, and its dynamic effect is also consistent with the user's listening to the audio signal. The action does not match.

Based on this, as shown in Figure 2, it is a flow chart of an audio processing method according to an exemplary embodiment of the present application. This method can be applied to an electronic device. The electronic device includes at least two players and a motion sensor. , the method includes the following steps:

In step 202, the orientation information of the virtual sound source in the audio to be played in the virtual sound field is obtained. The virtual sound field is established based on the positional relationship between the at least two players and the user's left and right hearing organs.

In step 204, the motion sensor is used to obtain the user's motion information, and a sound source orientation adjustment instruction is generated based on the motion information.

In step 206, based on the sound source position adjustment instruction, the orientation information of the virtual sound source and the audio to be played, an audio signal corresponding to each of the at least two players is determined, and The audio signal is played through the corresponding player.

The electronic device in this embodiment can be any device with audio processing capabilities, such as a smartphone, a camera device, a VR device, a wearable device or a computer that supports audio playback, and so on. Wherein, the electronic device is connected to an audio player, that is, at least two players, such as ordinary headphones, bone conduction headphones or speakers, etc. The motion sensor includes any sensor that can collect data and the collected data can detect the user's motion, such as inertial measurement sensors, visual sensors (such as monocular or binocular) or laser radar, etc.

In some examples, the at least two players and motion sensors can be mounted on the device that performs the above method embodiments. For example, a wearable device is equipped with a player and a motion sensor, and the wearable device can obtain the audio to be played, Obtain the data collected by the motion sensor to determine the user's motion information, and after processing the audio to be played, control the player to play the processed audio signal.

In other examples, the player and the motion sensor may be respectively connected to the device executing the above method embodiment. For example, the player and the motion sensor may be respectively connected to a device such as a mobile terminal, and the mobile terminal obtains the audio to be played and obtains the action. The data collected by the sensor determines the user's action information, and after processing the audio to be played, the processed audio signal is played through the player; the playback method can be that the audio is processed and then played by the player, or it can be The processed audio is stored and then played through the player when needed; or, if the device executing the above method embodiment is not directly connected to the player, the device can send the processed audio to other devices connected to the player. Control player playback from other devices, etc.

In this embodiment, a virtual sound field can be established based on the positional relationship between the at least two players and the user's left and right hearing organs, thereby obtaining the orientation information of the virtual sound source in the audio to be played in the virtual sound field; and, the action can be used The sensor obtains the user's action information, and generates a sound source orientation adjustment instruction based on the action information, so that the sound source orientation adjustment instruction can be used to adjust the orientation of the virtual sound source, determine a new audio signal, and pass the at least two The player plays; the user can hear the sound source with changing position when moving, and the user can feel that the position of the sound source in the sound changes with the user's movement. Therefore, the audio processing solution of this embodiment can generate audio that matches the corresponding motion sound effects based on the user's movements, achieving a realistic spatial sound effect experience.

For example, when the user is listening to music with headphones, if the user turns right, the device obtains the user's action information of turning right, adjusts the orientation information of the virtual sound source in the audio signal through the above solution, and generates a dynamic audio signal in real time. , the position of the virtual sound source in the audio signal relative to the user changes with the user's movements, thus realizing a VR sound effect in which the sound follows the movement of the person, and the user can feel the sound source changing in real time with his/her movements. In addition, this embodiment can also provide users with personalized sound effect requirements. The user can control the azimuth changes of the virtual sound source through actions. That is, the user can customize the corresponding relationship between his actions and the azimuth changes of the virtual sound source, so that the user can control the azimuth changes of the virtual sound source according to the user's needs. Flexibly adjust the position of the virtual sound source according to its own needs.

Regarding step 202, in some examples, the sound field refers to the area where sound waves exist in the medium. The physical quantity describing the sound field can be sound pressure, particle vibration velocity, displacement or medium density, etc., and is generally a function of position and time. The relationship between changes in the spatial position and time of these physical quantities in the sound field is described by the acoustic wave equation. The sound source is in a uniform and isotropic medium, and the sound field in which the influence of the boundary is negligible can be called a free sound field. In this embodiment, the player plays the audio. The virtual sound field in this embodiment can be established based on the positional relationship between the at least two players and the user's left and right hearing organs. The orientation information describes the location of the virtual sound source in the virtual sound field. Position information, as an example, can establish a coordinate system based on the user. For example, the origin of the coordinate system is on the user. Alternatively, establishing a coordinate system with the player is also optional. In actual applications, the orientation of the virtual sound source can be flexibly determined according to needs. The information is described by the coordinate information of the virtual sound source in the coordinate system.

In some examples, the virtual sound source can be determined by signal recognition of the audio to be played. After identifying the virtual sound source in the audio to be played, the orientation information of the virtual sound source in the virtual sound field is determined. In practical applications, It is implemented through audio processing algorithms or trained neural networks. In other examples, it can be implemented based on signal frequency domain analysis. For example, it can be to obtain the frequency domain signal from the audio to be played, extract the orientation of each discrete frequency from the frequency domain signal, and adjust the orientation of each discrete frequency in subsequent steps. , so that the position of the virtual sound source in the virtual sound field changes with the user's actions. As an example, the orientation information of the virtual sound source in the virtual sound field in the audio to be played is obtained by extracting multiple discrete frequencies from the audio to be played, and based on the orientation information of the multiple discrete frequencies in the virtual sound field It is certain that in this embodiment, the sound signal generated by the real sound source can be decomposed into multiple discrete frequencies.

In some examples, the audio to be played is an audio signal of at least two channels; the orientation information of the multiple discrete frequencies in the virtual sound field is obtained in the following manner: according to the audio of the at least two channels The amplitude ratio and/or phase difference of the signal is used to obtain the orientation information of the multiple discrete frequencies in the virtual sound field based on the first coordinate system and the binaural transfer function. Wherein, the first coordinate system is established based on the positional relationship between the at least two players and the user's left and right hearing organs. Each discrete frequency sound source can be used as an independent sound source and is distributed in the coordinate system to represent the sound. Various directions of the source; there are usually at least two players, so the audio to be played is usually an audio signal of at least two channels. The location of each independent sound source in space can be based on the amplitude of the two-channel frequency domain signal. The ratio and phase difference are calculated by combining the propagation model of sound to both ears. Optionally, the sound propagation model to both ears can be implemented using the binaural transfer function. The binaural transfer function is also called Head Related Transfer Functions (HRTF). This function describes the flow of sound waves from the sound source to the binaural Transmission process; or you can also refer to the propagation model of sound in the free sound field, etc. If the audio to be played is a mono signal, it can be expanded into the same signal for the left and right channels through copying.

In practical applications, there are many ways to use motion sensors to obtain user motion information. For example, the motion sensor may include a sensor worn on the user. The electronic device is connected to the sensor. The data collected by the sensor may be sent to the electronic device, and the electronic device determines the user's motion information. In other examples, one or more of the following implementation methods may also be included:

For example, the electronic device is a wearable electronic device, and the motion sensor includes an inertial measurement sensor (IMU, Inertial Measurement Unit), that is, the motion sensor is built into the electronic device. The use of the motion sensor to obtain the user's motion information includes: : Determine the user's action information based on the measurement data of the inertial measurement sensor.

Alternatively, the electronic device is a wearable electronic device, and the motion sensor includes a first image sensor; that is, the electronic device is equipped with one or more first image sensors. When the wearable electronic device is worn on the user's head, the The first image sensor faces the user's eyes; and using the motion sensor to obtain the user's motion information includes: acquiring an image collected by the first image sensor, and acquiring the motion information of the user's eyeballs based on the image. Optionally, there can be multiple acquisition methods, for example, it can be implemented using an eyeball recognition algorithm or a trained neural network.

Alternatively, the electronic device is a wearable electronic device, and the motion sensor includes one or more second image sensors. When the wearable electronic device is worn on the user's head, the motion sensor of the one or more second image sensors The observation range covers the activity space of the user's hand; the use of the motion sensor to obtain the user's motion information includes: acquiring images collected by the one or more second image sensors. If the user's hand is detected from the image, part to obtain the movement information of the user's hand. Optionally, there can be multiple acquisition methods, for example, it can be implemented using hand recognition algorithms or trained neural networks.

In practical applications, the user's action information may be determined using any of the above methods, or the action information may be determined using a combination of at least two of the above methods, which is not limited in this embodiment. As an example, during implementation, the audio to be played has a certain playback duration. One of the implementation methods may be used to determine the user's action information in certain time periods, and other implementation methods may be used to determine the user's action information in other time periods. As an example, during implementation, when the user's eyeballs rotate, the first image sensor in the aforementioned embodiment can collect an image of the user's eyes, the electronic device determines the action information of the user's eyeballs based on the image, and adjusts the audio at the current time based on the action information. the direction of the virtual sound source; subsequently, the user moves his hand, and the second image sensor of the aforementioned embodiment can collect an image of the user's hand. The electronic device determines the action information of the user's hand based on the image, and adjusts the current time based on the action information. The position of the virtual sound source in the audio. Optionally, during the audio playback process, the user may have different types of action information. This embodiment may adopt a variety of embodiments for collecting user actions, so that the position of the virtual sound source in the virtual sound field can be changed according to the varies with different types of user actions.

Optionally, the user's action information can include actions of a variety of body parts, such as any one or more of hand movements, head movements, eye movements, or leg movements; in actual applications, it can be set as needed, or Set by user. Optionally, the user's action information may be a vector of information used to describe user action changes, which may include multiple types of information, such as any one or more of direction information, distance information, or speed information of the user's action. Kinds etc.

Optionally, the user's action information can be calculated based on a set coordinate system; or the user's action information can also include information describing changes in the user's actions, which changes can refer to the user's actions at the current moment. The change of an action relative to another action, which can be implemented in a variety of ways, for example, it can be the previous collection cycle, the set time before the current time, or the set collection cycle before the current time. etc., or it can also be calculated within a set coordinate system as mentioned above. This coordinate system can be flexibly determined according to needs. For example, it can be a coordinate system established based on the initial posture of the user when using the player, or it can be a coordinate system established based on the posture of the player when the player is used by the user; or it can also be It can be other customized coordinate systems, etc., which is not limited in this embodiment.

After obtaining the user's action information, a sound source orientation adjustment instruction can be generated based on the action information. In practical applications, in some examples, the corresponding relationship between the user's action information and the sound source orientation adjustment instruction can be preset by the technician. That is, what kind of action information generates what kind of sound source orientation adjustment instruction can be decided by the technician. . In other examples, it can also be decided by the user. For example, the user can be provided with a setting function. For example, the user can set the action information of one or more body parts to trigger the sound source orientation adjustment instruction, or set the direction information, Any one or more of distance information or speed information is used to describe the user's action information.

In some examples, the amount of change of the user's action can be determined based on the user's action information; based on the amount of change of the user's action and the first mapping relationship, the amount of change used to adjust the orientation of the virtual sound source in the virtual sound field is generated. The sound source orientation adjustment instruction; wherein, the first mapping relationship includes: the corresponding relationship between the change amount of the user's action and the change amount of the virtual sound source's orientation in the virtual sound field; that is, the change amount of the user action in this embodiment There is a corresponding relationship with the change amount of the virtual sound source's position in the virtual sound field. Through this first mapping relationship, the change amount of the user's action can be quickly used to determine the change amount of the virtual sound source's position in the virtual sound field, thereby improving the efficiency of audio processing. The speed also makes the change in the position of the virtual sound source in the virtual sound field correspond to the change in the user's action. For example, if the user's action is larger, the change in the virtual sound source's position in the virtual sound field will be larger. The two have Positive relationship.

In some examples, the orientation information of the virtual sound source in the virtual sound field in the audio to be played can be determined through the first coordinate system; the change amount of the user action can be determined through the first coordinate system, or That is, both are determined by the same coordinate system; in this embodiment, in order to make the position of the virtual sound source in the virtual sound field change with the user's action, the change amount of the user's action is equal to the change of the virtual sound source. The orientation information is determined using the same first coordinate system, and the relationship between the two can be determined quickly and accurately, which facilitates the establishment of the mapping relationship. Optionally, the first coordinate system may include: a coordinate system established based on the user's initial posture using the at least two players. As an example, the coordinate system represents the user's initial posture. Optionally, the virtual sound field and the virtual sound source can be defined to be stationary relative to the coordinate system, while the person moves relative to the coordinate system. Therefore, the user's actual action parameters in the space represent the person's movement relative to the virtual sound field, and the user's actual action information is converted into the person's movement relative to the virtual sound field. For example, the user's hand sliding to the right can be defined as the person's movement to the right. , which corresponds to the user dragging the sound field to the right, that is, the person moves to the left relative to the sound field.

In practical applications, the first mapping relationship can be implemented in multiple ways. In some examples, the first mapping relationship is preset. For example, one kind of first mapping relationship may be preset to apply to all users, or different first mapping relationships may be set according to different user types. Different first mapping relationships can also be set based on various factors such as different audio types and application scenarios. The electronic device can select from multiple preset first mapping relationships based on one or more of the aforementioned factors during implementation. Appropriate first mapping relationship. For example, if users of different ages, genders or heights perform the same action, the electronic device of this embodiment can generate different audio signals based on different first mapping relationships, that is, the virtual sounds in the audio heard by different users. The changes in the orientation of the source in the virtual sound field may be different.

In other examples, the first mapping relationship may be generated by obtaining the user's setting instructions. For example, the electronic device implementing this embodiment or other electronic devices connected to the electronic device of this embodiment may provide a user interface. A setting function is provided, through which the user's setting instructions can be obtained, and then the first mapping relationship is generated according to the user's setting instructions. Based on this, this embodiment provides the function of customizing the mapping relationship, which allows users to customize the mapping relationship they need. Therefore, different users can have different first mapping relationships, that is, for the same audio, different users can make the same Action, the generated audio signals are different, and the changes in the orientation of the virtual sound source in the virtual sound field in the audio heard by different users are different.

In other examples, the first mapping relationship may also be determined based on the user's historical behavior data. For example, with the user's authorization and consent, the user's historical behavior data is obtained. The historical behavior data may include the user's historical action information. and/or one or more information such as the user's historical played audio information, etc., to analyze the user's action characteristics or audio preferences and other user personalized information, and based on this, generate a first mapping relationship that conforms to the user's preferences.

In practical applications, the first mapping relationship may be determined using any of the above methods, or a combination of at least two of the above methods may be used, which is not limited in this embodiment. For example, for a certain user, if the user's historical behavior data is not obtained during the initial implementation of this embodiment, the first mapping relationship can be determined using the aforementioned preset method; after that, the user's audio information after adjusting the orientation is obtained. Preference data, such as obtaining the user's evaluation data of the audio after adjusting the position, etc., and continuously obtaining the user's historical behavior data, thereby generating a new first mapping relationship, and using the new first mapping relationship to implement the solution of this embodiment .

In this embodiment, the audio signal corresponding to each of the at least two players may be determined based on the sound source position adjustment instruction, the orientation information of the virtual sound source, and the audio to be played, The audio signal is played through the corresponding player, so that the position of the virtual sound source in the virtual sound field changes with the user's actions.

In some examples, the audio to be played has at least two virtual sound sources, and the type information of the at least two virtual sound sources is different; based on this, in this embodiment, the sound source is generated based on the action information. The orientation adjustment instruction may include: based on the action information, generating a sound source orientation adjustment instruction for adjusting the orientation of each virtual sound source in the at least two virtual sound sources, so that with the user's actions, different types of information The virtual sound source has different amounts of change in the orientation of the virtual sound field. As an example, the sound source position adjustment instruction can be used to adjust the position of each virtual sound source, and the change amount of the position of each virtual sound source is different. Therefore, an action of the user can make the played audio , the azimuth changes of each virtual sound source are different, so that the user can feel the different changes of different virtual sound sources in the audio. As an example, if the audio to be played includes two sound sources, but after the user takes an action, one of the sound sources moves from one direction to another by moving a larger amount of change, while the other sound source can move from one direction to another. , by moving a smaller amount of change to reach another direction. In practical applications, virtual sound sources with different types of information have different amounts of change in the orientation of the virtual sound field, which can be automatically determined by the electronic device. For example, the type information of the sound source can be identified, where the type information includes the following One or more: the orientation information, timbre information or volume information of the virtual sound source in the audio to be played in the virtual sound field, so that different sound sources can be accurately distinguished through one or more of the above information.

The solution of this embodiment can be applied to a video playback scene. The video playback scene includes audio to be played and multi-frame images synchronized with the audio to be played. Based on this, in this embodiment, after the user performs an action, the user can Changes in the image can also change the position of the virtual sound source in the audio. As an example, the electronic device includes a display area; generating a sound source azimuth adjustment instruction based on the action information includes: generating a sound source azimuth adjustment instruction and an image display instruction based on the action information; and passing the corresponding The player plays the audio signal, including: controlling the corresponding player to play the audio signal, and at the same time, based on the image display instruction, acquiring multiple frames of images displayed synchronously with the audio to be played and displaying them in the display area in; wherein the image includes a first pixel area related to the virtual sound source, and the changes in the first pixel area in the multi-frame image and the orientation of the virtual sound source in the virtual sound field all change with changes depending on the user's actions.

This embodiment can be applied to scenarios such as VR. As an example, the electronic device of this embodiment can be a VR device. When the user uses the VR device, the VR device displays multi-frame images of the video and simultaneously plays the images during the display. Picture-related audio, for example, the audio comes from a virtual object in the video. The virtual object occupies the first pixel area in the image. The VR device can adjust the position of the virtual object in the display area based on the user's actions and simultaneously adjust the virtual object. The orientation of the sound source, therefore, the image and the orientation changes of the sound source are adjusted at the same time, which can achieve both visual VR effects and audio VR effects at the same time. As an example, the user wears a VR device and the display area displays a concert stage. There is a performer playing in the center of the stage. The sound source in the audio is the performer. The user can move the performer to the display area through hand movements. Left side of the stage; after the VR device detects the user's action, it generates an image display instruction so that in the newly displayed image, the performer's pixel area is displayed on the left side of the display area, and, through the sound source orientation adjustment instruction, the The position of the virtual sound source in the virtual sound field also changes to the left along with the user's movements.

In the VR and other scenes of the above embodiments, user actions may not be involved, but only changes in the image. Based on this, in other examples, the electronic device may also adjust the virtual sound source based on changes in the image. Change, for example, the electronic device includes a display area, the display area is used to display a multi-frame image synchronized with the audio to be played, the image includes a first pixel area related to the virtual sound source; The method also includes: when displaying the current image, obtaining the position change of the first pixel area in the current image relative to the image before the current image; based on the position change, the orientation information of the virtual sound source and the For audio to be played, an audio signal corresponding to each of the at least two players is determined, and the corresponding player is controlled to play the audio signal. For example, based on the position change, the orientation information of the virtual sound source and the audio to be played, an audio signal corresponding to each of the at least two players is determined, and the corresponding playback is controlled. The audio signal is played by the device, so that the orientation of the virtual sound source in the virtual sound field changes as the position changes.

In practical applications, the embodiment in which the user changes the image and the orientation of the virtual sound source through actions, and the embodiment in which the image change causes the orientation of the virtual sound source to change. During implementation, one of them can be selected or used in combination, for example, During video playback, if the user has no action and only the image changes, the orientation of the virtual sound source can be changed based on the image change; if the user has actions, both the image and the orientation of the virtual sound source can be changed. etc., can be determined as needed in actual applications, and this embodiment does not limit this.

In some examples, the determining the audio signal corresponding to each of the at least two players includes: obtaining scene information of the current audio playback scene, determining a binaural transfer function corresponding to the scene information, according to The binaural transfer function determines an audio signal corresponding to each of the at least two players. In this embodiment, the binaural transfer function is also called Head Related Transfer Functions (HRTF), which describes the transmission process of sound waves from the sound source to both ears. In actual applications, electronic devices are in different scenarios, and the propagation process from the sound source to the user's left and right ear auditory systems has different characteristics. Based on this, a variety of binaural transfer functions can be preset, and the electronic device can select when determining the audio signal. The binaural transfer function is consistent with the current scene, so that the processed audio signal can be consistent with the current scene, so that the audio signal has better sound effects. As an example, the scene information includes one or more of the following: audio type information, user type information, time information or environmental information of the user's environment, etc. For example, different audio types can use different sound effect processing. Identify the type of audio through audio classification algorithms or trained neural networks; different users may have different preferences, so different user types can use different sound effects processing, and user type information can be passed through user information with the user's authorization and consent. Determine; the time information can be obtained by electronic devices through the network and other methods. When the user listens to the audio in different time periods, different sound effect processing can be used; the environmental information of the user's environment can be determined in a variety of ways, for example, through The image sensor of the electronic device can be determined by collecting images of the environment, or can also be determined by collecting sound information of the surrounding environment, or can also be determined by acquiring geographical location information from the electronic device, etc. This embodiment does not limit this; practical application , you can select any one or a combination of the above information to determine the scene information as needed. Moreover, a variety of binaural transfer functions can be preset, so that the electronic device can select an appropriate binaural transfer function for sound effect processing according to needs during processing.

In some examples, the audio to be played has at least two virtual sound sources; the determining the audio signal corresponding to each of the at least two players includes: determining the audio signal corresponding to each of the virtual sound sources. The corresponding binaural transfer function is used to determine the audio signal corresponding to each of the at least two players, so that different virtual sound sources in the audio signal have different sound effects. In this embodiment, when the audio to be played includes at least two virtual sound sources, different binaural transfer functions can also be used to process different virtual sound sources, so that different virtual sound sources have different sound effects. For example, the audio to be played includes the singing sound of a singer who is relatively close in front of the user, and the sound of footsteps from one side of the user to the other which is far behind the user. By using different binaural transfer functions for the two sound sources, , different sound effects can be achieved, for example, the footsteps can be enhanced through the binaural transfer function, etc.

Next, an embodiment will be used to illustrate.

The audio processing of this embodiment can automatically convert ordinary binaural audio signals into VR sound effects, that is, simulate sound effects such as translation and rotation of sounds. The audio stream can be any audio signal that supports binaural playback, and the VR sound effects can be controlled by user actions or user action information set by audio editing software. This embodiment can be applied to electronic devices and audio software with binaural playback functions, such as headphones, mobile phones, cameras, VR devices, wearable devices that support audio playback, etc., as well as software tools with audio editing functions, etc. This embodiment can realize VR sound effects in which the sound follows the movement of the human body. The user can hear different sounds according to different movements and postures, and the changes in the sound are consistent with the movement of the human body. For example, based on external motion settings, static audio signals are processed in real time or post-processing to generate dynamic audio signals for users to play through binaural devices. VR sound effects move with people. As an example, the movement of audio can be decomposed into six degrees of freedom: three degrees of freedom of rotation and three degrees of freedom of translation. The processing process of this embodiment may include the steps shown in Figure 3A:

Step 301: Obtain the original audio signal.

Step 302: Obtain the user's action information.

Step 303: Perform VR sound effect processing on the original audio signal according to the user's action information.

Step 304: Output audio signals with VR sound effects.

Regarding step 301, obtaining the original audio signal, the audio signal originally used for direct playback can be obtained. As an example, the audio signal can be derived from the decoded signal of the audio file, or the audio stream signal in the streaming media, etc. If the audio signal is a mono signal, it can be copied into a two-channel signal with the same left and right sides. If the audio signal is a two-channel signal, it can be used directly.

Regarding step 302, obtain the user's action information; as an example, the user's action information can be determined based on the data collected by the action sensor. The action information is related to VR sound effects, and optionally can be six degrees of freedom action information. Six degrees of freedom of motion can simulate human movement in the sound field. The six degrees of freedom include three rotational degrees of freedom (α, β, γ) and three translational degrees of freedom (x, y, z). In practical applications, the number of degrees of freedom may be less than six, for example, there may be only three rotational degrees of freedom, or only one rotational degree of freedom, etc., which is not limited in this embodiment. In practical applications, there are many ways to determine the user's action information based on the data collected by the action sensor. For example, the user's movement and posture can be obtained by inertial sensors (such as VR headsets, mobile phones, smart watches, etc.), image sensors can obtain The user's movements and postures (such as identifying head movements, hand movements, eye movements, etc. through images), or obtaining brain electrical signals through sensors to obtain movement instructions from the brain; or, movement methods can also be defined through software , and all other ways to obtain external motion and posture.

Regarding step 303, perform VR sound effect processing on the original audio signal according to the user's action information.

For step 304, output an audio signal with VR sound effects. The audio signal can be a two-channel or multi-channel audio signal. The output audio signal is used for playing through a player, or for storage, or can be sent to other devices. Players on other devices play etc.

Optionally, as shown in Figure 3B, is a flow chart of an embodiment of the aforementioned step 303, which may include the following embodiments:

(1) Obtain the two-channel frequency domain signal based on the audio to be played.

The audio to be played in this embodiment can be mono or dual-channel. If it is a mono signal, it can be copied into a dual-channel signal. Among them, the sampling frequency of the time domain signal of the audio to be played is f _s , and the time domain signal of each channel is x _m (t), where m is the microphone serial number, m=1, 2, t is the sampling discrete time sequence, t =1,2,…. Among them, this embodiment uses two examples for description.

Optionally, N sampling points can be extracted from the two-channel time domain signal x _m (t) at intervals of L sampling points as a frame signal, expressed as x _m (n) _l . Among them, n is the time sequence within a frame signal, n=1, 2,...,N; l is the frame sequence, l=1, 2,.... Among them, N is called the frame length, L is called the frame shift, 0<L≤N, and N can be a power of 2.

Windowing is performed on the l-th frame of the binaural original time-domain signal x _m (n) _l , then the l-th frame of the windowed time-domain signal x′ _m (n) _l is

x′ _m (n) _l =x _m (n) _l h _ana (n) n＝1, 2,…,N

Among them, ha _ana (n) is the N-point analysis window function. Commonly used window functions include sine window or Hamming window. Perform discrete Fourier transform (DFT) on each channel windowed time domain signal x′ _m (n) _l of the l-th frame to obtain the N-point complex spectrum X _m (k) _l of the m-th channel of the l-th frame. , where k is a discrete spectrum sequence, k=1, 2,...,N.

Among them, e is a natural constant,

Is the unit imaginary number. As an example, fast Fourier transform (FFT) or the like can be used to accelerate calculations during the implementation process.

(2) Extract the sound direction of each discrete frequency from the two-channel frequency domain signal.

In this embodiment, a virtual static coordinate system O-XYZ can be established to represent the initial posture of the person. The initial posture includes the user's initial posture when the solution of this embodiment starts to be executed, such as the moment when the user puts on the headset, or The moment when the device is triggered to start getting the audio to be played. Of course, in actual applications, the coordinate system can also be established based on other methods such as a player, which is not limited in this embodiment.

As an example, each sound source of discrete frequency can be used as an independent sound source, distributed in the coordinate system, representing the sound source in various directions of the person, as shown in Figure 3C, which shows the sound source of each discrete frequency in the coordinate system. Azimuth diagram, each dot in Figure 3C represents a discrete frequency, and the sound source azimuth of each frequency is expressed as

Usually expressed in the Cartesian coordinate system as

Expressed in the spherical coordinate system as

The position of each independent sound source in space can be calculated based on the amplitude ratio and phase difference of the two-channel frequency domain signal, combined with the propagation model of sound to both ears. Optionally, the sound propagation model to both ears can use the Head Related Transfer Function (HRTF), or you can refer to the sound propagation model in the free sound field. Since the mono signal is copied and expanded into the same signal for the left and right channels, all sound sources can be considered to come from directly in front of the user.

(3) Obtain the user's action information.

The user's action information in this embodiment includes postures with six degrees of freedom: three rotational degrees of freedom (α, β, γ), and three translational degrees of freedom (x, y, z). In practical applications, according to needs, It may not be six degrees of freedom. In this embodiment, the data collected by the motion sensor can be used to determine the user's action information, where the user's action information can refer to the user's actual action parameters, that is, the user's actual action information in space.

In this embodiment, for a more immersive experience, the virtual sound field and the virtual sound source can be defined to be stationary relative to the aforementioned coordinate system O-XYZ, while the person moves relative to the coordinate system. Therefore, the user's actual movement parameters represent the person's movement relative to the virtual sound field.

In this embodiment, the user's actual movement information needs to be converted into the movement of the person relative to the virtual sound field. For example, the user's hand sliding to the right can be defined as the person moving to the right. Correspondingly, it can also be defined as dragging the sound field to the right, that is, the person moves to the left relative to the sound field. Therefore, the same action information of the user can correspond to the coordinate system. different motion parameters. Based on this, this embodiment can predefine the relationship between the user's actual action information and the motion parameters in the coordinate system. For example, the detected user's actual action information in space can be converted into motion in the coordinate system, that is, the aforementioned six degrees of freedom (α, β, γ) and (x, y, z); specifically, The user's actual action information in the space is converted into corresponding six degrees of freedom, which can be customized as needed in actual applications. This embodiment does not limit this.

(4) Calculate the relative orientation and relative translation parameters of the sound. The movement of six degrees of freedom represents the movement of a person, and the person moves from the O-XYZ coordinate system to the O'-X'Y'Z' coordinate system, as shown in Figure 3D, which shows a schematic diagram showing the change of the user's action in the coordinate system. , frequency discrete sound source coordinates in the coordinate system O-XYZ

Mapping to the new coordinate system O'-X'Y'Z' is

(5) Calculate the binaural transfer function. In this embodiment, the azimuth direction of each discrete frequency sound source can be calculated based on the head-related transfer function or the propagation model of the sound source in the free field.

The transfer functions propagated to both ears respectively, the transfer functions of the left and right ears are T _L (k) and T _R (k) respectively, both of which are complex numbers and have amplitude and phase components. If the propagation model of the sound source in the free sound field is adopted, the equivalent distance between the two ears needs to be determined. The distance between the two ears can be preset, or it can be measured by the at least two players, etc.

(6) Reconstruct binaural frequency domain signals. The two-channel signal is synthesized into one main audio signal X _main (k), which can be obtained by linear superposition of the two-channel complex spectrum X _m (k) _l , as described in the following formula:

Then, combining the binaural transfer functions T _L (k) and T _R (k), the left channel spectrum signal X _L (k) _l and the right channel spectrum signal X _R (k) _l can be generated respectively. The generation method can be as follows The formula states:

X _L (k) _l =T _L (k)X _main (k) _l

X _R (k) _l =T _R (k)X _main (k) _l

(7) Output binaural time domain signals. _For the left _and right channel spectral signals X _L ₍ k) _l _and and x' _R (n) _l , n=1, 2,...,N. Optionally, methods such as inverse fast Fourier transform (IFFT) can be used to accelerate calculations.

Add a synthesis window to the left and right channel time domain signals of the l-th frame to obtain the windowed time-domain signals of the l-th frame of the left and right channels, respectively:

x″ _L (n) _l =x′ _L (n) _l h _syn (n)

x″ _R (n) _l =x′ _R (n) _l h _syn (n)

Among them, h _syn (n) is a synthetic window function. Optionally, the synthetic window function may include a sine window or a Hamming window.

Optionally, through the overlap-add method (Overlap-add), the N-point time domain signals of the l-th frame of the left and right channels are overlapped and accumulated, respectively:

Among them, x″′ _L (n+M) _l-1 and x″′ _R (n+M) _l-1 are the overlapped and accumulated N-point time domain signals of the l-1th frame of the left and right channels respectively. If l=1, x″′ _L (n+M) _l-1 =0, x″′ _R (n+M) _l-1 =0.

Therefore, the overlapping and accumulated M point output signals of the lth frame of the left and right channels are the first M elements of x″′ _L (n) _l and x″′ _R (n) _l respectively:

x _L (n) _l =x″′ _L (n) _l n＝1, 2,…,M

x _R (n) _l =x″′ _R (n) _l n＝1, 2,…,M

x _L (n) _l and x _R (n) _l are the left and right channel output audio signals of the l-th frame, which have VR sound effects that move with people.

It can be seen from the above embodiment that this embodiment can realize any mono/dual channel sound source to have a VR sound effect that follows the movement of the person during playback. For example, when the person is moving, he can experience an absolutely still virtual sound field. ; For example, if the sound is initially in the front, when the person turns to the right, the sound will also turn to the left relative to the person, giving the user the illusion that the sound is stationary in a virtual space.

The above audio processing method embodiments can be implemented by software, or can be implemented by hardware or a combination of software and hardware. Taking software implementation as an example, as a device in a logical sense, it is formed by reading the corresponding computer program instructions in the non-volatile memory into the memory and running it through the image processing processor where it is located. From the hardware level, as shown in Figure 4, it is a hardware structure diagram of the audio processing device 400 that implements the audio processing method of this embodiment. In addition to the processor 401 and the memory 402 shown in Figure 4, the embodiment The audio processing device used to implement the audio processing method usually may also include other hardware according to the actual functions of the audio processing device, which will not be described again.

In this embodiment, the processor 401 implements the following steps when executing the computer program:

Obtain the orientation information of the virtual sound source in the audio to be played in the virtual sound field, which is established based on the positional relationship between at least two players and the user's left and right hearing organs;

Use a motion sensor to obtain the user's motion information, and generate a sound source orientation adjustment instruction based on the motion information;

In some examples, the audio to be played has at least two virtual sound sources, and the type information of the at least two virtual sound sources is different;

The processor 401 executes the generation of sound source orientation adjustment instructions based on the action information, including:

Based on the action information, a sound source orientation adjustment instruction for adjusting the orientation of each virtual sound source in the at least two virtual sound sources is generated, so that along with the user's actions, virtual sound sources of different types of information are adjusted in the virtual sound source. The orientation of the sound field has different amounts of change.

In some examples, the type information includes one or more of the following:

The orientation information, timbre information or volume information of the virtual sound source in the virtual sound field in the audio to be played.

In some examples, the electronic device includes a display area;

The processor 401 executes the generation of a sound source orientation adjustment instruction based on the action information, including: generating a sound source orientation adjustment instruction and an image display instruction based on the action information;

The processor 401 executes the step of playing the audio signal through the corresponding player, including:

Control the corresponding player to play the audio signal, and at the same time obtain multiple frames of images displayed synchronously with the audio to be played based on the image display instruction and display them in the display area; wherein the image includes the The first pixel area related to the virtual sound source, the change of the first pixel area in the multi-frame images, and the position of the virtual sound source in the virtual sound field all change with the user's actions.

In some examples, the electronic device includes a display area, the display area is used to display a multi-frame image synchronized with the audio to be played, the image includes a first pixel area related to the virtual sound source; The processor 401 also performs:

When displaying the current image, obtain the position change of the first pixel area in the current image relative to the image before the current image;

Based on the position change, the orientation information of the virtual sound source and the audio to be played, an audio signal corresponding to each of the at least two players is determined, and the corresponding player is controlled to play the audio signal.

In some examples, the user's action information includes the change amount of the user's action. The processor 401 executes the generation of sound source orientation adjustment instructions based on the action information, including:

Based on the change amount of the user action and the first mapping relationship, a sound source orientation adjustment instruction for adjusting the change amount of the virtual sound source in the virtual sound field is generated; wherein the first mapping relationship includes: the change of the user action The corresponding relationship between the quantity and the change quantity of the virtual sound source's position in the virtual sound field.

In some examples, the first mapping relationship is preset, or generated by obtaining the user's setting instructions, or determined based on the user's historical behavior data.

In some examples, the orientation information of the virtual sound source in the virtual sound field in the audio to be played is determined through the first coordinate system;

The change amount of the user action is determined through the first coordinate system.

In some examples, the first coordinate system is a coordinate system established based on the user's initial posture using the at least two players.

In some examples, the electronic device is a wearable electronic device, and the motion sensor includes an inertial measurement sensor;

The processor 401 executes the method of using the motion sensor to obtain the user's motion information, including:

Based on the measurement data of the inertial measurement sensor, the user's action information is determined.

In some examples, the electronic device is a wearable electronic device, and the motion sensor includes a first image sensor; when the wearable electronic device is worn on the user's head, the first image sensor faces the user's eyes;

Obtain the image collected by the first image sensor, and obtain the movement information of the user's eyeball based on the image.

In some examples, the electronic device is a wearable electronic device, and the motion sensor includes one or more second image sensors. When the wearable electronic device is worn on the user's head, one or more of the second image sensors The observation range of the image sensor covers the activity space of the user's hand;

Obtain images collected by the one or more second image sensors, and if the user's hand is detected from the image, obtain action information of the user's hand.

In some examples, the processor 401 performs the determining the audio signal corresponding to each of the at least two players, including:

Scene information of the current audio playback scene is obtained, a binaural transfer function corresponding to the scene information is determined, and an audio signal corresponding to each of the at least two players is determined based on the binaural transfer function.

In some examples, the scene information includes one or more of the following: audio type information, user type information, time information, or environmental information of the user's environment.

In some examples, the audio to be played has at least two virtual sound sources;

The processor 401 performs the determination of the audio signal corresponding to each player in the at least two players, including:

Determine a binaural transfer function corresponding to each of the virtual sound sources, and use the determined binaural transfer function to determine an audio signal corresponding to each of the at least two players, so that different virtual sounds in the audio signal Sources have different sound effects.

In some examples, the orientation information of the virtual sound source in the virtual sound field in the audio to be played is obtained by extracting multiple discrete frequencies from the audio to be played, and based on the multiple discrete frequencies in the virtual sound field. The orientation information is determined.

In some examples, the audio to be played is an audio signal of at least two channels; the orientation information of the multiple discrete frequencies in the virtual sound field is obtained in the following manner:

According to the amplitude ratio and/or phase difference of the audio signals of the at least two channels, the orientation information of the multiple discrete frequencies in the virtual sound field is obtained based on the first coordinate system and the binaural transfer function.

As shown in Figure 5, an embodiment of the application further provides an electronic device 500, including: at least two players 510 and a motion sensor 520; the electronic device also includes a processor 530, a memory 540, and a computer program executable by said processor;

When the processor executes the computer program, the following steps are implemented:

The processor executes the generation of sound source orientation adjustment instructions based on the action information, including:

In some examples, the type information includes one or more of the following:

In some examples, the electronic device includes a display area;

The processor executes the generation of a sound source orientation adjustment instruction based on the action information, including: generating a sound source orientation adjustment instruction and an image display instruction based on the action information;

The processor executes the step of playing the audio signal through the corresponding player, including:

Control the corresponding player to play the audio signal, and at the same time obtain multiple frames of images displayed synchronously with the audio to be played based on the image display instruction and display them in the display area; wherein the image includes the The first pixel area related to the virtual sound source, the change of the first pixel area in the multi-frame image, and the position of the virtual sound source in the virtual sound field all change with the user's actions.

In some examples, the electronic device includes a display area, the display area is used to display a multi-frame image synchronized with the audio to be played, the image includes a first pixel area related to the virtual sound source; The processor 401 also executes:

In some examples, the user's action information includes the change amount of the user's action, and the processor executes the generation of sound source orientation adjustment instructions based on the action information, including:

The processor executes the method of using the motion sensor to obtain the user's motion information, including:

In some examples, the processor performing the determining an audio signal corresponding to each of the at least two players includes:

The processor performs the determining of an audio signal corresponding to each of the at least two players, including:

Embodiments of this specification also provide a computer-readable storage medium, which stores a number of computer instructions. When executed, the computer instructions implement the steps of the audio processing method described in any embodiment.

Embodiments of the present description may take the form of a computer program product implemented on one or more storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) having program code embodied therein. Storage media available for computers include permanent and non-permanent, removable and non-removable media, and can be implemented by any method or technology to store information. Information may be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to: phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, Magnetic tape cassettes, tape magnetic disk storage or other magnetic storage devices or any other non-transmission medium can be used to store information that can be accessed by a computing device.

As for the device embodiment, since it basically corresponds to the method embodiment, please refer to the partial description of the method embodiment for relevant details. The device embodiments described above are only illustrative. The units described as separate components may or may not be physically separated. The components shown as units may or may not be physical units, that is, they may be located in One location, or it can be distributed across multiple network units. Some or all of the modules can be selected according to actual needs to achieve the purpose of the solution of this embodiment. Persons of ordinary skill in the art can understand and implement the method without any creative effort.

The steps of the various methods above are divided just for the purpose of clear description. During implementation, they can be combined into one step or some steps can be split into multiple steps. As long as they include the same logical relationship, they are all within the scope of protection of this patent. ; Adding insignificant modifications or introducing insignificant designs to the algorithm or process, but not changing the core design of the algorithm and process, are within the scope of protection of this application.

The description of "specific examples" or "some examples" means that the specific features, structures, materials or characteristics described in connection with the embodiments or examples are included in at least one embodiment or example of this specification. In this specification, schematic representations of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials or characteristics described may be combined in any suitable manner in any one or more embodiments or examples.

It should be noted that in this article, relational terms such as first and second are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply that these entities or operations are mutually exclusive. any such actual relationship or sequence exists between them. The terms "comprises," "comprises," or any other variation thereof are intended to cover a non-exclusive inclusion such that a process, method, article or apparatus including a list of elements includes not only those elements but also others not expressly listed elements, or elements inherent to such process, method, article or equipment. Without further limitation, an element defined by the statement "comprises a..." does not exclude the presence of additional identical elements in a process, method, article, or apparatus that includes the stated element.

The methods and devices provided by the embodiments of the present invention have been introduced in detail above. Specific examples are used in this article to illustrate the principles and implementations of the present invention. The description of the above embodiments is only used to help understand the method and its implementation of the present invention. Core idea; at the same time, for those of ordinary skill in the art, there will be changes in the specific implementation and application scope based on the idea of the present invention. In summary, the content of this description should not be understood as a limitation of the present invention. .

Claims

An audio processing method, characterized in that the method is applied to an electronic device, the electronic device includes at least two players and a motion sensor, and the method includes:

Obtaining the orientation information of the virtual sound source in the audio to be played in the virtual sound field, the virtual sound field is established based on the positional relationship between the at least two players and the user's left and right hearing organs;

Using the motion sensor to obtain the user's motion information, and generating a sound source orientation adjustment instruction based on the motion information;

Based on the sound source position adjustment instruction, the orientation information of the virtual sound source and the audio to be played, an audio signal corresponding to each of the at least two players is determined, and passed through the corresponding The player plays the audio signal.
The method according to claim 1, characterized in that there are at least two virtual sound sources in the audio to be played, and the type information of the at least two virtual sound sources is different;

Generating sound source orientation adjustment instructions based on the action information includes:

Based on the action information, a sound source orientation adjustment instruction for adjusting the orientation of each virtual sound source in the at least two virtual sound sources is generated, so that along with the user's actions, virtual sound sources of different types of information are adjusted in the virtual sound source. The orientation of the sound field has different amounts of change.
The method according to claim 2, characterized in that the type information includes one or more of the following:

The orientation information, timbre information or volume information of the virtual sound source in the virtual sound field in the audio to be played.
The method according to claim 1, wherein the electronic device includes a display area;

Generating a sound source orientation adjustment instruction based on the action information includes: generating a sound source orientation adjustment instruction and an image display instruction based on the action information;

Playing the audio signal through the corresponding player includes:

Control the corresponding player to play the audio signal, and at the same time obtain multiple frames of images displayed synchronously with the audio to be played based on the image display instruction and display them in the display area; wherein the image includes the The first pixel area related to the virtual sound source, the change of the first pixel area in the multi-frame image, and the position of the virtual sound source in the virtual sound field all change with the user's actions.
The method according to claim 1, characterized in that the electronic device includes a display area, the display area is used to display multiple frames of images synchronized with the audio to be played, the images include images that are synchronized with the virtual sound source. The relevant first pixel area; the method further includes:

When displaying the current image, obtain the position change of the first pixel area in the current image relative to the image before the current image;

Based on the position change, the orientation information of the virtual sound source and the audio to be played, an audio signal corresponding to each of the at least two players is determined, and the corresponding player is controlled to play the audio signal.
The method according to claim 1, wherein the user's action information includes a change amount of the user's action; and generating a sound source orientation adjustment instruction based on the action information includes:

Based on the change amount of the user action and the first mapping relationship, a sound source orientation adjustment instruction for adjusting the change amount of the virtual sound source in the virtual sound field is generated; wherein the first mapping relationship includes: the change of the user action The corresponding relationship between the quantity and the change quantity of the virtual sound source's position in the virtual sound field.
The method according to claim 6, characterized in that the first mapping relationship is preset, or generated by obtaining the user's setting instructions, or determined based on the user's historical behavior data.
The method according to claim 6, characterized in that the orientation information of the virtual sound source in the virtual sound field in the audio to be played is determined through the first coordinate system;

The change amount of the user action is determined through the first coordinate system.
The method of claim 8, wherein the first coordinate system includes: a coordinate system established based on a user's initial posture using the at least two players.
The method of claim 1, wherein the electronic device is a wearable electronic device, and the motion sensor includes an inertial measurement sensor;

The use of the motion sensor to obtain the user's motion information includes:

Based on the measurement data of the inertial measurement sensor, the user's action information is determined.
The method of claim 1, wherein the electronic device is a wearable electronic device, and the motion sensor includes a first image sensor; when the wearable electronic device is worn on the user's head, the first image sensor The image sensor faces the user's eyes;

The use of the motion sensor to obtain the user's motion information includes:

Obtain the image collected by the first image sensor, and obtain the movement information of the user's eyeball based on the image.
The method of claim 1, wherein the electronic device is a wearable electronic device, the motion sensor includes one or more second image sensors, and when the wearable electronic device is worn on the user's head, The observation range of one or more second image sensors covers the activity space of the user's hand;

The use of the motion sensor to obtain the user's motion information includes:

Obtain images collected by the one or more second image sensors, and if the user's hand is detected from the image, obtain action information of the user's hand.
The method of claim 1, wherein determining the audio signal corresponding to each of the at least two players includes:

Scene information of the current audio playback scene is obtained, a binaural transfer function corresponding to the scene information is determined, and an audio signal corresponding to each of the at least two players is determined based on the binaural transfer function.
The method according to claim 13, wherein the scene information includes one or more of the following: audio type information, user type information, time information or environmental information of the user's environment.
The method according to claim 1, characterized in that the audio to be played has at least two virtual sound sources;

The determination of an audio signal corresponding to each of the at least two players includes:

Determine a binaural transfer function corresponding to each of the virtual sound sources, and use the determined binaural transfer function to determine an audio signal corresponding to each of the at least two players, so that different virtual sounds in the audio signal Sources have different sound effects.
The method according to claim 1 or 8, characterized in that the orientation information of the virtual sound source in the virtual sound field in the audio to be played is obtained by extracting a plurality of discrete frequencies from the audio to be played, according to the A plurality of discrete frequencies are determined by the azimuth information in the virtual sound field.
The method according to claim 16, characterized in that the audio to be played is an audio signal of at least two channels; the orientation information of the multiple discrete frequencies in the virtual sound field is obtained in the following manner:

According to the amplitude ratio and/or phase difference of the audio signals of the at least two channels, the orientation information of the multiple discrete frequencies in the virtual sound field is obtained based on the first coordinate system and the binaural transfer function.
An audio processing device, characterized in that the device includes a processor, a memory, and a computer program stored on the memory and executable by the processor. When the processor executes the computer program, claim 1 is realized. to any of the methods described in 17.
An electronic device, characterized in that the electronic device includes at least two players and a motion sensor; the electronic device further includes a processor, a memory, and a computer program stored on the memory and executable by the processor. ;

When the processor executes the computer program, the method of any one of claims 1 to 17 is implemented.
A computer-readable storage medium, characterized in that a number of computer instructions are stored on the computer-readable storage medium, and when the computer instructions are executed, the steps of the method described in any one of claims 1 to 17 are implemented.