WO2022033236A1

WO2022033236A1 - Audio enhancement method and apparatus, storage medium, and wearable device

Info

Publication number: WO2022033236A1
Application number: PCT/CN2021/104553
Authority: WO
Inventors: 林贻鸿
Original assignee: Oppo广东移动通信有限公司
Priority date: 2020-08-11
Filing date: 2021-07-05
Publication date: 2022-02-17
Also published as: CN111935573B; CN111935573A

Abstract

Disclosed in embodiments of the present application are an audio enhancement method and apparatus, a storage medium, and a wearable device. The method comprises: acquiring a first audio signal by a bone conduction microphone, and acquiring a second audio signal by a non-bone conduction microphone; according to the first audio signal, detecting whether the wearer produces voice; if yes, performing enhancement on the first audio signal to obtain a voice-enhanced signal; and if not, performing enhancement on the second audio signal to obtain an environment sound-enhanced signal of the real environment.

Description

Audio enhancement method, device, storage medium and wearable device

This application claims the priority of the Chinese patent application with the application number 202010802651.8 and the invention titled "Audio Enhancement Method, Device, Storage Medium and Wearable Device" filed with the China Patent Office on August 11, 2020, the entire contents of which are by reference Incorporated in this application.

technical field

The present application relates to the technical field of audio processing, and in particular, to an audio enhancement method, device, storage medium and wearable device.

Background technique

At present, wearable devices such as smart glasses and smart helmets have gradually entered people's lives. By installing different applications on the wearable device, the wearable device can realize various functions. For example, by installing an application with audio collection capability on the wearable device, and then combining with the audio collection hardware (such as a microphone) configured by the wearable device itself, the audio collection function is provided to the user.

SUMMARY OF THE INVENTION

Embodiments of the present application provide an audio enhancement method, apparatus, storage medium, and wearable device, which can improve the flexibility of the wearable device for audio enhancement.

In a first aspect, an embodiment of the present application provides an audio enhancement method, which is applied to a wearable device, where the wearable device includes a bone conduction microphone and a non-bone conduction microphone, and the audio enhancement method includes:

The first audio signal is acquired by collecting the bone conduction microphone, and the second audio signal is simultaneously acquired by the non-bone conduction microphone;

When the wearer's pronunciation is detected according to the first audio signal, the first audio signal is enhanced to obtain the wearer's pronunciation enhancement signal;

When it is detected that the wearer does not speak according to the first audio signal, enhancement processing is performed on the second audio signal to obtain an ambient sound enhancement signal of the real environment where the wearer is located.

In a second aspect, an embodiment of the present application provides an audio enhancement device, which is applied to a wearable device, where the wearable device includes a bone conduction microphone and a non-bone conduction microphone, and the audio enhancement device includes:

an audio acquisition module, configured to acquire a first audio signal through the bone conduction microphone, and acquire a second audio signal through the non-bone conduction microphone synchronously;

A pronunciation enhancement module, configured to perform enhancement processing on the first audio signal when the wearer's pronunciation is detected according to the first audio signal, to obtain the wearer's pronunciation enhancement signal;

An ambient sound enhancement module, configured to perform enhancement processing on the second audio signal when it is detected that the wearer has not spoken according to the first audio signal, to obtain an ambient sound enhancement signal of the actual environment where the wearer is located .

In a third aspect, the embodiments of the present application provide a storage medium on which a computer program is stored, and when the computer program is loaded by a wearable device including a bone conduction microphone and a non-bone conduction microphone, the storage medium provided by the embodiments of the present application is executed. Audio enhancement method.

In a fourth aspect, an embodiment of the present application provides a wearable device, the wearable device includes a bone conduction microphone, a non-bone conduction microphone, a processor, and a memory, the memory stores a computer program, and the processor loads the The computer program executes the audio enhancement method provided by the embodiments of the present application.

Description of drawings

In order to illustrate the technical solutions in the embodiments of the present application more clearly, the following briefly introduces the drawings that are used in the description of the embodiments. Obviously, the drawings in the following description are only some embodiments of the present application. For those of ordinary skill in the art, other drawings can also be obtained from these drawings without creative effort.

FIG. 1 is a schematic flowchart of an audio enhancement method provided by an embodiment of the present application.

FIG. 2 is a schematic diagram of the arrangement positions of the bone conduction microphone and the non-bone conduction microphone when the physical representation form of the wearable device is smart glasses according to the embodiment of the present application.

FIG. 3 is a schematic view of wearing the smart glasses in FIG. 2 .

FIG. 4 is a schematic diagram of enhancing processing of a first audio signal in an embodiment of the present application.

FIG. 5 is another schematic diagram of performing enhancement processing on the first audio signal in an embodiment of the present application.

FIG. 6 is a schematic diagram of the arrangement positions of multiple non-bone conduction microphones when the physical representation form of the wearable device is smart glasses according to an embodiment of the present application.

FIG. 7 is a schematic diagram of predicting the direction of a sound source in a real environment in an embodiment of the present application.

FIG. 8 is a schematic diagram of the setting position of the camera module when the physical presentation form of the wearable device is smart glasses according to the embodiment of the present application.

FIG. 9 is another schematic flowchart of an audio enhancement method provided by an embodiment of the present application.

FIG. 10 is a schematic structural diagram of an audio enhancement apparatus provided by an embodiment of the present application.

FIG. 11 is a schematic structural diagram of a wearable device provided by an embodiment of the present application.

detailed description

Please refer to the drawings, wherein the same component symbols represent the same components, and the principles of the present application are exemplified by being implemented in a suitable computing environment. The following description is based on illustrated specific embodiments of the present application and should not be construed as limiting other specific embodiments of the present application not detailed herein.

It should be noted that the relational terms such as first and second involved in the following embodiments of this application are only used to distinguish one entity or operation from another entity or operation, and are not used to limit these entities or operations. There is an actual sequential relationship between them.

In related technologies, such as smart glasses and smart helmets, in order to better enhance the user's voice, in the design stage, the position of the user and the microphone needs to be designed and measured, and the position of the user's mouth and the position of the microphone are relatively fixed. . After that, the relative positions of the mouth and the microphone are further used to adjust the parameters of the algorithm, so that the microphone has a good enhancement effect on the user's voice and a good suppression effect on the environmental noise. However, in the related art, the wearable device only enhances the user's voice, and suppresses other sounds. When the user wants to collect non-personal voices, these sounds will be suppressed as noise.

To this end, the present application provides an audio enhancement method, an audio enhancement device, a storage medium, and a wearable device, and the executive body of the audio enhancement method may be the audio enhancement device provided in the embodiment of the present application, or an audio enhancement device integrated with the audio enhancement device. Wearable devices, in which the audio enhancement device can be implemented in hardware or software. It should be noted that the embodiments of the present application do not specifically limit the physical presentation form of the wearable device, for example, the physical presentation form of the wearable device may be smart glasses, smart helmets, and the like.

An embodiment of the present application provides an audio enhancement method, which is applied to a wearable device, where the wearable device includes a bone conduction microphone and a non-bone conduction microphone, and the audio enhancement method includes:

In one embodiment, the performing enhancement processing on the first audio signal to obtain the wearer's pronunciation enhancement signal includes:

acquiring a historical bone conduction noise signal of the wearer before the first audio signal is collected;

Acquiring a bone conduction noise signal of the wearer during the collection of the first audio signal according to the historical bone conduction noise signal;

The anti-phase superposition of the bone conduction noise signal and the first audio signal is performed to obtain the pronunciation enhancement signal.

In one embodiment, the acquiring the bone conduction noise signal of the wearer during the collection of the first audio signal according to the historical bone conduction noise signal includes:

Perform model training according to the historical bone conduction noise signal to obtain a noise prediction model corresponding to the wearer;

The bone conduction noise signal during the acquisition of the first audio signal is predicted according to the noise prediction model.

The pronunciation enhancement signal is obtained by performing pronunciation enhancement processing on the first audio signal by using a pre-trained pronunciation enhancement model.

In one embodiment, there are multiple non-bone conduction microphones, and the second audio signal is enhanced to obtain an enhanced ambient sound signal of the actual environment where the wearer is located, including:

predicting the sound source direction of the real environment according to the plurality of second audio signals collected by the plurality of non-bone conduction microphones;

The second audio signal is subjected to beamforming processing according to the sound source direction to obtain the ambient sound enhancement signal.

In one embodiment, the audio enhancement method further includes:

extracting spectral features of the first audio signal;

Pronunciation detection is performed on the spectral feature by using a pre-trained pronunciation detection model to obtain a detection result of whether the wearer has spoken.

In one embodiment, the synchronization performs audio collection through the bone conduction microphone and the non-bone conduction microphone to obtain the first audio signal collected by the bone conduction microphone, and after obtaining the second audio signal collected by the non-bone conduction microphone. ,Also includes:

When the current configuration is to enhance the pronunciation of the wearer, the first audio signal is enhanced according to the second audio signal; or,

When the current configuration is to enhance the ambient sound, the second audio signal is enhanced according to the first audio signal.

In one embodiment, the audio enhancement method further includes:

When the wearer is in motion, three-dimensional reconstruction is performed on the real environment to obtain a virtual reality environment;

Identifying whether the wearer is at risk of exercise according to the virtual reality environment;

When it is recognized that the wearer has a movement risk, a safety prompt is given to the wearer.

In one embodiment, the wearable device further includes a camera module, and the three-dimensional reconstruction of the real environment to obtain a virtual reality environment includes:

Obtain the depth image and color image of the real environment through the camera module;

The virtual reality environment is obtained by performing three-dimensional reconstruction of the real environment according to the depth image and the color image. Please refer to FIG. 1 , which is a schematic flowchart of an audio enhancement method provided by an embodiment of the present application. The following description takes the wearable device provided by the present application as an example of the execution subject of the audio enhancement method, where the wearable device includes a bone conduction microphone and a non-bone conduction microphone. As shown in FIG. 1 , the process of the audio enhancement method provided by the embodiment of the present application may be as follows:

In 101, a first audio signal is acquired through a bone conduction microphone, and a second audio signal is simultaneously acquired through a non-bone conduction microphone.

It should be noted that the transmission medium of sound includes solid, liquid and air, and bone conduction of sound belongs to the solid conduction of sound wave. Bone conduction is a very common physiological phenomenon. For example, we hear the sound of chewing food, which is transmitted to the inner ear through the jawbone. When the user speaks, the vibration is generated through the vocal organ, which is then transmitted to other parts, such as the bridge of the nose and the ear bone, through the body tissues such as bones, muscles and skin. Using this principle, the bone conduction microphone can convert the vibration of the body part caused by the user's pronunciation into sound signals, thereby restoring the sound made by the user.

In the embodiment of the present application, the wearable device includes a bone conduction microphone and a non-bone conduction microphone. Wherein, the non-bone conduction microphone includes any type of microphone for audio collection based on the principle of sound air transmission and/or liquid transmission, including but not limited to electric type, condenser type, piezoelectric type, electromagnetic type, carbon particle type and semiconductor type and other types of microphones.

2, taking the physical representation of the wearable device as smart glasses as an example, the non-bone conduction microphone is arranged at one end of the temple near the frame, and the bone conduction microphone is arranged at one end of the temple frame. Referring to FIG. 3 , when the smart glasses are in a wearing state, the bone conduction microphone will directly contact the user's ear bones, while the non-bone conduction microphone will not directly contact the user.

It should be noted that FIG. 2 shows only an optional arrangement of the bone conduction microphone and the non-bone conduction microphone, and the bone conduction microphone and the non-bone conduction microphone are not limited to this arrangement in a specific implementation. For example, the bone conduction microphone can also be arranged on the frame of the smart glasses and contact the user's nose bridge when wearing it.

In the embodiment of the present application, when audio collection is triggered, the wearable device synchronously performs audio collection through the bone conduction microphone and the non-bone conduction microphone, and the audio signal collected by the bone conduction microphone is recorded as the first audio signal, and the non-bone conduction microphone is recorded as the first audio signal. The audio signal collected by the microphone is recorded as the second audio signal.

It should be noted that the embodiments of this application do not specifically limit how to trigger audio collection. For example, it may be when the user operates the wearable device to start the audio collection application for recording, or when the user operates the wearable device to start the audio collection. When an instant messaging application conducts an audio and video call, it can also trigger audio collection when the wearable device performs voice wake-up or voice recognition.

In 102, when the voice of the wearer is detected according to the first audio signal, the first audio signal is enhanced to obtain a voice enhancement signal of the wearer.

According to the audio collection principle of the bone conduction microphone, there is less interference from the wearer when the bone conduction microphone collects audio. Therefore, the first audio signal collected by the bone conduction microphone can be effectively used to detect whether the wearer of the wearable device is pronounce.

The wearable device may detect whether the wearer of the wearable device has spoken according to the first audio signal according to the configured pronunciation detection strategy. It should be noted that the embodiment of the present application does not specifically limit the configuration of the pronunciation detection strategy, which can be configured by those of ordinary skill in the art according to actual needs. For example, the pronunciation detection strategy can be configured as follows: using a voice activity detection (Voice Activity Detection, VAD) algorithm to detect whether the wearer of the wearable device is vocal.

In the embodiment of the present application, if the pronunciation of the wearer of the wearable device is detected according to the first audio signal, the pronunciation of the wearer is used as the enhancement object, and the first audio signal is enhanced according to the configured pronunciation enhancement strategy, and the corresponding result is obtained The wearer's pronunciation enhances the signal. The enhancement processing on the first audio signal can be regarded as a process of eliminating the non-wearer's pronunciation component in the first audio signal, and the purpose of enhancing the wearer's pronunciation is achieved by eliminating the non-wearer's pronunciation component.

The embodiments of the present application do not specifically limit the configuration of pronunciation enhancement strategies, which can be configured by those of ordinary skill in the art according to actual needs, including artificial intelligence-based pronunciation enhancement strategies and non-artificial intelligence pronunciation enhancement strategies.

In 103, when it is detected that the wearer does not speak according to the first audio signal, an enhancement process is performed on the second audio signal to obtain an ambient sound enhancement signal of the real environment where the wearer is located.

In this embodiment of the present application, if it is detected that the wearer of the wearable device does not speak according to the first audio signal, the ambient sound of the real environment where the wearer is located is used as the enhancement object, and the second sound is enhanced according to the configured ambient sound enhancement strategy. The audio signal is enhanced and processed to correspondingly obtain an enhanced signal of the ambient sound of the actual environment where the wearer is located.

The configuration of the corresponding environmental sound enhancement strategy in the embodiments of the present application is not specifically limited, and can be configured by those of ordinary skill in the art according to actual needs, including artificial intelligence-based pronunciation enhancement strategies and non-artificial intelligence pronunciation enhancement strategies.

It can be seen from the above that the audio enhancement method provided by the present application is suitable for wearable devices including bone conduction microphones and non-bone conduction microphones. signal; according to the first audio signal to detect whether the wearer of the wearable device pronounces; if the wearer's pronunciation is detected, then the wearer's pronunciation is used as an enhancement object, and the first audio signal is enhanced to obtain the wearer's pronunciation enhancement. If it is detected that the wearer does not speak, the ambient sound of the wearer's real environment is used as the enhancement object, and the second audio signal is enhanced to obtain the corresponding ambient sound enhancement signal of the wearer's real environment. Therefore, the present application flexibly performs audio enhancement processing by arranging bone conduction microphones and non-bone conduction microphones in the wearable device, and using the audio collection results of the bone conduction microphones to dynamically determine audio enhancement objects.

Optionally, in an embodiment, the enhancement processing is performed on the first audio signal to obtain the wearer's pronunciation enhancement signal, including:

(1) Obtain the historical bone conduction noise signal of the wearer before the first audio signal is collected;

(2) acquiring the wearer's bone conduction noise signal during the first audio signal collection period according to the historical bone conduction noise signal;

(3) The anti-phase superposition of the bone conduction noise signal and the first audio signal is performed to obtain a pronunciation enhancement signal.

The embodiments of the present application further provide an optional pronunciation enhancement strategy.

It is easy to understand that, just as there are various noises in the environment (for example, there are noises generated by computer operation in the office, noises generated by tapping on the keyboard, etc.), there are also various noises in the human body, such as the human body. breath sounds, heartbeat sounds, chewing sounds, and cough sounds. Then, when the wearable device collects audio through the bone conduction microphone, it is obviously difficult to collect the pure pronunciation signal. The first audio signal collected by the wearable device includes not only the pronunciation components of the wearer, but also the breath sound and heartbeat of the speaker. sounds, chewing sounds and/or cough sounds.

As mentioned above, since the external environment has less interference with the bone conduction microphone, that is, the interference of the bone conduction microphone mainly comes from the wearer himself. In the embodiment of the present application, the first audio signal is regarded as composed of two parts, that is, the pure wearer's voice signal and the wearer's bone conduction noise (including but not limited to breath sound, heartbeat sound, chewing sound, coughing sound, etc. ). Therefore, the enhancement of the wearer's voice signal can be achieved by eliminating the bone conduction noise in the first audio signal.

It can be understood that if the wearer of the wearable device speaks, the wearable device will collect the mixed signal of the wearer's voice signal and the bone conduction noise signal through the bone conduction microphone. If the wearer does not speak, the wearable device will only The wearer's bone conduction noise signal is collected through the bone conduction microphone. In the embodiment of the present application, when the wearer does not speak, the wearable device buffers the collected bone conduction noise signal.

When the first audio signal is enhanced, the wearable device first acquires the wearer's bone conduction noise signal before the first audio signal is collected, which is recorded as a historical bone conduction noise signal. For example, the wearable device takes the collection start time of the first audio signal as the end time, and obtains the preset time length collected before the end time (the preset time length can be selected by a person of ordinary skill in the art according to actual needs. Appropriate value, this application implements For example, there is no specific limitation on this, for example, it can be set to a historical bone conduction noise signal of 500ms).

For example, if the preset duration is configured to be 500 milliseconds, and the start time of the aforementioned first audio signal is 09:13:56 and 500 milliseconds on July 28, 2020, then the wearable device obtains the first audio signal at 09:00 on July 28, 2020. From 13 minutes 56 seconds to 09:13:56 seconds and 500 milliseconds on July 28, 2020, the historical bone conduction noise signal with a duration of 500 milliseconds was buffered by the aforementioned bone conduction microphone.

After acquiring the wearer's historical bone conduction noise signal before the first audio signal acquisition, the wearable device further acquires the wearer's bone conduction noise signal during the first audio signal acquisition according to the historical bone conduction noise signal.

For example, the wearable device can obtain the noise distribution characteristics of historical bone conduction noise signals, and then generate a noise signal with the same duration as the first audio signal according to the noise distribution characteristics, as the bone conduction noise signal of the wearer during the collection of the first audio signal .

For another example, considering the stability of noise, its variation in continuous time is usually small, and the wearable device can also directly use the acquired historical bone conduction noise signal as the bone conduction noise signal during the aforementioned first audio signal acquisition, wherein, If the duration of the historical bone conduction noise signal is greater than the duration of the aforementioned first audio signal, a portion of the same duration as the aforementioned first audio signal may be cut from the historic bone conduction noise signal, as the wearer’s data during the collection of the aforementioned first audio signal. Bone conduction noise signal; if the duration of the historical bone conduction noise signal is less than the duration of the first audio signal, the historical bone conduction noise signal can be copied to obtain multiple historical bone conduction noises, and then multiple historical bone conduction noise signals can be spliced A noise signal with the same duration as the aforementioned first audio signal is obtained as a bone conduction noise signal of the wearer during the aforementioned first audio signal collection period.

Referring to FIG. 4 , after acquiring the bone conduction noise signal of the wearer during the first audio signal collection period, the wearable device first performs anti-phase processing on the aforementioned bone conduction noise signal to obtain an anti-phase bone conduction noise signal; Then, the wearable device further superimposes the anti-phase bone conduction noise signal and the first audio signal, thereby eliminating the bone conduction noise part in the first audio signal, and obtaining a pure voice part of the wearer, that is, a voice enhancement signal.

Optionally, in an embodiment, acquiring the wearer's bone conduction noise signal during the first audio signal acquisition period according to the historical bone conduction noise signal, including:

(1) Perform model training according to historical bone conduction noise signals to obtain a noise prediction model corresponding to the wearer;

(2) Predicting the bone conduction noise signal during the first audio signal acquisition period according to the noise prediction model.

The embodiments of the present application further provide a solution for optionally acquiring the wearer's bone conduction noise signal during the first audio signal acquisition period according to the historical bone conduction noise signal.

Wherein, after acquiring the historical bone conduction noise signal, the wearable device uses the historical bone conduction noise signal as sample data, and performs model training according to a preset training algorithm to obtain a noise prediction model corresponding to the wearer.

It should be noted that the training algorithm is a machine learning algorithm, and the machine learning algorithm can predict the data by continuously learning features. For example, a wearable device can predict the current noise distribution based on the historical noise distribution. Among them, the machine learning algorithm may include: decision tree algorithm, regression algorithm, Bayesian algorithm, neural network algorithm (may include deep neural network algorithm, convolutional neural network algorithm and recurrent neural network algorithm, etc.), clustering algorithm, etc., Which training algorithm is selected as the preset training algorithm for model training can be selected by those of ordinary skill in the art according to actual needs.

For example, the training algorithm configured by the wearable device is a Gaussian mixture model algorithm (which is a regression algorithm). After the wearable device obtains the historical bone conduction noise signal, the historical bone conduction noise signal is used as sample data, and the wearable device uses the Gaussian mixture as the sample data. The model algorithm performs model training to obtain a Gaussian mixture model (the Gaussian mixture model includes a plurality of Gaussian units for describing noise distribution), and the Gaussian mixture model is used as a noise prediction model. After that, the wearable device uses the start time and end time of the first audio signal collection period as the input of the noise prediction model, and inputs it into the noise prediction model for prediction, and obtains the predicted noise signal output by the noise prediction model, which is used as the wearer in the aforementioned first noise prediction model. Bone conduction noise signal during audio signal acquisition.

(1) Obtain a pre-trained general bone conduction noise model;

(2) adaptively process the general bone conduction noise model according to the historical bone conduction noise signal, and obtain the noise prediction model corresponding to the wearer;

(3) Predicting the bone conduction noise signal during the first audio signal acquisition period according to the noise prediction model.

The wearable device further acquires a pre-trained general bone conduction noise model after acquiring the wearer's historical bone conduction noise signal before the first audio signal is collected.

Then, the wearable device extracts the acoustic features of the historical bone conduction noise signal, and performs adaptive processing on the general bone conduction noise model based on the acoustic features to obtain a noise prediction model corresponding to the wearer.

Among them, the adaptive processing refers to the processing method of using a part of the non-specific user's acoustic characteristics in the general bone conduction noise model that is similar to the wearer's bone conduction noise as the wearer's acoustic characteristics, and the adaptive processing can use the maximum a posteriori Estimation algorithm implementation. The maximum a posteriori estimation is to obtain the estimation of the quantity that is difficult to observe based on empirical data. In the estimation process, the prior probability and Bayes' theorem are used to obtain the posterior probability, and the objective function (that is, the expression of the wearer's bone conduction noise model) Formula) is the likelihood function of the posterior probability, and the parameter value when the likelihood function is the maximum can be obtained (for example, the gradient descent algorithm can be used to obtain the maximum value of the likelihood function), which also realizes the general bone conduction noise model. A part of the non-specific speaker's acoustic feature features in the wearer are used as the effect of training with the wearer's acoustic features, so that the bone conduction corresponding to the wearer can be obtained according to the parameter value when the obtained likelihood function is the largest. noise model. After that, the bone conduction noise signal of the wearer during the first audio signal collection period can be predicted and obtained according to the noise prediction model.

The pronunciation enhancement signal is obtained by performing the pronunciation enhancement processing on the first audio signal through the pretrained pronunciation enhancement model.

An optional pronunciation enhancement strategy is further provided in the embodiment of the present application.

It should be noted that, in the embodiment of the present application, a pronunciation enhancement model is pre-trained, and the pronunciation enhancement model is configured to enhance the user's pronunciation part in the audio signal.

Correspondingly, when the first audio signal is enhanced, the wearable device calls the pre-trained pronunciation enhancement model, and inputs the first audio signal into the called pronunciation enhancement model for pronunciation enhancement processing, so that the corresponding pronunciation enhancement can be obtained. Signal.

For example, please refer to FIG. 5, the first audio signal is composed of two parts, which are respectively the pronunciation signal of the wearer's pronunciation and other noise signals, and the first audio signal is subjected to pronunciation enhancement processing through the pre-trained pronunciation enhancement model to obtain the noise signal. Suppressed articulation enhances the signal.

Optionally, in one embodiment, the pronunciation enhancement process is performed on the first audio signal through a pre-trained pronunciation enhancement model to obtain a pronunciation enhancement signal, including:

(1) Identify the wearer's pronunciation type to obtain the wearer's pronunciation type;

(2) according to the first preset correspondence of pronunciation type and pronunciation enhancement model, determine the target pronunciation enhancement model of the pronunciation type of corresponding wearer;

(3) Perform pronunciation enhancement processing on the first audio signal through the target pronunciation enhancement model to obtain a pronunciation enhancement signal.

Among them, the pronunciation types include but are not limited to speaking, singing, humming, and the like. In the embodiment of the present application, for different pronunciation types, a pronunciation enhancement model suitable for performing pronunciation enhancement processing on the pronunciation type audio signal is respectively trained, and a first preset correspondence between the pronunciation type and the pronunciation enhancement model is obtained accordingly.

In the embodiment of the present application, when the pronunciation enhancement processing is performed on the first audio signal by the pre-trained pronunciation enhancement model, the wearable device first identifies the pronunciation type of the wearer to obtain the pronunciation type of the wearer. Wherein, in the embodiment of the present application, the recognition mode of pronunciation type is not specifically limited, and can be configured by those of ordinary skill in the art according to actual needs. For example, a pronunciation type recognition model for pronunciation type recognition can be pre-trained. The pronunciation type recognition model is called to identify the pronunciation type.

After recognizing the wearer's pronunciation type, the wearable device determines the pronunciation enhancement model corresponding to the wearer's pronunciation type according to the first preset correspondence between the pronunciation type and the pronunciation enhancement model, which is recorded as the target pronunciation enhancement model. For example, the pronunciation enhancement model A suitable for the pronunciation enhancement processing of the pronunciation type "speaking", the pronunciation enhancement model B suitable for the pronunciation enhancement processing for the pronunciation type "singing", and the pronunciation enhancement model B suitable for the pronunciation enhancement processing for the pronunciation type "humming" are pre-trained. If the pronunciation enhancement model C that performs pronunciation enhancement processing recognizes that the wearer's pronunciation type is "speaking", the pronunciation enhancement model A can be determined as the target pronunciation enhancement model.

After determining the target pronunciation enhancement model, the wearable device calls the target pronunciation enhancement model, and inputs the first audio signal into the called target pronunciation enhancement model for pronunciation enhancement processing to obtain a corresponding pronunciation enhancement signal.

Optionally, in an embodiment, there are multiple non-bone conduction microphones, and the second audio signal is enhanced to obtain an enhanced ambient sound signal of the actual environment where the wearer is located, including:

(1) Predict the sound source direction of the real environment according to multiple second audio signals collected by multiple non-bone conduction microphones;

(2) Performing beamforming processing on the second audio signal according to the sound source direction to obtain an ambient sound enhancement signal.

In the embodiment of the present application, the wearable device includes a plurality of non-bone conduction microphones, and a microphone array is formed by the plurality of bone conduction microphones.

For example, referring to FIG. 6 , the wearable device is provided with two non-bone conduction microphones at the end of the temple near the frame. It should be noted that the arrangement of the non-bone conduction microphones shown in FIG. 6 is only an optional arrangement, and those skilled in the art can select the number and arrangement of the non-bone conduction microphones according to actual needs.

In this embodiment of the present application, the wearable device will collect multiple second audio signals through multiple non-bone conduction microphones. When the second audio signal is enhanced, the wearable device may collect the first audio signal according to the non-bone conduction microphones. The time difference between the two audio signals predicts the sound source direction of the wearer's real environment.

The following takes the setting method of the non-bone conduction microphone shown in FIG. 6 as an example to describe the method for the wearable device to predict the direction of the sound source in the real environment:

Please refer to FIG. 7 , there is another speaker "User B" in the real environment, and the sound signal sent by User B will be collected by the non-bone conduction microphone 1 and the non-bone conduction microphone 2 successively, and the non-bone conduction microphone 1 and the non-bone conduction microphone 2 The time difference between the sound signal collected by the conduction microphone 2 is t, and the distance between the non-bone conduction microphone 1 and the non-bone conduction microphone 2 is recorded as L1. It is assumed that the incident direction of the sound signal is the same as that of the non-bone conduction microphone 1 and the non-bone conduction microphone 2. The included angle of the connection line is θ. Since the propagation speed C of the sound signal in the air is known, then the sound path difference L2=C*t between the user B and the non-bone conduction microphone 1 and the non-bone conduction microphone 2 is calculated according to The principle of trigonometric functions has the following formula:

θ=cos-1(L2/L1);

Thus, the calculated angle θ represents the direction of user B compared to the wearable device, that is, the direction of the sound source.

After predicting the sound source direction of the actual environment where the wearer is located, the beamforming process can be performed on the second audio signal in the sound source direction according to the configured beamforming algorithm to eliminate the noise signal outside the sound source direction, and the corresponding Ambient sound enhances the signal. The embodiments of the present application do not specifically limit which beamforming algorithm is used, and can be configured by those of ordinary skill in the art according to actual needs. For example, a generalized sidelobe canceler (GSC) algorithm may be used to perform beamforming processing on the second audio signal.

Optionally, in an embodiment, performing enhancement processing on the second audio signal to obtain an ambient sound enhancement signal of the real environment where the wearer is located, including:

An ambient sound enhancement signal is obtained by performing an ambient sound enhancement process on the second audio signal by using the pre-trained ambient sound enhancement model.

An optional ambient sound enhancement strategy is further provided in the embodiment of the present application.

It should be noted that, in the embodiment of the present application, an ambient sound enhancement model is pre-trained, and the ambient sound enhancement model is configured to enhance the ambient sound part in the audio signal.

Correspondingly, when performing enhancement processing on the second audio signal, the wearable device invokes the pre-trained ambient sound enhancement model, and inputs the second audio signal into the invoked ambient sound enhancement model for ambient sound enhancement processing, so as to obtain the corresponding The ambient sound enhances the signal.

(1) Identify the environment type of the real environment, and obtain the environment type of the real environment;

(2) According to the second preset correspondence between the environmental type and the environmental sound enhancement model, determine the target environmental sound enhancement model corresponding to the environmental type of the real environment;

(3) The ambient sound enhancement processing is performed on the second audio signal by the target ambient sound enhancement model to obtain an ambient sound enhancement signal.

It should be noted that, in the embodiment of the present application, there is no specific limitation on the division of the environment types, and the division can be performed by those of ordinary skill in the art according to actual needs. For example, the types of environments can be divided into indoor environments and outdoor environments. In the embodiment of the present application, for different environmental types, an environmental sound enhancement model suitable for performing environmental sound enhancement processing on the audio signal of the environmental type is respectively trained, and the second preset correspondence between the environmental type and the environmental sound enhancement model is obtained accordingly. relation.

In the embodiment of the present application, when the environmental sound enhancement processing is performed on the second audio signal through the pre-trained environmental sound enhancement model, the wearable device first identifies the environmental type of the actual environment where the wearer is located, and obtains the wearer's environmental type . There is no specific limitation on the identification method of the environment type in the embodiments of the present application, which can be configured by those of ordinary skill in the art according to actual needs. The environment type identification model is called to identify the environment type.

After recognizing the environmental type of the wearer's real environment, the wearable device determines the environmental sound enhancement model corresponding to the wearer's real environment according to the preset correspondence between the environmental type and the environmental sound enhancement model. Enhance the model for the target ambient sound. For example, an ambient sound enhancement model D suitable for performing ambient sound enhancement processing on the environment type "indoor environment" and an ambient sound enhancement model E suitable for performing ambient sound enhancement processing on the environment type "outdoor environment" are pre-trained. If the environment type of the real scene where the wearer is located is "outdoor environment", the ambient sound enhancement model E may be determined as the target ambient sound enhancement model.

After determining the target ambient sound enhancement model, the wearable device invokes the target ambient sound enhancement model, and inputs the second audio signal into the invoked target ambient sound enhancement model for ambient sound enhancement processing to obtain a corresponding ambient sound enhancement signal.

Optionally, in an embodiment, the audio enhancement method provided by this application further includes:

(1) extracting the spectral feature of the first audio signal;

(2) Pronunciation detection is performed on the spectral features through the pre-trained pronunciation detection model to obtain the detection result of whether the wearer pronounces it.

It should be noted that, in the embodiment of the present application, a pronunciation detection model is also pre-trained, and the pronunciation detection model is configured to detect whether the wearer speaks. For example, a sample pronunciation signal of the wearer's pronunciation can be collected in advance, and the spectral feature of the sample pronunciation signal can be extracted as a training sample, and then the training sample can be used for model training to obtain a pronunciation detection model.

Correspondingly, when detecting whether the wearer of the wearable device pronounces according to the first audio signal, the wearable device can extract the spectral features of the first audio signal, and call the pre-trained pronunciation detection model, and input the extracted spectral features into the pronunciation detection model. The model performs pronunciation detection to obtain the detection result of whether the wearer has spoken.

Optionally, in an embodiment, after acquiring the first audio signal through the acquisition of the bone conduction microphone, and synchronously acquiring the second audio signal through the acquisition of the non-bone conduction microphone, the method further includes:

It should be noted that, in the embodiments of the present application, both the first audio signal and the second audio signal are regarded as composed of two parts, that is, the wearer's pronunciation part and the ambient sound part. Correspondingly, the wearable device can perform enhancement processing on the wearer's pronunciation part or the ambient sound part according to the configured priority level.

Wherein, when the current configuration is to enhance the pronunciation of the wearer, that is, when the priority of the pronunciation of the wearer is higher than that of the ambient sound, the wearable device performs enhancement processing on the first audio signal according to the second audio signal, and suppresses the ambient sound.

When the current configuration is to enhance the ambient sound, that is, when the priority of the ambient sound is higher than that of the wearer's pronunciation, the wearable device enhances the second audio signal according to the first audio signal to suppress the wearer's pronunciation.

(1) When the wearer is in motion, 3D reconstruction is performed on the actual environment where the wearer is located to obtain a virtual reality environment;

(2) Identify whether the wearer has exercise risks according to the virtual reality environment;

(3) When it is recognized that the wearer has a movement risk, a safety reminder is given to the wearer.

In the embodiment of the present application, in order to prevent the wearer of the wearable device from being too focused on pronunciation (such as speaking) and ignoring the risks in the real environment, the wearable device also provides safety prompts to the wearer.

Among them, the wearable device firstly identifies the state of the wearer to determine whether the wearer is in a state of movement. It should be noted that the embodiments of the present application do not specifically limit the manner of how to identify whether the wearer is in a motion state, and can be selected by those of ordinary skill in the art according to actual needs. For example, the wearable device can identify whether its current speed in any direction has reached the preset speed. If so, it can determine that it is in a state of motion, so as to determine that the wearer is also in a state of exercise. If not, it is determined that it is in a state of non-exercise. In this way, it is determined that the wearer is in a non-exercise state; for another example, the wearable device can identify whether its current displacement in any direction has reached the preset displacement, and if so, it is determined that it is in a movement state, thereby determining that the wearer is also in a movement state. If not, it is determined that the wearer is in a non-exercise state, thereby determining that the wearer is also in a non-exercise state.

If it is recognized that the wearer is in motion, the wearable device performs three-dimensional reconstruction of the real environment where the wearer is located according to the configured three-dimensional reconstruction strategy, and obtains a virtual reality environment corresponding to the real environment. Among them, the three-dimensional reconstruction of the real environment can be regarded as the process of digitizing the real existing environment, that is, the process of the wearable device recognizing the real environment in which it is located. It should be noted that the configuration of the 3D reconstruction strategy is not specifically limited in the embodiments of the present application, and a person of ordinary skill in the art can configure an appropriate 3D reconstruction strategy according to actual needs. For example, a 3D reconstruction strategy that achieves a certain balance between reconstruction accuracy and reconstruction efficiency can be selected according to the processing capability of the wearable device.

After reconstructing the virtual reality environment corresponding to the real environment, the wearable device identifies whether the wearer has a movement risk according to the virtual reality environment and current movement data (including but not limited to movement direction, movement speed, movement acceleration and movement trend, etc.). , where the movement risk includes but is not limited to the risk of falling, the risk of collision, etc.

If it is recognized that the wearer has a movement risk, the wearable device will give a safety reminder to the wearer, so as to avoid possible damage to the wearer caused by the movement risk. It should be noted that, in the embodiments of the present application, there are no specific restrictions on the manner in which the wearable device performs safety prompts, including but not limited to audio prompting methods, text prompting methods, video prompting methods, and fusion prompting methods (such as the fusion of audio and text). , the fusion of video and text), etc.

Optionally, in an embodiment, the wearable device further includes a camera module, which performs three-dimensional reconstruction on the real environment where the wearer is located to obtain a virtual reality environment, including:

(1) Obtain the depth image and color image of the wearer's real environment through the camera module;

(2) 3D reconstruction of the real environment according to the depth image and color image to obtain a virtual reality environment.

In the embodiment of the present application, the wearable device further includes a camera module. For example, the camera module includes a depth camera and an RGB camera. Please refer to FIG. 8 , which shows an optional setting method of the depth camera and the RGB camera. It should be noted that the depth camera and the RGB camera are not limited to the setting manner shown in FIG. 8 , and may also be set by those of ordinary skill in the art according to actual needs.

The type of the depth camera is not specifically limited in the embodiments of the present application, and can be selected by those of ordinary skill in the art according to actual needs. For example, the depth camera selected in the embodiment of the present application is a time-of-flight camera. The time-of-flight camera uses the reflection characteristics of light to calculate the depth (or say, the distance of the object to the time-of-flight camera). Correspondingly, in the embodiment of the present application, when the time-of-flight camera is aimed at the real scene, the electronic device can obtain a depth image of the real scene through the time-of-flight camera, and the depth image includes the depth values (depth values) at different positions in the real scene. The value reflects the distance from a location in the real scene to the time-of-flight camera, the smaller the value, the closer to the time-of-flight camera, and the larger the value, the farther away from the time-of-flight camera).

An RGB camera receives light reflected (or emitted) by an object and converts the light into a color image that carries the color characteristics of the object. Correspondingly, in this embodiment of the present application, the electronic device may obtain a color image of the real scene through the RGB camera when the RGB camera is aimed at the real scene, and the color image includes color data at different positions in the real scene.

It should be noted that the acquired depth image includes the depth value of the real scene, but does not include the color data of the real scene, and the acquired color image includes the color data of the real scene, but does not include the depth value of the real scene, In order to ensure the consistency of the acquired depth image and color image of the real scene, the electronic device can synchronize the depth image and color image of the real scene acquired by the time-of-flight camera and the RGB camera.

After acquiring the depth image and color image of the real scene, the wearable device performs three-dimensional reconstruction of the real scene according to the aforementioned depth image and color image according to the configured three-dimensional reconstruction algorithm, and correspondingly obtains a virtual reality scene corresponding to the real scene. It should be noted that. There is no specific limitation on which three-dimensional reconstruction algorithm is adopted in the embodiments of the present application, and can be selected by those of ordinary skill in the art according to actual needs.

FIG. 9 is another schematic flowchart of an audio enhancement method provided by an embodiment of the present application. In the following, the wearable device is smart glasses as an example for description. As shown in FIG. 9 , the process of the audio enhancement method provided by the embodiment of the present application may be as follows:

In 201, the smart glasses acquire a first audio signal through a bone conduction microphone, and synchronously acquire a second audio signal through a non-bone conduction microphone.

In the embodiment of the present application, the smart eye includes a bone conduction microphone and a non-bone conduction microphone. Wherein, the non-bone conduction microphone includes any type of microphone for audio collection based on the principle of sound air transmission and/or liquid transmission, including but not limited to electric type, condenser type, piezoelectric type, electromagnetic type, carbon particle type and semiconductor type and other types of microphones.

Exemplarily, please refer to FIG. 2 , the non-bone conduction microphone is arranged at one end of the temple of the smart glasses close to the frame, and the bone conduction microphone is arranged at one end of the mirror frame in principle of the temple. Referring to FIG. 3 , when the smart glasses are in a wearing state, the bone conduction microphone will directly contact the user's ear bones, while the non-bone conduction microphone will not directly contact the user.

In the embodiment of the present application, when triggering to perform audio collection, the smart eye synchronously performs audio collection through the bone conduction microphone and the non-bone conduction microphone, the audio signal collected by the bone conduction microphone is recorded as the first audio signal, and the non-bone conduction microphone is recorded as the first audio signal. The collected audio signal is denoted as the second audio signal.

It should be noted that the embodiments of this application do not specifically limit how to trigger audio collection. For example, it may be when the user operates the smart eye to start an audio collection application to record, or the user operates the smart eye to start instant messaging. When a similar application makes an audio and video call, it can also trigger audio collection when the smart eye performs voice wake-up or voice recognition.

In 202, the smart glasses extract spectral features of the first audio signal.

According to the audio collection principle of the bone conduction microphone, there is less interference from the wearer when the bone conduction microphone collects audio. Therefore, the first audio signal collected by the bone conduction microphone can be effectively used to detect whether the wearer of the smart eye pronounces the sound. .

Correspondingly, when detecting whether the wearer of the smart eye speaks according to the first audio signal, the smart eye can extract the spectral feature of the first audio signal.

In 203 , the smart glasses perform pronunciation detection on the spectral features through the pre-trained pronunciation detection model to determine whether the wearer has spoken, if yes, go to 204 , otherwise go to 207 .

In the embodiment of the present application, in addition to extracting the spectral features of the first audio signal, the smart glasses also call a pre-trained pronunciation detection model, and input the extracted spectral features into the pronunciation detection model for pronunciation detection, so as to obtain the detection of whether the wearer has spoken. result. Among them, if the detection result of the wearer's speech is obtained, go to 204 , and if the detection result that the wearer does not speak is obtained, go to 207 .

In 204, the smart glasses identify the pronunciation type of the wearer to obtain the pronunciation type of the wearer.

In the embodiment of the present application, when the pronunciation enhancement processing is performed on the first audio signal by the pre-trained pronunciation enhancement model, the smart eye first identifies the pronunciation type of the wearer to obtain the pronunciation type of the wearer. Wherein, in the embodiment of the present application, the recognition mode of pronunciation type is not specifically limited, and can be configured by those of ordinary skill in the art according to actual needs. For example, a pronunciation type recognition model for pronunciation type recognition can be pre-trained. The pronunciation type recognition model is called to identify the pronunciation type.

In 205, the smart glasses determine a target pronunciation enhancement model corresponding to the wearer's pronunciation type according to the preset correspondence between the pronunciation type and the pronunciation enhancement model.

After recognizing the wearer's pronunciation type, the smart eye determines the pronunciation enhancement model corresponding to the wearer's pronunciation type according to the first preset correspondence between the pronunciation type and the pronunciation enhancement model, which is recorded as the target pronunciation enhancement model. For example, the pronunciation enhancement model A suitable for the pronunciation enhancement processing of the pronunciation type "speaking", the pronunciation enhancement model B suitable for the pronunciation enhancement processing for the pronunciation type "singing", and the pronunciation enhancement model B suitable for the pronunciation enhancement processing for the pronunciation type "humming" are pre-trained. If the pronunciation enhancement model C that performs the pronunciation enhancement processing recognizes that the wearer's pronunciation type is "speaking", the pronunciation enhancement model A can be determined as the target pronunciation enhancement model.

In 206, the smart glasses perform pronunciation enhancement processing on the first audio signal through the target pronunciation enhancement model to obtain a pronunciation enhancement signal.

After determining the target pronunciation enhancement model, the intelligent eye calls the target pronunciation enhancement model, and inputs the first audio signal into the called target pronunciation enhancement model for pronunciation enhancement processing, and obtains the corresponding pronunciation enhancement signal.

In 207, the smart glasses identify the environment type of the real environment to obtain the environment type of the real environment.

In the embodiment of the present application, the smart eye first identifies the environment type of the real environment where the wearer is located, and obtains the environment type of the real scene where the wearer is located. There is no specific limitation on the identification method of the environment type in the embodiments of the present application, which can be configured by those of ordinary skill in the art according to actual needs. The intelligent eye invokes the environment type identification model to identify the environment type.

In 208, the smart glasses determine a target ambient sound enhancement model corresponding to the environment type of the real environment according to the second preset correspondence between the environment type and the ambient sound enhancement model.

After recognizing the environmental type of the wearer's real environment, the smart eye determines the environmental sound enhancement model corresponding to the wearer's real environment according to the preset correspondence between the environmental type and the environmental sound enhancement model, which is denoted as Target ambient sound enhancement model. For example, an ambient sound enhancement model D suitable for performing ambient sound enhancement processing on the environment type "indoor environment" and an ambient sound enhancement model E suitable for performing ambient sound enhancement processing on the environment type "outdoor environment" are pre-trained. If the environment type of the real scene where the wearer is located is "outdoor environment", the ambient sound enhancement model E may be determined as the target ambient sound enhancement model.

In 209, the smart glasses perform ambient sound enhancement processing on the second audio signal through the target ambient sound enhancement model to obtain an ambient sound enhancement signal.

After determining the target ambient sound enhancement model, the smart eye invokes the target ambient sound enhancement model, and inputs the second audio signal into the invoked target ambient sound enhancement model for ambient sound enhancement processing to obtain a corresponding ambient sound enhancement signal.

Please refer to FIG. 10 , which is a schematic structural diagram of an audio enhancement apparatus provided by an embodiment of the present application. The audio enhancement device is applied to the wearable device provided in the present application, and the wearable device includes a bone conduction microphone and a non-bone conduction microphone. As shown in Figure 10, the audio enhancement device may include:

The audio collection module 301 is used to collect the first audio signal through the bone conduction microphone, and obtain the second audio signal synchronously through the non-bone conduction microphone;

The pronunciation enhancement module 302 is configured to perform enhancement processing on the first audio signal when the wearer's pronunciation is detected according to the first audio signal to obtain the wearer's pronunciation enhancement signal;

The ambient sound enhancement module 303 is configured to perform enhancement processing on the second audio signal when it is detected that the wearer does not speak according to the first audio signal, so as to obtain an ambient sound enhancement signal of the actual environment where the wearer is located.

Optionally, in an embodiment, when performing enhancement processing on the first audio signal to obtain the wearer's pronunciation enhancement signal, the pronunciation enhancement module 302 is configured to:

Obtain the wearer's historical bone conduction noise signal before the first audio signal is collected;

Acquiring the wearer's bone conduction noise signal during the first audio signal acquisition period according to the historical bone conduction noise signal;

The anti-phase superposition of the bone conduction noise signal and the first audio signal is performed to obtain a pronunciation enhancement signal.

Optionally, in an embodiment, when acquiring the wearer's bone conduction noise signal during the first audio signal acquisition period according to the historical bone conduction noise signal, the pronunciation enhancement module 302 is configured to:

Carry out model training according to the historical bone conduction noise signal to obtain the noise prediction model corresponding to the wearer;

The pronunciation enhancement signal is obtained by performing pronunciation enhancement processing on the first audio signal by using the pre-trained pronunciation enhancement model.

Optionally, in an embodiment, there are multiple non-bone conduction microphones, and when the second audio signal is enhanced to obtain an enhanced ambient sound signal of the actual environment where the wearer is located, the ambient sound enhancement module 303:

According to multiple second audio signals collected by multiple non-bone conduction microphones, predict the sound source direction of the real environment;

The second audio signal is subjected to beamforming processing according to the sound source direction to obtain an ambient sound enhancement signal.

Optionally, in an embodiment, the audio enhancement device provided by the present application further includes a pronunciation detection module for:

extracting spectral features of the first audio signal;

The pre-trained pronunciation detection model is used to detect the pronunciation of the spectral features, and the detection result of whether the wearer is pronounced is obtained.

Optionally, in an embodiment, after the first audio signal is acquired by the bone conduction microphone, and the second audio signal is acquired by the non-bone conduction microphone simultaneously,

The pronunciation enhancement module 302 is further configured to perform enhancement processing on the first audio signal according to the second audio signal when the current configuration is to enhance the wearer's pronunciation; or,

The ambient sound enhancement module 303 is further configured to perform enhancement processing on the second audio signal according to the first audio signal when the current configuration is to enhance the ambient sound.

Optionally, in an embodiment, the audio enhancement device provided by the present application further includes a safety prompt module for:

When the wearer is in motion, three-dimensional reconstruction of the real environment is performed to obtain a virtual reality environment;

Identify whether the wearer is at risk of exercise based on the virtual reality environment;

When it is recognized that the wearer is at risk of exercise, the wearer is reminded of safety.

Optionally, in an embodiment, the wearable device further includes a camera module, and when performing three-dimensional reconstruction of the real environment to obtain a virtual reality environment, the safety prompt module is used for:

The three-dimensional reconstruction of the real environment is carried out according to the depth image and the color image, and the virtual reality environment is obtained.

It should be noted that the audio enhancement apparatus provided in the embodiment of the present application and the audio enhancement method in the above embodiment belong to the same concept, and the audio enhancement apparatus can execute any of the methods provided in the audio enhancement method embodiment, and the specific implementation process is described in detail. See the above related embodiments, and details are not repeated here.

The embodiments of the present application provide a storage medium on which a computer program is stored. When the stored computer program is loaded by a wearable device including a bone conduction microphone and a non-bone conduction microphone, the audio enhancement method provided by the embodiments of the present application is executed. steps in . The storage medium may be a magnetic disk, an optical disk, a read only memory (Read Only Memory, ROM), or a random access device (Random Access Memory, RAM), and the like.

An embodiment of the present application further provides a wearable device. Please refer to FIG. 11 . The wearable device includes a processor 401 , a memory 402 , a bone conduction microphone 403 and a non-bone conduction microphone 404 .

The processor in the embodiment of the present application is a general-purpose processor, such as a processor of an ARM architecture.

A computer program is stored in the memory 402, which may be a high-speed random access memory, or a non-volatile memory, such as at least one magnetic disk storage device, flash memory device, or other volatile solid-state storage devices.

The bone conduction microphone 403 includes a microphone for audio collection based on the principle of sound bone conduction.

The non-bone conduction microphone 404 includes any type of microphone for audio collection based on the principle of sound air transmission and/or liquid transmission, including but not limited to electrodynamic, capacitive, piezoelectric, electromagnetic, carbon particle and semiconductor type, etc. type microphone.

In this embodiment of the present application, the memory 402 may further include a memory controller to provide access to the memory 402 by the processor 401, and the processor 401 implements the following functions by loading a computer program in the memory 402:

The first audio signal is acquired by the bone conduction microphone 403, and the second audio signal is acquired by the non-bone conduction microphone 404 simultaneously;

Optionally, in an embodiment, when performing enhancement processing on the first audio signal to obtain the wearer's pronunciation enhancement signal, the processor 401 executes:

Optionally, in an embodiment, when acquiring the wearer's bone conduction noise signal during the first audio signal acquisition period according to the historical bone conduction noise signal, the processor 401 executes:

Optionally, in an embodiment, there are multiple non-bone conduction microphones 404, and when the second audio signal is enhanced to obtain an ambient sound enhancement signal of the actual environment where the wearer is located, the processor 401 executes:

predicting the sound source direction of the real environment according to the plurality of second audio signals collected by the plurality of non-bone conduction microphones 404;

Optionally, in an embodiment, the processor 401 further executes:

extracting spectral features of the first audio signal;

Optionally, in an embodiment, audio collection is performed simultaneously through the bone conduction microphone 403 and the non-bone conduction microphone 404 to obtain the first audio signal collected by the bone conduction microphone 403 and the second audio signal collected by the non-bone conduction microphone 404. After the signal, the processor 401 executes:

Optionally, in an embodiment, the processor 401 further executes:

Optionally, in an embodiment, the wearable device further includes a camera module, and when performing three-dimensional reconstruction on the real environment to obtain a virtual reality environment, the processor 401 executes:

It should be noted that the wearable device provided by the embodiment of the present application and the audio enhancement method in the above embodiment belong to the same concept, and any method provided in the audio enhancement method embodiment can be executed on the wearable device, and its specific implementation For details of the process, please refer to the above embodiments, which will not be repeated here.

It should be noted that, for the audio enhancement method of the embodiment of the present application, ordinary testers in the art can understand that all or part of the process of implementing the audio enhancement method of the embodiment of the present application can be completed by controlling the relevant hardware through a computer program. , the computer program can be stored in a computer-readable storage medium, such as in the memory of the wearable device provided by this application, and executed by the processor in the wearable device, and the execution process can include such as Flow of an embodiment of an audio enhancement method. The storage medium may be a magnetic disk, an optical disk, a read-only memory, a random access memory, or the like.

The audio enhancement method, device, storage medium, and wearable device provided by the embodiments of the present application have been described in detail above. The principles and implementations of the present application are described with specific examples in this article. It is only used to help understand the method of the present application and its core idea; at the same time, for those skilled in the art, according to the idea of the present application, there will be changes in the specific implementation and application scope. The contents of the description should not be construed as limiting the application.

Claims

An audio enhancement method is applied to a wearable device, wherein the wearable device includes a bone conduction microphone and a non-bone conduction microphone, and the audio enhancement method includes:

The first audio signal is acquired by collecting the bone conduction microphone, and the second audio signal is simultaneously acquired by the non-bone conduction microphone;

When the wearer's pronunciation is detected according to the first audio signal, the first audio signal is enhanced to obtain the wearer's pronunciation enhancement signal;

When it is detected that the wearer does not speak according to the first audio signal, enhancement processing is performed on the second audio signal to obtain an ambient sound enhancement signal of the real environment where the wearer is located.
The audio enhancement method according to claim 1, wherein the enhancement processing is performed on the first audio signal to obtain the pronunciation enhancement signal of the wearer, comprising:

acquiring a historical bone conduction noise signal of the wearer before the first audio signal is collected;

Acquiring a bone conduction noise signal of the wearer during the collection of the first audio signal according to the historical bone conduction noise signal;

The anti-phase superposition of the bone conduction noise signal and the first audio signal is performed to obtain the pronunciation enhancement signal.
The audio enhancement method according to claim 2, wherein the acquiring the wearer's bone conduction noise signal during the collection of the first audio signal according to the historical bone conduction noise signal comprises:

Perform model training according to the historical bone conduction noise signal to obtain a noise prediction model corresponding to the wearer;

The bone conduction noise signal during the acquisition of the first audio signal is predicted according to the noise prediction model.
The audio enhancement method according to claim 1, wherein the performing enhancement processing on the first audio signal to obtain the wearer's pronunciation enhancement signal, comprising:

The pronunciation enhancement signal is obtained by performing pronunciation enhancement processing on the first audio signal by using a pre-trained pronunciation enhancement model.
The audio enhancement method according to claim 1, wherein there are multiple non-bone conduction microphones, and the enhancement processing is performed on the second audio signal to obtain an ambient sound enhancement signal of the actual environment where the wearer is located ,include:

predicting the sound source direction of the real environment according to the plurality of second audio signals collected by the plurality of non-bone conduction microphones;

The second audio signal is subjected to beamforming processing according to the sound source direction to obtain the ambient sound enhancement signal.
The audio enhancement method of claim 1, further comprising:

extracting spectral features of the first audio signal;

Pronunciation detection is performed on the spectral feature by using a pre-trained pronunciation detection model to obtain a detection result of whether the wearer has spoken.
The audio enhancement method according to claim 1, wherein, in the synchronization, audio collection is performed by the bone conduction microphone and the non-bone conduction microphone to obtain the first audio signal collected by the bone conduction microphone, and the non-bone conduction microphone is obtained. After the collected second audio signal, it further includes:

When the current configuration is to enhance the pronunciation of the wearer, the first audio signal is enhanced according to the second audio signal; or,

When the current configuration is to enhance the ambient sound, the second audio signal is enhanced according to the first audio signal.
The audio enhancement method according to any one of claims 1-7, further comprising:

When the wearer is in motion, three-dimensional reconstruction is performed on the real environment to obtain a virtual reality environment;

Identifying whether the wearer is at risk of exercise according to the virtual reality environment;

When it is recognized that the wearer has a movement risk, a safety prompt is given to the wearer.
The audio enhancement method according to claim 8, wherein the wearable device further comprises a camera module, and the three-dimensional reconstruction of the real environment to obtain a virtual reality environment comprises:

Obtain the depth image and color image of the real environment through the camera module;

The virtual reality environment is obtained by performing three-dimensional reconstruction of the real environment according to the depth image and the color image.
An audio enhancement device is applied to a wearable device, wherein the wearable device includes a bone conduction microphone and a non-bone conduction microphone, and the audio enhancement device includes:

an audio acquisition module, configured to acquire a first audio signal through the bone conduction microphone, and acquire a second audio signal through the non-bone conduction microphone synchronously;

A pronunciation enhancement module, configured to perform enhancement processing on the first audio signal when the wearer's pronunciation is detected according to the first audio signal, to obtain the wearer's pronunciation enhancement signal;

An ambient sound enhancement module, configured to perform enhancement processing on the second audio signal when it is detected that the wearer has not spoken according to the first audio signal, to obtain an ambient sound enhancement signal of the actual environment where the wearer is located .
A storage medium having a computer program stored thereon, wherein the audio enhancement according to any one of claims 1-9 is performed when the computer program is loaded by a wearable device comprising a bone conduction microphone and a non-bone conduction microphone method.
A wearable device, comprising a bone conduction microphone, a non-bone conduction microphone, a processor and a memory, wherein the memory stores a computer program, wherein the processor is configured to execute by calling the computer program:

The first audio signal is acquired by collecting the bone conduction microphone, and the second audio signal is simultaneously acquired by the non-bone conduction microphone;

When the wearer's pronunciation is detected according to the first audio signal, the first audio signal is enhanced to obtain the wearer's pronunciation enhancement signal;

When it is detected that the wearer does not speak according to the first audio signal, enhancement processing is performed on the second audio signal to obtain an ambient sound enhancement signal of the real environment where the wearer is located.
The electronic device according to claim 12, wherein, when the first audio signal is enhanced to obtain the wearer's voice enhancement signal, the processor is configured to execute:

acquiring a historical bone conduction noise signal of the wearer before the first audio signal is collected;

Acquiring a bone conduction noise signal of the wearer during the collection of the first audio signal according to the historical bone conduction noise signal;

The anti-phase superposition of the bone conduction noise signal and the first audio signal is performed to obtain the pronunciation enhancement signal.
14. The electronic device of claim 13, wherein, when acquiring the wearer's bone conduction noise signal during the first audio signal acquisition based on the historical bone conduction noise signal, the processor is configured to execute:

Perform model training according to the historical bone conduction noise signal to obtain a noise prediction model corresponding to the wearer;

The bone conduction noise signal during the acquisition of the first audio signal is predicted according to the noise prediction model.
The electronic device according to claim 12, wherein, when performing enhancement processing on the first audio signal to obtain the wearer's pronunciation enhancement signal, the processor is configured to execute:

The pronunciation enhancement signal is obtained by performing pronunciation enhancement processing on the first audio signal by using a pre-trained pronunciation enhancement model.
The electronic device according to claim 12, wherein there are multiple non-bone conduction microphones, and when the second audio signal is enhanced to obtain an ambient sound enhancement signal of the real environment where the wearer is located, The processor is used to execute:

predicting the sound source direction of the real environment according to the plurality of second audio signals collected by the plurality of non-bone conduction microphones;

The second audio signal is subjected to beamforming processing according to the sound source direction to obtain the ambient sound enhancement signal.
The electronic device of claim 12, wherein the processor is further configured to perform:

extracting spectral features of the first audio signal;

Pronunciation detection is performed on the spectral feature by using a pre-trained pronunciation detection model to obtain a detection result of whether the wearer has spoken.
The electronic device according to claim 12, wherein, when audio is collected by the bone conduction microphone and the non-bone conduction microphone, a first audio signal collected by the bone conduction microphone is obtained, and a first audio signal collected by the non-bone conduction microphone is obtained. After the two audio signals, the processor is further configured to execute:

When the current configuration is to enhance the pronunciation of the wearer, the first audio signal is enhanced according to the second audio signal; or,

When the current configuration is to enhance the ambient sound, the second audio signal is enhanced according to the first audio signal.
The electronic device according to any one of claims 12-18, wherein the processor is further configured to execute:

When the wearer is in motion, three-dimensional reconstruction is performed on the real environment to obtain a virtual reality environment;

Identifying whether the wearer is at risk of exercise according to the virtual reality environment;

When it is recognized that the wearer has a movement risk, a safety prompt is given to the wearer.
The electronic device according to claim 19, wherein the wearable device further comprises a camera module, and when performing three-dimensional reconstruction of the real environment to obtain a virtual reality environment, the processor is configured to execute:

Obtain the depth image and color image of the real environment through the camera module;

The virtual reality environment is obtained by performing three-dimensional reconstruction of the real environment according to the depth image and the color image.