CN116386589A - A Deep Learning Speech Reconstruction Method Based on Smartphone Acceleration Sensor - Google Patents

Abstract

The invention discloses a deep learning voice reconstruction method based on a smart phone acceleration sensor, which comprises the steps of firstly collecting data, and collecting mainboard vibration signals caused by a smart phone loudspeaker in a plurality of frequency sampling modes by combining a plurality of motion sensors; then, data processing is carried out, and sensor signals are processed through the steps of linear interpolation, noise rejection, feature extraction and the like; then, carrying out voice reconstruction, and providing a voice reconstruction algorithm for generating an countermeasure network based on the multi-scale time-frequency domain of the wavelet to convert the preprocessed motion sensor data into voice waveform data; and finally, evaluating the effect, and evaluating the generated voice through subjective and objective indexes. The invention improves the high-frequency performance and the robustness of the voice synthesis, so that the synthesized voice is more similar to the original voice.

Description

A Deep Learning Speech Reconstruction Method Based on Smartphone Acceleration Sensor

technical field

The invention belongs to the technical field of artificial intelligence, and in particular relates to a deep learning speech reconstruction method.

Background technique

With the advancement of research on basic technologies of the mobile Internet, e-commerce, social networking and new media industries have flourished in the past ten years, and the mobile intelligent terminal consumer market represented by smartphones has opened up nearly a decade of prosperity. According to the latest data from the statistical agency Statista, as of the third quarter of 2022, the number of smartphone users worldwide will reach 6.64 billion, accounting for about 83.32% of the world's population. Simultaneously with the explosive growth of the number of smartphone users, is the increasingly tight coupling between human daily life and smart devices. As an indispensable and important part of the modern mobile smart device design paradigm, the motion sensor undertakes the key responsibilities of sensing the external environment of the device, identifying the motion state of the device, and reading user interaction input, and is widely used in a variety of mobile terminals. Motion sensors represented by acceleration sensors are generally mounted on the motherboard of smartphones, and are tightly coupled with core components including smartphone processors, speakers, microphones, etc., and jointly serve the operation of the core system.

Inside the phone, the speaker and numerous sensors are integrated on a circuit board, which can be regarded as an efficient solid transmission medium. When the speaker is working, sound vibrations are transmitted through the entire motherboard, which allows a motion sensor placed on the same surface to capture the solid-state vibration caused by the speaker. And since the motion sensor and the speaker are integrated on the same circuit board in physical contact, in close proximity to each other, no matter how the smartphone is placed (on a table or in the hand), the voice signal from the speaker will always respond to the motion sensor ( such as gyroscopes and accelerometers) have a significant impact. These motion sensors are more sensitive to vibration signals, so the signals captured by them always contain the acoustic vibrations brought to the motherboard by the speakers of the mobile phone.

In general, previous studies formulated it as a classification problem and applied machine learning-based solutions to construct a mapping between features extracted from non-acoustic signals and words. A large number of studies have demonstrated the feasibility of recognizing numbers, words and even key words from sensor vibration signals. For example, the Zhejiang University team found that the sampling frequency of the built-in accelerometer of the smart phone released after 2018 is as high as 500Hz, which almost covers the entire base frequency band (85-255Hz) of adult speech. They proposed a speech recognition system based on deep learning. This system collects the speech signal from the speaker through a low-privilege spy application, converts the speech signal into a spectrogram, and uses DenseNet as the basic network to analyze the acceleration signal spectrogram. The carried voice information (text) is classified and recognized. Han et al. proposed a distributed side-channel attack by using the vibration signals captured by the sensor network (including geophones, accelerometers, and gyroscopes). In order to solve the problem of low sensor sampling frequency, they adopted TI-ADC ( Time-Interleaved analog-to-digital converter) distribution form to approach the overall high sampling frequency, while maintaining a low node sampling rate. However, existing work can only recognize a few words from a very limited vocabulary.

Contents of the invention

In order to overcome the deficiencies in the prior art, the present invention provides a deep learning voice reconstruction method based on the acceleration sensor of the smart phone. Firstly, the data is collected, and the vibration of the motherboard caused by the speaker of the smart phone is analyzed in combination with a variety of motion sensors in a variety of frequency sampling modes. The signal is collected; then the data is processed, and the sensor signal is processed through linear interpolation, noise elimination, feature extraction and other steps; then the speech reconstruction is carried out, and a speech reconstruction algorithm based on wavelet-based multi-scale time-frequency domain generative confrontation network is proposed. The preprocessed motion sensor data is converted into speech waveform data; finally, the effect evaluation is carried out, and the generated speech is evaluated through subjective and objective indicators. The invention improves the high-frequency performance and robustness of speech synthesis, making the synthesized speech closer to the original speech.

The technical solution adopted by the present invention to solve its technical problems comprises the steps:

Step 1: Data collection;

Use the mobile phone to play audio files;

Collect the signal of the acceleration sensor, and record the signal and the corresponding time stamp;

Step 2: Data processing; perform linear interpolation, noise processing and feature extraction on sensor signals;

Step 2-1: Linear interpolation;

Locate all time points without acceleration data by timestamp, and use linear interpolation to fill in the missing data; for acceleration data with the same timestamp, use the means of taking the mean value, and use this mean value to represent the acceleration data of this timestamp, so that Each timestamp has one and only one acceleration data;

Step 2-2: noise processing;

Use the reference data under silent conditions to remove the noise caused by gravity and hardware factors in the sensor signal; use a high-pass filter with a cut-off frequency of aHz for filtering;

Step 2-3: Feature Extraction

The sensor signal is divided into multiple segments with fixed overlap, the segment length and the overlap length are set to 256 and 64 respectively, and the Hamming window is used to window each segment, and the spectrum is calculated by the short-time Fourier transform STFT to obtain the STFT matrix, which records the amplitude and phase of each time and frequency, and converts it into a corresponding spectrogram according to formula (1):

spectrogram{x(n)}＝|STFT{x(n)} ² (1)

Where x(n) represents the acceleration sensor signal, and STFT{x(n)} represents the STFT matrix corresponding to the acceleration sensor signal;

The spectrogram is converted into a Mel spectrogram, and the mutual conversion between the frequency f on the spectrogram corresponding to the acceleration sensor signal and the Mel scale f _mel is realized by formulas (2) and (3):

Finally, the formula (2) is used to convert the spectrogram to the Mel spectrogram to obtain the feature - the Mel spectrogram;

Step 3: Speech reconstruction;

The speech reconstruction model WMelGAN based on wavelet-based multi-scale time-frequency domain generative confrontation network is used to convert the sensor data processed in step 2 into a synthetic speech signal;

Step 3-1: The speech reconstruction model of wavelet-based multi-scale time-frequency domain generative adversarial network includes three components: generator, multi-scale discriminator and wavelet discriminator;

The generator converts the mel spectrogram into a synthetic speech signal. The generator consists of a series of upsampled transposed convolutional layers, and each transposed convolutional layer is followed by a residual network with a dilated convolutional layer;

The different sub-discriminators of the multi-scale discriminator have a model structure based on a convolutional network, and work on different data scales to judge the output results of generators of different scales; the network structure of each sub-discriminator is sequentially composed of a layer 1 One-dimensional convolution, 4-layer group convolution and a one-dimensional and one-dimensional downsampling structure, the input of which includes the original speech signal and the synthetic speech signal generated by the generator network;

The wavelet discriminator decomposes the input speech signal into four sub-signals of different frequency bands through cubic wavelet decomposition, and uses the stacked convolutional neural network to evaluate and judge the generation effect; in the WMelGAN model, the generator and multiple discriminators use The training is conducted in an adversarial way, so that the audio generated by the generator reaches the effect that the discriminator cannot judge whether it is true or false, and finally the generator is used to generate the final synthesized speech signal;

Step 3-2: loss function;

By setting a series of loss functions for confrontation training between the generator and the discriminator, the goals are shown in formulas (4) and (5):

loss _D = loss _{disc_TD} + loss _{disc_WD} (4)

loss _G ＝loss _{gen_TD} +loss _{gen_WD} +loss _mel *45+(loss _{feature_TD} +loss _{feature_WD} )*2 (5)

Among them, loss _D represents the overall loss function of the two discriminators, and loss _G represents the loss function of the generator; the loss of the generator is composed of five parts, namely multi-scale discriminator loss loss _{gen_TD} , wavelet discriminator loss loss _{gen_WD} , Mel Loss loss _mel , multi-scale time-domain discriminator feature map loss loss _{feature_TD} , wavelet discriminator feature map loss loss _{feature_WD} ;

The overall loss of the two discriminators is divided into multi-scale discriminator loss loss _{disc_TD} and wavelet discriminator loss loss _{disc_WD} , defined as:

表示期望；Where x represents the original speech signal waveform, s represents the Mel spectrogram, z represents the Gaussian noise vector, TD and WD represent the multi-scale discriminator and wavelet discriminator respectively, the subscript k represents different scales; G(s, z) represents the generated speech signal;

Express expectations;

The generator's multi-scale discriminator loss loss _{gen_TD} and the generator's wavelet discriminator loss loss _{gen_WD} are defined as:

分别对应不同尺度的三个多尺度判别器；in

Three multi-scale discriminators corresponding to different scales;

Mel loss loss _mel is the use of multi-scale Mel spectrogram to quantify the gap between the original speech waveform and the synthesized speech waveform, which is defined as:

loss _mel =||MEL(x)-MEL(G(s,z))|| _F (10)

Wherein ||·|| _F represents the F norm, wherein MEL( ) represents the Mel spectrogram transformation for a given speech signal;

Step 4: form a speech evaluation system; evaluate the intelligibility and naturalness indicators of the reconstructed speech signal through the subjective evaluation system and the objective evaluation system;

Step 4-1: The subjective evaluation system uses the mean opinion score MOS as the evaluation index

Step 4-2: Objective evaluation system;

Step 4-2-1: Measure the difference between the synthesized speech and the original speech signal using the three objective indicators of peak signal-to-noise ratio PSNR, Mel cepstrum distortion MCD and fundamental frequency root mean square error F0RMSE; use dictation test accuracy rate ADT to measure intelligibility of synthesized speech;

Peak Signal-to-Noise Ratio PSNR measures the ratio between the maximum possible power of a signal and the noise power affecting its quality:

Wherein, S _peak represents the peak speech signal, S represents the speech signal, and N represents the noise signal;

Step 4-2-2: Use the Mel cepstrum distortion MCD to quantify the gap between the original speech signal and the synthesized speech signal in the Mel cepstrum feature MFCC, specifically, the Mel cepstrum distortion of the kth frame is expressed as:

Where r represents the original speech signal, M is the number of Mel filters, MC _s (i, k) and MC _r (i, k) are the MFCC coefficients of the synthesized speech signal s and the original speech signal r;

Omit the subscript of MC, expressed as:

where X _k,n represents the logarithmic power output of the nth triangular filter, namely:

Wherein X(k, m) represents the kth frame Fourier transform result of the input speech frame whose frequency index is m, and w _n (m) represents the nth Mel filter;

Step 4-2-3: Use the root mean square error F0 RMSE of the fundamental frequency to compare the gap between the fundamental frequency of the original speech and the fundamental frequency of the synthesized speech, expressed as:

代表合成语音信号的基频特征。Where f0 represents the fundamental frequency feature of the original speech signal,

Represents the fundamental frequency characteristics of the synthesized speech signal.

Preferably, the a=20.

Preferably, the number of mel filters M=10.

The beneficial effects of the present invention are as follows:

The data acquisition of the present invention can acquire multi-audio file motion sensor sampling signals in multiple frequency sampling modes in batches. Data processing enables a speaker-independent general sensor-based speech synthesis framework, reducing the dependency on speaker-specific datasets. Speech reconstruction improves the high-frequency performance and robustness of speech synthesis by optimizing the model architecture and introducing wavelet discriminators, making the synthesized speech closer to the original speech. Generating Speech Reviews demonstrates the perceptual ability of smartphone motion sensors to reproduce speaker speech.

Description of drawings

Fig. 1 is a schematic framework diagram of the deep learning speech reconstruction method based on the smartphone acceleration sensor of the present invention.

Fig. 2 is a schematic diagram of the voice reconstruction algorithm based on the wavelet-based multi-scale time-frequency domain generative adversarial network of the present invention.

Detailed ways

The present invention will be further described below in conjunction with the accompanying drawings and embodiments.

The purpose of the present invention is to provide a voice reconstruction method based on the acceleration sensor of a smart phone, aiming at solving the privacy security problem contained in the motion sensor data in the prior art, and improving the quality and efficiency of voice synthesis.

The present invention carries out voice reconstruction for the signal of the built-in acceleration sensor of the smart phone, establishes the internal correlation between the state of the built-in acceleration sensor and the speaker of the mobile phone, establishes a propagation model of the vibration signal between the acceleration sensor and the speaker of the mobile phone, and realizes the vibration caused by the speaker. Functions for reconstructing raw speech signals in accelerometer vibrations.

As shown in Figure 1, the present invention adopts following technical scheme:

One: Data collection; use the self-developed Android APP to play audio files, and select a variety of motion sensors and frequency sampling modes to collect the vibration signals of the main board caused by the speaker of the smart phone according to the needs. The beneficial effect is that the motion sensor sampling signals of multiple audio files in multiple frequency sampling modes can be acquired in batches.

Two: Data processing; integrate multiple digital signal processing schemes, perform data preprocessing, noise analysis and elimination, and data feature (Mel spectrogram) extraction on motion sensor signals, perform noise modeling on acceleration sensor signals, and separate perception Noise component in the data. Its beneficial effect is to realize a speaker-independent general sensor speech synthesis framework and reduce the dependence on specific speaker data sets.

Three: Speech reconstruction; as shown in Figure 2, a speech reconstruction algorithm based on wavelet-based multi-scale time-frequency domain generative confrontation network is proposed. This sensor-speech mapping model based on the structure design of generative confrontation network converts the preprocessed motion sensor data into speech waveform data, uses the confrontation generation network to establish the mapping function between the denoising perception data and the original speech data, and introduces wavelet Discriminator to improve the high-frequency effect of speech synthesis. The beneficial effect is that by optimizing the model architecture and introducing a wavelet discriminator, the high-frequency performance and robustness of speech synthesis are improved, making the synthesized speech closer to the original speech.

Four: Generate a speech evaluation system. The intelligibility and naturalness of the reconstructed speech signal are evaluated by subjective evaluation system and objective evaluation system. The beneficial effect is to demonstrate that the motion sensor of the smart phone has the perception ability to restore the voice of the speaker.

Specific examples:

Concrete steps of the present invention are as follows:

Step 1: Data collection.

Use the self-developed Android APP application SensorListener to play audio files, and select a variety of motion sensors and frequency sampling modes to collect vibration signals of the motherboard caused by smartphone speakers and save them locally. Specifically, this application will obtain all audio files locally stored in the mobile phone through the interface provided by Android, and display them on the application home page. The user can select the audio of the sensor data to be collected on the main page. After clicking to start collecting, the application will play the audio and record the sensor signal. Specifically, for each piece of audio, the application will obtain linear acceleration sensor objects, acceleration sensor objects, and gyroscope sensor objects, and register them as listeners (Listener), and then create a media player object (MediaPlayer) to start playing audio; During this period, the sensor will record each vibration signal and the corresponding time stamp; when the audio is played, MediaPlayer will return an end signal to end the acquisition of the sensor signal by canceling the listener identity of the motion sensor object; finally, the linear acceleration Signal, acceleration signal, and gyroscope signal are written to the CSV file with the same name as the audio. This application will obtain all audio files stored locally on the phone through the interface provided by Android, and display them on the application home page. The user can select the audio of the sensor data to be collected on the main page. After clicking to start collecting, the application will play the audio and record the sensor signal. Specifically, for each piece of audio, the application will obtain linear acceleration sensor objects, acceleration sensor objects, and gyroscope sensor objects, and register them as listeners (Listener), and then create a media player object (MediaPlayer) to start playing audio; During this period, the sensor will record each vibration signal and the corresponding time stamp; when the audio is played, MediaPlayer will return an end signal to end the acquisition of the sensor signal by canceling the listener identity of the motion sensor object; finally, the linear acceleration Signal, acceleration signal, and gyroscope signal are written to the CSV file with the same name as the audio. Users can choose a variety of frequency sampling modes to capture the output signal of the motion sensor.

Step 2: Data processing.

Integrate a variety of digital signal processing solutions to perform data preprocessing, noise analysis and elimination, and data feature (Mel spectrogram) extraction on motion sensor signals. 1. linear interpolation, the present invention locates all time points without acceleration data by timestamp, and uses linear interpolation to fill in missing data; Represents the acceleration data of this timestamp, so that each timestamp has one and only one acceleration data. ②Noise processing, use the benchmark data under silent conditions to eliminate the noise caused by gravity and hardware factors in the sensor signal; use a high-pass filter with a cutoff frequency of 50Hz to remove the influence of human activities. ③ feature extraction, the present invention divides the sensor signal into a plurality of short segments with fixed overlap, the lengths of the segments and overlaps are set to 256 and 64 respectively, and each segment is windowed using the Hamming window, through the fast Fourier transform Transform (STFT) to calculate its spectrum to get STFT matrix. This matrix records the magnitude and phase for each time and frequency according to the following equation:

spectrogram{x(n)}＝|STFT{x(n)} ² (1)

Convert to the corresponding spectrogram, where x(n) represents the acceleration sensor signal, and STFT{x(n)} represents the STFT matrix corresponding to the acceleration sensor signal. The horizontal axis x of the spectrogram is time, the vertical axis y is frequency, and the value corresponding to (x, y) represents the amplitude of frequency y at time x. In order to make the frequency of the spectrogram conform to the logarithmic distribution pattern of the frequency of the human ear, the present invention converts the spectrogram into a Mel spectrogram. The mutual conversion between the frequency (Hz) on the spectrogram corresponding to the acceleration sensor signal and the Mel scale (mel) is realized by using the following two formulas:

Finally, the feature is obtained by using the Mel filter - the Mel spectrogram.

Step 3: Speech reconstruction.

The sensor-speech mapping model based on the generative adversarial network structure design converts the preprocessed motion sensor data into speech waveform data. As shown in Figure 2, the WMelGAN model constructed by the present invention based on the idea of generative confrontation network includes three core components: generator, multi-scale discriminator and wavelet discriminator. The generator is designed to convert the mel spectrogram into an intelligible speech audio waveform signal, and convert the mel spectrogram sequence into a speech waveform signal with a higher amount of data through a series of upsampled transposed convolutional networks, each conversion After setting the convolution module, a residual network with a dilated convolution layer is set to obtain a larger receptive field. The wider receptive field helps to capture the cross-space correlation between long vectors and helps to learn the acoustic structure. sexual characteristics. Different sub-discriminators of the multi-scale discriminator have similar convolutional network-based model structures, and work on different data scales to judge the fitting effect of generators at different scales. The network structure of each time-domain discriminator is a downsampling structure composed of a layer of 1-dimensional convolution and a 4-layer group convolution. Its input is two parts: the original speech signal and the generated speech signal generated by the generation network. constitute. The wavelet discriminator decomposes the input speech signal into four sub-signals of different frequency bands through cubic wavelet decomposition, and uses the stacked convolutional neural network to evaluate and judge the generation effect. In the framework of WMelGAN, the generator and multiple discriminators are trained in an adversarial manner, so that the audio generated by the generator reaches the effect that the discriminator group cannot judge the authenticity, and finally the generator is used to generate high-quality speech with high intelligibility Signal.

The voice reconstruction algorithm based on the wavelet-based multi-scale time-frequency domain generative confrontation network proposed by the present invention performs confrontation training between the generator and the discriminator by setting a series of loss functions, and the target is shown in formula (4)(5):

loss _D = loss _{disc_TD} + loss _{disc_WD} (4)

loss _G ＝loss _{gen_TD} +loss _{gen_WD} +loss _mel *45+(loss _{feature_TD} +loss _{feature_WD} )*2 (5)

Among them, loss _D represents the loss function of the discriminant network, and loss _G represents the loss function of the generation network. The loss function of the discriminator is calculated from the features output by the multi-scale temporal discriminative network and the wavelet discriminative network. This loss function minimizes the L1 distance between the original speech signal and the feature map of the discriminative network that generates the speech signal. The loss of the generation network consists of five parts, namely multi-scale time-domain discriminator loss loss _{gen_TD} , wavelet discriminator loss loss _{gen_WD} , Mel loss _mel , multi-scale time-domain discriminator feature map loss loss _{feature_TD} , wavelet discriminator feature Graph loss loss _{feature_WD} . The multi-scale time domain discriminant network loss loss _{disc_TD} of the discriminator and the wavelet discriminant network loss _{disc_WD} of the discriminator are defined as:

where x represents the original waveform, s represents the acoustic features (e.g., mel spectrogram), z represents the Gaussian noise vector, and TD and WD represent the multiscale temporal discriminant network and wavelet discriminative network, respectively. The generator's multi-scale time-domain discriminant network loss loss _{gen_TD} and the generator's wavelet discriminant network loss _{gen_WD} are defined as:

分别代表不同尺度的三个时域判别器所产生的损失。梅尔损失loss_mel是利用多尺度的梅尔语谱图量化原始语音波形和合成语音波形之间的差距，其定义为：in

Represent the losses produced by the three temporal discriminators at different scales, respectively. Mel loss loss _mel is the use of multi-scale Mel spectrogram to quantify the gap between the original speech waveform and the synthesized speech waveform, which is defined as:

loss _mel =||MEL(x)-MEL(G(s,z))|| _F (10)

where ||·|| _F represents the F-norm, and MEL(·) represents the Mel spectrogram transform for a given speech signal.

Step 4: Generate voice reviews.

The intelligibility and naturalness of the reconstructed speech signal are evaluated by subjective evaluation system and objective evaluation system. The subjective evaluation system refers to the field of speech synthesis. Since the final service object of speech synthesis is human beings, the general speech synthesis model uses Mean Opinion Score (MOS) as an evaluation index. This research system refers to the subjective evaluation system in the evaluation of speech effects; the objective evaluation system refers to the field of human behavior recognition and signal processing. Specifically, the present invention uses the three objective parameters of Peak Signal-to-Noise Ratio (PSNR), Mel-Cepstral Distortion (Mel-Cepstral Distortion, MCD) and Root Mean Square Error (F0 Root Mean Square Error, F0 RMSE) of the fundamental frequency. metric to measure the difference between the synthesized speech and the reference speech signal. In addition, the present invention also uses Accuracy of Dictation Test (ADT) to measure the intelligibility of synthesized speech. To quantify the quality of a synthetic speech signal reconstructed from an accelerometer signal, the peak signal-to-noise ratio (PSNR) measures the ratio between the maximum possible power of the signal and the noise power affecting its quality:

In order to measure the distance between the original speech signal and the synthesized speech signal, the present invention proposes to use Mel Cepstrum Distortion (MCD) to quantify the difference between the two signals in Mel Cepstrum Characteristic (MFCC). Specifically, the Mel cepstrum distortion of the kth frame can be expressed as:

Wherein s represents the synthesized speech signal, r represents the original speech signal, M is the number of the Mel filter, M=10 among the present invention, MC _s (i, k) and MC _r (i, k) are synthesized speech signals s and the MFCC coefficients of the original speech signal r. For simplicity, after omitting the subscript of MC, it can be expressed as:

where X _k,n represents the logarithmic power output of the nth triangular filter, namely:

Among them, X(k,m) represents the Fourier transform result of frame k of the input speech frame with frequency index m, and w _n (m) represents the nth Mel filter. Obviously, the lower the value of Mel Cepstral Distortion (MCD), the closer the synthetic audio reconstructed from the built-in accelerometer is to the original speech signal. The present invention uses the root mean square error (F0 RMSE) of the base frequency to compare the gap between the original voice base frequency and the synthesized voice base frequency. the better. Specifically, the root mean square error of the fundamental frequency is expressed as:

则代表合成语音信号的基频特征。Where f0 represents the fundamental frequency feature of the original speech signal,

represents the fundamental frequency characteristics of the synthesized speech signal.

The present invention uses the correct rate of dictation test and the average opinion score as the supervisory index, and the peak signal-to-noise ratio, the Mel cepstrum distortion and the root mean square error of the fundamental frequency as the objective index to construct and generate the speech effect evaluation model, thereby measuring the generation effect of the reconstructed speech .

Claims (3)

1. The deep learning voice reconstruction method based on the smart phone acceleration sensor is characterized by comprising the following steps of:

step 1: collecting data;

playing the audio file by using a mobile phone;

collecting signals of an acceleration sensor, and recording the signals and corresponding time stamps;

step 2: data processing; performing linear interpolation, noise processing and feature extraction on the sensor signals;

step 2-1: linear interpolation;

locating all time points without acceleration data through time stamps, and filling the missing data by using linear interpolation; for acceleration data with the same time stamp, adopting a mean value taking means, wherein the mean value represents the acceleration data of the time stamp, so that each time stamp has only one acceleration data;

step 2-2: noise processing;

removing noise caused by gravity factors and hardware factors from sensor signals by using reference data under a silent condition; filtering with a high pass filter having a cut-off frequency of aHz;

step 2-3: feature extraction

Dividing the sensor signal into a plurality of segments with fixed overlap, setting the segment length and the overlap length as 256 and 64 respectively, windowing each segment by using a hamming window, calculating a frequency spectrum by short-time fourier transform (STFT) to obtain an STFT matrix, recording the amplitude and phase of each time and frequency, and converting into a corresponding spectrogram according to formula (1):

spectrogram{x(n)}＝|STFT{x(n)}| ² (1)

wherein x (n) represents an acceleration sensor signal, and STFT { x (n) } represents an STFT matrix corresponding to the acceleration sensor signal;

converting the spectrogram into a Mel spectrogram, and realizing the frequency f and Mel scale f on the spectrogram corresponding to the acceleration sensor signal through the steps (2) and (3) _mel Interconversion between:

finally, converting the spectrogram into a Mel spectrogram by using a formula (2) to obtain a characteristic-Mel spectrogram;

step 3: reconstructing voice;

generating a voice reconstruction model WMelGAN of an countermeasure network by adopting a multi-scale time-frequency domain based on wavelets, and converting the sensor data processed in the step 2 into a synthetic voice signal;

step 3-1: the wavelet-based multi-scale time-frequency domain generation of a speech reconstruction model of an countermeasure network includes three components: a generator, a multi-scale discriminator, and a wavelet discriminator;

the generator converts the Mel spectrogram into a synthetic voice signal, the generator is formed by a series of up-sampling transposed convolution layers, and a residual error network with a cavity convolution layer is arranged behind each transposed convolution layer;

different sub-discriminants of the multi-scale discriminant have model structures based on a convolution network and work on different data scales to judge the output results of generators of different scales; the network structure of each sub-arbiter is a down-sampling structure consisting of a layer 1-dimensional convolution, a layer 4-dimensional grouping convolution and a layer one-dimensional sum in sequence, and the input of the sub-arbiter comprises an original voice signal and a synthesized voice signal generated by a generating network device;

the wavelet discriminator decomposes the input voice signal into four sub-signals with different frequency bands through three times of wavelet decomposition, and evaluates and judges the generation effect by using a stacked convolutional neural network; in the WMelGAN model, the generator and the plurality of discriminators train in a countermeasure mode, so that the audio generated by the generator reaches the effect that the discriminators cannot judge true or false, and finally, the generator is utilized to generate a final synthesized voice signal;

step 3-2: a loss function;

the countermeasure training between the generator and the arbiter is performed by setting a series of loss functions, the targets are as shown in formulas (4) and (5):

loss _D ＝loss _{disc_TD} +loss _{disc_WD} (4)

loss _G ＝loss _{gen_TD} +loss _{gen_WD} +loss _mel *45+(loss _{feature_TD} +loss _{feature_WD} )*2 (5)

wherein loss is _D Representing the overall loss function of two discriminators, loss _G Representing a loss function of the generator; the loss of the generator is composed of five parts, namely a loss of the multi-scale discriminator _{gen_TD} Loss of wavelet discriminators loss _{gen_WD} Mel loss _mel Loss of feature map loss of multi-scale time domain discriminator _{feature_TD} Loss of wavelet discriminant feature map loss _{feature_WD} ；

The overall loss of two discriminators is divided into multi-scale discriminators loss _{disc_TD} And wavelet discriminant loss _{disc_WD} The definition is:

wherein x represents an original voice signal waveform, s represents a Mel spectrogram, z represents a Gaussian noise vector, TD and WD respectively represent a multi-scale discriminator and a wavelet discriminator, and subscript k represents different scales; g (s, z) represents the generated speech signal;

representing the desire;

multi-scale discriminant loss of generator _{gen_TD} Wavelet discriminant loss of sum generator _{gen_WD} Is defined as:

wherein the method comprises the steps of

Three multi-scale discriminators corresponding to different scales respectively;

mel loss _mel The difference between the original voice waveform and the synthesized voice waveform is quantized by utilizing a multi-scale mel spectrogram, which is defined as:

loss _mel ＝||MEL(x)-MEL(G(s,z))|| _F (10)

wherein I II _F Representing the F-norm, wherein MEL (·) represents the MEL-gram transform for a given speech signal;

step 4: a idiomatic evaluation system; evaluating the intelligibility and naturalness index of the reconstructed voice signal through a subjective evaluation system and an objective evaluation system;

step 4-1: subjective evaluation system uses mean opinion score MOS as evaluation index

Step 4-2: an objective evaluation system;

step 4-2-1: measuring the difference between the synthesized voice and the original voice signal by adopting three objective indexes of peak signal-to-noise ratio PSNR, mel cepstrum distortion MCD and root mean square error F0RMSE of fundamental frequency; the intelligibility of the synthesized voice is measured by adopting dictation test accuracy ADT;

peak signal-to-noise ratio PSNR measures the ratio between the maximum possible power of a signal and the noise power affecting its quality:

wherein S is _peak Representing peak speech signals, S representing speech signals, N representing noise signals;

step 4-2-2: the difference between the original speech signal and the synthesized speech signal between the mel-cepstrum features MFCCs is quantized using the mel-cepstrum distortion MCD, specifically, the mel-cepstrum distortion of the k-th frame is expressed as:

where r represents the original speech signal, M is the number of Mel filters, MC _s (i, k) and MC _r (i, k) are MFCC coefficients of the synthesized speech signal s and the original speech signal r;

omitting the subscript of MC, expressed as:

wherein X is _k,n Representing the logarithmic power output of the nth triangular filter, namely:

where X (k, m) represents the k-th frame fourier transform result, w, of an input speech frame with frequency index m _n (m) represents an nth mel filter;

step 4-2-3: the difference between the fundamental frequency of the original speech and the fundamental frequency of the synthesized speech is compared using the root mean square error F0RMSE of the fundamental frequency, expressed as:

where f0 represents the fundamental frequency characteristic of the original speech signal,

representing the fundamental frequency characteristics of the synthesized speech signal.

2. The smart phone acceleration sensor-based deep learning speech reconstruction method according to claim 1, wherein a=20.

3. The deep learning voice reconstruction method based on the smart phone acceleration sensor according to claim 1, wherein the number m=10 of mel filters.

Images