WO2022121799A1 - 声音信号处理方法、装置和电子设备 - Google Patents

声音信号处理方法、装置和电子设备 Download PDF

Info

Publication number
WO2022121799A1
WO2022121799A1 PCT/CN2021/135398 CN2021135398W WO2022121799A1 WO 2022121799 A1 WO2022121799 A1 WO 2022121799A1 CN 2021135398 W CN2021135398 W CN 2021135398W WO 2022121799 A1 WO2022121799 A1 WO 2022121799A1
Authority
WO
WIPO (PCT)
Prior art keywords
feature map
convolution kernel
convolution
data
sound
Prior art date
Application number
PCT/CN2021/135398
Other languages
English (en)
French (fr)
Inventor
范文之
孔凡留
徐杨飞
张志飞
Original Assignee
北京有竹居网络技术有限公司
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 北京有竹居网络技术有限公司 filed Critical 北京有竹居网络技术有限公司
Priority to US18/256,285 priority Critical patent/US20240038252A1/en
Publication of WO2022121799A1 publication Critical patent/WO2022121799A1/zh

Links

Images

Classifications

    • G—PHYSICS
    • G10—MUSICAL INSTRUMENTS; ACOUSTICS
    • G10L—SPEECH ANALYSIS OR SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING; SPEECH OR AUDIO CODING OR DECODING
    • G10L21/00—Processing of the speech or voice signal to produce another audible or non-audible signal, e.g. visual or tactile, in order to modify its quality or its intelligibility
    • G10L21/02—Speech enhancement, e.g. noise reduction or echo cancellation
    • G10L21/0208—Noise filtering
    • G—PHYSICS
    • G10—MUSICAL INSTRUMENTS; ACOUSTICS
    • G10L—SPEECH ANALYSIS OR SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING; SPEECH OR AUDIO CODING OR DECODING
    • G10L21/00—Processing of the speech or voice signal to produce another audible or non-audible signal, e.g. visual or tactile, in order to modify its quality or its intelligibility
    • G10L21/02—Speech enhancement, e.g. noise reduction or echo cancellation
    • G—PHYSICS
    • G10—MUSICAL INSTRUMENTS; ACOUSTICS
    • G10L—SPEECH ANALYSIS OR SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING; SPEECH OR AUDIO CODING OR DECODING
    • G10L21/00—Processing of the speech or voice signal to produce another audible or non-audible signal, e.g. visual or tactile, in order to modify its quality or its intelligibility
    • G10L21/02—Speech enhancement, e.g. noise reduction or echo cancellation
    • G10L21/0208—Noise filtering
    • G10L21/0216—Noise filtering characterised by the method used for estimating noise
    • G—PHYSICS
    • G10—MUSICAL INSTRUMENTS; ACOUSTICS
    • G10L—SPEECH ANALYSIS OR SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING; SPEECH OR AUDIO CODING OR DECODING
    • G10L21/00—Processing of the speech or voice signal to produce another audible or non-audible signal, e.g. visual or tactile, in order to modify its quality or its intelligibility
    • G10L21/02—Speech enhancement, e.g. noise reduction or echo cancellation
    • G10L21/0208—Noise filtering
    • G10L21/0216—Noise filtering characterised by the method used for estimating noise
    • G10L21/0232—Processing in the frequency domain
    • G—PHYSICS
    • G10—MUSICAL INSTRUMENTS; ACOUSTICS
    • G10L—SPEECH ANALYSIS OR SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING; SPEECH OR AUDIO CODING OR DECODING
    • G10L25/00—Speech or voice analysis techniques not restricted to a single one of groups G10L15/00 - G10L21/00
    • G10L25/27—Speech or voice analysis techniques not restricted to a single one of groups G10L15/00 - G10L21/00 characterised by the analysis technique
    • G—PHYSICS
    • G10—MUSICAL INSTRUMENTS; ACOUSTICS
    • G10L—SPEECH ANALYSIS OR SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING; SPEECH OR AUDIO CODING OR DECODING
    • G10L25/00—Speech or voice analysis techniques not restricted to a single one of groups G10L15/00 - G10L21/00
    • G10L25/27—Speech or voice analysis techniques not restricted to a single one of groups G10L15/00 - G10L21/00 characterised by the analysis technique
    • G10L25/30—Speech or voice analysis techniques not restricted to a single one of groups G10L15/00 - G10L21/00 characterised by the analysis technique using neural networks

Definitions

  • the present disclosure relates to the field of Internet technologies, and in particular, to a sound signal processing method, apparatus, and electronic device.
  • the collected sound signal usually contains various kinds of noise, such as ambient noise and noise from other interfering sound sources.
  • noise In communication applications, the existence of noise will reduce the clarity and intelligibility of speech, and seriously affect the quality of calls; in intelligent human-computer interaction systems, noise will significantly reduce the recognition rate of the speech recognition system and seriously affect the user's product experience.
  • an embodiment of the present disclosure provides a sound signal processing method, the method includes: importing first spectrum data corresponding to first audio data into a pre-trained sound processing model to obtain a processing result; and based on the processing result , generating pure audio data corresponding to the first audio data; wherein the sound processing model includes at least one preset convolution layer, and the operations performed in the preset convolution layer include: based on the first convolution kernel group , perform a convolution operation on the corresponding first spectral feature map of the input preset convolution layer to obtain a second spectral feature map; based on the second convolution kernel group, combine the obtained second spectral feature maps to obtain The third spectral feature map corresponding to the second convolution kernel group.
  • an embodiment of the present disclosure provides a sound signal processing apparatus, including: a first generating unit, configured to import first spectrum data corresponding to the first audio data into a pre-trained sound processing model to obtain a processing result; a second generating unit, configured to generate pure audio data corresponding to the first audio data based on the processing result; wherein, the sound processing model includes at least one preset convolution layer, where the preset convolution layer
  • the operations performed include: based on the first convolution kernel group, performing a convolution operation on the corresponding first sound spectrum feature map of the input preset convolution layer to obtain a second sound spectrum feature map; based on the second convolution kernel group, performing a convolution operation on The obtained second sound spectrum feature maps are combined to obtain a third sound spectrum feature map corresponding to the second convolution kernel group.
  • embodiments of the present disclosure provide an electronic device, including: one or more processors; and a storage device for storing one or more programs, when the one or more programs are stored by the one or more programs Each processor executes, so that the one or more processors implement the sound signal processing method as described in the first aspect.
  • an embodiment of the present disclosure provides a computer-readable medium on which a computer program is stored, and when the program is executed by a processor, implements the steps of the sound signal processing method described in the first aspect.
  • FIG. 1 is a flowchart of an embodiment of a sound signal processing method according to the present disclosure
  • Fig. 2 is a schematic diagram of an operation flow performed at a preset convolutional layer
  • Fig. 3 is an exemplary sound spectrum characteristic map
  • FIG. 4 is an exemplary flowchart of step 201
  • FIG. 5 is an exemplary flowchart of step 202
  • FIG. 6 is an exemplary scene diagram of step 201
  • FIG. 7A and 7B are exemplary scene diagrams of step 202;
  • 8A and 8B are exemplary scene graphs of receptive field changes
  • FIG. 9 is a schematic structural diagram of an embodiment of a sound signal processing apparatus according to the present disclosure.
  • FIG. 10 is an exemplary system architecture to which the sound signal processing method of an embodiment of the present disclosure may be applied;
  • FIG. 11 is a schematic diagram of a basic structure of an electronic device provided according to an embodiment of the present disclosure.
  • the term “including” and variations thereof are open to include, i.e., “including but not limited to”.
  • the term “based on” is “based at least in part on.”
  • the term “one embodiment” means “at least one embodiment”; the term “another embodiment” means “at least one additional embodiment”; the term “some embodiments” means “at least some embodiments”. Relevant definitions of other terms will be given in the description below.
  • FIG. 1 shows a flow of an embodiment of a sound signal processing method according to the present disclosure.
  • the sound signal processing method is applied to a terminal device.
  • the sound signal processing method includes the following steps:
  • Step 101 Import the first spectrum data corresponding to the first audio data into a pre-trained sound processing model to obtain a processing result.
  • an execution body eg, a terminal device of the sound signal processing method may import the first spectrum data corresponding to the first audio data into a pre-trained sound processing model to obtain a processing result.
  • the above-mentioned first audio data may be a sound signal in the form of a digital signal.
  • a sound signal in the form of an analog signal can be converted into a sound signal in the form of a digital signal.
  • the above-mentioned first audio data may be a time-domain signal.
  • the first audio data may be time-frequency transformed to obtain the first spectrum data.
  • the specific transformation method for performing the time-frequency transformation can be set according to the actual application scenario, which is not limited here.
  • the first spectral data may form a two-dimensional matrix, one direction of the matrix represents the frequency dimension, the other direction of the matrix represents the time dimension, and the value of the matrix element in the matrix represents the magnitude of the frequency.
  • the original signal (2-second time domain signal) can be divided into frames and windowed, and many frames can be obtained, and FFT (Fast Fourier) can be performed on each frame.
  • Leaf transform convert the time domain signal into frequency domain signal, and stack the frequency domain signal (spectrogram) after FFT of each frame in time to obtain the spectrogram.
  • This spectrogram can be understood as an intuitive interpretation of the first spectral data.
  • Step 102 based on the processing result, generate pure audio data corresponding to the first audio data.
  • the above-mentioned execution body may generate pure audio data corresponding to the first audio data based on the processing result.
  • the specific data items included in the processing result may be set according to the actual application scenario, which is not limited here.
  • pure audio data corresponding to the first audio data may be generated in a manner suitable for the specific data item according to different specific data items included in the processing result.
  • the above-mentioned sound processing model may be pre-trained.
  • the parameters in the sound processing model can be predetermined by training.
  • the above-mentioned sound processing model may include at least one preset convolution layer.
  • the number of preset convolutional layers in the above sound processing model may be set according to an actual application scenario, which is not limited herein. It can be understood that other types of network layers can also be set in the sound processing model according to actual application scenarios.
  • FIG. 2 shows the operation flow performed in the preset convolution layer.
  • Step 201 based on the first convolution kernel group, perform a convolution operation on the first sound spectrum feature map input to the preset convolution layer to obtain a second sound spectrum feature map.
  • each first convolution kernel group corresponds to each first sound spectrum feature map input to the preset convolution layer.
  • the number of first convolution kernel groups matches the number of first spectral feature maps input to the preset convolution layer.
  • Step 202 based on the second convolution kernel group, combine the obtained second sound spectrum feature maps to obtain a third sound spectrum feature map corresponding to the second convolution kernel group.
  • the number of second convolution kernel groups matches the number of output channels.
  • FIG. 3 shows an exemplary spectrogram.
  • the frequency dimension and the time dimension of the spectrogram are exemplarily marked in FIG. 3 .
  • the first spectrum data can be understood as the original spectrogram.
  • a spectrogram feature map can be obtained.
  • the preset convolution layer after the first preset convolution layer, the input is the sound spectrum feature map, and the output can also be called the sound spectrum feature map.
  • this application takes a preset convolution layer as an example for description.
  • the input of the preset convolutional layer may be referred to as the first spectral feature map.
  • the original spectrogram can also understand the spectrogram feature map
  • the preset convolution layer may include at least two first convolution kernel groups.
  • the first convolution kernel group has a one-to-one correspondence with the first spectral feature map.
  • each first convolution kernel group can process a first spectral feature map to obtain a second spectral feature map.
  • the number of convolution kernels in the first convolution kernel group may be one or at least two.
  • each second convolution kernel group involves all the second sound spectrum feature maps, and the calculation result of each second convolution kernel group can be used as an output of the preset convolution layer.
  • the input of the preset convolution layer may be 3 channels, that is, the first sound spectrum feature map No. 1, the first sound spectrum feature map No. 2, and the first sound spectrum feature map No. 3.
  • the number of the first convolution kernel group may be the same as the number of input channels, that is, the number of the first convolution kernel group may be 3.
  • Each first convolution kernel group may have a corresponding first spectral feature map.
  • the first convolution kernel group No. 1 may convolve the first acoustic spectrum feature map No. 1 to obtain the second acoustic spectrum feature map No. 1.
  • the first convolution kernel group No. 2 can convolve the first sound spectrum feature map No. 2 to obtain the second sound spectrum feature map No. 2.
  • the first convolution kernel group No. 3 can convolve the first sound spectrum feature map No. 3 to obtain a third sound spectrum feature map No. 3.
  • the number of output channels of the preset convolutional layer can be 2.
  • the number of the second convolution kernel group may be the same as the number of output channels, that is, the number of the second convolution kernel group is 2.
  • the second convolution kernel group No. 1 may combine the second acoustic spectral feature map No. 1, the second acoustic spectral feature map No. 2, and the second acoustic spectral feature map No. 3 to obtain the third acoustic spectral feature map No. 1.
  • the second convolution kernel group No. 2 may combine the second acoustic spectrum feature map No. 1, the second acoustic spectrum feature map No. 2, and the second acoustic spectrum feature map No. 3 to obtain the third acoustic spectrum feature map No. 2.
  • the second convolution kernel in the second convolution kernel group may be a three-dimensional convolution kernel.
  • the depth of the second convolution kernel may be the same as the number of the second spectral feature maps.
  • the first spectrum data is processed by using a sound processing model including at least one preset convolution layer to obtain a processing result, and based on the processing result, pure audio data is obtained, It can reduce the amount of calculation consumed to obtain pure audio data and improve the processing speed.
  • the number of multiplication calculations for a single preset convolution layer of the present application is C1+C2.
  • C1 is the multiplication calculation amount in step 201, that is, the length of the first convolution kernel * the width of the first convolution kernel * the length of the frequency dimension * the length of the time dimension * the number of input channels.
  • C2 is the multiplication calculation amount in step 201, that is, the number of input channels*frequency dimension length*time dimension length*the number of output channels. It can be understood that the size of the second convolution kernel is usually 1*1*number of input channels when merging.
  • the number of multiplication calculations is C3, that is, the number of input channels * frequency dimension length * time dimension length * first convolution kernel length * first convolution kernel width * number of output channels .
  • the above-mentioned sound processing model is provided in the terminal device.
  • the sound signal processing methods provided by some embodiments of the present application not only reduce the amount of calculation, but also ensure a better processing accuracy, that is, have a better noise suppression effect. Due to the small amount of computation, the methods and sound processing models provided in some embodiments of this application are suitable for implementation in terminal devices.
  • the sound processing model provided by some embodiments of the present application is implemented in a terminal device, and the collected sound can be processed in time, which can not only improve the user's sound experience, but also reduce the amount of data transmission in remote interaction tasks.
  • the number of the first convolution kernels in the first convolution kernel group is at least two.
  • the above step 201 may include: according to the first correspondence, using the first convolution kernel in the first convolution kernel group to perform a convolution operation on the first acoustic spectrum feature map to obtain the second acoustic spectrum feature picture.
  • the first correspondence may indicate a correspondence between the first convolution kernel and the frequency of the first spectral feature map.
  • a first convolution kernel may be set at every other frequency.
  • a first convolution kernel a, a first convolution kernel b, a first convolution kernel c, a first convolution kernel d, and a first convolution kernel e may be set.
  • the number of convolution kernels in the first convolution kernel group can be set according to the actual application scenario, which is not limited here.
  • each of the first convolution kernels in the first convolution kernel group may be convolution kernels with the same size and different weights.
  • the weight of each first convolution kernel may be a value learned by adjusting during the training process of the sound processing model.
  • the first convolution kernel group including at least two first convolution kernels
  • a different convolution kernel is learned for different frequency dimensions of the output, which increases the amount of network parameters and does not increase the calculation quantity.
  • the processing accuracy of the sound processing model can be improved while ensuring the processing efficiency.
  • the number of second convolution kernels in the second convolution kernel group is at least two.
  • the above step 204 may include: according to the second correspondence, using the second acoustic spectrum feature map obtained by merging the second convolution kernels in the second convolution kernel group to obtain the second convolution kernel group The corresponding third spectral feature map.
  • the second correspondence is used to indicate the correspondence between the second convolution kernel and the frequency of the second spectral feature map.
  • the second correspondence is used to indicate the correspondence between the second convolution kernel and the frequency of the second spectral feature map.
  • Figures 7A and 7B please refer to Figures 7A and 7B.
  • Fig. 7A shows the second convolution kernel f corresponding to the first frequency in the frequency dimension.
  • the second convolution kernel f can perform a calculation on the values in the same positions (ie, the first row and the first column) of the second acoustic spectrum feature map No. 1, the second acoustic spectrum feature map No. 2, and the second acoustic spectrum feature map No. 3 Merge (for example, take a weighted sum) to obtain the value in the corresponding position (ie, the first row and the first column) in No. 1 of the third spectral feature map.
  • Fig. 7B shows the second convolution kernel g corresponding to the first frequency in the frequency dimension.
  • the second convolution kernel g can combine the values in the same position (ie, the first row and last column) of the second acoustic spectrum feature map No. 1, the second acoustic spectrum feature map No. 2, and the second acoustic spectrum feature map No. 3 (for example, taking a weighted sum) to obtain the value in the corresponding position (ie, the first row and last column) in No. 1 of the third spectral feature map.
  • the second convolution group No. 1 may include a second convolution kernel f and a second convolution kernel g, and may also include a second convolution kernel corresponding to other frequencies of the frequency dimension of the second acoustic spectrum feature map. .
  • the second convolution kernel group including at least two second convolution kernels
  • different convolution kernels can be learned for different frequencies, the amount of network parameters is increased, and the amount of calculation is not increased. Thereby, the processing accuracy of the sound processing model can be improved while ensuring the processing efficiency.
  • the number of convolution kernels of the first convolution kernel group is determined according to the length and stride of the frequency dimension of the first spectral feature map.
  • the stride can be used to characterize the sparsity of the convolution operation.
  • the length of the frequency dimension is 10
  • the stride is 2, and the number of convolution kernels is 5. If the stride in Figure 6 is changed to 1, the number of convolution kernels can be 10.
  • the number of convolution kernels in the first convolution kernel group is the same length as the frequency.
  • step size as the adjustment basis for adjusting the number of convolution kernels can reduce the number of calculations and improve processing efficiency.
  • the receptive field of the first convolution kernel is determined based on the sampling position and a preset position offset parameter.
  • the receptive field of the first convolution kernel may be determined based on candidate sampling positions and preset position offset parameters.
  • FIGS. 8A and 8B illustrate exemplary schematic diagrams of receptive field changes.
  • the candidate sampling position of the convolution kernel is shown by the shaded part of FIG. 8A; if the set position offset parameter indicates that the sampling position is changed on the basis of the candidate sampling position, for example, it is changed to FIG. 8B.
  • the position of the shaded part in Figure 8B, the final receptive field of the convolution kernel is the position of the shaded part in Figure 8B.
  • the sound processing model includes at least one self-attention layer disposed after the at least one preset convolutional layer.
  • the operations performed in the self-attention layer include: for each spectral feature map output by the preset convolution layer, according to the value of each position in the spectral feature map and the spectral feature map The value of other positions in the value is re-valued at this position.
  • the self-attention layer is to re-value the value of each position of the sound spectrum feature map
  • the specific implementation of the self-attention layer can be set according to the actual application scenario. Do limit.
  • the processing results can be made more accurate.
  • the above-described sound processing model includes masking data.
  • Mask data also referred to as mask data, is used to extract the target signal from the mixed signal.
  • a masking signal is used to process the mixed signal, and the speech signal can be extracted from the mixed signal.
  • the spectrogram corresponding to the pure speech data can be obtained by multiplying the masking data by the spectrogram corresponding to the mixed signal.
  • the above step 102 may include generating second spectral data according to the masking data and the first spectral data; converting the second spectral data into time domain data to obtain the pure audio data.
  • the product of the first spectral data and the masking data may be used as the second spectral data.
  • outputting a sound processing model including masking data may be trained in the following ways: acquiring mixed audio samples; importing mixed audio samples into an untrained sound processing model to generate candidate masking data; label and the candidate masking data, generate a first loss value; based on the first loss value, adjust the parameters in the untrained voice processing model;
  • the labels of the training samples are generated by performing time-frequency transformation on the pure audio samples and the mixed audio samples respectively, generating training masking data according to the transformed data, and determining the training masking data as labels.
  • the frequency domain data corresponding to the pure audio sample and the frequency domain data corresponding to the mixed audio sample can be taken as a ratio, and the ratio can be determined as the masking data for training.
  • a set of pure audio samples and a set of noise samples can be set.
  • the pure audio samples may be selected from the set of pure audio samples in various ways, and the noise samples may also be selected from the set of noise samples in various ways. Then the selected pure audio sample and the selected noise sample are combined to obtain a mixed audio sample.
  • the sound processing model trained based on the intermediate processing result has relatively high processing accuracy. Therefore, by using the masking data as the intermediate processing result, the accuracy of the audio signal processing can be improved.
  • the processing results may include clean spectral data.
  • the pure spectral data may be frequency domain data corresponding to the pure audio data.
  • the above step 102 may include: converting pure spectral data into time domain data to obtain pure audio data.
  • outputting a sound processing model including pure audio data can be trained by the following methods: obtaining mixed audio samples; importing mixed audio samples into an untrained sound processing model to generate candidate pure spectral data; and the candidate pure spectrum data to generate a second loss; based on the second loss value, parameters in the untrained sound processing model.
  • the labels of the mixed audio samples include the pure spectral samples corresponding to the pure audio samples.
  • pure spectral data can be obtained.
  • the present disclosure provides an embodiment of a sound signal processing apparatus.
  • the apparatus embodiment corresponds to the method embodiment shown in FIG. 1 .
  • the sound signal processing apparatus of this embodiment includes: a first generating unit 901 and a second generating unit 902 .
  • the first generating unit is used to import the first spectrum data corresponding to the first audio data into a pre-trained sound processing model to obtain a processing result;
  • the second generating unit is used to generate the first frequency spectrum based on the processing result Pure audio data corresponding to audio data;
  • the sound processing model includes at least one preset convolution layer, and the operations performed in the preset convolution layer include: based on the first convolution kernel group, preset the input A convolution operation is performed on the corresponding first spectral feature map of the convolutional layer to obtain a second spectral feature map, wherein the number of the first convolution kernel group and the first spectral feature input to the preset convolution layer The number of maps is matched; based on the second convolution kernel group, the obtained second spectral feature maps are merged to obtain a third spectral feature map corresponding to the second convolution kernel
  • the number of the first convolution kernels in the first convolution kernel group is at least two; and based on the first convolution kernel group, inputting the corresponding first acoustic spectral features of the preset convolution layer Perform a convolution operation on the image to obtain a second spectral feature map, including: according to the first correspondence, using the first convolution kernel in the first convolution kernel group to perform a convolution operation on the first spectral feature map to obtain the second spectral feature map.
  • the spectral feature map wherein the first correspondence is used to indicate the corresponding relationship between the first convolution kernel and the frequency of the first spectral feature map.
  • the number of the second convolution kernels in the second convolution kernel group is at least two; and the obtained second acoustic spectrum feature maps are combined based on the second convolution kernel group to obtain the same
  • the third sound spectrum feature map corresponding to the second convolution kernel group includes: according to the second correspondence, the second sound spectrum feature map obtained by merging the second convolution kernels in the second convolution kernel group is used to obtain the same
  • the number of convolution kernels of the first convolution kernel group is determined according to the length of the frequency dimension of the first spectral feature map and the first step length.
  • the receptive field of the first convolution kernel is determined based on candidate sampling positions and preset position offset parameters.
  • the sound processing model includes at least one self-attention layer disposed after the at least one preset convolutional layer; wherein the execution of the self-attention layer The operation includes: for each sound spectrum feature map output by the preset convolution layer, re-acquiring the position according to the value of each position in the sound spectrum feature map and the values of other positions in the sound spectrum feature map value.
  • the sound processing model is set in the terminal device.
  • the processing result includes masking data; and the generating, based on the processing result, the pure audio data corresponding to the first audio data includes: according to the masking data and the first spectrum data , generating second spectrum data; converting the second spectrum data into time domain data to obtain the pure audio data.
  • the sound processing model is trained by: acquiring mixed audio samples; importing mixed audio samples into an untrained sound processing model to generate candidate masking data;
  • the candidate masking data is used to generate a first loss value; based on the first loss value, the parameters in the untrained sound processing model are adjusted; wherein the labels of the training samples are generated in the following manner: for pure audio samples and mixed audio samples Time-frequency transformation is performed separately, masking data for training is generated according to the transformed data, and masking data for training is determined as a label.
  • the processing result includes pure spectral data; and the generating, based on the processing result, the pure audio data corresponding to the first audio data includes: converting the pure spectral data into time domain data to obtain the pure audio data.
  • the sound processing model is trained by the following methods: acquiring mixed audio samples, wherein the labels of the mixed audio samples include pure spectral samples corresponding to the pure audio samples; importing the mixed audio samples into untrained sound processing the model to generate candidate pure spectrum data; generating a second loss value according to the pure spectrum samples and the candidate pure spectrum data; adjusting parameters in the untrained sound processing model based on the second loss value.
  • FIG. 10 illustrates an exemplary system architecture to which the sound signal processing method according to an embodiment of the present disclosure may be applied.
  • the system architecture may include terminal devices 1001 , 1002 , and 1003 , a network 1004 , and a server 1005 .
  • the network 1004 is a medium used to provide a communication link between the terminal devices 1001 , 1002 , 1003 and the server 1005 .
  • the network 1004 may include various connection types, such as wired, wireless communication links, or fiber optic cables, among others.
  • the terminal devices 1001, 1002, and 1003 can interact with the server 1005 through the network 1004 to receive or send messages and the like.
  • Various client applications may be installed on the terminal devices 1001 , 1002 and 1003 , such as web browser applications, search applications, and news information applications.
  • the client applications in the terminal devices 1001, 1002, and 1003 can receive the user's instruction, and complete corresponding functions according to the user's instruction, for example, add corresponding information to the information according to the user's instruction.
  • the terminal devices 1001, 1002, and 1003 may be hardware or software.
  • the terminal devices 1001, 1002, and 1003 can be various electronic devices that have a display screen and support web browsing, including but not limited to smart phones, tablet computers, e-book readers, MP3 players (Moving Picture Experts Group Audio Layer III, Moving Picture Experts Compression Standard Audio Layer 3), MP4 (Moving Picture Experts Group Audio Layer IV, Moving Picture Experts Compression Standard Audio Layer 4) Players, Laptops and Desktops, etc.
  • the terminal devices 1001, 1002, and 1003 are software, they can be installed in the electronic devices listed above. It can be implemented as a plurality of software or software modules (eg, software or software modules for providing distributed services), or can be implemented as a single software or software module. There is no specific limitation here.
  • the server 1005 may be a server that provides various services, such as receiving information acquisition requests sent by terminal devices 1001, 1002, and 1003, and acquiring display information corresponding to the information acquisition requests in various ways according to the information acquisition requests. And the related data of the displayed information is sent to the terminal devices 1001 , 1002 and 1003 .
  • the sound signal processing method provided by the embodiment of the present disclosure may be executed by a terminal device, and correspondingly, the sound signal processing apparatus may be set in the terminal devices 1001 , 1002 , and 1003 .
  • the sound signal processing method provided by the embodiment of the present disclosure may also be executed by the server 1005 , and correspondingly, the sound signal processing apparatus may be provided in the server 1005 .
  • terminal devices, networks and servers in FIG. 10 are only illustrative. There can be any number of terminal devices, networks and servers according to implementation needs.
  • FIG. 11 it shows a schematic structural diagram of an electronic device (eg, the terminal device or the server in FIG. 10 ) suitable for implementing an embodiment of the present disclosure.
  • Terminal devices in the embodiments of the present disclosure may include, but are not limited to, such as mobile phones, notebook computers, digital broadcast receivers, PDAs (personal digital assistants), PADs (tablets), PMPs (portable multimedia players), vehicle-mounted terminals (eg, mobile terminals such as in-vehicle navigation terminals), etc., and stationary terminals such as digital TVs, desktop computers, and the like.
  • the electronic device shown in FIG. 11 is only an example, and should not impose any limitation on the function and scope of use of the embodiments of the present disclosure.
  • the electronic device may include a processing device (eg, a central processing unit, a graphics processor, etc.) 1101 that may be loaded into a random access memory according to a program stored in a read only memory (ROM) 1102 or from a storage device 1108
  • the program in the (RAM) 1103 executes various appropriate operations and processes.
  • various programs and data necessary for the operation of the electronic device 1100 are also stored.
  • the processing device 1101, the ROM 1102, and the RAM 1103 are connected to each other through a bus 1104.
  • An input/output (I/O) interface 1105 is also connected to the bus 1104 .
  • I/O interface 1105 input devices 1106 including, for example, a touch screen, touchpad, keyboard, mouse, camera, microphone, accelerometer, gyroscope, etc.; including, for example, a liquid crystal display (LCD), speakers, vibration
  • An output device 1107 such as a computer
  • a storage device 1108 including, for example, a magnetic tape, a hard disk, etc.
  • Communication means 1109 may allow electronic devices to communicate wirelessly or by wire with other devices to exchange data. While FIG. 11 shows an electronic device having various means, it should be understood that not all of the illustrated means are required to be implemented or available. More or fewer devices may alternatively be implemented or provided.
  • embodiments of the present disclosure include a computer program product comprising a computer program carried on a non-transitory computer readable medium, the computer program containing program code for performing the method illustrated in the flowchart.
  • the computer program may be downloaded and installed from the network via the communication device 1109 , or from the storage device 1108 , or from the ROM 1102 .
  • the processing apparatus 1101 the above-mentioned functions defined in the methods of the embodiments of the present disclosure are executed.
  • the computer-readable medium mentioned above in the present disclosure may be a computer-readable signal medium or a computer-readable storage medium, or any combination of the above two.
  • the computer-readable storage medium can be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus or device, or a combination of any of the above. More specific examples of computer readable storage media may include, but are not limited to, electrical connections with one or more wires, portable computer disks, hard disks, random access memory (RAM), read only memory (ROM), erasable Programmable read only memory (EPROM or flash memory), fiber optics, portable compact disk read only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.
  • a computer-readable storage medium may be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system, apparatus, or device.
  • a computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave with computer-readable program code embodied thereon. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the foregoing.
  • a computer-readable signal medium can also be any computer-readable medium other than a computer-readable storage medium that can transmit, propagate, or transport the program for use by or in connection with the instruction execution system, apparatus, or device .
  • Program code embodied on a computer readable medium may be transmitted using any suitable medium including, but not limited to, electrical wire, optical fiber cable, RF (radio frequency), etc., or any suitable combination of the foregoing.
  • clients and servers can communicate using any currently known or future developed network protocols such as HTTP (HyperText Transfer Protocol), and can communicate with digital data in any form or medium.
  • Communication eg, a communication network
  • Examples of communication networks include local area networks (“LAN”), wide area networks (“WAN”), the Internet (eg, the Internet), and peer-to-peer networks (eg, ad hoc peer-to-peer networks), as well as any currently known or future development network of.
  • the above-mentioned computer-readable medium may be included in the above-mentioned electronic device; or may exist alone without being assembled into the electronic device.
  • the above-mentioned computer-readable medium carries one or more programs, and when the above-mentioned one or more programs are executed by the electronic device, the electronic device: import the first spectrum data corresponding to the first audio data into the pre-trained sound processing model, and obtain a processing result; based on the processing result, generate pure audio data corresponding to the first audio data; wherein, the sound processing model includes at least one preset convolution layer, and the preset convolution layer executes The operation includes: based on the first convolution kernel group, performing a convolution operation on the corresponding first sound spectrum feature map of the input preset convolution layer to obtain a second sound spectrum feature map, wherein the number of the first convolution kernel group Matching the number of the first sound spectrum feature maps input to the preset convolution layer; based on the second convolution kernel group, the obtained second sound spectrum feature maps are merged to obtain a corresponding second convolution kernel group.
  • the third spectral feature map of where the number of second convolution kernel groups matches the number of output channels.
  • Computer program code for performing operations of the present disclosure may be written in one or more programming languages, including but not limited to object-oriented programming languages—such as Java, Smalltalk, C++, and This includes conventional procedural programming languages - such as the "C" language or similar programming languages.
  • the program code may execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer, or entirely on the remote computer or server.
  • the remote computer may be connected to the user's computer through any kind of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (eg, using an Internet service provider through Internet connection).
  • LAN local area network
  • WAN wide area network
  • each block in the flowchart or block diagrams may represent a module, segment, or portion of code that contains one or more logical functions for implementing the specified functions executable instructions.
  • the functions noted in the blocks may occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved.
  • each block of the block diagrams and/or flowchart illustrations, and combinations of blocks in the block diagrams and/or flowchart illustrations can be implemented in dedicated hardware-based systems that perform the specified functions or operations , or can be implemented in a combination of dedicated hardware and computer instructions.
  • the units involved in the embodiments of the present disclosure may be implemented in a software manner, and may also be implemented in a hardware manner.
  • the name of the unit does not constitute a limitation of the unit itself under certain circumstances, for example, the first generating unit may also be described as a "unit for generating a processing result".
  • exemplary types of hardware logic components include: Field Programmable Gate Arrays (FPGAs), Application Specific Integrated Circuits (ASICs), Application Specific Standard Products (ASSPs), Systems on Chips (SOCs), Complex Programmable Logical Devices (CPLDs) and more.
  • FPGAs Field Programmable Gate Arrays
  • ASICs Application Specific Integrated Circuits
  • ASSPs Application Specific Standard Products
  • SOCs Systems on Chips
  • CPLDs Complex Programmable Logical Devices
  • a machine-readable medium may be a tangible medium that may contain or store a program for use by or in connection with the instruction execution system, apparatus or device.
  • the machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium.
  • Machine-readable media may include, but are not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices, or devices, or any suitable combination of the foregoing.
  • machine-readable storage media would include one or more wire-based electrical connections, portable computer disks, hard disks, random access memory (RAM), read only memory (ROM), erasable programmable read only memory (EPROM or flash memory), fiber optics, compact disk read only memory (CD-ROM), optical storage, magnetic storage, or any suitable combination of the foregoing.
  • RAM random access memory
  • ROM read only memory
  • EPROM or flash memory erasable programmable read only memory
  • CD-ROM compact disk read only memory
  • magnetic storage or any suitable combination of the foregoing.
  • the number of the first convolution kernel group matches the number of the first spectral feature maps input to the preset convolution layer, and the second convolution kernel group The number matches the number of output channels.
  • the number of the first convolution kernels in the first convolution kernel group is at least two; and the corresponding input preset convolution layer based on the first convolution kernel group Performing a convolution operation on the first sound spectrum feature map to obtain a second sound spectrum feature map, including: according to the first correspondence, using the first convolution kernel in the first convolution kernel group to convolve the first sound spectrum feature map An operation is performed to obtain a second spectral feature map, wherein the first correspondence is used to indicate a corresponding relationship between the first convolution kernel and the frequency of the first spectral feature map.
  • the number of second convolution kernels in the second convolution kernel group is at least two; and the second spectral feature map obtained based on the second convolution kernel group Merging to obtain a third acoustic spectrum feature map corresponding to the second convolution kernel group, including: according to the second correspondence, using the second convolution kernel in the second convolution kernel group to merge the obtained second acoustic spectrum feature to obtain a third spectral feature map corresponding to the second convolution kernel group, wherein the second correspondence is used to indicate a corresponding relationship between the second convolution kernel and the frequency of the second spectral feature map.
  • the number of convolution kernels of the first convolution kernel group is determined according to the length of the frequency dimension of the first spectral feature map and the first step length.
  • the receptive field of the first convolution kernel is determined based on candidate sampling positions and preset position offset parameters.
  • the sound processing model includes at least one self-attention layer disposed after the at least one preset convolutional layer; wherein, in the self-attention
  • the operations performed in the force layer include: for each sound spectrum feature map output by the preset convolution layer, according to the value of each position in the sound spectrum feature map and the values of other positions in the sound spectrum feature map, to This position is re-valued.
  • the sound processing model is set in the terminal device.
  • the processing result includes masking data; and the generating, based on the processing result, the pure audio data corresponding to the first audio data includes: according to the masking data and the The first spectrum data is generated to generate second spectrum data; the second spectrum data is converted into time domain data to obtain the pure audio data.
  • the sound processing model is trained by the following methods: obtaining mixed audio samples; importing the mixed audio samples into an untrained sound processing model to generate candidate masking data; according to the mixed audio samples and the candidate masking data, generate a first loss value; based on the first loss value, adjust the parameters in the untrained sound processing model; wherein the label of the training sample is generated in the following way: for pure audio
  • the samples and the mixed audio samples are respectively subjected to time-frequency transformation, and masking data for training is generated according to the transformed data, and the masking data for training is determined as a label.
  • the processing result includes pure spectral data; and the generating, based on the processing result, the pure audio data corresponding to the first audio data includes: converting the pure spectral data into time domain data to obtain the pure audio data.
  • the sound processing model is trained by: acquiring mixed audio samples, wherein the labels of the mixed audio samples include pure spectral samples corresponding to the pure audio samples; importing the mixed audio samples into The untrained sound processing model generates candidate pure spectral data; according to the pure spectral sample and the candidate pure spectral data, a second loss value is generated; based on the second loss value, the untrained sound processing model is adjusted. parameter.
  • a sound signal processing apparatus includes: a first generating unit, configured to import first spectrum data corresponding to first audio data into a pre-trained sound processing model to obtain a processing result ; a second generating unit for generating pure audio data corresponding to the first audio data based on the processing result; wherein, the sound processing model includes at least one preset convolution layer, in which the preset convolution layer
  • the operations performed by the layer include: based on the first convolution kernel group, performing a convolution operation on the corresponding first sound spectrum feature map input to the preset convolution layer to obtain a second sound spectrum feature map; based on the second convolution kernel group, The obtained second sound spectrum feature maps are combined to obtain a third sound spectrum feature map corresponding to the second convolution kernel group.
  • the number of the first convolution kernels in the first convolution kernel group is at least two; and the corresponding input preset convolution layer based on the first convolution kernel group Performing a convolution operation on the first sound spectrum feature map to obtain a second sound spectrum feature map, including: according to the first correspondence, using the first convolution kernel in the first convolution kernel group to convolve the first sound spectrum feature map An operation is performed to obtain a second spectral feature map, wherein the first correspondence is used to indicate a corresponding relationship between the first convolution kernel and the frequency of the first spectral feature map.
  • the number of second convolution kernels in the second convolution kernel group is at least two; and the second spectral feature map obtained based on the second convolution kernel group Merging to obtain a third acoustic spectrum feature map corresponding to the second convolution kernel group, including: according to the second correspondence, using the second convolution kernel in the second convolution kernel group to merge the obtained second acoustic spectrum feature to obtain a third spectral feature map corresponding to the second convolution kernel group, wherein the second correspondence is used to indicate a corresponding relationship between the second convolution kernel and the frequency of the second spectral feature map.
  • the number of convolution kernels of the first convolution kernel group is determined according to the length of the frequency dimension of the first spectral feature map and the first step length.
  • the receptive field of the first convolution kernel is determined based on candidate sampling positions and preset position offset parameters.
  • the sound processing model includes at least one self-attention layer disposed after the at least one preset convolutional layer; wherein, in the self-attention
  • the operations performed in the force layer include: for each sound spectrum feature map output by the preset convolution layer, according to the value of each position in the sound spectrum feature map and the values of other positions in the sound spectrum feature map, to This position is re-valued.
  • the sound processing model is set in the terminal device.
  • the processing result includes masking data; and the generating, based on the processing result, the pure audio data corresponding to the first audio data includes: according to the masking data and the The first spectrum data is generated to generate second spectrum data; the second spectrum data is converted into time domain data to obtain the pure audio data.
  • the sound processing model is trained by the following methods: obtaining mixed audio samples; importing the mixed audio samples into an untrained sound processing model to generate candidate masking data; according to the mixed audio samples and the candidate masking data, generate a first loss value; based on the first loss value, adjust the parameters in the untrained sound processing model; wherein the label of the training sample is generated in the following way: for pure audio
  • the samples and the mixed audio samples are respectively subjected to time-frequency transformation, and masking data for training is generated according to the transformed data, and the masking data for training is determined as a label.
  • the processing result includes pure spectral data; and the generating, based on the processing result, the pure audio data corresponding to the first audio data includes: converting the pure spectral data into time domain data to obtain the pure audio data.
  • the sound processing model is trained by: acquiring mixed audio samples, wherein the labels of the mixed audio samples include pure spectral samples corresponding to the pure audio samples; importing the mixed audio samples into The untrained sound processing model generates candidate pure spectral data; according to the pure spectral sample and the candidate pure spectral data, a second loss value is generated; based on the second loss value, the untrained sound processing model is adjusted. parameter.
  • an electronic device includes: one or more processors; a storage device for storing one or more programs, when the one or more programs are stored by the one or more programs
  • the processor executes such that the one or more processors implement any of the methods described in this application.
  • a computer-readable medium having a computer program stored thereon, the program, when executed by a processor, implements the method as described in any one of the present application.

Abstract

声音信号处理方法、装置和电子设备。该方法包括：将第一音频数据对应的第一频谱数据，导入预先训练的声音处理模型，得到处理结果（S101）；基于该处理结果，生成第一音频数据对应的纯净音频数据（S102）；声音处理模型包括至少一个预设卷积层，在预设卷积层执行的操作包括：基于第一卷积核组，对输入预设卷积层的对应第一声谱特征图进行卷积操作，得到第二声谱特征图（S201）；基于第二卷积核组，对得到的第二声谱特征图进行合并，得到与第二卷积核组对应的第三声谱特征图（S202）。由此，可以提供新的声音信号处理方式。

Description

声音信号处理方法、装置和电子设备
相关申请的交叉引用
本申请要求于2020年12月08日提交的，申请号为202011462091.2、发明名称为“声音信号处理方法、装置和电子设备”的中国专利申请的优先权，该申请的全文通过引用结合在本申请中。
技术领域
本公开涉及互联网技术领域，尤其涉及一种声音信号处理方法、装置和电子设备。
背景技术
随着互联网的发展，用户越来越多的使用终端设备实现各种功能。例如，在日常通信、智能语音交互系统等应用中，需要由终端采集声音信号。采集的声音信号中通常包含各种噪声，如环境噪声和来自其他干扰声源的噪声等。在通信应用中，噪声的存在会降低语音的清晰度和可懂度，严重影响通话质量；在智能人机交互系统中，噪声会显著降低语音识别系统识别率，严重影响用户的产品体验。
发明内容
提供该公开内容部分以便以简要的形式介绍构思，这些构思将在后面的具体实施方式部分被详细描述。该公开内容部分并不旨在标识要求保护的技术方案的关键特征或必要特征，也不旨在 用于限制所要求的保护的技术方案的范围。
第一方面，本公开实施例提供了一种声音信号处理方法，该方法包括：将第一音频数据对应的第一频谱数据，导入预先训练的声音处理模型，得到处理结果；基于所述处理结果，生成所述第一音频数据对应的纯净音频数据；其中，所述声音处理模型包括至少一个预设卷积层，在所述预设卷积层执行的操作包括：基于第一卷积核组，对输入预设卷积层的对应第一声谱特征图进行卷积操作，得到第二声谱特征图；基于第二卷积核组，对得到的第二声谱特征图进行合并，得到与第二卷积核组对应的第三声谱特征图。
第二方面，本公开实施例提供了一种声音信号处理装置，包括：第一生成单元，用于将第一音频数据对应的第一频谱数据，导入预先训练的声音处理模型，得到处理结果；第二生成单元，用于基于所述处理结果，生成所述第一音频数据对应的纯净音频数据；其中，所述声音处理模型包括至少一个预设卷积层，在所述预设卷积层执行的操作包括：基于第一卷积核组，对输入预设卷积层的对应第一声谱特征图进行卷积操作，得到第二声谱特征图；基于第二卷积核组，对得到的第二声谱特征图进行合并，得到与第二卷积核组对应的第三声谱特征图。
第三方面，本公开实施例提供了一种电子设备，包括：一个或多个处理器；存储装置，用于存储一个或多个程序，当所述一个或多个程序被所述一个或多个处理器执行，使得所述一个或多个处理器实现如第一方面所述的声音信号处理方法。
第四方面，本公开实施例提供了一种计算机可读介质，其上存储有计算机程序，该程序被处理器执行时实现如第一方面所述的声音信号处理方法的步骤。
附图说明
结合附图并参考以下具体实施方式，本公开各实施例的上述和其他特征、优点及方面将变得更加明显。贯穿附图中，相同或 相似的附图标记表示相同或相似的元素。应当理解附图是示意性的，原件和元素不一定按照比例绘制。
图1是根据本公开的声音信号处理方法的一个实施例的流程图；
图2是在预设卷积层执行的操作流程示意图；
图3是示例性的声谱特征图；
图4是步骤201的示例性流程图；
图5是步骤202的示例性流程图；
图6是步骤201的示例性场景图；
图7A和图7B是步骤202的示例性场景图；
图8A和图8B是感受野变化的示例性场景图；
图9是根据本公开的声音信号处理装置的一个实施例的结构示意图；
图10是本公开的一个实施例的声音信号处理方法可以应用于其中的示例性系统架构；
图11是根据本公开实施例提供的电子设备的基本结构的示意图。
具体实施方式
下面将参照附图更详细地描述本公开的实施例。虽然附图中显示了本公开的某些实施例，然而应当理解的是，本公开可以通过各种形式来实现，而且不应该被解释为限于这里阐述的实施例，相反提供这些实施例是为了更加透彻和完整地理解本公开。应当理解的是，本公开的附图及实施例仅用于示例性作用，并非用于限制本公开的保护范围。
应当理解，本公开的方法实施方式中记载的各个步骤可以按照不同的顺序执行，和/或并行执行。此外，方法实施方式可以包括附加的步骤和/或省略执行示出的步骤。本公开的范围在此方面不受限制。
本文使用的术语“包括”及其变形是开放性包括，即“包括但不 限于”。术语“基于”是“至少部分地基于”。术语“一个实施例”表示“至少一个实施例”；术语“另一实施例”表示“至少一个另外的实施例”；术语“一些实施例”表示“至少一些实施例”。其他术语的相关定义将在下文描述中给出。
需要注意，本公开中提及的“第一”、“第二”等概念仅用于对不同的装置、模块或单元进行区分，并非用于限定这些装置、模块或单元所执行的功能的顺序或者相互依存关系。
需要注意，本公开中提及的“一个”、“多个”的修饰是示意性而非限制性的，本领域技术人员应当理解，除非在上下文另有明确指出，否则应该理解为“一个或多个”。
本公开实施方式中的多个装置之间所交互的消息或者信息的名称仅用于说明性的目的，而并不是用于对这些消息或信息的范围进行限制。
请参考图1，其示出了根据本公开的声音信号处理方法的一个实施例的流程。该声音信号处理方法应用于终端设备。如图1所示该声音信号处理方法，包括以下步骤：
步骤101，将第一音频数据对应的第一频谱数据，导入预先训练的声音处理模型，得到处理结果。
在本实施例中，声音信号处理方法的执行主体(例如终端设备)可以将将第一音频数据对应的第一频谱数据，导入预先训练的声音处理模型，得到处理结果。
在本实施例中，上述第一音频数据可以是数字信号形式的声音信号。通常，模拟信号形式的声音信号可以转换成数字信号形式的声音信号。
在一些应用场景中，上述第一音频数据可以是时域信号，为了方便处理，可以将第一音频数据进行时频变换，得到第一频谱数据。在这里，进行时频变换的具体变换方式，可以根据实际的应用场景设置，在此不做限定。
在一些应用场景中，第一频谱数据可以形成二维矩阵，矩阵的一个方向表示频率维，矩阵的另一个方向表示时间维，矩阵中 的矩阵元素值表征频率的幅度。
作为示例，对于一段2秒的音频数据进行时频变换，可以将对原始信号(2秒的时域信号)进行分帧加窗后，可以得到很多帧，对每一帧做FFT(快速傅里叶变换)，把时域信号转为频域信号，把每一帧FFT后的频域信号(频谱图)在时间上堆叠起来就可以得到声谱图。这个声谱图可以理解作为第一频谱数据的直观解释。
步骤102，基于处理结果，生成第一音频数据对应的纯净音频数据。
在本实施例中，上述执行主体可以基于处理结果，生成第一音频数据对应的纯净音频数据。
在本实施例中，处理结果所包括的具体数据项，可以根据实际应用场景设置，在此不做限定。步骤102中，可以根据处理结果中所包括的具体数据项的不同，采用适合该具体数据项的方式，生成第一音频数据对应的纯净音频数据。
在本实施例中，上述声音处理模型，可以是预先训练的。换句话说，声音处理模型中的参数，可以是经过训练预先确定的。
在本实施例中，上述声音处理模型可以包括至少一个预设卷积层。
在本实施例中，上述声音处理模型中的预设卷积层的数量，可以根据实际应用场景设置，在此不做限定。可以理解，声音处理模型中还可以根据实际应用场景设置其它类型的网络层。
在本实施例中，请参考图2，其示出了在预设卷积层执行的操作流程。
步骤201，基于第一卷积核组，对输入预设卷积层的第一声谱特征图进行卷积操作，得到第二声谱特征图。
在本实施例中，各个第一卷积核组与输入至预设卷积层的各个第一声谱特征图对应。
在一些实施例中，第一卷积核组的数量与输入至预设卷积层的第一声谱特征图的数量匹配。
步骤202，基于第二卷积核组，对得到的第二声谱特征图进行 合并，得到与第二卷积核组对应的第三声谱特征图。
在一些实施例中，第二卷积核组的数量与输出通道数量匹配。
请参考图3，其示出了示例性的声谱特征图。图3中示例性地标出了声谱特征图的频率维和时间维。
在本实施例中，第一频谱数据可以理解为原始声谱图。原始声谱图在经过声音处理模型的第一个预设卷积层进行特征提取之后，可以得到声谱特征图。第一个预设卷积层之后的预设卷积层，输入的为声谱特征图，输出的也可以称为声谱特征图。
为了方便说明，本申请中以一个预设卷积层为例进行说明。预设卷积层的输入可以称为第一声谱特征图。(原始声谱图也可以理解声谱特征图)
在本实施例中，预设卷积层可以包括至少两个第一卷积核组。第一卷积核组与第一声谱特征图一一对应。换句话说，每个第一卷积核组可以处理一个第一声谱特征图，得到一个第二声谱特征图。
在本实施例中，第一卷积核组中的卷积核数量可以是一个或者至少两个。
在本实施例中，每个第二卷积核组的计算均涉及所有第二声谱特征图，每个第二卷积核组的计算结果可以作为预设卷积层的一个输出。
请参考图4，图4示出了步骤201的示意图。预设卷积层的输入可以为3通道，即第一声谱特征图一号、第一声谱特征图二号和第一声谱特征图三号。第一卷积核组的数量可以与输入通道数相同，即第一卷积核组的数量可以为3。每个第一卷积核组可以具有对应的第一声谱特征图。具体的，第一卷积核组一号可以对第一声谱特征图一号进行卷积，得到第二声谱特征图一号。第一卷积核组二号可以对第一声谱特征图二号进行卷积，得到第二声谱特征图二号。第一卷积核组三号可以对第一声谱特征图三号进行卷积，得到第三声谱特征图三号。
请参考图5，图5示出了步骤202的示意图。预设卷积层的输 出通道数可以是2。第二卷积核组的数量可以与输出通道数相同，即第二卷积核组的数量为2。第二卷积核组一号可以对第二声谱特征图一号、第二声谱特征图二号和第二声谱特征图三号进行合并，得到第三声谱特征图一号。第二卷积核组二号可以对第二声谱特征图一号、第二声谱特征图二号和第二声谱特征图三号进行合并，得到第三声谱特征图二号。
在一些应用场景中，第二卷积核组中的第二卷积核可以为三维卷积核。第二卷积核的深度可以与第二声谱特征图的数量相同。
需要说明的是，本实施例提供的声音信号处理方法，通过将第一频谱数据采用包括至少一个预设卷积层的声音处理模型进行处理，得到处理结果，以及基于处理结果得到纯净音频数据，可以减少得到纯净音频数据所耗费的计算量，提高处理速度。
具体对比分析如下：如果卷积的步长为1，本申请的单个预设卷积层，乘法的计算次数为C1+C2。C1是步骤201的乘法计算量，即第一卷积核长度*第一卷积核宽度*频率维长度*时间维长度*输入通道数。C2是步骤201的乘法计算量，即输入通道数*频率维长度*时间维长度*输出通道数，可以理解，合并时候第二卷积核的尺寸通常是1*1*输入通道数。在相关技术中，通常情况下的卷积层，乘法的计算次数为C3，即输入通道数*频率维长度*时间维长度*第一卷积核长度*第一卷积核宽度*输出通道数。由此，可以得出，本申请提供的方式，大大减少了计算量，使得声音处理模型处理声音信号时所消耗的计算资源大大减少。
在一些实施例中，上述声音处理模型设置于终端设备。
需要说明的是，本申请一些实施例提供的声音信号处理方法，在降低了计算量的同时，保证了较好的处理准确率，即具有较好的噪声抑制效果。由于计算量较小，本申请中一些实施例中提供的方法和声音处理模型，适用于在终端设备实施。本申请一些实施例提供的声音处理模型实施在终端设备，可以及时对于采集到的声音进行处理，不仅可以提升用户声音体验，还可以减少在远端交互任务中的数据传输量。
在一些实施例中，上述第一卷积核组中的第一卷积核数量为至少两个。
在一些实施例中，上述步骤201，可以包括：根据第一对应关系，采用第一卷积核组中的第一卷积核对第一声谱特征图进行卷积操作，得到第二声谱特征图。
在这里，第一对应关系可以指示第一卷积核与第一声谱特征图的频率之间的对应关系。作为示例，请参考图6，在第一声谱特征图一号的频率维上，可以每隔一个频率设置一个第一卷积核。具体的，可以设置第一卷积核a、第一卷积核b、第一卷积核c、第一卷积核d和第一卷积核e。
可以理解，第一卷积核组中卷积核的数量可以根据实际应用场景设置，在此不做限定。
在本实施例中，第一卷积核组中的各个第一卷积核，可以是大小相同、权重不同的卷积核。各个第一卷积核的权重可以是在对声音处理模型的训练过程中，调整学习到的数值。
需要说明的是，通过设置包括至少两个第一卷积核的第一卷积核组，对输出的不同频率维学习一个不同的卷积核，这样使得网络参数量增大，并且没有增加计算量。由此，可以在保证处理效率的同时，提高声音处理模型的处理准确率。
在一些实施例中，第二卷积核组中的第二卷积核数量为至少两个。
在一些实施例中，上述步骤204可以包括：根据第二对应关系，采用第二卷积核组中的第二卷积核合并得到的第二声谱特征图，得到与第二卷积核组对应的第三声谱特征图。
在这里，所述第二对应关系用于指示第二卷积核与第二声谱特征图的频率之间的对应关系。作为示例，请参考图7A和图7B。
图7A中示出了频率维中第一个频率对应的第二卷积核f。第二卷积核f可以对第二声谱特征图一号、第二声谱特征图二号、第二声谱特征图三号的相同位置(即第一行第一列)中的值进行合并(例如取加权和)，得到第三声谱特征图一号中对应位置(即第 一行第一列)中的值。
图7B中示出了频率维中第一个频率对应的第二卷积核g。第二卷积核g可以对第二声谱特征图一号、第二声谱特征图二号、第二声谱特征图三号的相同位置(即第一行最后一列)中的值进行合并(例如取加权和)，得到第三声谱特征图一号中对应位置(即第一行最后一列)中的值。
可以理解，第二卷积组一号中可以包括第二卷积核f和第二卷积核g，还可以包括与第二声谱特征图的频率维的其它频率对应的第二卷积核。
需要说明的是，通过设置包括至少两个第二卷积核的第二卷积核组，可以对不同的频率学习不同的卷积核，增大网络参数量，并且没有增加计算量。由此，可以在保证处理效率的同时，提高声音处理模型的处理准确率。
在一些实施例中，第一卷积核组的卷积核数量根据第一声谱特征图频率维的长度和步长确定。
在这里，步长可以用于表征进行卷积操作的稀疏程度。作为示例，请参考图6，图6中频率维长度为10，步长为2，卷积核数量为5。如果图6中的步长改为1，则卷积核数量可以是10。
在一些实施例中，第一卷积核组的卷积核数量与频率为长度相同。
需要说明的是，设置步长作为调整卷积核数量的调整依据，可以减少计算次数，提高处理效率。
在一些实施例中，第一卷积核的感受野基于采样位置和预设的位置偏移参数确定。
在这里，第一卷积核的感受野可以基于候选采样位置和预设的位置偏移参数确定。
作为示例，请参考图8A和图8B，图8A和图8B示出了感受野变化的示例性示意图。第一卷积核计算过程中，卷积核的候选采样位置通过图8A的阴影部分示出；如果设置的位置偏移参数指示将采样位置在候选采样位置的基础上改变，例如改变为图8B中 的阴影部分的位置，则卷积核最终的感受野为图8B中的阴影部分的位置。
需要说明的是，通过感受野的变化，不改变参数数量和运算成本，却可以观察大的感受野。由此，可以保证处理效率的同时，提高处理的准确率。
在一些实施例中，所述声音处理模型包括至少一个自注意力层，所述自注意力层设置在所述至少一个预设卷积层之后。
在这里，在所述自注意力层中执行的操作包括：对于预设卷积层输出的每个声谱特征图，根据该声谱特征图中每个位置的取值与该声谱特征图中其它位置的取值，对该位置进行重新取值。
需要说明的是，在说明了自注意力层是对于声谱特征图每个位置的取值进行重新取值的情况下，自注意力层的具体实现方式可以根据实际应用场景设置，在此不做限定。
需要说明的是，通过设置自注意力层(Self-Attention)，可以使得处理结果，尤其是包括掩蔽数据的处理结果更为准确。
在一些实施例中，上述声音处理模型包括掩蔽数据。掩蔽(mask)数据，也可以称为掩膜数据，用于将目标信号从混合信号中抽离。作为示例，语音信号与背景噪声混合的混合信号中，采用掩蔽信号对混合信号进行处理，可以从混合信号中抽离出来语音信号。
通常，可以将掩蔽数据与混合信号对应的声谱图对位相乘，得到纯净的语音数据对应的声谱图。
在一些实施例中，上述步骤102，可以包括根据所述掩蔽数据和所述第一频谱数据，生成第二频谱数据；将第二频谱数据转换为时域数据，得到所述纯净音频数据。
在一些应用场景中，可以将第一频谱数据与掩蔽数据的乘积，作为第二频谱数据。
在一些实施例中，输出包括掩蔽数据的声音处理模型，可以通过以下方式训练：获取混合音频样本；将混合音频样本导入未训练完成的声音处理模型，生成候选掩蔽数据；根据所述混合音 频样本的标签和所述候选掩蔽数据，生成第一损失值；基于所述第一损失值，调整未训练完成的声音处理模型中的参数；
在这里，所述训练样本的标签通过如下方式生成：对纯净音频样本和混合音频样本分别进行时频变换，根据变换得到数据生成训练用掩蔽数据，以及将训练用掩蔽数据确定为标签。
作为示例，可以将纯净音频样本对应的频域数据，与混合音频样本对应的频域数据，取比值，以及将比值确定为上述训练用掩蔽数据。
在一些应用场景中，可以设置纯净音频样本集合和噪声样本集合。可以采用各种方式从纯净音频样本集合中选取纯净音频样本，也可以采用各种方式从噪声样本集合中选取噪声样本。然后将所选取的纯净音频样本和所选取的噪声样本进行君合，得到混合音频样本。
需要说明的是，基于中间处理结果训练得到的声音处理模型，相对具有较高的处理准确率。由此，采用掩蔽数据作为中间处理结果的处理方式，可以提高对于声音信号处理的准确率。
在一些实施例中，所述处理结果可以包括纯净频谱数据。纯净频谱数据可以是纯净音频数据对应的频域数据。
在一些实施例中，上述步骤102，可以包括：将纯净频谱数据转换为时域数据，得到纯净音频数据。
在一些实施例中，输出包括纯净音频数据的声音处理模型，可以通过如下方式训练：获取混合音频样本；将混合音频样本导入未训练完成的声音处理模型，生成候选纯净频谱数据；根据纯净频谱样本和所述候选纯净频谱数据，生成第二损失；基于第二损失值，未训练完成的声音处理模型中的参数。
在这里，混合音频样本的标签包括纯净音频样本对应的纯净频谱样本。作为示例，对纯净音频样本进行时频变换，可以得到纯净频谱数据。
进一步参考图9，作为对上述各图所示方法的实现，本公开提 供了一种声音信号处理装置的一个实施例，该装置实施例与图1所示的方法实施例相对应，该装置具体可以应用于各种电子设备中。
如图9所示，本实施例的声音信号处理装置包括：第一生成单元901和第二生成单元902。其中，第一生成单元，用于将第一音频数据对应的第一频谱数据，导入预先训练的声音处理模型，得到处理结果；第二生成单元，用于基于所述处理结果，生成所述第一音频数据对应的纯净音频数据；其中，所述声音处理模型包括至少一个预设卷积层，在所述预设卷积层执行的操作包括：基于第一卷积核组，对输入预设卷积层的对应第一声谱特征图进行卷积操作，得到第二声谱特征图，其中，第一卷积核组的数量与输入至所述预设卷积层的第一声谱特征图的数量匹配；基于第二卷积核组，对得到的第二声谱特征图进行合并，得到与第二卷积核组对应的第三声谱特征图，其中，第二卷积核组的数量与输出通道数量匹配。
在本实施例中，声音信号处理装置的第一生成单元901和第二生成单元902的具体处理及其所带来的技术效果可分别参考图1对应实施例中步骤101和步骤102的相关说明，在此不再赘述。
在一些实施例中，第一卷积核组中的第一卷积核数量为至少两个；以及所述基于第一卷积核组，对输入预设卷积层的对应第一声谱特征图进行卷积操作，得到第二声谱特征图，包括：根据第一对应关系，采用第一卷积核组中的第一卷积核对第一声谱特征图进行卷积操作，得到第二声谱特征图，其中，第一对应关系用于指示第一卷积核与第一声谱特征图的频率之间的对应关系。
在一些实施例中，第二卷积核组中的第二卷积核数量为至少两个；以及所述基于第二卷积核组，对得到的第二声谱特征图进行合并，得到与第二卷积核组对应的第三声谱特征图，包括：根据第二对应关系，采用第二卷积核组中的第二卷积核合并得到的第二声谱特征图，得到与第二卷积核组对应的第三声谱特征图，其中，所述第二对应关系用于指示第二卷积核与第二声谱特征图 的频率之间的对应关系。
在一些实施例中，第一卷积核组的卷积核数量根据第一声谱特征图频率维的长度和第一步长确定。
在一些实施例中，第一卷积核的感受野基于候选采样位置和预设的位置偏移参数确定。
在一些实施例中，所述声音处理模型包括至少一个自注意力层，所述自注意力层设置在所述至少一个预设卷积层之后；其中，在所述自注意力层中执行的操作包括：对于预设卷积层输出的每个声谱特征图，根据该声谱特征图中每个位置的取值与该声谱特征图中其它位置的取值，对该位置进行重新取值。
在一些实施例中，应用于终端设备，所述声音处理模型设置于所述终端设备。
在一些实施例中，所述处理结果包括掩蔽数据；以及所述基于所述处理结果，生成所述第一音频数据对应的纯净音频数据，包括：根据所述掩蔽数据和所述第一频谱数据，生成第二频谱数据；将第二频谱数据转换为时域数据，得到所述纯净音频数据。
在一些实施例中，所述声音处理模型通过以下方式训练：获取混合音频样本；将混合音频样本导入未训练完成的声音处理模型，生成候选掩蔽数据；根据所述混合音频样本的标签和所述候选掩蔽数据，生成第一损失值；基于所述第一损失值，调整未训练完成的声音处理模型中的参数；其中所述训练样本的标签通过如下方式生成：对纯净音频样本和混合音频样本分别进行时频变换，根据变换得到数据生成训练用掩蔽数据，以及将训练用掩蔽数据确定为标签。
在一些实施例中，所述处理结果包括纯净频谱数据；以及所述基于所述处理结果，生成所述第一音频数据对应的纯净音频数据，包括：将纯净频谱数据转换为时域数据，得到所述纯净音频数据。
在一些实施例中，所述声音处理模型通过以下方式训练：获取混合音频样本，其中，所述混合音频样本的标签包括纯净音频 样本对应的纯净频谱样本；将混合音频样本导入未训练完成的声音处理模型，生成候选纯净频谱数据；根据纯净频谱样本和所述候选纯净频谱数据，生成第二损失值；基于所述第二损失值，调整未训练完成的声音处理模型中的参数。
请参考图10，图10示出了本公开的一个实施例的声音信号处理方法可以应用于其中的示例性系统架构。
如图10所示，系统架构可以包括终端设备1001、1002、1003，网络1004，服务器1005。网络1004用以在终端设备1001、1002、1003和服务器1005之间提供通信链路的介质。网络1004可以包括各种连接类型，例如有线、无线通信链路或者光纤电缆等等。
终端设备1001、1002、1003可以通过网络1004与服务器1005交互，以接收或发送消息等。终端设备1001、1002、1003上可以安装有各种客户端应用，例如网页浏览器应用、搜索类应用、新闻资讯类应用。终端设备1001、1002、1003中的客户端应用可以接收用户的指令，并根据用户的指令完成相应的功能，例如根据用户的指令在信息中添加相应信息。
终端设备1001、1002、1003可以是硬件，也可以是软件。当终端设备1001、1002、1003为硬件时，可以是具有显示屏并且支持网页浏览的各种电子设备，包括但不限于智能手机、平板电脑、电子书阅读器、MP3播放器(Moving Picture Experts Group Audio Layer III，动态影像专家压缩标准音频层面3)、MP4(Moving Picture Experts Group Audio Layer IV，动态影像专家压缩标准音频层面4)播放器、膝上型便携计算机和台式计算机等等。当终端设备1001、1002、1003为软件时，可以安装在上述所列举的电子设备中。其可以实现成多个软件或软件模块(例如用来提供分布式服务的软件或软件模块)，也可以实现成单个软件或软件模块。在此不做具体限定。
服务器1005可以是提供各种服务的服务器，例如接收终端设备1001、1002、1003发送的信息获取请求，根据信息获取请求通 过各种方式获取信息获取请求对应的展示信息。并展示信息的相关数据发送给终端设备1001、1002、1003。
需要说明的是，本公开实施例所提供的声音信号处理方法可以由终端设备执行，相应地，声音信号处理装置可以设置在终端设备1001、1002、1003中。此外，本公开实施例所提供的声音信号处理方法还可以由服务器1005执行，相应地，声音信号处理装置可以设置于服务器1005中。
应该理解，图10中的终端设备、网络和服务器的数目仅仅是示意性的。根据实现需要，可以具有任意数目的终端设备、网络和服务器。
下面参考图11，其示出了适于用来实现本公开实施例的电子设备(例如图10中的终端设备或服务器)的结构示意图。本公开实施例中的终端设备可以包括但不限于诸如移动电话、笔记本电脑、数字广播接收器、PDA(个人数字助理)、PAD(平板电脑)、PMP(便携式多媒体播放器)、车载终端(例如车载导航终端)等等的移动终端以及诸如数字TV、台式计算机等等的固定终端。图11示出的电子设备仅仅是一个示例，不应对本公开实施例的功能和使用范围带来任何限制。
如图11所示，电子设备可以包括处理装置(例如中央处理器、图形处理器等)1101，其可以根据存储在只读存储器(ROM)1102中的程序或者从存储装置1108加载到随机访问存储器(RAM)1103中的程序而执行各种适当的动作和处理。在RAM 1103中，还存储有电子设备1100操作所需的各种程序和数据。处理装置1101、ROM 1102以及RAM 1103通过总线1104彼此相连。输入/输出(I/O)接口1105也连接至总线1104。
通常，以下装置可以连接至I/O接口1105：包括例如触摸屏、触摸板、键盘、鼠标、摄像头、麦克风、加速度计、陀螺仪等的输入装置1106；包括例如液晶显示器(LCD)、扬声器、振动器等的输出装置1107；包括例如磁带、硬盘等的存储装置1108；以及 通信装置1109。通信装置1109可以允许电子设备与其他设备进行无线或有线通信以交换数据。虽然图11示出了具有各种装置的电子设备，但是应理解的是，并不要求实施或具备所有示出的装置。可以替代地实施或具备更多或更少的装置。
特别地，根据本公开的实施例，上文参考流程图描述的过程可以被实现为计算机软件程序。例如，本公开的实施例包括一种计算机程序产品，其包括承载在非暂态计算机可读介质上的计算机程序，该计算机程序包含用于执行流程图所示的方法的程序代码。在这样的实施例中，该计算机程序可以通过通信装置1109从网络上被下载和安装，或者从存储装置1108被安装，或者从ROM1102被安装。在该计算机程序被处理装置1101执行时，执行本公开实施例的方法中限定的上述功能。
需要说明的是，本公开上述的计算机可读介质可以是计算机可读信号介质或者计算机可读存储介质或者是上述两者的任意组合。计算机可读存储介质例如可以是——但不限于——电、磁、光、电磁、红外线、或半导体的系统、装置或器件，或者任意以上的组合。计算机可读存储介质的更具体的例子可以包括但不限于：具有一个或多个导线的电连接、便携式计算机磁盘、硬盘、随机访问存储器(RAM)、只读存储器(ROM)、可擦式可编程只读存储器(EPROM或闪存)、光纤、便携式紧凑磁盘只读存储器(CD-ROM)、光存储器件、磁存储器件、或者上述的任意合适的组合。在本公开中，计算机可读存储介质可以是任何包含或存储程序的有形介质，该程序可以被指令执行系统、装置或者器件使用或者与其结合使用。而在本公开中，计算机可读信号介质可以包括在基带中或者作为载波一部分传播的数据信号，其中承载了计算机可读的程序代码。这种传播的数据信号可以采用多种形式，包括但不限于电磁信号、光信号或上述的任意合适的组合。计算机可读信号介质还可以是计算机可读存储介质以外的任何计算机可读介质，该计算机可读信号介质可以发送、传播或者传输用于由指令执行系统、装置或者器件使用或者与其结合使用的程序。 计算机可读介质上包含的程序代码可以用任何适当的介质传输，包括但不限于：电线、光缆、RF(射频)等等，或者上述的任意合适的组合。
在一些实施方式中，客户端、服务器可以利用诸如HTTP(HyperText Transfer Protocol，超文本传输协议)之类的任何当前已知或未来研发的网络协议进行通信，并且可以与任意形式或介质的数字数据通信(例如，通信网络)互连。通信网络的示例包括局域网(“LAN”)，广域网(“WAN”)，网际网(例如，互联网)以及端对端网络(例如，ad hoc端对端网络)，以及任何当前已知或未来研发的网络。
上述计算机可读介质可以是上述电子设备中所包含的；也可以是单独存在，而未装配入该电子设备中。
上述计算机可读介质承载有一个或者多个程序，当上述一个或者多个程序被该电子设备执行时，使得该电子设备：将第一音频数据对应的第一频谱数据，导入预先训练的声音处理模型，得到处理结果；基于所述处理结果，生成所述第一音频数据对应的纯净音频数据；其中，所述声音处理模型包括至少一个预设卷积层，在所述预设卷积层执行的操作包括：基于第一卷积核组，对输入预设卷积层的对应第一声谱特征图进行卷积操作，得到第二声谱特征图，其中，第一卷积核组的数量与输入至所述预设卷积层的第一声谱特征图的数量匹配；基于第二卷积核组，对得到的第二声谱特征图进行合并，得到与第二卷积核组对应的第三声谱特征图，其中，第二卷积核组的数量与输出通道数量匹配。
可以以一种或多种程序设计语言或其组合来编写用于执行本公开的操作的计算机程序代码，上述程序设计语言包括但不限于面向对象的程序设计语言—诸如Java、Smalltalk、C++，还包括常规的过程式程序设计语言—诸如“C”语言或类似的程序设计语言。程序代码可以完全地在用户计算机上执行、部分地在用户计算机上执行、作为一个独立的软件包执行、部分在用户计算机上部分在远程计算机上执行、或者完全在远程计算机或服务器上执行。 在涉及远程计算机的情形中，远程计算机可以通过任意种类的网络——包括局域网(LAN)或广域网(WAN)—连接到用户计算机，或者，可以连接到外部计算机(例如利用因特网服务提供商来通过因特网连接)。
附图中的流程图和框图，图示了按照本公开各种实施例的系统、方法和计算机程序产品的可能实现的体系架构、功能和操作。在这点上，流程图或框图中的每个方框可以代表一个模块、程序段、或代码的一部分，该模块、程序段、或代码的一部分包含一个或多个用于实现规定的逻辑功能的可执行指令。也应当注意，在有些作为替换的实现中，方框中所标注的功能也可以以不同于附图中所标注的顺序发生。例如，两个接连地表示的方框实际上可以基本并行地执行，它们有时也可以按相反的顺序执行，这依所涉及的功能而定。也要注意的是，框图和/或流程图中的每个方框、以及框图和/或流程图中的方框的组合，可以用执行规定的功能或操作的专用的基于硬件的系统来实现，或者可以用专用硬件与计算机指令的组合来实现。
描述于本公开实施例中所涉及到的单元可以通过软件的方式实现，也可以通过硬件的方式来实现。其中，单元的名称在某种情况下并不构成对该单元本身的限定，例如，第一生成单元还可以被描述为“生成处理结果的单元”。
本文中以上描述的功能可以至少部分地由一个或多个硬件逻辑部件来执行。例如，非限制性地，可以使用的示范类型的硬件逻辑部件包括：现场可编程门阵列(FPGA)、专用集成电路(ASIC)、专用标准产品(ASSP)、片上系统(SOC)、复杂可编程逻辑设备(CPLD)等等。
在本公开的上下文中，机器可读介质可以是有形的介质，其可以包含或存储以供指令执行系统、装置或设备使用或与指令执行系统、装置或设备结合地使用的程序。机器可读介质可以是机器可读信号介质或机器可读储存介质。机器可读介质可以包括但不限于电子的、磁性的、光学的、电磁的、红外的、或半导体系 统、装置或设备，或者上述内容的任何合适组合。机器可读存储介质的更具体示例会包括基于一个或多个线的电气连接、便携式计算机盘、硬盘、随机存取存储器(RAM)、只读存储器(ROM)、可擦除可编程只读存储器(EPROM或快闪存储器)、光纤、便捷式紧凑盘只读存储器(CD-ROM)、光学储存设备、磁储存设备、或上述内容的任何合适组合。
根据本公开的一个或多个实施例，所述第一卷积核组的数量与输入至所述预设卷积层的第一声谱特征图的数量匹配，所述第二卷积核组的数量与输出通道数量匹配。
根据本公开的一个或多个实施例，第一卷积核组中的第一卷积核数量为至少两个；以及所述基于第一卷积核组，对输入预设卷积层的对应第一声谱特征图进行卷积操作，得到第二声谱特征图，包括：根据第一对应关系，采用第一卷积核组中的第一卷积核对第一声谱特征图进行卷积操作，得到第二声谱特征图，其中，第一对应关系用于指示第一卷积核与第一声谱特征图的频率之间的对应关系。
根据本公开的一个或多个实施例，第二卷积核组中的第二卷积核数量为至少两个；以及所述基于第二卷积核组，对得到的第二声谱特征图进行合并，得到与第二卷积核组对应的第三声谱特征图，包括：根据第二对应关系，采用第二卷积核组中的第二卷积核合并得到的第二声谱特征图，得到与第二卷积核组对应的第三声谱特征图，其中，所述第二对应关系用于指示第二卷积核与第二声谱特征图的频率之间的对应关系。
根据本公开的一个或多个实施例，第一卷积核组的卷积核数量根据第一声谱特征图频率维的长度和第一步长确定。
根据本公开的一个或多个实施例，第一卷积核的感受野基于候选采样位置和预设的位置偏移参数确定。
根据本公开的一个或多个实施例，所述声音处理模型包括至少一个自注意力层，所述自注意力层设置在所述至少一个预设卷积层之后；其中，在所述自注意力层中执行的操作包括：对于预 设卷积层输出的每个声谱特征图，根据该声谱特征图中每个位置的取值与该声谱特征图中其它位置的取值，对该位置进行重新取值。
根据本公开的一个或多个实施例，应用于终端设备，所述声音处理模型设置于所述终端设备。
根据本公开的一个或多个实施例，所述处理结果包括掩蔽数据；以及所述基于所述处理结果，生成所述第一音频数据对应的纯净音频数据，包括：根据所述掩蔽数据和所述第一频谱数据，生成第二频谱数据；将第二频谱数据转换为时域数据，得到所述纯净音频数据。
根据本公开的一个或多个实施例，所述声音处理模型通过以下方式训练：获取混合音频样本；将混合音频样本导入未训练完成的声音处理模型，生成候选掩蔽数据；根据所述混合音频样本的标签和所述候选掩蔽数据，生成第一损失值；基于所述第一损失值，调整未训练完成的声音处理模型中的参数；其中所述训练样本的标签通过如下方式生成：对纯净音频样本和混合音频样本分别进行时频变换，根据变换得到数据生成训练用掩蔽数据，以及将训练用掩蔽数据确定为标签。
根据本公开的一个或多个实施例，所述处理结果包括纯净频谱数据；以及所述基于所述处理结果，生成所述第一音频数据对应的纯净音频数据，包括：将纯净频谱数据转换为时域数据，得到所述纯净音频数据。
根据本公开的一个或多个实施例，所述声音处理模型通过以下方式训练：获取混合音频样本，其中，所述混合音频样本的标签包括纯净音频样本对应的纯净频谱样本；将混合音频样本导入未训练完成的声音处理模型，生成候选纯净频谱数据；根据纯净频谱样本和所述候选纯净频谱数据，生成第二损失值；基于所述第二损失值，调整未训练完成的声音处理模型中的参数。
根据本公开的一个或多个实施例，一种声音信号处理装置，包括：第一生成单元，用于将第一音频数据对应的第一频谱数据， 导入预先训练的声音处理模型，得到处理结果；第二生成单元，用于基于所述处理结果，生成所述第一音频数据对应的纯净音频数据；其中，所述声音处理模型包括至少一个预设卷积层，在所述预设卷积层执行的操作包括：基于第一卷积核组，对输入预设卷积层的对应第一声谱特征图进行卷积操作，得到第二声谱特征图；基于第二卷积核组，对得到的第二声谱特征图进行合并，得到与第二卷积核组对应的第三声谱特征图。
根据本公开的一个或多个实施例，第一卷积核组中的第一卷积核数量为至少两个；以及所述基于第一卷积核组，对输入预设卷积层的对应第一声谱特征图进行卷积操作，得到第二声谱特征图，包括：根据第一对应关系，采用第一卷积核组中的第一卷积核对第一声谱特征图进行卷积操作，得到第二声谱特征图，其中，第一对应关系用于指示第一卷积核与第一声谱特征图的频率之间的对应关系。
根据本公开的一个或多个实施例，第二卷积核组中的第二卷积核数量为至少两个；以及所述基于第二卷积核组，对得到的第二声谱特征图进行合并，得到与第二卷积核组对应的第三声谱特征图，包括：根据第二对应关系，采用第二卷积核组中的第二卷积核合并得到的第二声谱特征图，得到与第二卷积核组对应的第三声谱特征图，其中，所述第二对应关系用于指示第二卷积核与第二声谱特征图的频率之间的对应关系。
根据本公开的一个或多个实施例，第一卷积核组的卷积核数量根据第一声谱特征图频率维的长度和第一步长确定。
根据本公开的一个或多个实施例，第一卷积核的感受野基于候选采样位置和预设的位置偏移参数确定。
根据本公开的一个或多个实施例，所述声音处理模型包括至少一个自注意力层，所述自注意力层设置在所述至少一个预设卷积层之后；其中，在所述自注意力层中执行的操作包括：对于预设卷积层输出的每个声谱特征图，根据该声谱特征图中每个位置的取值与该声谱特征图中其它位置的取值，对该位置进行重新取 值。
根据本公开的一个或多个实施例，应用于终端设备，所述声音处理模型设置于所述终端设备。
根据本公开的一个或多个实施例，所述处理结果包括掩蔽数据；以及所述基于所述处理结果，生成所述第一音频数据对应的纯净音频数据，包括：根据所述掩蔽数据和所述第一频谱数据，生成第二频谱数据；将第二频谱数据转换为时域数据，得到所述纯净音频数据。
根据本公开的一个或多个实施例，所述声音处理模型通过以下方式训练：获取混合音频样本；将混合音频样本导入未训练完成的声音处理模型，生成候选掩蔽数据；根据所述混合音频样本的标签和所述候选掩蔽数据，生成第一损失值；基于所述第一损失值，调整未训练完成的声音处理模型中的参数；其中所述训练样本的标签通过如下方式生成：对纯净音频样本和混合音频样本分别进行时频变换，根据变换得到数据生成训练用掩蔽数据，以及将训练用掩蔽数据确定为标签。
根据本公开的一个或多个实施例，所述处理结果包括纯净频谱数据；以及所述基于所述处理结果，生成所述第一音频数据对应的纯净音频数据，包括：将纯净频谱数据转换为时域数据，得到所述纯净音频数据。
根据本公开的一个或多个实施例，所述声音处理模型通过以下方式训练：获取混合音频样本，其中，所述混合音频样本的标签包括纯净音频样本对应的纯净频谱样本；将混合音频样本导入未训练完成的声音处理模型，生成候选纯净频谱数据；根据纯净频谱样本和所述候选纯净频谱数据，生成第二损失值；基于所述第二损失值，调整未训练完成的声音处理模型中的参数。
根据本公开的一个或多个实施例，电子设备，包括：一个或多个处理器；存储装置，用于存储一个或多个程序，当所述一个或多个程序被所述一个或多个处理器执行，使得所述一个或多个处理器实现本申请中任一所述的方法。
根据本公开的一个或多个实施例，计算机可读介质，其上存储有计算机程序，，该程序被处理器执行时实现如本申请中任一所述的方法。
以上描述仅为本公开的较佳实施例以及对所运用技术原理的说明。本领域技术人员应当理解，本公开中所涉及的公开范围，并不限于上述技术特征的特定组合而成的技术方案，同时也应涵盖在不脱离上述公开构思的情况下，由上述技术特征或其等同特征进行任意组合而形成的其它技术方案。例如上述特征与本公开中公开的(但不限于)具有类似功能的技术特征进行互相替换而形成的技术方案。
此外，虽然采用特定次序描绘了各操作，但是这不应当理解为要求这些操作以所示出的特定次序或以顺序次序执行来执行。在一定环境下，多任务和并行处理可能是有利的。同样地，虽然在上面论述中包含了若干具体实现细节，但是这些不应当被解释为对本公开的范围的限制。在单独的实施例的上下文中描述的某些特征还可以组合地实现在单个实施例中。相反地，在单个实施例的上下文中描述的各种特征也可以单独地或以任何合适的子组合的方式实现在多个实施例中。
尽管已经采用特定于结构特征和/或方法逻辑动作的语言描述了本主题，但是应当理解所附权利要求书中所限定的主题未必局限于上面描述的特定特征或动作。相反，上面所描述的特定特征和动作仅仅是实现权利要求书的示例形式。

Claims (15)

  1. 一种声音信号处理方法，其特征在于，包括：
    将第一音频数据对应的第一频谱数据，导入预先训练的声音处理模型，得到处理结果；
    基于所述处理结果，生成所述第一音频数据对应的纯净音频数据；其中，
    所述声音处理模型包括至少一个预设卷积层，在所述预设卷积层执行的操作包括：
    基于第一卷积核组，对输入预设卷积层的对应第一声谱特征图进行卷积操作，得到第二声谱特征图；
    基于第二卷积核组，对得到的第二声谱特征图进行合并，得到与第二卷积核组对应的第三声谱特征图。
  2. 根据权利要求1所述的方法，其特征在于，所述第一卷积核组的数量与输入至所述预设卷积层的第一声谱特征图的数量匹配，所述第二卷积核组的数量与输出通道数量匹配。
  3. 根据权利要求1所述的方法，其特征在于，第一卷积核组中的第一卷积核数量为至少两个；以及
    所述基于第一卷积核组，对输入预设卷积层的对应第一声谱特征图进行卷积操作，得到第二声谱特征图，包括：
    根据第一对应关系，采用第一卷积核组中的第一卷积核对第一声谱特征图进行卷积操作，得到第二声谱特征图，其中，第一对应关系用于指示第一卷积核与第一声谱特征图的频率之间的对应关系。
  4. 根据权利要求1所述的方法，其特征在于，第二卷积核组中的第二卷积核数量为至少两个；以及
    所述基于第二卷积核组，对得到的第二声谱特征图进行合并，得到与第二卷积核组对应的第三声谱特征图，包括：
    根据第二对应关系，采用第二卷积核组中的第二卷积核合并得到的第二声谱特征图，得到与第二卷积核组对应的第三声谱特征图，其中，所述第二对应关系用于指示第二卷积核与第二声谱特征图的频率之间的对应关系。
  5. 根据权利要求1所述的方法，其特征在于，第一卷积核组的卷积核数量根据第一声谱特征图频率维的长度和第一步长确定。
  6. 根据权利要求1所述的方法，其特征在于，第一卷积核的感受野基于候选采样位置和预设的位置偏移参数确定。
  7. 根据权利要求1所述的方法，其特征在于，所述声音处理模型包括至少一个自注意力层，所述自注意力层设置在所述至少一个预设卷积层之后；其中，
    在所述自注意力层中执行的操作包括：对于预设卷积层输出的每个声谱特征图，根据该声谱特征图中每个位置的取值与该声谱特征图中其它位置的取值，对该位置进行重新取值。
  8. 根据权利要求1所述的方法，其特征在于，应用于终端设备，所述声音处理模型设置于所述终端设备。
  9. 根据权利要求1-8中任一项所述的方法，其特征在于，所述处理结果包括掩蔽数据；以及
    所述基于所述处理结果，生成所述第一音频数据对应的纯净音频数据，包括：
    根据所述掩蔽数据和所述第一频谱数据，生成第二频谱数据；
    将第二频谱数据转换为时域数据，得到所述纯净音频数据。
  10. 根据权利要求9所述的方法，其特征在于，所述声音处理模型通过以下方式训练：
    获取混合音频样本；
    将混合音频样本导入未训练完成的声音处理模型，生成候选掩蔽数据；
    根据所述混合音频样本的标签和所述候选掩蔽数据，生成第一损失值；
    基于所述第一损失值，调整未训练完成的声音处理模型中的参数；其中
    所述训练样本的标签通过如下方式生成：对纯净音频样本和混合音频样本分别进行时频变换，根据变换得到数据生成训练用掩蔽数据，以及将训练用掩蔽数据确定为标签。
  11. 根据权利要求1-8中任一项所述的方法，其特征在于，所述处理结果包括纯净频谱数据；以及
    所述基于所述处理结果，生成所述第一音频数据对应的纯净音频数据，包括：
    将纯净频谱数据转换为时域数据，得到所述纯净音频数据。
  12. 根据权利要求11所述的方法，其特征在于，所述声音处理模型通过以下方式训练：
    获取混合音频样本，其中，所述混合音频样本的标签包括纯净音频样本对应的纯净频谱样本；
    将混合音频样本导入未训练完成的声音处理模型，生成候选纯净频谱数据；
    根据纯净频谱样本和所述候选纯净频谱数据，生成第二损失值；
    基于所述第二损失值，调整未训练完成的声音处理模型中的参数。
  13. 一种声音信号处理装置，其特征在于，包括：
    第一生成单元，用于将第一音频数据对应的第一频谱数据， 导入预先训练的声音处理模型，得到处理结果；
    第二生成单元，用于基于所述处理结果，生成所述第一音频数据对应的纯净音频数据；其中，
    所述声音处理模型包括至少一个预设卷积层，在所述预设卷积层执行的操作包括：
    基于第一卷积核组，对输入预设卷积层的对应第一声谱特征图进行卷积操作，得到第二声谱特征图；
    基于第二卷积核组，对得到的第二声谱特征图进行合并，得到与第二卷积核组对应的第三声谱特征图。
  14. 一种电子设备，其特征在于，包括：
    至少一个处理器；
    存储装置，用于存储至少一个程序，
    当所述至少一个程序被所述至少一个处理器执行，使得所述至少一个处理器实现如权利要求1-12中任一所述的方法。
  15. 一种计算机可读介质，其上存储有计算机程序，其特征在于，该程序被处理器执行时实现如权利要求1-12中任一所述的方法。
PCT/CN2021/135398 2020-12-08 2021-12-03 声音信号处理方法、装置和电子设备 WO2022121799A1 (zh)

Priority Applications (1)

Application Number Priority Date Filing Date Title
US18/256,285 US20240038252A1 (en) 2020-12-08 2021-12-03 Sound signal processing method and apparatus, and electronic device

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
CN202011462091.2A CN112634928B (zh) 2020-12-08 2020-12-08 声音信号处理方法、装置和电子设备
CN202011462091.2 2020-12-08

Publications (1)

Publication Number Publication Date
WO2022121799A1 true WO2022121799A1 (zh) 2022-06-16

Family

ID=75312383

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2021/135398 WO2022121799A1 (zh) 2020-12-08 2021-12-03 声音信号处理方法、装置和电子设备

Country Status (3)

Country Link
US (1) US20240038252A1 (zh)
CN (1) CN112634928B (zh)
WO (1) WO2022121799A1 (zh)

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN115810364A (zh) * 2023-02-07 2023-03-17 海纳科德（湖北）科技有限公司 混音环境中的端到端目标声信号提取方法及系统

Families Citing this family (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN112634928B (zh) * 2020-12-08 2023-09-29 北京有竹居网络技术有限公司 声音信号处理方法、装置和电子设备
CN113506581B (zh) * 2021-07-08 2024-04-05 京东科技控股股份有限公司 一种语音增强方法和装置
CN113938749B (zh) * 2021-11-30 2023-05-05 北京百度网讯科技有限公司 音频数据处理方法、装置、电子设备和存储介质
CN114171038B (zh) * 2021-12-10 2023-07-28 北京百度网讯科技有限公司 语音降噪方法、装置、设备及存储介质
CN116030793B (zh) * 2023-03-30 2023-06-16 北京建筑大学 方言识别系统及其训练方法

Citations (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20140288928A1 (en) * 2013-03-25 2014-09-25 Gerald Bradley PENN System and method for applying a convolutional neural network to speech recognition
CN109065030A (zh) * 2018-08-01 2018-12-21 上海大学 基于卷积神经网络的环境声音识别方法及系统
CN110796027A (zh) * 2019-10-10 2020-02-14 天津大学 一种基于紧密卷积的神经网络模型的声音场景识别方法
CN111460932A (zh) * 2020-03-17 2020-07-28 哈尔滨工程大学 基于自适应卷积的水声信号分类识别方法
CN111582454A (zh) * 2020-05-09 2020-08-25 北京百度网讯科技有限公司 生成神经网络模型的方法和装置
CN112634928A (zh) * 2020-12-08 2021-04-09 北京有竹居网络技术有限公司 声音信号处理方法、装置和电子设备

Family Cites Families (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN106710589B (zh) * 2016-12-28 2019-07-30 百度在线网络技术（北京）有限公司 基于人工智能的语音特征提取方法及装置
CN109308913A (zh) * 2018-08-02 2019-02-05 平安科技（深圳）有限公司 音乐质量评价方法、装置、计算机设备及存储介质

Patent Citations (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20140288928A1 (en) * 2013-03-25 2014-09-25 Gerald Bradley PENN System and method for applying a convolutional neural network to speech recognition
CN109065030A (zh) * 2018-08-01 2018-12-21 上海大学 基于卷积神经网络的环境声音识别方法及系统
CN110796027A (zh) * 2019-10-10 2020-02-14 天津大学 一种基于紧密卷积的神经网络模型的声音场景识别方法
CN111460932A (zh) * 2020-03-17 2020-07-28 哈尔滨工程大学 基于自适应卷积的水声信号分类识别方法
CN111582454A (zh) * 2020-05-09 2020-08-25 北京百度网讯科技有限公司 生成神经网络模型的方法和装置
CN112634928A (zh) * 2020-12-08 2021-04-09 北京有竹居网络技术有限公司 声音信号处理方法、装置和电子设备

Cited By (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN115810364A (zh) * 2023-02-07 2023-03-17 海纳科德（湖北）科技有限公司 混音环境中的端到端目标声信号提取方法及系统
CN115810364B (zh) * 2023-02-07 2023-04-28 海纳科德（湖北）科技有限公司 混音环境中的端到端目标声信号提取方法及系统

Also Published As

Publication number Publication date
US20240038252A1 (en) 2024-02-01
CN112634928A (zh) 2021-04-09
CN112634928B (zh) 2023-09-29

Similar Documents

Publication Publication Date Title
WO2022121799A1 (zh) 声音信号处理方法、装置和电子设备
WO2022105545A1 (zh) 语音合成方法、装置、可读介质及电子设备
CN111583903B (zh) 语音合成方法、声码器训练方法、装置、介质及电子设备
WO2020207174A1 (zh) 用于生成量化神经网络的方法和装置
WO2023273985A1 (zh) 一种语音识别模型的训练方法、装置及设备
CN111597825B (zh) 语音翻译方法、装置、可读介质及电子设备
WO2022121801A1 (zh) 信息处理方法、装置和电子设备
CN110516159B (zh) 一种信息推荐方法、装置、电子设备及存储介质
WO2022135130A1 (zh) 语音提取方法、装置和电子设备
CN112259116B (zh) 一种音频数据的降噪方法、装置、电子设备及存储介质
CN112364860A (zh) 字符识别模型的训练方法、装置和电子设备
CN111462728A (zh) 用于生成语音的方法、装置、电子设备和计算机可读介质
CN111724807A (zh) 音频分离方法、装置、电子设备及计算机可读存储介质
US20230298611A1 (en) Speech enhancement
WO2023005729A1 (zh) 语音信息处理方法、装置和电子设备
CN111008213A (zh) 用于生成语言转换模型的方法和装置
CN114420135A (zh) 基于注意力机制的声纹识别方法及装置
CN112257459B (zh) 语言翻译模型的训练方法、翻译方法、装置和电子设备
CN112562633A (zh) 一种歌唱合成方法、装置、电子设备及存储介质
WO2023011397A1 (zh) 一种生成声学特征、语音模型训练、语音识别方法及装置
WO2022227932A1 (zh) 声音信号处理方法、装置和电子设备
WO2022134968A1 (zh) 模型的训练方法、语音识别方法、装置、介质及设备
CN110598049A (zh) 用于检索视频的方法、装置、电子设备和计算机可读介质
WO2022121800A1 (zh) 声源定位方法、装置和电子设备
CN113706663B (zh) 图像生成方法、装置、设备及存储介质

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 21902506

Country of ref document: EP

Kind code of ref document: A1

WWE Wipo information: entry into national phase

Ref document number: 18256285

Country of ref document: US

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 21902506

Country of ref document: EP

Kind code of ref document: A1