JP7465992B2 - Audio data processing method, device, equipment, storage medium, and program - Google Patents

Description

This application claims priority to a Chinese patent application filed on August 24, 2021, bearing application number 202110978065.3, the entire contents of which are incorporated herein by reference.

This disclosure relates to the field of audio processing technology, and in particular to the field of voice synthesis technology.

As an audio filter, electric acoustic effects are used to adjust and beautify audio, and are widely used in scenes such as KTV works and short video works. Good electric acoustic effects can improve the audio quality of the works. If good electric acoustic effects can be provided for application products, it can improve the competitiveness of the products, enrich the play of the products, and increase the fun for users.

The present disclosure provides an audio data processing method, device, apparatus, storage medium, and program .

According to one aspect of the present disclosure, there is provided an audio data processing method, comprising: decomposing original audio data to obtain voice audio data and background audio data; performing electroacoustic processing on the voice audio data to obtain electroacoustic voice data; and synthesizing the electroacoustic voice data and the background audio data to obtain target audio data.

According to another aspect of the present disclosure, an audio data processing device is provided, comprising: a decomposition module for decomposing original audio data to obtain voice audio data and background audio data; an electroacoustic processing module for performing electroacoustic processing on the voice audio data to obtain electroacoustic voice data; and a synthesis module for synthesizing the electroacoustic voice data and the background audio data to obtain target audio data.

Another aspect of the present disclosure provides an electronic device, comprising at least one processor and a memory communicatively coupled to the at least one processor, the memory storing instructions executable by the at least one processor, the instructions being executed by the at least one processor such that the at least one processor can perform a method as illustrated in an embodiment of the present disclosure.

According to another aspect of an embodiment of the present disclosure, a non-transitory computer-readable storage medium is provided having computer instructions stored thereon, the computer instructions being used to cause the computer to perform a method as set forth in an embodiment of the present disclosure.

According to another aspect of an embodiment of the present disclosure, a computer program is provided, comprising computer programs/instructions which, when executed by a processor, implement the methods illustrated in the embodiments of the present disclosure.

It should be understood that the contents described in this section are not intended to identify key or important features of the embodiments of the present disclosure, and are not intended to limit the scope of the present disclosure. Other features of the present disclosure will be readily understood from the following description.

The drawings are used to provide a better understanding of the present solution and are not intended to limit the present disclosure.

FIG. 1 shows a schematic flow chart of an audio data processing method according to an embodiment of the present disclosure. FIG. 2 illustrates a schematic flow chart of a method for decomposing original audio data according to an embodiment of the present disclosure. FIG. 3 shows a schematic flow chart of a method for performing electrosonification processing on voice audio data according to an embodiment of the present disclosure. FIG. 4 illustrates a schematic flow chart of an audio data processing method according to another embodiment of the present disclosure. FIG. 5 shows a schematic block diagram of an audio data processing apparatus according to an embodiment of the present disclosure. FIG. 6 illustrates generally a block diagram of an exemplary electronic device for implementing an embodiment of the present invention.

Hereinafter, exemplary embodiments of the present disclosure will be described with reference to the drawings, which include various details of the embodiments of the present disclosure for ease of understanding and should be considered as illustrative. Therefore, as can be understood by those skilled in the art, various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. Similarly, for clarity and simplicity, descriptions of known functions and structures are omitted in the following description.

The audio data processing method of the embodiment of the present disclosure will be described below with reference to FIG. 1. It should be noted that in the technical solution of the present disclosure, the collection, storage, use, processing, transmission, provision, disclosure, etc. of related data such as audio data all comply with the provisions of the relevant law rules and are not contrary to public order and morals.

Figure 1 is a flowchart of an audio data processing method according to an embodiment of the present disclosure.

As shown in FIG. 1, this audio data processing method 100 includes the following:

In operation S110, the original audio data is decomposed to obtain voice audio data and background audio data.

In operation S120, electroacoustic processing is performed on the human voice audio data to obtain electroacoustic human voice data.

In operation S130, the electroacoustic voice data and the background audio data are synthesized to obtain the target audio data.

According to an embodiment of the present disclosure, the original audio data may include, for example, human voice information and background audio information, where the human voice may be, for example, a singing voice, and the background audio may be, for example, accompaniment music. In this embodiment, for example, a sound source separation algorithm may be used to separate the human voice information and the background information in the original audio data, and human voice audio data including the human voice information and background audio data including the background audio information may be obtained.

According to an embodiment of the present disclosure, by separating the voice information and background audio information in the original audio data, the voice information is converted into an electric voice, and the electric voice information is synthesized with the background audio information, thereby achieving electroacoustization of audio data that simultaneously contains background audio information and voice information.

According to an embodiment of the present disclosure, the original audio data can be decomposed by implementing a sound source separation algorithm using a neural network. The input of the neural network can be audio data having background sound information and voice information, and the output of the neural network can be voice audio data including voice information and background audio data including background sound information.

According to an embodiment of the present disclosure, a music file and a voice file can be obtained in advance, and the music file and the voice file can be cut into equal-length segments to obtain a plurality of music segments X and a plurality of voice segments Y. Each music segment X can be synthesized with a corresponding voice segment Y to obtain original audio data Z. Each original audio data Z is used as the input of a neural network, and the music segment X and the voice segment Y corresponding to the original audio data Z are used as the expected output to train the neural network. In addition, in order to improve the training effect and accelerate the convergence of the network, the music segment X, the voice segment Y, and the original audio data Z can all be preprocessed into a Mel spectrum. Accordingly, the output result of the neural network is also based on the Mel spectrum. For example, the output result in the Mel spectrum format can be synthesized into the corresponding original audio data by an algorithm such as the Griffin-Lim algorithm.

Based on this, the method for decomposing the above original audio data is further described below with reference to FIG. 2 and by combining specific examples. As can be understood by those skilled in the art, the following exemplary examples are used to understand the present disclosure, and do not limit the present disclosure.

Figure 2 shows a schematic flow chart of a method for decomposing original audio data according to an embodiment of the present disclosure.

As shown in FIG. 2, a method 210 for decomposing original audio data includes:

In operation S211, the original mel spectrum data corresponding to the original audio data is determined.

Next, in operation S212, background mel spectrum data and voice mel spectrum data corresponding to the original mel spectrum data are determined using a neural network.

According to an embodiment of the present disclosure, the background mel spectrum data can include background audio information in the original mel spectrum data, and the voice mel spectrum data can include voice information in the original mel spectrum data.

In operation S213, background audio data is generated based on the background mel spectrum data, and human voice audio data is generated based on the human voice mel spectrum data.

According to an embodiment of the present disclosure, background audio data can be generated based on background mel spectrum data using an algorithm such as the Griffin-Lim algorithm, and human voice audio data can be generated based on human voice mel spectrum data.

According to an embodiment of the present disclosure, electro-acoustic processing of human voice audio data can be realized by quantizing the fundamental frequency of the human voice data. For example, the fundamental frequency, spectral envelope, and aperiodic parameters of the human voice data can be determined. Here, the fundamental frequency indicates the vibration frequency of the vocal cords when pronouncing a sound, and is the pitch of the tone when embodied in audio. Next, the fundamental frequency is quantized, and the human voice data is resynthesized based on the quantized fundamental frequency, spectral envelope, and aperiodic parameters, thereby realizing electro-acoustic processing of the human voice audio data. Here, this resynthesized human voice data is electro-acoustic human voice data, and includes human voice information having electro-sound effects.

The method of electroacoustic processing of the above-mentioned human voice audio data will be further described below with reference to FIG. 3 and in combination with specific examples. As will be understood by those skilled in the art, the following exemplary examples are used to understand the present disclosure, and the present disclosure is not limited thereto.

Figure 3 is a schematic flow chart of a method for performing electrosonification processing on human voice audio data according to an embodiment of the present disclosure.

As shown in FIG. 3, a method 320 for performing electrosonification processing on this human voice audio data can include:

In operation S321, the original fundamental frequency of the voice audio data is extracted.

According to an embodiment of the present disclosure, the original fundamental frequency can be extracted from human voice audio data based on algorithms such as DIO and Harvest.

In operation S322, the original fundamental frequency is corrected to obtain the first fundamental frequency.

According to an embodiment of the present disclosure, the electroacoustic effect can be improved by correcting the fundamental frequency. For example, in this embodiment, human voice audio data can be divided into a plurality of audio segments. Then, for each audio segment in the plurality of audio segments, an energy and a zero cross rate of the audio segment are determined. Based on the energy and the zero cross rate, it is determined whether the audio segment is a dull audio segment. Then, a linear interpolation algorithm is used to correct the fundamental frequency of the dull audio segment.

According to an embodiment of the present disclosure, the voice audio data is divided into a plurality of audio segments with a predetermined unit length, and the length of each audio segment is one predetermined unit length. Here, the predetermined unit length can be set according to actual needs. For example, in this embodiment, the predetermined unit length may be any value between 10 ms and 40 ms.

According to an embodiment of the present disclosure, multiple sampling points are provided in each audio segment. Based on the numerical values of each sampling point in the audio segment, the energy of the audio segment can be determined. For example, the energy of the audio segment can be calculated based on the following formula:

ここで、ｘ_ｉはｉ番目のサンプリングポイントの数値を示し、ｎはサンプリングポイントの数である。

Here, x _i denotes the numerical value of the i-th sampling point, and n is the number of sampling points.

ここで、ｓｒは、オーディオのサンプリングレートを表す。 According to an embodiment of the present disclosure, the number of sampling points n can be determined based on the length and sampling rate of the audio segment. Taking the predetermined unit length as an example of 10 ms, the number of sampling points n can be calculated based on the following formula:

Here, sr represents the audio sampling rate.

According to an embodiment of the present disclosure, it is possible to determine whether the numerical values of every two adjacent sampling points in an audio segment are opposite in sign to each other. Next, the ratio of the number of times adjacent sampling points in the audio segment have opposite signs to the number of all sampling points is determined as the zero cross rate.

ここで、ＺＣＲは、オーディオセグメントのゼロクロスレートであり、ｎは、オーディオセグメントにおけるサンプリングポイントの数であり、ｘ_ｉは、オーディオセグメントにおけるｉ番目のサンプルポイントの数値を表し、ｘ_i－１ は、オーディオセグメントにおけるｉ-１番目のサンプリングポイントの数値を表す。 According to an embodiment of the present disclosure, the zero crossing rate of an audio segment can be calculated based on the following formula:

where ZCR is the zero crossing rate of the audio segment, n is the number of sampling points in the audio segment, x _i represents the numerical value of the i-th sample point in the audio segment, and x _i-1 represents the numerical value of the i-1-th sampling point in the audio segment.

ここで、ｓｒはオーディオのサンプリングレートを表す。 According to an embodiment of the present disclosure, the number of sampling points n can be determined based on the length and sampling rate of the audio segment. Taking the predetermined unit length as an example of 10 ms, the number of sampling points n can be calculated based on the following formula:

Here, sr represents the audio sampling rate.

When the human body produces sound, the vocal cords do not vibrate for the production of clear sounds, so the corresponding fundamental frequency is 0. For the production of voiced sounds, the vocal cords vibrate, so the corresponding fundamental frequency is not 0. Based on this, in this embodiment, the fundamental frequency can be corrected by utilizing the above characteristics.

For example, for each audio segment, if the energy E of the audio segment is less than the threshold e_min and the zero cross rate ZCR of the audio segment is greater than the threshold zcr_max, then the audio segment is a clear audio segment and its fundamental frequency is 0; otherwise, the audio segment is a voiced audio segment and its fundamental frequency is not 0. Here, e_min and zcr_max can be set according to actual needs.

For each clear sound audio segment, the fundamental frequency of the audio segment can be set to 0. For the dull sound audio segments, the fundamental frequency of each dull sound audio segment can be extracted based on an algorithm such as DIO or Harvest, and then it can be detected one by one whether the fundamental frequency value of each dull sound audio segment is 0 or not. For a dull sound audio segment whose fundamental frequency value is 0, a non-zero fundamental frequency value can be obtained as the fundamental frequency value of the dull sound audio segment by performing linear interpolation based on the dull sound audio segment values in the vicinity of the dull sound audio segment based on a linear interpolation algorithm.

For example, there are six dull sound audio segments, and the fundamental frequency values are 100, 100, 0, 0, 160, and 100, respectively. That is, the fundamental frequency value of the third and fourth dull sound audio segments is 0. Therefore, linear interpolation can be performed based on non-zero fundamental frequency values near the fundamental frequency values of the third and fourth dull sound audio segments, that is, based on the second fundamental frequency value 100 and the fifth fundamental frequency value 160, linear interpolation is performed to obtain that the fundamental frequency values of the third and fourth dull sound audio segments are 120 and 140. That is, the six fundamental frequency values after correction are 100, 100, 120, 140, 160, and 100.

Next, in operation S323, the first fundamental frequency is adjusted based on the predetermined electroacoustic parameters to obtain the second fundamental frequency.

According to an embodiment of the present disclosure, the predetermined electroacoustic parameters may include, for example, an electroacoustic degree parameter and/or an electroacoustic tone parameter. Here, the electroacoustic degree parameter may be used to control the degree of electroacoustic. The electroacoustic tone parameter may be used to control the tone. Exemplarily, in this embodiment, the electroacoustic degree parameter may include, for example, 1, 1.2, 1.4, and the larger the electroacoustic degree parameter, the more prominent the electroacoustic effect. The electroacoustic tone parameter may include, for example, -3, -2, -1, +1, +2, +3, where -1, -2, -3 indicate a decrease in tone by one, two, and three, respectively, and +1, +2, and +3 indicate an increase in tone by one, two, and three, respectively.

In the related art, the electroacoustic effect has a single effect, with no adjustable parameters. According to the embodiment of the present disclosure, two parameters, an electroacoustic degree parameter and an electroacoustic tone parameter, are set based on the characteristics of the electroacoustic effect and are used to control the electroacoustic effect, which can meet the needs of different users.

According to an embodiment of the present disclosure, a fundamental frequency variance and/or a fundamental frequency mean value can be determined based on the fundamental frequencies of all the dull audio segments. A corrected fundamental frequency variance is determined based on the electroacoustic degree parameter and the fundamental frequency variance, and/or a corrected fundamental frequency mean value is determined based on the electroacoustic degree parameter and the fundamental frequency mean value. Then, a first fundamental frequency is adjusted based on the corrected fundamental frequency variance and/or the corrected fundamental frequency mean value to obtain a second fundamental frequency.

For example, in this embodiment, the variance of the fundamental frequencies of all the voiced audio segments can be calculated, and the average value of the fundamental frequencies of all the voiced audio segments is calculated as the fundamental frequency variance, which is the fundamental frequency average value.

ここで、ｎｅｗ＿ｖａｒは、補正基本周波数分散であり、ｖａｒは、基本周波数分散であり、ａは、電気音響程度パラメータである。 The corrected fundamental dispersion can then be calculated based on the following formula:

where new_var is the corrected fundamental frequency variance, var is the fundamental frequency variance, and a is the electroacoustic degree parameter.

ここで、ｎｅｗ＿ｍｅａｎは、補正基本周波数平均値であり、ｍｅａｎは、基本周波数平均値であり、ｂは電気音響トーンパラメータである。 The corrected mean fundamental frequency can be calculated based on the following formula:

where new_mean is the corrected fundamental frequency mean value, mean is the fundamental frequency mean value, and b is an electroacoustic tone parameter.

ここで、Ｆ０’ は、第二基本周波数である。 The second fundamental frequency can then be calculated based on the following formula:

Here, F0' is the second fundamental frequency.

In operation S324, a quantization process is performed on the second fundamental frequency to obtain a third fundamental frequency.

In natural audio, voice tones are inflected and change gradually, whereas electroacoustics quantizes tones into specific scales, where the tones change discontinuously, similar to the tones transmitted from electronic musical instruments. Based on this, according to an embodiment of the present disclosure, the fundamental frequency of human voice data can be quantized with each key frequency of a piano as a target frequency.

ここで、 ｓｃａｌｅは、周波数範囲であり、Ｆ０´は、第二基本周波数である。 Illustratively, in this embodiment, the frequency range can be determined based on the following formula:

where scale is the frequency range and F0' is the second fundamental frequency.

ここで、Ｆ０’’ は、第三基本周波数である。 Then, based on the frequency range, the third fundamental frequency can be determined based on the following formula:

Here, F0'' is the third fundamental frequency.

In operation S325, electroacoustic voice data is determined based on the third fundamental frequency.

According to an embodiment of the present disclosure, a spectral envelope and aperiodic parameters can be determined based on the voice audio data and the first fundamental frequency. Electroacoustic voice data can then be determined based on the third fundamental frequency, the spectral envelope and the aperiodic parameters.

The above-mentioned audio data processing method will be further described below with reference to FIG. 4 in combination with specific examples. As will be understood by those skilled in the art, the following exemplary examples are used to understand the present disclosure, and the present disclosure is not limited thereto.

Figure 4 shows a schematic flow chart of an audio data processing method according to another embodiment of the present disclosure.

As shown in FIG. 4, this audio data processing method 400 includes the following: In operation S401, it is determined whether or not the audio data (abbreviated as audio) includes accompaniment music (abbreviated as accompaniment). If the audio data includes accompaniment, operation S402 is executed. If the audio data includes only human voices and does not include accompaniment, operation S403 is executed.

In operation S402, a sound source separation algorithm is used to separate the human voice from the accompaniment. Then, operation S403 is performed on the human voice obtained by the separation.

In operation S403, the zero-crossing rate, fundamental frequency f0, and energy are extracted for the human voice.

In operation S404, the fundamental frequency is corrected based on the zero cross rate and energy to obtain F0.

In operation S405, the spectral envelope SP and the aperiodic parameters AP are calculated using the voice and the corrected fundamental frequency F0.

In operation S406, the fundamental frequency is adjusted to obtain F0' based on the electroacoustic degree parameter a and electroacoustic tone parameter b set by the user.

In operation S407, the fundamental frequency F0' is quantized to obtain F0''.

In operation S408, the fundamental frequency F0'', the spectral envelope SP and the aperiodic parameters AP are used to synthesize a human voice with electrical sound effects.

In operation S409, if the audio has accompaniment, perform operation S410. Otherwise, perform operation S411.

In operation S410, the accompaniment is also matched to the human voice to generate the final audio with electronic sound effects.

In operation S411, audio with electric sound effects is output.

The audio data processing method according to the embodiment of the present disclosure can flexibly and efficiently increase the electric acoustic effects in the audio data, thereby improving the entertainment enjoyment of the user.

Figure 5 shows a schematic block diagram of an audio data processing device according to an embodiment of the present invention.

As shown in FIG. 5, the audio data processing device 500 includes a decomposition module 510, an electroacoustic processing module 520, and a synthesis module 530.

The decomposition module 510 is used to decompose the original audio data to obtain voice audio data and background audio data.

The electroacoustic processing module 520 is used to perform electroacoustic processing on the human voice audio data to obtain electroacoustic human voice data.

The synthesis module 530 is used to synthesize the electroacoustic voice data and the background audio data to obtain the target audio data.

According to an embodiment of the present disclosure, the decomposition module may include a mel spectrum determination submodule, a decomposition submodule, and a generation submodule. Here, the mel spectrum determination submodule is used to determine original mel spectrum data corresponding to the original audio data. The decomposition submodule is used to determine background mel spectrum data and voice mel spectrum data corresponding to the original mel spectrum data using a neural network. The generation submodule is used to generate background audio data based on the background mel spectrum data, and generate voice audio data based on the voice mel spectrum data.

According to an embodiment of the present disclosure, the electroacoustic processing module may include an extraction submodule, a correction submodule, an adjustment submodule, a quantization submodule, and an electroacoustic determination submodule. Here, the extraction submodule is used to extract an original fundamental frequency of the human voice audio data. The correction submodule is used to correct the original fundamental frequency to obtain a first fundamental frequency. The adjustment submodule is used to adjust the first fundamental frequency based on a predetermined electroacoustic parameter to obtain a second fundamental frequency. The quantization submodule is used to perform a quantization process on the second fundamental frequency to obtain a third fundamental frequency. The electroacoustic determination submodule is used to determine the electroacoustic human voice data based on the third fundamental frequency.

According to an embodiment of the present disclosure, the correction submodule may include a segmentation unit, an energy determination unit, a zero cross rate determination unit, a dullness judgment unit, and a correction unit. Here, the segmentation unit is used to divide the voice audio data into a plurality of audio segments. The energy determination unit is used for each audio segment in the plurality of audio segments to determine an energy of the audio segment. The zero cross rate determination unit is used for each audio segment in the plurality of audio segments to determine a zero cross rate of the audio segment. The dullness judgment unit is used to determine whether a type of the audio segment is a dullness audio segment based on the energy and the zero cross rate. The correction unit is used to correct the fundamental frequency of the dullness audio segment using a linear interpolation algorithm.

According to an embodiment of the present disclosure, a plurality of sampling points are provided in the audio segment. The energy determination unit is further used to determine an energy of the audio segment based on a numerical value of each sampling point in the audio segment.

According to an embodiment of the present disclosure, the zero-cross rate determination unit is further used to determine whether the numerical values of every two adjacent sampling points in the audio segment are opposite in sign to each other, and then determines the ratio of the number of times that adjacent sampling points in the audio segment are opposite in sign to the number of all sampling points as the zero-cross rate.

According to an embodiment of the present disclosure, the predetermined electroacoustic parameters may include an electroacoustic degree parameter and/or an electroacoustic tone parameter. The adjustment submodule may include a first determination unit, a second determination unit, and an adjustment unit. Here, the first determination unit is used to determine a fundamental frequency variance and/or a fundamental frequency average value based on a fundamental frequency of the dull audio segment. The second determination unit is used to determine a corrected fundamental frequency variance based on the electroacoustic degree parameter and the fundamental frequency variance, and/or to determine a corrected fundamental frequency average value based on the electroacoustic degree parameter and the fundamental frequency average value. The adjustment unit is used to adjust the first fundamental frequency based on the corrected fundamental frequency variance and/or the corrected fundamental frequency average value to obtain a second fundamental frequency.

According to an embodiment of the present disclosure, the quantization submodule may include a frequency range determination unit and a third fundamental frequency determination unit.

ここで、ｓｃａｌｅ は、周波数範囲であり、Ｆ０’ は、第二基本周波数である。 Here, the frequency range determination unit is used to determine the frequency range based on the following formula:

where scale is the frequency range and F0' is the second fundamental frequency.

ここで、Ｆ０’’ は、第三基本周波数である。 The third fundamental frequency determining unit is used to determine the third fundamental frequency based on the frequency range and according to the following formula:

Here, F0'' is the third fundamental frequency.

According to an embodiment of the present disclosure, the audio data processing device may further include a determination module, which is used to determine the spectral envelope and the aperiodic parameters based on the voice audio data and the first fundamental frequency.

According to an embodiment of the present disclosure, the electroacoustic voice determination submodule is further used to determine electroacoustic voice data based on the third fundamental frequency, the spectral envelope and the non-periodic parameters.

According to an embodiment of the present disclosure, the present disclosure further provides an electronic device, a readable storage medium, and a computer program .

FIG. 6 is a schematic block diagram illustrating an example of an electronic device 600 capable of implementing embodiments of the present disclosure. The electronic device is intended to represent various types of digital computers, such as laptop computers, desktop computers, workbenches, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various types of mobile devices, such as personal digital assistants, mobile phones, smart phones, wearable devices, and other similar computing devices. The components, their connections and relationships, and their functions shown herein are merely exemplary and are not intended to limit the implementation of the present disclosure as described and/or claimed herein.

As shown in FIG. 6, the device 600 includes a computing unit 601, which can perform various appropriate operations and processes based on a computer program stored in a read-only memory (ROM) 602 or loaded from a storage unit 608 into a random access memory (RAM) 603. The RAM 603 can further store various programs and data required for the operation of the device 600. The computing unit 601, the ROM 602, and the RAM 603 are interconnected via a bus 604. An input/output interface 605 is also connected to the bus 604.

The components of the device 600 are connected to an I/O interface 605, and include an input unit 606, such as a keyboard, a mouse, etc., an output unit 607, such as various types of displays, speakers, etc., a storage unit 608, such as a magnetic disk, an optical disk, etc., and a communication unit 609, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 609 enables the device 600 to exchange information/data with other devices via a computer network such as the Internet and/or various types of electrical communication networks.

The computing unit 601 may be any of a variety of general-purpose and/or dedicated processing modules having processing and computing capabilities. Some examples of the computing unit 601 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various operational machine learning model algorithm computing units, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, and the like. The computing unit 601 performs each of the methods and processes described above, such as the audio data processing method. For example, in some embodiments, the audio data processing method may be realized as a computer software program tangibly included in a machine-readable medium, such as the storage unit 608. In some embodiments, some or all of the computer program may be loaded and/or installed in the electronic device 600 via the ROM 602 and/or the communication unit 609. When the computer program is loaded into RAM 603 and executed by the computing unit 601, it may perform one or more steps of the audio data processing method described above. Alternatively, in other embodiments, the computing unit 601 may be configured to perform the audio data processing method in any other suitable manner (e.g., via firmware).

Various embodiments of the systems and techniques described herein may be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and/or combinations thereof. These various embodiments may be implemented in one or more computer programs that may be executed and/or interpreted by a programmable system that includes at least one programmable processor, which may be a dedicated or general purpose programmable processor, and may include a processor that may receive data and instructions from and transmit data and instructions to a storage system, at least one input device, and at least one output device.

Program codes for implementing the methods of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general purpose computer, a special purpose computer, or other programmable data processing apparatus, so that when the program code is executed by the processor or controller, the functions/operations specified in the flowcharts and/or block diagrams are implemented. The program codes may be fully executed on the device, partially executed on the device, partially executed on the device as a separate software package and partially executed on a remote device, or fully executed on a remote device or server.

In the context of this disclosure, a machine-readable medium may be a tangible medium, and may contain or store a program for use in or in combination with an instruction execution system, device, or appliance. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or appliance, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections by one or more wires, portable computer disks, hard disks, random access memories (RAMs), read-only memories (ROMs), erasable programmable read-only memories (EPROMs or flash memories), optical fibers, portable compact disk read-only memories (CD-ROMs), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

A computer may implement the systems and techniques described herein to provide interaction with a user, and the computer may include a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to a user, and a keyboard and pointing device (e.g., a mouse or trackball) through which the user may provide input to the computer. Other types of devices may also provide interaction with a user, for example, the feedback provided to the user may be any form of sensing feedback (e.g., visual feedback, auditory feedback, or tactile feedback) and may receive input from the user in any form (including voice input, audio input, or tactile input).

The systems and techniques described herein may be implemented in a computing system that includes background components (e.g., a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer having a graphical user interface or a web browser through which a user can interact with embodiments of the systems and techniques described herein), or a computing system that includes any combination of such background, middleware, or front-end components. The components of the system may be connected to each other by any form or medium of digital data communication (e.g., a communications network). Examples of communications networks illustratively include a local area network (LAN), a wide area network (WAN), and the Internet.

The computer system may include clients and servers. The clients and servers are generally remote and typically interact via a communications network. The relationship between the clients and servers is created by computer programs running on the corresponding computers and having a client-server relationship. The servers may be cloud servers, servers in a distributed system, or servers coupled to a blockchain.

It should be understood that various types of flows shown above may be used, and steps may be rearranged, added, or removed. For example, each step described in the present invention may be performed in parallel, sequentially, or in a different order, and the present specification is not limited thereto, as long as the desired results of the technical proposal of the present disclosure can be achieved.

The specific embodiments described above do not limit the scope of protection of the present disclosure. Those skilled in the art should understand that various modifications, combinations, subcombinations and substitutions can be made according to design requirements and other factors. Any modifications, equivalent replacements and improvements made within the spirit and principles of the present disclosure should be included within the scope of protection of the present disclosure.

Claims (17)

1. An audio data processing method by an audio data processing device, comprising:
Decomposing the original audio data to obtain voice audio data and background audio data;
performing electroacoustic processing on the human voice audio data to obtain electroacoustic human voice data;
and combining the electroacoustic voice data and the background audio data to obtain target audio data.
performing electroacoustic processing on the voice audio data to obtain electroacoustic voice data;
Extracting an original fundamental frequency of the voice audio data;
correcting the original fundamental frequency to obtain a first fundamental frequency;
adjusting the first fundamental frequency based on a predetermined electroacoustic parameter to obtain a second fundamental frequency;
performing a quantization process on the second fundamental frequency to obtain a third fundamental frequency;
determining the electroacoustic voice data based on the third fundamental frequency;
correcting the original fundamental frequency to obtain a first fundamental frequency,
Segmenting the voice audio data into a plurality of audio segments;
For each audio segment in the plurality of audio segments, determining an energy and a zero crossing rate for the audio segment;
determining whether the audio segment is a dull audio segment based on the energy and the zero crossing rate;
and compensating for the fundamental frequency of the voiced audio segment utilizing a linear interpolation algorithm.
A method for processing audio data. Decomposing the original audio data to obtain background audio data and voice audio data includes:
determining original mel spectrum data corresponding to the original audio data;
determining background mel spectrum data and voice mel spectrum data corresponding to the original mel spectrum data using a neural network;
The method of claim 1 , further comprising: generating the background audio data based on the background mel spectrum data; and generating the voice audio data based on the voice mel spectrum data. A plurality of sampling points are provided in the audio segment, and determining an energy of the audio segment comprises:
2. A method as claimed in claim 1 , comprising determining an energy of the audio segment based on the numerical values of each sampling point in the audio segment. The audio segment includes a plurality of sampling points, and determining a zero crossing rate for the audio segment includes:
determining whether the signs of the numerical values of two adjacent sample points in the audio segment are opposite to each other;
2. The method of claim 1 , further comprising: determining a percentage of the total number of sample points in the audio segment that have different codes for adjacent sample points; and determining the zero crossing rate as the percentage of the total number of sample points in the audio segment that have different codes. The predetermined electroacoustic parameter includes an electroacoustic degree parameter and/or an electroacoustic tone parameter, and adjusting the first fundamental frequency based on the predetermined electroacoustic parameter to obtain a second fundamental frequency includes:
determining a fundamental frequency variance and/or a fundamental frequency mean based on a fundamental frequency of the dull audio segment;
determining a corrected fundamental frequency variance based on said electroacoustic degree parameter and said fundamental frequency variance and/or determining a corrected fundamental frequency mean value based on said electroacoustic tone parameter and said fundamental frequency mean value;
and adjusting the first fundamental frequency based on the corrected fundamental frequency variance and/or the corrected fundamental frequency average value to obtain the second fundamental frequency. Performing a quantization process on the second fundamental frequency to obtain a third fundamental frequency includes determining a frequency range based on the following formula:

where scale is the frequency range, F0′ is the second fundamental frequency,
determining the third fundamental frequency based on the frequency range according to the following formula:

The method according to any one of claims 1 to 5 , wherein F0'' is the third fundamental frequency. determining a spectral envelope and aperiodic parameters based on the voice audio data and the first fundamental frequency;
wherein determining the electroacoustic voice data based on the third fundamental frequency comprises:
The method of claim 1 , 3 or 5 , comprising determining the electroacoustic voice data based on the third fundamental frequency, the spectral envelope and the non-periodic parameters. A decomposition module for decomposing the original audio data to obtain voice audio data and background audio data;
an electroacoustic processing module for performing electroacoustic processing on the human voice audio data to obtain electroacoustic human voice data;
a synthesis module for synthesizing the electroacoustic voice data and the background audio data to obtain target audio data.
The electroacoustic processing module includes:
an extraction sub-module for extracting an original fundamental frequency of the voice audio data;
a correction sub-module for correcting the original fundamental frequency to obtain a first fundamental frequency;
an adjusting sub-module for adjusting the first fundamental frequency based on a predetermined electroacoustic parameter to obtain a second fundamental frequency;
a quantization submodule for performing a quantization process on the second fundamental frequency to obtain a third fundamental frequency;
an electroacoustic determination submodule for determining the electroacoustic voice data based on the third fundamental frequency;
The correction submodule:
a segmentation unit for dividing the voice audio data into a plurality of audio segments;
an energy determination unit for determining, for each audio segment in the plurality of audio segments, an energy of the audio segment;
a zero cross rate determination unit for determining, for each audio segment in the plurality of audio segments, a zero cross rate determination unit for determining a zero cross rate for the audio segment;
a voiced sound determining unit for determining whether a type of the audio segment is a voiced sound audio segment based on the energy and the zero cross rate;
a correction unit for correcting the fundamental frequency of the dull audio segment using a linear interpolation algorithm.
Audio data processing device. The decomposition module includes:
a mel spectrum determination sub-module for determining original mel spectrum data corresponding to the original audio data;
a decomposition submodule for determining background mel-spectrum data and voice mel-spectrum data corresponding to the original mel-spectrum data using a neural network;
The apparatus of claim 8 , further comprising: a generating sub-module for generating the background audio data based on the background mel spectrum data, and for generating the voice audio data based on the voice mel spectrum data. A plurality of sampling points are provided in the audio segment, and the energy determining unit further comprises:
The apparatus of claim 8 , further comprising: determining an energy of the audio segment based on a numerical value of each sampling point in the audio segment. The audio segment includes a plurality of sampling points, and the zero crossing rate determination unit further comprises:
determining whether the signs of the numerical values of every two adjacent sample points in the audio segment are opposite to each other;
The apparatus of claim 8 , further comprising: determining a ratio of the number of times adjacent sample points in the audio segment have different codes to the total number of sample points, and determining the zero crossing rate as the ratio. The predetermined electroacoustic parameters include electroacoustic degree parameters and/or electroacoustic tone parameters, and the adjustment sub-module
a first determining unit for determining a fundamental frequency variance and/or a fundamental frequency mean value based on a fundamental frequency of the dull audio segment;
a second determination unit for determining a corrected fundamental frequency variance based on said electroacoustic degree parameter and said fundamental frequency variance and/or for determining a corrected fundamental frequency mean value based on said electroacoustic degree parameter and said fundamental frequency mean value;
The apparatus according to claim 8 , further comprising: an adjusting unit for adjusting the first fundamental frequency based on the corrected fundamental frequency variance and/or the corrected fundamental frequency average value to obtain the second fundamental frequency. The quantization sub-module includes a frequency range determining unit and a third fundamental frequency determining unit;
The frequency range determining unit is used to determine a frequency range according to the following formula:

where scale is the frequency range, F0′ is the second fundamental frequency,
The third fundamental frequency determining unit is used to determine the third fundamental frequency based on the frequency range according to the following formula:

13. The apparatus according to claim 8 , wherein F0'' is the third fundamental frequency. a determining module for determining a spectral envelope and a non-periodic parameter based on the voice audio data and the first fundamental frequency;
wherein the electroacoustic determination submodule further comprises:
13. Apparatus according to any one of claims 8, 10 to 12 , further comprising: determining the electroacoustic voice data based on the third fundamental frequency, the spectral envelope and the non-periodic parameters. At least one processor;
a memory in communication with the at least one processor;
The memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor such that the at least one processor can perform the method according to any one of claims 1 to 7 .
Electronics. A non-transitory computer-readable storage medium having computer instructions stored thereon, comprising:
A storage medium in which the computer instructions are used to cause a computer to carry out the method according to any one of claims 1 to 7 . A computer program comprising instructions which, when executed by a processor, implements the method according to any one of claims 1 to 7 .

Images