WO2025190098A1 - 音频数据处理方法、装置、电子设备、介质和程序产品 - Google Patents

音频数据处理方法、装置、电子设备、介质和程序产品

Info

Publication number
WO2025190098A1
WO2025190098A1 PCT/CN2025/080169 CN2025080169W WO2025190098A1 WO 2025190098 A1 WO2025190098 A1 WO 2025190098A1 CN 2025080169 W CN2025080169 W CN 2025080169W WO 2025190098 A1 WO2025190098 A1 WO 2025190098A1
Authority
WO
WIPO (PCT)
Prior art keywords
audio data
playback
time
storage area
collected
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Pending
Application number
PCT/CN2025/080169
Other languages
English (en)
French (fr)
Inventor
李浩然
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Tencent Technology (Shenzhen) Co Ltd
Original Assignee
Tencent Technology (Shenzhen) Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Tencent Technology (Shenzhen) Co Ltd filed Critical Tencent Technology (Shenzhen) Co Ltd
Publication of WO2025190098A1 publication Critical patent/WO2025190098A1/zh
Anticipated expiration legal-status Critical
Pending legal-status Critical Current

Links

Classifications

    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/16—Sound input; Sound output
    • G—PHYSICS
    • G10—MUSICAL INSTRUMENTS; ACOUSTICS
    • G10L—SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING
    • G10L21/00—Speech or voice signal processing techniques to produce another audible or non-audible signal, e.g. visual or tactile, in order to modify its quality or its intelligibility
    • G10L21/02—Speech enhancement, e.g. noise reduction or echo cancellation
    • G10L21/0208—Noise filtering
    • G—PHYSICS
    • G10—MUSICAL INSTRUMENTS; ACOUSTICS
    • G10L—SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING
    • G10L21/00—Speech or voice signal processing techniques to produce another audible or non-audible signal, e.g. visual or tactile, in order to modify its quality or its intelligibility
    • G10L21/02—Speech enhancement, e.g. noise reduction or echo cancellation
    • G10L21/0208—Noise filtering
    • G10L2021/02082—Noise filtering the noise being echo, reverberation of the speech

Definitions

  • the present application relates to the field of computers, and in particular to an audio data processing method, device, electronic device, storage medium, and program product.
  • an audio acquisition device e.g., a microphone
  • an audio playback device e.g., a speaker
  • the sound played by the audio playback device propagates through the air and enters the audio acquisition device, where it is picked up, resulting in an echo.
  • echo cancellation can be performed on the audio data collected by the audio acquisition device.
  • the echo cancellation method provided in the related art still leaves a large amount of echo in the processed audio data, which reduces the accuracy of the echo cancellation processing and the quality of the audio data.
  • a method for processing audio data comprising:
  • echo cancellation processing is performed on the collected audio data.
  • an audio data processing apparatus comprising:
  • An acquisition module is configured to acquire the audio data to be played and store the acquired audio data in a set storage area
  • a processing module is configured to play the acquired playback audio data and collect audio data through an audio collection device to obtain collected audio data;
  • An extraction module configured to extract, from the storage area, the playback audio data whose corresponding playback duration matches the playback duration of the collected audio data in order of the corresponding storage time from earliest to latest;
  • the cancellation module is configured to perform echo cancellation processing on the collected audio data according to the extracted playback audio data.
  • an electronic device including:
  • processors one or more processors
  • the storage device is used to store one or more computer-readable instructions, and when the one or more computer-readable instructions are executed by the one or more processors, the electronic device implements the method as described above.
  • a computer-readable storage medium on which computer-readable instructions are stored.
  • the computer-readable instructions are executed by a processor of an electronic device, the electronic device implements the method as described above.
  • a computer program product comprising computer-readable instructions, which implement the aforementioned method when executed by a processor.
  • FIG1 is a schematic diagram of an implementation environment shown in an exemplary embodiment of the present application.
  • FIG2 is a flow chart of an audio data processing method shown in an exemplary embodiment of the present application.
  • FIG3 is a flowchart of an audio data processing method shown in another exemplary embodiment of the present application.
  • FIG4 is a flowchart of an audio data processing method shown in another exemplary embodiment of the present application.
  • FIG5 is a flowchart of an audio data processing method shown in another exemplary embodiment of the present application.
  • FIG6 is a flowchart of an audio data processing method shown in another exemplary embodiment of the present application.
  • FIG7 is a flowchart of an audio data processing method shown in another exemplary embodiment of the present application.
  • FIG8 is a flowchart of an audio data processing method shown in another exemplary embodiment of the present application.
  • FIG9 is a flowchart of an audio data processing method shown in another exemplary embodiment of the present application.
  • FIG10 is a flowchart of an audio data processing method shown in another exemplary embodiment of the present application.
  • FIG11 is a flowchart of an audio data processing method shown in another exemplary embodiment of the present application.
  • FIG12 is a flowchart of an audio data processing method shown in another exemplary embodiment of the present application.
  • FIG13 is a schematic diagram of an implementation environment shown in another exemplary embodiment of the present application.
  • FIG14 is a schematic diagram of a data feeding module shown in an exemplary embodiment of the present application.
  • FIG15 is a flowchart of a voice data caching process shown in an exemplary embodiment of the present application.
  • FIG16 is a block diagram of a data feeding process shown in an exemplary embodiment of the present application.
  • FIG17 is a flow chart of a data feeding process shown in an exemplary embodiment of the present application.
  • FIG18 is a structural diagram of an audio data processing apparatus according to an exemplary embodiment of the present application.
  • FIG19 shows a schematic structural diagram of a computer system suitable for implementing an electronic device according to an embodiment of the present application.
  • module refers to a computer-readable instruction or a portion of a computer-readable instruction having a predetermined function, and works together with other related parts to achieve a predetermined goal, and can be implemented in whole or in part by using software, hardware (such as processing circuits or memories), or a combination thereof.
  • a processor or multiple processors or memories
  • each module or unit can be part of an overall module or unit that includes the function of the module or unit.
  • echo cancellation processing has low accuracy, which reduces the quality of audio data.
  • embodiments of the present application provide an audio data processing method and device, electronic device, storage medium, and program product that can improve the accuracy of echo cancellation processing and the quality of audio data.
  • Figure 1 is a schematic diagram of an implementation environment involved in this application, which includes a terminal device 110 and a server 120.
  • the terminal device 110 and the server 120 communicate with each other via a wired or wireless network.
  • the terminal device 110 can upload its own data to the server 120 and also obtain data from the server 120.
  • the terminal device 110 may include but is not limited to mobile phones, tablets, laptops, computers, voice interaction devices, home appliances, vehicle-mounted terminals, aircraft, remote driving terminals, payment devices (for example, palm-swiping devices, etc.), etc.;
  • the server 120 may be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN (Content Delivery Network), and big data and artificial intelligence platforms.
  • This section does not limit the specific forms of terminal devices and servers.
  • terminal devices 110 and servers 120 in FIG. 1 is merely illustrative, and any number of terminal devices 110 and servers 120 may be provided according to actual needs.
  • the audio data processing method provided by the embodiments of the present application may be executed by the terminal device 110 .
  • the terminal device 110 can store the obtained playback audio data in a set storage area, play the obtained playback audio data, and collect audio data through the audio collection device to obtain collected audio data; then, according to the corresponding storage time from early to late, the playback audio data whose corresponding playback duration matches the playback duration of the collected audio data is extracted from the storage area, so that the collected audio data is echo-cancelled according to the extracted playback audio data. That is, using the playback audio data as reference data for echo-cancelling the collected audio data can improve the accuracy of the echo cancellation processing.
  • the playback audio data is first stored in the storage area, and then after the collected audio data is obtained, the playback audio data for echo cancellation is extracted from the storage area, thereby ensuring that there is sufficient playback audio data for echo cancellation, so that the playback audio data whose corresponding playback duration matches the playback duration of the collected audio data can be extracted according to the corresponding storage time from early to late.
  • the playback audio data is directly extracted according to the obtained playback audio data. In this approach, echo cancellation is performed on the collected audio data using the playback audio data.
  • This approach may cause errors in the alignment of the collected audio data with the playback audio data because the playback durations of the acquired playback audio data and the collected audio data may not match. Furthermore, if the difference between the playback duration of the acquired playback audio data and the playback audio data is greater than a certain threshold, some of the playback audio data will be deleted, resulting in a lack of reference data for the subsequently acquired collected audio data, reducing the accuracy of the echo cancellation solution. (For example, at a certain moment, 7 frames of playback audio data and 3 frames of collected audio data may be acquired. The 3 frames of playback audio data extracted from the 7 frames of playback audio data and used for echo cancellation of the collected audio data may be unrelated to the collected audio data.
  • the subsequently acquired collected audio data will lack the playback audio data that serves as reference data for echo cancellation.
  • the solution provided by the embodiments of the present application can improve the temporal and duration matching between the extracted playback audio data and the collected audio data, improve the accuracy of the playback audio data used for echo cancellation, thereby improving the accuracy of the echo cancellation processing and, in turn, improving the quality of the audio data.
  • the server 120 may have similar functions to the terminal device 110, thereby executing the audio data processing method provided in the embodiment of the present application.
  • the server 120 may store the acquired playback audio data in a set storage area, play the acquired playback audio data, and collect the audio data through an audio collection device to obtain collected audio data; then, in the order of the corresponding storage time from earliest to latest, extract the playback audio data whose corresponding playback duration matches the playback duration of the collected audio data from the storage area, thereby performing echo cancellation processing on the collected audio data based on the extracted playback audio data.
  • the terminal device 110 and the server 120 can also jointly execute the audio data processing method provided in the embodiment of the present application.
  • the terminal device 110 can obtain the playback audio data to be played, play the obtained playback audio data, and send the obtained playback audio data to the server 120, so that the server 120 stores the playback audio data from the terminal device 110 in a set storage area; the terminal device 110 can also collect audio data through an audio collection device to obtain collected audio data, and transmit the collected audio data to the server 120.
  • the server 120 After the server 120 receives the collected audio data from the terminal device 110, it can extract the playback audio data whose corresponding playback duration matches the playback duration of the collected audio data from the storage area in the order of the corresponding storage time from early to late, thereby performing echo cancellation processing on the collected audio data based on the extracted playback audio data.
  • the audio data processing method in the embodiments of the present application can be applied to various scenarios requiring echo cancellation, such as voice calls, video calls, live interactions, online conferences, etc.
  • the embodiments of the present application involve playing audio data, collecting audio data, and other user-related data.
  • the method of the present application is applied to a specific product or technology, it is necessary to obtain the user's permission or consent, and the extraction, use, and processing of the relevant data comply with local safety standards and local laws and regulations.
  • FIG. 2 is a flow chart illustrating an audio data processing method according to an exemplary embodiment of the present application.
  • the method can be performed by an electronic device.
  • the method can be applied to the implementation environment shown in FIG. 1 .
  • the electronic device can be the terminal device 110 in the implementation environment shown in FIG. 1 , or the server 120 in the implementation environment shown in FIG. 1 .
  • the method can be performed jointly by the terminal device 110 and the server 120 in the implementation environment shown in FIG. 1 .
  • the audio data processing method may include steps S210 to S240 , which are described in detail as follows:
  • Step S210 Acquire the audio data to be played, and store the acquired audio data in a set storage area.
  • playing audio data refers to audio data that needs to be played, and its types include but are not limited to audio data in the voice communication process, music, audio contained in the video, etc., among which voice communication includes but is not limited to voice calls, live broadcast interactions and other communication scenarios.
  • the designated storage area is used to store the playback audio data for subsequent echo cancellation processing.
  • the specific form of the storage area can be flexibly set according to actual needs, including but not limited to cache, message queue, etc.
  • acoustic echo cancellation refers to the elimination of noise generated by the feedback path (feedback path) generated by air propagation between the audio acquisition device and the audio playback device.
  • the audio played by the audio playback device is eliminated from the audio collected by the audio acquisition device.
  • the audio acquisition device is used to collect sounds in the environment, and its types include but are not limited to microphones, etc.
  • the audio playback device is used to play audio, and its types include but are not limited to speakers, horns, etc.
  • an electronic device can play audio and collect audio at the same time.
  • the audio collection device of the electronic device if the audio collection device of the electronic device is collecting audio data, the audio collection device will directly or indirectly collect the playback audio data.
  • the played sound is indirectly collected after being reflected by the room, etc.
  • the audio data collected by the audio collection device not only includes the sound in the environment, but also includes the sound corresponding to the playback audio data.
  • the playback audio data can be first stored in the storage area of the electronic device as reference data for subsequent echo cancellation processing.
  • Step S220 playing the acquired audio data, and collecting audio data through an audio collection device to obtain collected audio data.
  • the electronic device can play the audio data through an audio playback device (such as a speaker, a loudspeaker, etc.).
  • an audio playback device such as a speaker, a loudspeaker, etc.
  • the storage and playback of the audio data can be performed simultaneously, or the audio data can be played first and then stored, or the audio data can be stored first and then played.
  • the sound of the surrounding environment can be collected by an audio collection device (such as a microphone, etc.) to obtain collected audio data.
  • an audio collection device such as a microphone, etc.
  • the playing of the audio data and the collection of the audio data can be started synchronously or asynchronously.
  • Step S230 extracting the playback audio data whose corresponding playback duration matches the playback duration of the collected audio data from the storage area in the order of the corresponding storage time from earliest to latest.
  • the storage time refers to the time when the played audio data is stored in the storage area, and can represent the storage order of the played audio data.
  • the played audio data may be acquired from the storage area to serve as reference data for recycling and elimination processing.
  • the playback audio data is extracted from the storage area in order of the corresponding storage time from early to late until the playback duration of the extracted playback audio data matches the playback duration of the collected audio data.
  • the storage time of the first frame of playback audio data is earlier than the storage time of the second frame of playback audio data
  • the storage time of the second frame of playback audio data is earlier than the storage time of the third frame of playback audio data, that is, the first frame of playback audio data is stored first, then the second frame of playback audio data is stored, and then the third frame of playback audio data is stored.
  • 1 frame of collected audio data is collected
  • the first frame of playback audio data is extracted from the storage area.
  • 2 frames of collected audio data are collected, the first and second frames of playback audio data are extracted from the storage area.
  • one frame of audio data refers to audio data of a short period of time (ie, unit playback duration), wherein the unit playback duration may be 10 milliseconds, 20 milliseconds, etc.
  • Step S240 performing echo cancellation processing on the collected audio data according to the extracted played audio data.
  • the electronic device can perform echo cancellation processing on the collected audio data based on the extracted playback audio data.
  • the specific method of the echo cancellation processing can be flexibly set according to actual needs.
  • the size of the echo signal contained in the collected audio data can be predicted based on the extracted playback audio data, and the predicted echo signal can be subtracted from the audio signal corresponding to the collected audio data to obtain the collected audio data after echo cancellation processing.
  • the playback audio data is used as reference data for performing echo cancellation processing on the collected audio data, which can improve the accuracy of the echo cancellation processing.
  • the playback audio data is first stored in a storage area, and then, after the collected audio data is obtained, the playback audio data for echo cancellation is extracted from the storage area, thereby ensuring that there is sufficient playback audio data for echo cancellation.
  • the playback data is extracted in order from early to late according to the corresponding storage time, and the playback duration of the extracted playback audio data matches the playback duration of the collected audio data, thereby improving the matching degree between the extracted playback audio data and the collected audio data in time and duration, improving the accuracy of the playback audio data used for echo cancellation, thereby improving the accuracy of the echo cancellation processing, and further improving the quality of the audio data.
  • FIG. 3 is a flow chart of an audio data processing method proposed based on the method shown in FIG. 2 .
  • the audio data processing method further includes steps S310 to S330 , which are described in detail as follows:
  • Step S310 Detect the total playing time corresponding to the playing audio data contained in the storage area.
  • the storage area contains playback audio data added to the storage area at different storage times.
  • the electronic device can detect the total playback time of the playback audio data contained in the storage area in real time or periodically, wherein the total playback time can be calculated based on the number of frames of the playback audio data contained in the storage area and the unit playback time corresponding to one frame of audio data. For example, assuming that the storage area contains 10 frames of playback audio data and the playback time of one frame of audio data is 10 milliseconds, the total playback time is 100 milliseconds.
  • Step S320 If the total playback time is greater than or equal to the playback time threshold, the target playback audio data to be deleted is extracted from the storage area in the order of the corresponding storage time from earliest to latest.
  • the total playback time corresponding to the playback audio data contained in the storage area is too long, it indicates that the time difference between the playback time of some playback audio data in the storage area and the current time is too long, and the probability of the playback audio data collected subsequently containing the playback audio data is low.
  • the total playback time of the playback audio data contained in the storage area is greater than 1 minute, it indicates that some playback audio data was played 1 minute ago, and the probability of the subsequently collected audio data containing the playback audio data played 1 minute ago is low. Therefore, in order to save storage resources and improve the accuracy of echo cancellation, a playback time threshold is set to delete useless playback audio data in the storage area.
  • the electronic device detects that the total playback time is greater than or equal to the playback time threshold, it extracts the target playback audio data from the storage area in order of the corresponding storage time from early to late, wherein the target playback audio data refers to the playback audio data that needs to be deleted.
  • the specific value of the playback time threshold can be flexibly set according to actual needs. In an optional example, it can be set to 30 seconds, 20 seconds, etc. based on experience. In another optional example, there will be a time difference between obtaining the playback audio data and playing the playback audio data, that is, there is a playback link delay, which is recorded as the first delay. If the total playback time exceeds the first delay, it means that the time difference between the playback time of part of the playback audio data in the storage area and the current time exceeds the first delay, and the probability of the subsequent collected audio data containing the playback audio data is small.
  • the electronic device can calculate the playback time threshold based on the first delay, wherein the playback time threshold is positively correlated with the first delay, that is, the larger the first delay, the larger the playback time threshold.
  • the electronic device can use the first delay directly as the playback time threshold, or the electronic device can also use the product of the first delay and the set coefficient as the playback delay threshold, and the set coefficient is a value greater than 1.
  • the electronic device can calculate the playback time threshold based on the second delay, wherein the second delay can be directly used as the playback time threshold, or the product of the second delay and the set coefficient can be used as the playback delay threshold.
  • the electronic device can also determine the playback time threshold based on the first delay and the second delay, wherein the playback time threshold is positively correlated with the first delay and positively correlated with the second delay, wherein the playback time threshold can be calculated based on the total delay of the first delay and the second delay.
  • the total delay can be directly used as the playback time threshold, or the product of the total delay and the set coefficient can be used as the playback delay threshold. It can be seen that, in this embodiment, the playback duration threshold can be determined according to at least one of the first delay and the second delay.
  • the electronic device can extract a specified number of frames of playback audio data from the storage area as the target playback audio data according to the corresponding storage time from early to late, wherein the specified number can be 1, 2, 3, etc.
  • Step S330 Delete the target playback audio data from the storage area.
  • the electronic device After the electronic device finds the target playback audio data from the storage area, it deletes the target playback audio data from the storage area so that the target playback audio data is not used for echo cancellation processing of subsequent collected audio data.
  • steps S210 to S240 shown in FIG. 3 may refer to steps S210 to S240 shown in FIG. 2 .
  • the target playback audio data to be deleted is extracted from the storage area in order of the corresponding storage time from early to late, and the target playback audio data is deleted from the storage area, which can save storage resources and improve the accuracy of echo cancellation processing.
  • FIG. 4 is a flow chart of an audio data processing method proposed based on the method shown in FIG. 3 .
  • the audio data processing method may further include steps S410 to S420 , which are described in detail as follows:
  • Step S410 Calculate the time difference between the acquisition time and the playback time of the audio data to obtain a first delay.
  • the first delay refers to the playback link delay, that is, the delay from acquiring the playback audio data to playing the playback audio data, which can represent the time difference between the acquisition time and the playback time of the playback audio data.
  • the specific method for calculating the first delay can be flexibly configured based on actual needs.
  • the audio data is played by an audio playback device, which is located in the playback chain and affects the first delay. Therefore, the electronic device can calculate the first delay based on the attribute parameters of the audio playback device.
  • the electronic device can stamp the played audio data with an acquisition timestamp and a playback timestamp, and calculate the difference between the acquisition timestamp and the playback timestamp to obtain the first delay.
  • the first delay may be recalculated and updated every set period.
  • Step S420 Calculate a playback duration threshold according to the first time delay; wherein the playback duration threshold is positively correlated with the first time delay.
  • the electronic device may calculate a playback delay threshold, wherein the longer the first delay is, the longer the playback delay threshold is, and the shorter the first delay is, the shorter the playback delay threshold is.
  • the electronic device may directly use the first delay as the playback delay threshold, or the electronic device may use the product of the first delay and a set coefficient as the playback delay threshold.
  • steps S210 to S240 shown in Figure 4 can refer to steps S210 to S240 shown in Figure 2
  • steps S310 to S330 shown in Figure 4 can refer to steps S310 to S330 shown in Figure 3.
  • the playback duration threshold is determined based on the time difference between the acquisition time and the playback time of the audio data, which can improve the accuracy of the echo cancellation process.
  • FIG. 5 is a flow chart of an audio data processing method proposed based on the method shown in FIG. 4 .
  • the audio data processing method may further include steps S510 to S520, which are described in detail as follows:
  • Step S510 Calculate the time difference between the time of collecting the audio data and the time of performing the echo cancellation process to obtain a second delay.
  • the second delay refers to the acquisition link delay, that is, the delay from the acquisition of the acquired audio data to the echo cancellation processing of the played audio data, which can represent the time difference between the acquisition time of the acquired audio data and the time of echo cancellation processing.
  • the specific method for calculating the second delay can be flexibly set according to actual needs.
  • the electronic device can calculate the second delay based on the attribute parameters of the audio collection device.
  • the electronic device can stamp the collected audio data with a collection timestamp and a processing timestamp (i.e., the time for performing echo cancellation processing), calculate the difference between the collection timestamp and the processing timestamp, and obtain the second delay.
  • the second delay may be recalculated and updated every set period.
  • Step S520 Calculate a playback duration threshold according to the total delay of the first delay and the second delay; wherein the playback duration threshold is positively correlated with the total delay.
  • the total playback time corresponding to the playback audio data contained in the storage area exceeds the total delay of the first delay and the second delay (i.e., the sum of the first delay and the second delay), it indicates that the time difference between the playback time of some of the playback audio data in the storage area and the current time exceeds the total delay, and the probability that the subsequently collected audio data contains the playback audio data is low.
  • the first delay is 3 seconds
  • the second delay is 5 seconds
  • the current time is 11:00:30
  • the subsequent collected audio data that needs to be echo canceled is collected after 11:00:25.
  • the collected audio data will only include the audio data played after 11:00:25.
  • the collected audio data will only include the audio data played after 11:00:22. That is, the time difference between the acquisition time of the useful audio data and the current time is less than or equal to 8 seconds (i.e., the total delay). Therefore, if the total playback time of the audio data contained in the storage area exceeds 8 seconds, it indicates that the time difference between the acquisition time of some audio data and the current time is greater than 8 seconds, and the storage area contains useless audio data. Therefore, after calculating the first delay and the second delay, the electronic device can calculate the playback time threshold based on the total delay of the first delay and the second delay, wherein the longer the total delay, the larger the playback time threshold, and the shorter the total delay, the smaller the playback time threshold.
  • the electronic device may directly use the total delay as the playback duration threshold, or may use the product of the total delay and a set coefficient as the playback delay threshold.
  • steps S210 to S240 shown in Figure 5 can refer to steps S210 to S240 shown in Figure 2
  • the specific implementation details of steps S310 to S330 shown in Figure 5 can refer to steps S310 to S330 shown in Figure 3
  • the specific implementation details of step S410 shown in Figure 5 can refer to step S410 shown in Figure 4.
  • the playback duration threshold is determined based on the time difference between the acquisition time and the playback time of the audio data, and the time difference between the acquisition time of the audio data and the time of performing the echo cancellation processing, thereby improving the accuracy of the echo cancellation processing.
  • FIG. 6 is a flow chart of an audio data processing method proposed based on the method shown in FIG. 2 .
  • the audio data processing method further includes steps S610 to S620, which are described in detail as follows:
  • Step S610 searching for the earliest stored audio data from the multiple frames of audio data stored in the storage area.
  • the storage area contains multiple frames of playback audio data, and the frame of playback audio data with the earliest storage time can be searched therefrom, that is, the playback audio data stored earliest in the storage area among the multiple frames of playback audio data contained in the storage area. For example, assuming that the storage area contains 3 frames of playback audio data, which are stored in the storage area at 11:00:01, 11:00:02, and 11:00:03 respectively, then the playback audio data stored in the storage area at 11:00:01 is the playback audio data with the earliest storage time.
  • Step S620 If the storage duration of the audio data with the earliest storage time in the storage area is greater than or equal to the storage duration threshold, the audio data with the earliest storage time is deleted from the storage area.
  • the storage duration of the played audio data in the storage area represents the length of time from the time the played audio data is added to the storage area to the current time. If the storage duration of the played audio data is too long, it means that the time difference between the playback time of the played audio data and the current time is too long. The probability that the collected audio data that needs to be echo eliminated later contains the played audio data is low. Therefore, a storage duration threshold is set. If the storage duration of the earliest played audio data in the storage area is greater than or equal to the storage duration threshold, the played audio data is deleted.
  • the specific value of the storage duration threshold can be flexibly set according to actual needs, for example, it can be set to 10 milliseconds, 20 milliseconds, etc. based on experience; or it can be determined based on at least one of the first delay and the second delay.
  • the specific method of determining the storage duration threshold based on at least one of the first delay and the second delay is similar to the method of determining the playback duration threshold based on at least one of the first delay and the second delay.
  • the storage duration threshold is the same as the playback duration threshold.
  • steps S210 to S240 shown in FIG6 may refer to steps S210 to S240 shown in FIG2 .
  • the corresponding played audio data with a longer storage duration is deleted from the storage area, which can save storage resources and improve the accuracy of echo cancellation.
  • FIG. 7 is a flow chart of an audio data processing method proposed based on the method shown in FIG. 2 .
  • step S210 may include steps S710 to S730, which are described in detail as follows:
  • Step S710 Acquire audio data to be played.
  • the type of played audio data is different, and the method of obtaining the played audio data is different. For example, if the played audio data is music stored locally, the electronic device searches for the played audio data from the local storage. If the played audio data is audio data collected by the target communication end during voice communication with the target communication end, the electronic device receives the played audio data from the target communication end during the voice communication process.
  • Step S720 Detect the working status of the audio acquisition device and obtain a status detection result.
  • the electronic device does not need to store the played audio data in the storage area. Therefore, after obtaining the played audio data, it is still necessary to detect the working status of the audio acquisition device to obtain a status detection result indicating whether the audio acquisition device is collecting audio data.
  • Step S730 If the status detection result indicates that the audio acquisition device is acquiring audio data, the acquired playback audio data is stored in the storage area.
  • the electronic device stores the playback audio data in the storage area.
  • the electronic device can detect whether the audio collection device collects audio data within a period of time starting from the time when the playback audio data is obtained and lasting for a set time.
  • the set time can be positively correlated with the first delay.
  • the first delay can be directly used as the set time, or the sum of the first delay and the obtained playback time of the playback audio data can be used as the set time.
  • steps S220 to S240 shown in FIG. 7 may refer to steps S220 to S240 shown in FIG. 2 .
  • the acquired audio data to be played is stored in the storage area only when the audio acquisition device is acquiring audio data, which not only saves storage resources but also improves the accuracy of the echo cancellation process.
  • FIG. 8 is a flow chart of an audio data processing method proposed based on the method shown in FIG. 2 .
  • steps S810 to S820 may be included, which are described in detail as follows:
  • Step S810 If the storage area does not contain the playback audio data, generate silent audio data whose corresponding playback duration matches the playback duration of the collected audio data.
  • the electronic device After obtaining the collected audio data, if the storage area is empty, that is, the storage area does not contain the played audio data, in order to adapt to the echo cancellation processing process, the electronic device can generate silent audio data whose playback duration matches the playback duration of the collected audio data. That is, the silent audio data does not contain any sound, and the playback duration of the silent audio data matches the playback duration of the collected audio data.
  • Step S820 performing echo cancellation processing on the collected audio data according to the silent audio data.
  • the electronic device uses the silent audio data as reference data and performs echo cancellation processing on the collected audio data.
  • steps S210 to S240 shown in FIG. 8 may refer to steps S210 to S240 shown in FIG. 2 .
  • the electronic device after obtaining the collected audio data, if the storage area does not contain the played audio data, the electronic device generates silent audio data whose corresponding playback duration matches the playback duration of the collected audio data, as reference data for performing echo cancellation processing on the collected audio data, thereby eliminating the need to change the echo cancellation processing process and saving processing resources.
  • FIG. 9 is a flow chart of an audio data processing method proposed based on the method shown in FIG. 2 .
  • step S230 may include step S910, which is described in detail as follows:
  • Step S910 If a set number of frames of audio data are collected, the set number of frames of audio data are extracted from the storage area in the order of the corresponding storage time from earliest to latest.
  • a set number can be set, where the set number is an integer greater than or equal to 1, for example, it can be set to 1, 2, 3, etc., and its specific value can be negatively correlated with the real-time performance requirement of playing audio data. That is, if the real-time performance requirement of playing audio data is high, the set number is lower; if the real-time performance requirement of playing audio data is low, the set number can be higher.
  • the electronic device can trigger an echo cancellation processing flow for the collected audio data.
  • the echo cancellation processing multiple frames of playback audio data matching the set number are first extracted from the storage area in the order of the corresponding storage time from earliest to latest. That is, the set number of frames of playback audio data are extracted. For example, assuming the set number is 1, after collecting one frame of collected audio data, the frame of playback audio data with the earliest storage time can be extracted from the storage area.
  • step S240 may include step S920, which is described in detail as follows:
  • Step S920 playing audio data according to the extracted set number of frames, and performing echo cancellation processing on the collected audio data of the set number of frames.
  • the electronic device After extracting a set number of frames of playback audio data, the electronic device plays the audio data according to the set number of frames and performs echo cancellation processing on the audio data collected by the set number of frames. That is, the amount of playback audio data and collection audio data for each echo cancellation processing is the set number of frames, thereby avoiding the accumulation of playback audio data that needs to be processed by echo cancellation, reducing the efficiency of echo cancellation processing and the real-time performance of playback audio data.
  • steps S210 to S220 shown in FIG. 9 may refer to steps S210 to S220 shown in FIG. 2 .
  • the amount of audio data played and audio data collected for each echo cancellation process is a set number of frames, thereby avoiding accumulation of audio data that requires echo cancellation processing, improving the efficiency of echo cancellation processing and the real-time performance of audio data played.
  • FIG. 10 is a flow chart of an audio data processing method proposed based on the method shown in FIG. 2 .
  • step S230 may include step S1010, which is described in detail as follows:
  • Step S1010 extracting, from the head of the message queue, the playback audio data whose corresponding playback duration matches the playback duration of the collected audio data.
  • the message queue adopts a first-in-first-out mode, which only allows data to be inserted from the end of the queue and extracted from the head of the queue.
  • the playback audio data is stored in the message queue, and the storage time of the playback audio data at the head of the queue is earlier than the storage time of the playback audio data at the end of the queue.
  • the electronic device can extract the playback audio data in sequence starting from the head of the message queue until the playback time of the extracted playback audio data matches the playback time of the collected audio data, thereby realizing the extraction of playback audio data from the storage area in the order of the corresponding storage time from early to late.
  • steps S210 to S220 and step S240 shown in FIG10 may refer to steps S210 to S220 and step S240 shown in FIG2 .
  • storing the played audio data in the message queue can improve the efficiency of extracting the played audio data from the message queue.
  • FIG. 11 is a flow chart of an audio data processing method proposed based on the method shown in FIG. 2 .
  • the audio data processing method may further include step S1110, which is described in detail as follows:
  • Step S1110 sending the collected audio data after the echo cancellation processing to an external communication terminal.
  • the external communication terminal refers to a communication terminal other than the electronic device serving as the local terminal, which is capable of conducting voice communication with the local terminal, and its types include but are not limited to terminal devices, servers, etc., wherein, for the specific introduction of terminal devices and servers, please refer to the above records.
  • the audio acquisition device of the local terminal will play the sound from the external communication terminal during the process of collecting sound, so that the sound collected by the local terminal contains the sound from the external communication terminal. If the sound collected by the local terminal is directly sent to the external communication terminal for playback, it will cause the user of the external communication terminal to hear his own voice, that is, echo.
  • voice communication includes but is not limited to instant voice communication and non-instant voice communication, wherein, instant voice communication includes but is not limited to voice calls, video calls, live broadcasts, online meetings, etc.
  • the audio data processing method of this embodiment can be applied to voice communication scenarios to eliminate echoes in common voice scenarios.
  • the played audio data includes audio data sent by an external communication terminal during the voice communication process, which may be audio data collected by an audio collection device corresponding to the external communication terminal; the collected audio data includes audio data collected by the audio collection device of the local terminal during the voice communication process, which needs to be sent to the external communication terminal. Therefore, after performing echo cancellation processing on the collected audio data, the collected audio data after echo cancellation processing can be sent to the external communication terminal.
  • the played audio data may further include other audio data played by the local end during the voice communication process, except for the audio data from the external communication end.
  • steps S210-S240 shown in FIG11 can refer to steps S210-S240 shown in FIG2 .
  • the audio data processing method can also be applied to other scenarios requiring echo cancellation. For example, if a user records their own voice while playing music, the recording contains not only the user's voice but also the music being played, resulting in a relatively unclear user voice in the recording. Therefore, the played music can be used as the playback audio data, and the recording as the collected audio data. Using the audio data processing method, echo cancellation can be performed to remove the music from the recording, improving the clarity of the user's voice in the recording.
  • the audio collected by the conference member's terminal contains not only the conference member's voice but also the host's voice, resulting in a relatively unclear voice in the audio collected by the conference member's terminal. Therefore, the host's voice can be used as the playback audio data, and the audio collected by the conference member's terminal as the collected audio data.
  • echo cancellation can be performed to remove the conference host's voice from the audio, improving the clarity of the conference member's voice.
  • the audio data processing method is used to perform echo cancellation processing on the audio data that needs to be sent to the external communication terminal during the voice communication process, thereby improving the voice communication quality.
  • FIG. 12 is a flow chart of an audio data processing method proposed based on the method shown in FIG. 2 .
  • step S210 may include steps S1210 to S1230, which are described in detail as follows:
  • Step S1210 Acquire audio data to be played.
  • the specific method for obtaining the played audio data may refer to the description of the aforementioned embodiment.
  • Step S1220 If the storage area does not contain the playback audio data, the storage time corresponding to the acquired playback audio data is calculated according to the waiting time and the acquisition time corresponding to the acquired playback audio data; wherein the storage time is later than the acquisition time.
  • the playback audio data After obtaining the playback audio data to be played, if the storage area does not contain the playback audio data, if the playback audio data is immediately added to the storage area, the playback audio data will be used for echo cancellation processing of the currently obtained collected audio data. Due to the existence of the first delay and the second delay, the probability that the currently obtained collected audio data contains the playback audio data is small. For example, assuming that the first delay is 1 second and the second delay is 2 seconds, the storage area does not contain the playback audio data, and the playback audio data is obtained at 11:30:20, then the playback audio data will be played at 11:30:21, and the collected audio data that needs to be echo cancelled at 11:30:20 is collected at 11:30:18.
  • the waiting period can be determined based on at least one of the first delay and the second delay.
  • the waiting period can be positively correlated with the first delay, positively correlated with the second delay, or positively correlated with the total delay of the first delay and the second delay.
  • any one of the first delay, the second delay, and the total delay can be used as the waiting period.
  • Step S1230 storing the acquired playback audio data in a storage area according to the storage time.
  • the electronic device stores the acquired playback audio data in the storage area according to the calculated storage time. That is, after acquiring the playback audio data to be played, if the storage area does not contain the playback audio data, the electronic device can add the playback audio data to the storage area after waiting for a certain period of time.
  • the electronic device may directly add the to-be-played audio data to the storage area.
  • steps S220 to S240 shown in FIG. 12 may refer to steps S220 to S240 shown in FIG. 2 .
  • the playback audio data is added to the storage area after waiting for a certain period of time, so that the playback audio data is aligned with the collected audio data, thereby improving the accuracy of the echo cancellation processing.
  • Figure 13 is a schematic diagram of an implementation environment involved in the present application, which includes a first terminal device and a second terminal device.
  • the audio data processing method is executed by the first terminal device, and the second terminal device is an external terminal device for instant voice communication with the first terminal device.
  • the first terminal device includes a speaker, a microphone, a data feeding module, and an echo cancellation module.
  • the first terminal device will receive external voice data from the second terminal device, wherein the external voice data refers to the voice data collected by the second terminal device during the instant voice communication.
  • the first terminal device After receiving the external voice data, the first terminal device can play the external voice data through the speaker contained in it, that is, the external voice data is the playback audio data of the first terminal device, and the first terminal device can send the external voice data to the data feeding module.
  • the microphone of the first terminal device will also collect the user's voice to obtain local voice data.
  • the local voice data Since the speaker is playing the external voice data during the process of collecting the user's voice, the local voice data may contain external voice data.
  • the local voice data can be transmitted to the data feeding module.
  • the data feeding module feeds the local voice data and the external voice data to the echo cancellation module.
  • the echo cancellation module performs echo cancellation processing on the local voice data according to the fed external voice data to obtain outgoing voice data, and sends the outgoing voice data to the second terminal device.
  • the data feeding module includes a message queue and a data feeding decision module, wherein the message queue is used to cache external voice data, and the data feeding decision module is used to determine how to feed external voice data and local voice data to the echo cancellation module.
  • the data feeding module can first determine whether the echo cancellation function is enabled. If the echo cancellation function is not enabled, the external voice data is not needed for echo cancellation, and therefore, the external voice data is deleted. If the echo cancellation function is enabled, it indicates that the external voice data needs to be cached as reference data for subsequent echo cancellation. Therefore, the external voice data is added to the message queue. Because the duration of the external voice data contained in the message queue is too long, it indicates that some of the external voice data was played too early, and the probability of including the external voice data in the local voice data that needs to be processed as an echo message is low.
  • the data feeding module can also obtain the total delay and determine whether the total playback duration of the multiple frames of external voice data contained in the message queue is greater than the total delay. If the total playback duration is greater than the total delay, the external voice data frame at the head of the message queue is deleted, that is, the external voice data frame with the oldest storage time in the message queue is deleted. If the total playback duration is less than or equal to the total delay, it is re-determined whether the total playback duration of the multiple frames of external voice data contained in the message queue is greater than the total delay.
  • the total delay refers to the sum of the playback link delay and the acquisition link delay
  • the playback link refers to the transmission link of external voice data from the receipt of external voice data to the playback of external voice through the speaker.
  • the link includes a speaker, and the delay of this link is the playback link delay. That is, the playback link delay refers to the delay from the receipt of external voice data to the playback of external voice data.
  • the acquisition link refers to the transmission link of local voice data from the collection of local voice data to the echo cancellation processing of local voice data.
  • the link includes a microphone.
  • the acquisition link may include the transmission link of local voice data from the input end of the microphone to the input end of the data feeding module.
  • the delay of this link is the acquisition link delay. That is, the acquisition link delay refers to the delay from the collection of local voice data to the echo cancellation processing of local voice data.
  • the data feeding decision module can obtain a frame of reference voice data and feed the frame of reference voice data and the frame of local voice data to the echo cancellation module for processing, wherein the reference voice data can be silent voice data or from the message queue.
  • the data feeding decision module can determine whether the message queue contains external voice data. If so, it extracts a frame of external voice data from the head of the message queue, that is, extracts the frame of external voice data with the earliest storage time, and uses the external voice data as the reference voice data.
  • a frame of silent voice data can be generated, and the silent voice data can be used as reference voice data.
  • the reference voice data and local voice data are fed to the echo message module.
  • the echo cancellation module performs echo cancellation processing on the local voice data through the reference voice data to obtain the outgoing voice data output by the echo cancellation module. In this way, the outgoing voice data will not contain the sound from the second terminal device, and the outgoing voice data will be sent to the second terminal device for playback, and the user of the second terminal device will not hear his own echo.
  • the feeding is strictly carried out according to one frame of playback audio data and one frame of acquisition audio data, thereby achieving alignment of the acquisition audio data and the playback audio data, and improving the effect of echo cancellation.
  • the acquisition link delay and the playback link delay are used to manage the playback audio data contained in the message queue, so that too old playback audio data can be safely deleted without worrying about the problem of echo leakage, saving storage resources and further improving the effect of echo cancellation.
  • Figure 18 is a block diagram of an audio data processing device according to an exemplary embodiment of the present application. As shown in Figure 18, the device includes:
  • the acquisition module 1801 is configured to acquire the audio data to be played and store the acquired audio data in a set storage area;
  • the processing module 1802 is configured to play the acquired playback audio data and collect audio data through an audio collection device to obtain collected audio data;
  • the extraction module 1803 is configured to extract the playback audio data whose corresponding playback duration matches the playback duration of the collected audio data from the storage area in the order of the corresponding storage time from earliest to latest;
  • the cancellation module 1804 is configured to perform echo cancellation processing on the collected audio data according to the extracted played audio data.
  • the apparatus further includes a first deletion module configured to:
  • the target playback audio data to be deleted is extracted from the storage area in the order of the corresponding storage time from earliest to latest;
  • the apparatus further includes a computing module configured to:
  • a playback duration threshold is calculated based on the first time delay; wherein the playback duration threshold is positively correlated with the first time delay.
  • the calculation module is specifically configured as follows:
  • a playback time threshold is calculated according to the total delay of the first delay and the second delay; wherein the playback time threshold is positively correlated with the total delay.
  • the apparatus further includes a second deletion module configured to:
  • the playback audio data with the earliest storage time in the storage area is greater than or equal to the storage time threshold, the playback audio data with the earliest storage time is deleted from the storage area.
  • the acquisition module 1801 is specifically configured as follows:
  • the acquired playback audio data is stored in the storage area.
  • the apparatus further includes a generating module configured to:
  • the storage area does not contain the playback audio data, generating silent audio data whose corresponding playback duration matches the playback duration of the collected audio data;
  • the extraction module 1803 is specifically configured as follows:
  • the set number of frames of audio data are collected, the set number of frames of audio data are extracted from the storage area in the order of the corresponding storage time from earliest to latest, and the audio data is played;
  • echo cancellation processing is performed on the collected audio data, including:
  • the audio data is played according to the extracted set number of frames, and the echo cancellation processing is performed on the collected audio data of the set number of frames.
  • the extraction module 1803 is specifically configured as follows:
  • the playback audio data whose corresponding playback duration matches the playback duration of the collected audio data is extracted from the head of the message queue.
  • the apparatus under the condition that the played audio data includes audio data sent by the external communication terminal during voice communication with the external communication terminal, and the collected audio data includes audio data collected during the voice communication, the apparatus further includes a sending module configured to:
  • the collected audio data after echo cancellation processing is sent to the external communication terminal.
  • the acquisition module 1801 is specifically configured as follows:
  • the storage area does not contain the playback audio data, then calculating the storage time corresponding to the acquired playback audio data according to the waiting time and the acquisition time corresponding to the acquired playback audio data; wherein the storage time is later than the acquisition time;
  • the acquired playback audio data is stored in the storage area.
  • An embodiment of the present application also provides an electronic device, comprising: one or more processors; a storage device for storing one or more computer-readable instructions, which, when the one or more computer-readable instructions are executed by the one or more processors, enables the electronic device to implement the audio data processing method provided in the above-mentioned embodiments.
  • FIG19 shows a schematic structural diagram of a computer system of an electronic device suitable for implementing an embodiment of the present application.
  • the electronic device may be the terminal device 110 or the server 120 shown in FIG1 .
  • computer system 1900 includes a central processing unit (CPU) 1901, which can perform various appropriate actions and processes, such as executing the audio data processing method described in the above-described embodiment, based on computer-readable instructions stored in a read-only memory (ROM) 1902 or loaded from a storage unit 1908 into a random access memory (RAM) 1903.
  • ROM read-only memory
  • RAM random access memory
  • CPU 1901, ROM 1902, and RAM 1903 are interconnected via a bus 1904.
  • An input/output (I/O) interface 1905 is also connected to bus 1904.
  • I/O interface 1905 an input section 1906 including a keyboard, a mouse, and the like; an output section 1907 including devices such as a cathode ray tube (CRT), a liquid crystal display (LCD), and a speaker; a storage section 1908 including a hard disk; and a communication section 1909 including a network interface card such as a LAN (Local Area Network) card or a modem. Communication section 1909 performs communication processing via a network such as the Internet.
  • a drive 1910 is also connected to I/O interface 1905 as needed.
  • Removable media 1911 such as a magnetic disk, an optical disk, a magneto-optical disk, or a semiconductor memory, is installed in drive 1910 as needed, so that computer-readable instructions read from the media can be installed in storage section 1908 as needed.
  • the computer system 1900 includes an audio playback device and an audio collection device.
  • the audio playback device may be, for example, a speaker, a loudspeaker, etc.
  • the audio collection device may be, for example, a microphone, etc.
  • computer-readable instructions for implementing the audio data processing method can be carried on a computer-readable medium, which can be downloaded and installed from the network through the communication part 1909, and/or installed from the removable medium 1911.
  • the computer-readable medium shown in the embodiments of the present application may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two.
  • the computer-readable storage medium may be, for example, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or component, or any combination of the above.
  • Computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM), a flash memory, an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.
  • a computer-readable storage medium may be any tangible medium containing or storing computer-readable instructions that can be used by or in combination with an instruction execution system, device or device.
  • a computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable instructions.
  • Such a propagated data signal may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof.
  • the computer-readable instructions contained in a computer-readable medium may be transmitted using any suitable medium, including but not limited to wireless, wired, or any suitable combination thereof.
  • each box in the flowchart or block diagram can represent a module, program segment, or a part of the code, and the above-mentioned module, program segment, or a part of the code contains one or more executable instructions for implementing the specified logical function.
  • the functions marked in the box can also occur in an order different from that marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, which depends on the functions involved.
  • each box in the block diagram or flowchart, and the combination of boxes in the block diagram or flowchart can be implemented with a dedicated hardware-based system that performs the specified function or operation, or can be implemented with a combination of dedicated hardware and computer-readable instructions.
  • the units involved in the embodiments described in this application may be implemented by software or hardware, and the units described may also be set in a processor. In some cases, the names of these units do not constitute limitations on the units themselves.
  • Another aspect of the present application provides a non-volatile computer-readable storage medium having computer-readable instructions stored thereon.
  • the computer-readable instructions When executed by a processor of an electronic device, the electronic device implements the aforementioned audio data processing method.
  • the computer-readable storage medium may be included in the electronic device described in the above embodiments, or may exist independently and not be incorporated into the electronic device.
  • the computer-readable instructions may be stored in a computer-readable storage medium.
  • the computer program product may be a product, such as an APP (Application), a webpage, or a mini-program; or, the computer program product may be a storage medium, device, terminal, or virtual machine containing computer-readable instructions.

Landscapes

  • Engineering & Computer Science (AREA)
  • Human Computer Interaction (AREA)
  • Theoretical Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • Health & Medical Sciences (AREA)
  • Audiology, Speech & Language Pathology (AREA)
  • Signal Processing (AREA)
  • Quality & Reliability (AREA)
  • Acoustics & Sound (AREA)
  • Multimedia (AREA)
  • General Health & Medical Sciences (AREA)
  • General Engineering & Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Computational Linguistics (AREA)
  • Circuit For Audible Band Transducer (AREA)

Abstract

一种音频数据处理方法，包括：获取到待播放的播放音频数据后，将获取到的播放音频数据存储至设定的存储区域(S210)，播放获取到的播放音频数据，并通过音频采集装置采集音频数据，得到采集音频数据(S220)；然后，按照所对应的存储时间由早到晚的顺序，从存储区域中提取所对应的播放时长与采集音频数据的播放时长相匹配的播放音频数据(S230)，从而根据提取出的播放音频数据，对采集音频数据进行回声消除处理(S240)。

Description

音频数据处理方法、装置、电子设备、介质和程序产品
本申请要求于2024年03月12日提交中国专利局，申请号为202410294566.3，申请名称为“音频数据处理方法及装置、电子设备、存储介质”的中国专利申请的优先权，其全部内容通过引用结合在本申请中。
技术领域
本申请涉及计算机领域，具体而言，涉及一种音频数据处理方法、装置、电子设备、存储介质和程序产品。
背景技术
在音频采集装置(例如，麦克风等)和音频播放装置(例如，扬声器等)同时工作的条件下，音频播放装置播放的声音通过空气传播之后进入音频采集装置，被音频采集装置采集到，从而形成了回声。为了使音频采集装置采集到的音频数据中不包含音频播放装置播放的声音，可以对音频采集装置采集到的音频数据进行回声消除。
相关技术中提供的回声消除方式，处理后的音频数据还会残留大量的回声，降低了回声消除处理准确性以及音频数据质量。
发明内容
根据本申请的各种实施例，提供了一种音频数据处理方法，所述方法包括：
获取待播放的播放音频数据，将获取到的播放音频数据存储至设定的存储区域；
播放获取到的播放音频数据，并通过音频采集装置采集音频数据，得到采集音频数据；
按照所对应的存储时间由早到晚的顺序，从所述存储区域中提取所对应的播放时长与所述采集音频数据的播放时长相匹配的播放音频数据；
根据提取出的播放音频数据，对所述采集音频数据进行回声消除处理。
根据本申请的各种实施例，提供一种音频数据处理装置，所述装置包括：
获取模块，配置为获取待播放的播放音频数据，将获取到的播放音频数据存储至设定的存储区域；
处理模块，配置为播放获取到的播放音频数据，并通过音频采集装置采集音频数据，得到采集音频数据；
提取模块，配置为按照所对应的存储时间由早到晚的顺序，从所述存储区域中提取所对应的播放时长与所述采集音频数据的播放时长相匹配的播放音频数据；
消除模块，配置为根据提取出的播放音频数据，对所述采集音频数据进行回声消除处理。
根据本申请的各种实施例，提供一种电子设备，包括：
一个或多个处理器；
存储装置，用于存储一个或多个计算机可读指令，当所述一个或多个计算机可读指令被所述一个或多个处理器执行时，使得所述电子设备实现如前所述的方法。
根据本申请的各种实施例，提供一种计算机可读存储介质，其上存储有计算机可读指令，当所述计算机可读指令被电子设备的处理器执行时，使电子设备实现如前所述的方法。
根据本申请的各种实施例，提供一种计算机程序产品，包括计算机可读指令，所述计算机可读指令被处理器执行时实现如前所述的方法。
本申请的一个或多个实施例的细节在下面的附图和描述中提出。本申请的其它特征、目的和优点将从说明书、附图以及权利要求书变得明显。
附图说明
为了更清楚地说明本申请实施例中的技术方案，下面将对实施例描述中所需要使用的附图作简单地介绍，显而易见地，下面描述中的附图仅仅是本申请的一些实施例，对于本领域普通技术人员来讲，在不付出创造性劳动的前提下，还可以根据这些附图获得其他的附图。
图1是本申请的一示例性实施例示出的实施环境的示意图；
图2是本申请的一示例性实施例示出的音频数据处理方法的流程图；
图3是本申请的另一示例性实施例示出的音频数据处理方法的流程图；
图4是本申请的另一示例性实施例示出的音频数据处理方法的流程图；
图5是本申请的另一示例性实施例示出的音频数据处理方法的流程图；
图6是本申请的另一示例性实施例示出的音频数据处理方法的流程图；
图7是本申请的另一示例性实施例示出的音频数据处理方法的流程图；
图8是本申请的另一示例性实施例示出的音频数据处理方法的流程图；
图9是本申请的另一示例性实施例示出的音频数据处理方法的流程图；
图10是本申请的另一示例性实施例示出的音频数据处理方法的流程图；
图11是本申请的另一示例性实施例示出的音频数据处理方法的流程图；
图12是本申请的另一示例性实施例示出的音频数据处理方法的流程图；
图13是本申请的另一示例性实施例示出的实施环境的示意图；
图14是本申请的一示例性实施例示出的数据投喂模块的示意图；
图15是本申请的一示例性实施例示出的语音数据缓存过程的流程图；
图16是本申请的一示例性实施例示出的数据投喂流程的框图；
图17是本申请的一示例性实施例示出的数据投喂过程的流程图；
图18是本申请的一示例性实施例示出的音频数据处理装置的结构图；
图19示出了适于用来实现本申请实施例的电子设备的计算机系统的结构示意图。
具体实施方式
现在参考附图以更全面的方式描述示例实施方式。然而，示例的实施方式能够以各种形式实施，且不应被理解为仅限于这些范例；相反，提供这些实施方式的目的是使得本申请更加全面和完整，并将示例实施方式的构思全面地传达给本领域的技术人员。
此外，本申请所描述的特征、结构或特性可以以任何合适的方式结合在一个或更多实施例中。在下面的描述中，有许多具体细节从而可以充分理解本申请的实施例。然而，本领域技术人员应意识到，在实施本申请的技术方案时可以不需用到实施例中的所有细节特征，可以省略一个或更多特定细节，或者可以采用其它的方法、元件、装置、步骤等。
本申请实施例中，术语“模块”或“单元”是指有预定功能的计算机可读指令或计算机可读指令的一部分，并与其他相关部分一起工作以实现预定目标，并且可以通过使用软件、硬件(如处理电路或存储器)或其组合来全部或部分实现。同样的，一个处理器(或多个处理器或存储器)可以用来实现一个或多个模块或单元。此外，每个模块或单元都可以是包含该模块或单元功能的整体模块或单元的一部分。
附图中所示的方框图仅仅是功能实体，不一定必须与物理上独立的实体相对应。即，可以采用软件形式来实现这些功能实体，或在一个或多个硬件模块或集成电路中实现这些功能实体，或在不同网络和/或处理器装置和/或微控制器装置中实现这些功能实体。
附图中所示的流程图仅是示例性说明，不是必须包括所有的内容和操作/步骤，也不是必须按所描述的顺序执行。例如，有的操作/步骤还可以分解，而有的操作/步骤可以合并或部分合并，因此实际执行的顺序有可能根据实际情况改变。
还需要说明的是：在本申请中提及的“多个”是指两个或者两个以上。“和/或”描述关联对象的关联关系，表示可以存在三种关系，例如，A和/或B可以表示：单独存在A，同时存在A和B，单独存在B这三种情况。字符“/”一般表示前后关联对象是一种“或”的关系。
以下对本申请实施例的技术方案进行详细介绍：
相关技术中，回声消除处理的准确性较低，降低了音频数据质量。基于此，本申请的实施例提供了一种音频数据处理方法及装置、电子设备、存储介质、程序产品，可以提升回声消除处理准确性以及音频数据质量。
请参阅图1，图1是本申请涉及的一种实施环境的示意图，该实施环境包括终端设备110和服务器120。终端设备110和服务器120之间通过有线或者无线网络进行通信，终端设备110可以将自身的数据上传至服务器120，也可以从服务器120获取数据。
其中，终端设备110可以包括但不限于手机、平板、笔记本电脑、计算机、语音交互设备、家电、车载终端、飞行器、远程驾驶终端、支付设备(例如，刷掌设备等)等等；服务器120可以是独立的物理服务器，也可以是多个物理服务器构成的服务器集群或者分布式系统，还可以是提供云服务、云数据库、云计算、云函数、云存储、网络服务、云通信、中间件服务、域名服务、安全服务、CDN(Content Delivery Network，内容分发网络)以及大数据和人工智能平台等基础云计算服务的云服务器，本处不对终端设备和服务器的具体形式进行限制。
需要说明的是，图1中的终端设备110以及服务器120的数量仅仅是示意性的，根据实际需要，可以具有任意数量的终端设备110以及服务器120。
在一示例性实施例中，本申请的实施例提供的音频数据处理方法可以由终端设备110执行。示例性的，终端设备110可以获取到待播放的播放音频数据后，将获取到的播放音频数据存储至设定的存储区域，播放获取到的播放音频数据，并通过音频采集装置采集音频数据，得到采集音频数据；然后，按照所对应的存储时间由早到晚的顺序，从存储区域中提取所对应的播放时长与采集音频数据的播放时长相匹配的播放音频数据，从而根据提取出的播放音频数据，对采集音频数据进行回声消除处理，也就是说，将播放音频数据作为对采集音频数据进行回声消除处理的参考数据，可以提升回声消除处理的准确性，并且，先将播放音频数据存储在存储区域中，后续得到采集音频数据后，再从存储区域中用于回声消除的播放音频数据，从而保障有足量的播放音频数据以供回声消除，使得后续可以按照所对应的存储时间由早到晚的顺序，提取出所对应的播放时长与采集音频数据的播放时长相匹配的播放音频数据，相比相关技术中在获取到播放音频数据和采集音频数据后，直接根据获取到的播放音频数据对采集音频数据进行回声消除，这种方式由于获取到的播放音频数据和采集音频数据的播放时长可能不匹配，使得采集音频数据与播放音频数据的对齐容易出错，并且，若获取到的播放音频数据的播放时长与播放音频数据的播放时长之间的差值大于一定阈值时，会删除部分播放音频数据，导致后续获取到的采集音频数据没有参考数据，降低了回声消除准确性的方案(例如，某个时刻可能获取到7帧播放音频数据和3帧采集音频数据，从7帧播放音频数据中提取出的、用于采集音频数据的回声消除的3帧播放音频数据可能与采集音频数据无关，并且，对剩余4帧播放音频数据进行删除后，会导致后续获取到的采集音频数据缺少作为回声消除参考数据的播放音频数据)，本申请的实施例提供的方案可以提升提取出的播放音频数据与采集音频数据在时间和时长上的匹配度，提升用于回声消除的播放音频数据的准确度，从而提升了回声消除处理的准确性，进而提升了音频数据的质量。
在另一示例性实施例中，服务器120可以具有与终端设备110相似的功能，从而执行本申请实施例提供的音频数据处理方法。示例性的，服务器120可以获取到待播放的播放音频数据后，将获取到的播放音频数据存储至设定的存储区域，播放获取到的播放音频数据，并通过音频采集装置采集音频数据，得到采集音频数据；然后，按照所对应的存储时间由早到晚的顺序，从存储区域中提取所对应的播放时长与采集音频数据的播放时长相匹配的播放音频数据，从而根据提取出的播放音频数据，对采集音频数据进行回声消除处理。
在另一示例性实施例中，终端设备110和服务器120还可以共同执行本申请的实施例提供的音频数据处理方法。示例性的，终端设备110可以获取待播放的播放音频数据，以播放获取到的播放音频数据，并将获取到的播放音频数据发送至服务器120，使得服务器120将来自终端设备110的播放音频数据存储至设定的存储区域；终端设备110还可以通过音频采集装置采集音频数据，得到采集音频数据，并将采集音频数据传输至服务器120，服务器120接收到来自终端设备110的采集音频数据之后，可以按照所对应的存储时间由早到晚的顺序，从存储区域中提取所对应的播放时长与采集音频数据的播放时长相匹配的播放音频数据，从而根据提取出的播放音频数据，对采集音频数据进行回声消除处理。
本申请实施例中的音频数据处理方法可以应用于各种需要进行回声消除处理的场景，例如，语言通话、视频通话、直播互动、在线会议等等。本申请的实施例中涉及到播放音频数据、采集音频数据等与用户相关的数据，在本申请的方法应用于具体产品或技术中时，其均为获得用户许可或者同意，且相关的数据的提取、使用和处理均遵守当地安全标准和当地法律法规的规定。
参见图2，图2是本申请的一示例性实施例示出的一种音频数据处理方法的流程图。该方法可以由电子设备执行。该方法可以应用于图1所示的实施环境，具体地，电子设备可以是图1所示的实施环境中的终端设备110，也可以是图1所示的实施环境中的服务器120，或者，也可以由图1所示的实施环境中的终端设备110和服务器120共同执行该方法。
如图2所示，在一示例性实施例中，音频数据处理方法可以包括步骤S210-步骤S240，详细介绍如下：
步骤S210，获取待播放的播放音频数据，将获取到的播放音频数据存储至设定的存储区域。
需要说明的是，播放音频数据是指需要进行播放的音频数据，其类型包括但不限于语音通信过程中的音频数据、音乐、视频中包含的音频等，其中，语音通信包括但不限于语音通话、直播互动等通信场景。
设定的存储区域用于存储播放音频数据，以用于后续回声消除处理，其中，存储区域的具体形式可以根据实际需要灵活设置，包括但不限于缓存、消息队列等。需要说明的是，回声消除(Acoustic Echo Cancelling，AEC)是指消除音频采集装置和音频播放装置因空气传播产生回收路径(feedback path)而产生的杂音，也就是说，在音频采集装置采集到的音频中，消除音频播放装置播放的音频。其中，音频采集装置用于采集所处环境中的声音，其类型包括但不限于麦克风等，音频播放装置用于播放音频，其类型包括但不限于扬声器、喇叭等。
在一些情形中，电子设备可以同时播放音频和采集音频。在电子设备的音频播放装置播放获取到的播放音频数据的过程中，若电子设备的音频采集装置正在采集音频数据，则会使得音频采集装置直接或间接采集到该播放音频数据，例如，播放出的声音通过房间等反射后，间接被采集到。从而导致音频采集装置采集到的音频数据中，不仅包含所处环境中的声音，还包含播放音频数据对应的声音。为了从音频采集装置采集到的音频数据中去除播放音频数据对应的声音，在电子设备获取到待播放的播放音频数据后，可以将该播放音频数据先存储至电子设备的存储区域中，以作为后续回声消除处理的参考数据。
步骤S220，播放获取到的播放音频数据，并通过音频采集装置采集音频数据，得到采集音频数据。
在获取到待播放的播放音频数据之后，电子设备可以通过音频播放装置(诸如扬声器、喇叭等)播放该播放音频数据。可选的，播放音频数据的存储与播放可以同时进行，或者，可以先对播放音频数据进行播放，再对播放音频数据进行存储，或者，也可以先对播放音频数据进行存储，再对播放音频数据进行播放。
并且，还可以通过音频采集装置(诸如麦克风等)采集所处环境的声音，得到采集音频数据。其中，播放音频数据的播放和采集音频数据的采集可以是同步开启的，也可以是异步开启的。
步骤S230，按照所对应的存储时间由早到晚的顺序，从存储区域中提取所对应的播放时长与采集音频数据的播放时长相匹配的播放音频数据。
其中，存储时间是指播放音频数据存储至存储区域中的时间，能够表征播放音频数据的存储先后顺序。
在获取到采集音频数据后，由于采集音频数据中可能包含播放音频数据对应的声音，因此，可以从存储区域中获取播放音频数据，以作为回收消除处理的参考数据。考虑到播放音频数据存储至存储区域的时间越早，其越早被播放，从而越早被音频采集装置采集到，采集音频数据中包含该播放音频数据对应的声音的概率越高，并且，采集音频数据的播放时长越长，其包含的播放音频数据的播放时长越长，因此，按照所对应的存储时间由早到晚的顺序，依次从存储区域中提取播放音频数据，直至提取出的播放音频数据的播放时长与采集音频数据播放时长相匹配，例如，假设存储区域中包含3帧播放音频数据，第1帧播放音频数据的存储时间早于第2帧播放音频数据的存储时间，第2帧播放音频数据的存储时间早于第3帧播放音频数据的存储时间，也就是说，先存储第1帧播放音频数据，再存储第2帧播放音频数据，再存储第3帧播放音频数据，若采集到1帧采集音频数据，则从存储区域中提取第1帧播放音频数据，若采集到2帧采集音频数据，则从存储区域中提取第1、2帧播放音频数据。其中，一帧音频数据是指一小段时间(即，单位播放时长)的音频数据，其中，单位播放时长可以是10毫秒、20毫秒等。
步骤S240，根据提取出的播放音频数据，对采集音频数据进行回声消除处理。
在从存储区域中提取出播放音频数据后，电子设备可以基于提取出的播放音频数据，对采集音频数据进行回声消除处理，其中，回声消除处理的具体方式可以根据实际需要灵活设置，在一可选的实施方式中，可以根据提取出的播放音频数据，预测采集音频数据中包含的回声信号的大小，并将采集音频数据对应的音频信号减去预测出的回声信号，得到回声消除处理后的采集音频数据。
在图2所示的实施例中，将播放音频数据作为对采集音频数据进行回声消除处理的参考数据，可以提升回声消除处理的准确性，并且，先将播放音频数据存储在存储区域中，后续得到采集音频数据后，再从存储区域中提取用于回声消除的播放音频数据，从而保障有足量的播放音频数据以供回声消除，进一步地，在提取播放音频数据的过程中，是按照所对应的存储时间由早到晚的顺序提取的，且提取出的播放音频数据的播放时长与采集音频数据的播放时长是匹配的，从而提升了提取出的播放音频数据与采集音频数据在时间和时长上的匹配度，提升了用于回声消除的播放音频数据的准确度，从而提升了回声消除处理的准确性，进而提升了音频数据的质量。
在一示例性实施例中，参见图3，图3是在图2所示的基础上提出的一种音频数据处理方法的流程图。
如图3所示，音频数据处理方法还包括步骤S310-步骤S330，详细介绍如下：
步骤S310，检测存储区域中包含的播放音频数据所对应的总播放时长。
存储区域中包含不同存储时间添加至存储区域中的播放音频数据，电子设备可以实时或周期性检测存储区域中包含的播放音频数据的总播放时长，其中，总播放时长可以根据存储区域中包含的播放音频数据的帧数，以及一帧音频数据对应的单位播放时长计算，例如，假设存储区域中包含10帧播放音频数据，一帧音频数据的播放时长为10毫秒，则总播放时长为100毫秒。
步骤S320，若总播放时长大于或等于播放时长阈值，则按照所对应的存储时间由早到晚的顺序，从存储区域中提取出待删除的目标播放音频数据。
需要说明的是，当存储区域中包含的播放音频数据所对应的总播放时长太长时，则表明存储区域中部分播放音频数据的播放时间与当前时间之间的时差太长，后续采集到的播放音频数据中包含该播放音频数据的概率较低，例如，假设存储区域中包含的播放音频数据的总播放时长大于1分钟，则表明有些播放音频数据是1分钟之前播放的，则后续采集到的采集音频数据中包含1分钟之前播放的播放音频数据的概率较低，因此，为了节约存储资源，并提升回声消除的准确性，设置播放时长阈值，用于删除存储区域中无用的播放音频数据，若电子设备检测到总播放时长大于或等于播放时长阈值，则按照所对应的存储时间由早到晚的顺序，从存储区域中提取出目标播放音频数据，其中，目标播放音频数据是指需要删除的播放音频数据。
其中，播放时长阈值的具体取值可以根据实际需要灵活设置。在一可选的示例中，可以根据经验，设置为30秒、20秒等。在另一可选的示例中，获取到播放音频数据与播放该播放音频数据之间会存在时差，即，存在播放链路时延，记为第一时延，若总播放时长超过第一时延，则表明存储区域中部分播放音频数据的播放时间与当前时间之间的时差超过第一时延，后续采集到的采集音频数据中包含该播放音频数据的概率较小，因此，电子设备可以根据第一时延来计算播放时长阈值，其中，播放时长阈值与第一时延正相关，也就是说，第一时延越大，播放时长阈值越大，其中，电子设备可以将第一时延直接作为播放时长阈值，或者，电子设备也可以将第一时延与设定系数的乘积作为播放时延阈值，设定系数为大于1的数值。在另一可选的示例中，采集到播放音频数据与对播放音频数据进行回收消除处理之间会存在时差，即，存储采集链路时延，记为第二时延，若总播放时长超过第二时延，则表明存储区域中部分播放音频数据的播放时间与当前时间之间的时差超过第二时延，后续采集到的采集音频数据中包含该播放音频数据的概率较小，因此，电子设备可以根据第二时延来计算播放时长阈值，其中，可以将第二时延直接作为播放时长阈值，或者，也可以将第二时延与设定系数的乘积作为播放时延阈值。在另一可选的示例中，电子设备还可以根据第一时延和第二时延确定播放时长阈值，其中，播放时长阈值与第一时延正相关，与第二时延正相关，其中，可以根据第一时延与第二时延的总时延来计算播放时长阈值，例如，可以将总时延直接作为播放时长阈值，或者，也可以将总时延与设定系数的乘积作为播放时延阈值。由此可见，本实施例中，可以根据第一时延和第二时延中的至少一个来确定播放时长阈值。
提取目标播放音频数据的具体方式可以根据实际需要灵活设置。在一可选的示例中，电子设备可以按照所对应的存储时间由早到晚的顺序，从存储区域依次提取播放音频数据，直至提取出的播放音频数据的播放时长与目标播放时长相匹配，并将提取出的播放音频数据作为目标播放音频数据，其中，目标播放时长＝总播放时长-播放时长阈值，也就是说，目标播放音频数据的播放时长＝总播放时长-播放时长阈值，使得对目标播放音频数据进行删除后，存储区域中包含的播放音频数据的总播放时长与播放时长阈值相匹配。在另一可选的示例中，电子设备可以按照所对应的存储时间由早到晚的顺序，从存储区域中提取出指定数量帧播放音频数据作为目标播放音频数据，其中，指定数量可以是1、2、3等。
步骤S330，从存储区域中删除目标播放音频数据。
在电子设备从存储区域中查找出目标播放音频数据后，则从存储区域中删除目标播放音频数据，使得目标播放音频数据不用于后续采集音频数据的回声消除处理。
需要说明的是，图3所示的步骤S210-步骤S240的具体实现细节可以参照图2所示的步骤S210-步骤S240。
在图3所示的实施例中，若存储区域中包含的播放音频数据所对应的总播放时长大于或等于播放时长阈值，则按照所对应的存储时间由早到晚的顺序，从存储区域中提取出待删除的目标播放音频数据，并从存储区域中删除目标播放音频数据，可以节约存储资源，并提升回声消除处理的准确性。
在一示例性实施例中，参见图4，图4是在图3所示的基础上提出的一种音频数据处理方法的流程图。
如图4所示，音频数据处理方法还可以包括步骤S410-步骤S420，详细介绍如下：
步骤S410，计算播放音频数据的获取时间与播放时间之间的时差，得到第一时延。
第一时延是指播放链路时延，即，从获取到播放音频数据至播放该播放音频数据之间的时延，其能表征播放音频数据的获取时间与播放时间之间的时差。
其中，计算第一时延的具体方式可以根据实际需要灵活设置，在计算第一时延时，考虑播放音频数据是由音频播放装置进行播放的，音频播放装置位于播放链路中，音频播放装置会对第一时延造成影响，因此，电子设备可以根据音频播放装置的属性参数，计算第一时延。在另一可选的实施例中，电子设备可以为播放音频数据打上获取时间戳和播放时间戳，计算获取时间戳和播放时间戳之间的差值，得到第一时延。
可选的，为了提升准确性，可以每隔设定的周期重新计算第一时延，并更新第一时延。
步骤S420，根据第一时延，计算播放时长阈值；其中，播放时长阈值与第一时延正相关。
在确定第一时延之后，电子设备可以计算播放时延阈值，其中，第一时延越长，播放时延阈值越长，第一时延越短，播放时延阈值越短。
可选的，电子设备可以直接将第一时延作为播放时延阈值，或者，电子设备可以将第一时延与设定系数的乘积作为播放时延阈值。
需要说明的是，图4所示的步骤S210-步骤S240的具体实现细节可以参照图2所示的步骤S210-步骤S240，图4所示的步骤S310-步骤S330的具体实现细节可以参照图3所示的步骤S310-步骤S330。
在图4所示的实施例中，根据播放音频数据的获取时间与播放时间之间的时差确定播放时长阈值，可以提升回声消除处理的准确性。
在一示例性实施例中，参见图5，图5是在图4所示的基础上提出的一种音频数据处理方法的流程图。
如图5所示，音频数据处理方法还可以包括步骤S510-步骤S520，详细介绍如下：
步骤S510，计算采集音频数据的采集时间与进行回声消除处理的时间之间的时差，得到第二时延。
第二时延是指采集链路时延，即，从采集到采集音频数据至回声消除处理该播放音频数据之间的时延，其能表征采集音频数据的采集时间与进行回声消除处理的时间之间的时差。
其中，计算第二时延的具体方式可以根据实际需要灵活设置。在计算第二时延时，考虑采集音频数据是由音频采集装置采集到的，音频采集装置位于采集链路中，音频采集装置会对第二时延造成影响，因此，电子设备可以根据音频采集装置的属性参数，计算第二时延。在另一可选的实施例中，电子设备可以为采集音频数据打上采集时间戳和处理时间戳(即，进行回声消除处理的时间)，计算采集时间戳和处理时间戳之间的差值，得到第二时延。
可选的，为了提升准确性，可以每隔设定的周期重新计算第二时延，并更新第二时延。
步骤S520，根据第一时延与第二时延的总时延，计算播放时长阈值；其中，播放时长阈值与总时延正相关。
若存储区域中包含的播放音频数据所对应的总播放时长超过第一时延和第二时延的总时延(即，第一时延与第二时延的和)，则表明存储区域中部分播放音频数据的播放时间与当前时间之间的时差超过总时延，后续采集到的采集音频数据中包含该播放音频数据的概率较低，例如，假设第一时延为3秒，第二时延为5秒，当前时间为11:00：30，由于存在第二时延，因此，后续需要进行回声消除处理的采集音频数据是在11:00:25之后采集的，采集音频数据中只会包含11:00:25之后播放的播放音频数据，由于存在第一时延，因此，采集音频数据中只会包含11:00:22之后获取到的播放音频数据，也就是说，有用的播放音频数据的获取时间与当前时间之间的时差小于或等于8秒(即，总时延)，因此，若存储区域中包含的播放音频数据的总播放时长超过8秒，则表明有些播放音频数据的获取时间与当前时间之间的时差大于8秒，存储区域中包含无用的播放音频数据。因此，电子设备在计算出第一时延与第二时延之后，可以根据第一时延与第二时延的总时延计算播放时长阈值，其中，总时延越长，播放时长阈值越大，总时延越短，播放时长阈值越小。
可选的，电子设备可以直接将总时延作为播放时长阈值，也可以将总时延与设定系数的乘积作为播放时延阈值。
需要说明的是，图5所示的步骤S210-步骤S240的具体实现细节可以参照图2所示的步骤S210-步骤S240，图5所示的步骤S310-步骤S330的具体实现细节可以参照图3所示的步骤S310-步骤S330，图5所示的步骤S410的具体实现细节可以参照图4所示的步骤S410。
在图5所示的实施例中，根据播放音频数据的获取时间与播放时间之间的时差、以及采集音频数据的采集时间与进行回声消除处理的时间之间的时差，确定播放时长阈值，可以提升回声消除处理的准确性。
在一示例性实施例中，参见图6，图6是在图2所示的基础上提出的一种音频数据处理方法的流程图。
如图6所示，音频数据处理方法还包括步骤S610-步骤S620，详细介绍如下：
步骤S610，从存储区域包含的多帧播放音频数据中，查找存储时间最早的播放音频数据。
存储区域中包含多帧播放音频数据，可以从中查找存储时间最早的一帧播放音频数据，即，存储区域包含的多帧播放音频数据中，最早存储至存储区域中的播放音频数据，例如，假设存储区域中包含3帧播放音频数据，分别于11:00:01、11:00:02、11:00:03存储至存储区域，则11:00:01存储至存储区域中的播放音频数据为存储时间最早的播放音频数据。
步骤S620，若存储时间最早的播放音频数据在存储区域中的存储时长大于或等于存储时长阈值，则从存储区域中删除存储时间最早的播放音频数据。
播放音频数据在存储区域中的存储时长表征从播放音频数据添加至存储区域，到当前时间这一段时间的长短，播放音频数据的存储时长太长，则表明播放音频数据的播放时间与当前时间之间的时差太长，后续需要进行回声消除处理的采集音频数据包含该播放音频数据的概率较低，因此，设置存储时长阈值，若存储区域中，存储时间最早的播放音频数据的存储时长大于或等于存储时长阈值，则删除该播放音频数据。
其中，存储时长阈值的具体取值可以根据实际需要灵活设置，例如，可以根据经验设置为10毫秒、20毫秒等；或者，可以根据第一时延、第二时延中的至少一个来确定，其中，根据第一时延、第二时延中的至少一个来确定存储时长阈值的具体方式与根据第一时延、第二时延中的至少一个来确定播放时长阈值的方式类似。在一可选的示例中，存储时长阈值与播放时长阈值相同。
需要说明的是，图6所示的步骤S210-步骤S240的具体实现细节可以参照图2所示的步骤S210-步骤S240。
在图6所示的实施例中，从存储区域中删除所对应的存储时长较长的播放音频数据，可以节约存储资源，并提升回声消除的准确性。
在一示例性实施例中，参见图7，图7是在图2所示的基础上提出的一种音频数据处理方法的流程图。
如图7所示，步骤S210可以包括步骤S710-步骤S730，详细介绍如下：
步骤S710，获取待播放的播放音频数据。
需要说明的是，播放音频数据的类型不同，获取播放音频数据的方式不同，例如，若播放音频数据为本地存储的音乐，则电子设备从本地存储中查找播放音频数据，若播放音频数据为与目标通信端的语音通信过程中，目标通信端采集到的音频数据，则电子设备在语音通信过程中接收来自目标通信端的播放音频数据。
步骤S720，检测音频采集装置的工作状态，得到状态检测结果。
若在对播放音频数据进行播放的过程中，音频采集装置并未采集音频数据，则电子设备无需将播放音频数据存储至存储区域，因此，在获取到播放音频数据后，还需要检测音频采集装置的工作状态，从而得到表征音频采集装置是否正在采集音频数据的状态检测结果。
步骤S730，若状态检测结果表征音频采集装置正在采集音频数据，则将获取到的播放音频数据存储至存储区域。
若状态检测结果表征音频采集装置正在采集音频数据，则表明后续对播放音频数据进行播放的过程中，音频采集装置采集到的采集音频数据中可能会包含该播放音频数据，因此，电子设备将播放音频数据存储至存储区域。
可选的，由于播放音频数据的播放存在第一时延，在获取到播放音频数据后的一段时间内采集到的采集音频数据包含该播放音频数据的概率较高，例如，假设第一时延为9秒，获取到待播放的播放音频数据的时间为12:00:00，则播放音频数据的播放时间是12:00:09，在12:00:09采集的采集音频数据会包含该播放音频数据，因此，若状态检测结果表征音频采集装置当前没有采集音频数据，则电子设备可以检测以播放音频数据的获取时间为起始时间，时长为设定时长的这一段时间内，音频采集装置是否采集音频数据，若是，则将播放音频数据存储至存储区域，若否，则不将播放音频数据存储至存储区域中，其中，设定时长可以与第一时延正相关，可以直接将第一时延作为设定时长，也可以将第一时延与获取到的播放音频数据的播放时长之间的和作为设定时长。
需要说明的是，图7所示的步骤S220-步骤S240的具体实现细节可以参照图2所示的步骤S220-步骤S240。
在图7所示的实施例中，在音频采集装置正在采集音频数据的条件下，才将获取到的待播放的播放音频数据存储至存储区域，不仅节约了存储资源，还可以提升回声消除处理的准确性。
在一示例性实施例中，参见图8，图8是在图2所示的基础上提出的一种音频数据处理方法的流程图。
如图8所示，在通过音频采集装置采集音频数据，得到采集音频数据之后，还可以包括步骤S810-步骤S820，详细介绍如下：
步骤S810，若存储区域中不包含播放音频数据，则生成所对应的播放时长与采集音频数据的播放时长相匹配的静音音频数据。
在得到采集音频数据之后，如果存储区域为空，即，存储区域中不包含播放音频数据，为了适配回声消除处理过程，电子设备可以生成播放时长与采集音频数据的播放时长相匹配的静音音频数据，也就是说，静音音频数据中不包含声音，并且，静音音频数据的播放时长与采集音频数据的播放时长相匹配。
步骤S820，根据静音音频数据对采集音频数据进行回声消除处理。
电子设备以静音音频数据作为参考数据，对采集音频数据进行回声消除处理。
需要说明的是，图8所示的步骤S210-步骤S240的具体实现细节可以参照图2所示的步骤S210-步骤S240。
在图8所示的实施例中，在得到采集音频数据之后，若存储区域中不包含播放音频数据，则电子设备生成所对应的播放时长与采集音频数据的播放时长相匹配的静音音频数据，作为对采集音频数据进行回声消除处理的参考数据，使得无需更改回声消除处理过程，节约了处理资源。
在一示例性实施例中，参见图9，图9是在图2所示的基础上提出的一种音频数据处理方法的流程图。
如图9所示，步骤S230可以包括步骤S910，详细介绍如下：
步骤S910，若采集到设定数量帧采集音频数据，则按照所对应的存储时间由早到晚的顺序，从存储区域中提取设定数量帧播放音频数据。
为了提升播放音频数据的实时性，可以设置设定数量，其中，设定数量为大于等于1的整数，例如，可以设置为1、2、3等，其具体取值可以与播放音频数据的实时性需求负相关，也就是说，若对播放音频数据的实时性需求较高，则设定数量较低，若对播放音频数据的实时性需求较低，则设定数量可以较高。
在通过音频采集装置采集音频数据的过程中，若采集到设定数量帧采集音频数据，则电子设备可以触发对该采集音频数据的回声消除处理流程，在回声消除处理的过程中，先按照所对应的存储时间由早到晚的顺序，从存储区域中提取出与设定数量相匹配的多帧播放音频数据，也就是说，提取出设定数量帧播放音频数据，例如，假设设定数量为1，则可以在采集到一帧采集音频数据后，从存储区域中提取出存储时间最早的一帧播放音频数据。
在此条件下，步骤S240可以包括步骤S920，详细介绍如下：
步骤S920，根据提取出的设定数量帧播放音频数据，对采集到的设定数量帧采集音频数据进行回声消除处理。
电子设备在提取出设定数量帧播放音频数据后，根据设定数量帧播放音频数据，对设定数量帧采集音频数据进行回声消除处理，也就是说，每次进行回声消除处理的播放音频数据和采集音频数据的数据量都是设定数量帧，从而避免需要进行回声消除处理的播放音频数据堆积，降低回声消除处理效率和播放音频数据的实时性的情况。
需要说明的是，图9所示的步骤S210-步骤S220的具体实现细节可以参照图2所示的步骤S210-步骤S220。
在图9所示的实施例中，每次进行回声消除处理的播放音频数据和采集音频数据的数据量都是设定数量帧，从而避免需要进行回声消除处理的播放音频数据堆积，提升回声消除处理效率和播放音频数据的实时性。
在一示例性实施例中，参见图10，图10是在图2所示的基础上提出的一种音频数据处理方法的流程图。
如图10所示，在存储区域包含消息队列的条件下，步骤S230可以包括步骤S1010，详细介绍如下：
步骤S1010，从消息队列的队头提取所对应的播放时长与采集音频数据的播放时长相匹配的播放音频数据。
需要说明的是，消息队列采用先入先出模式，只允许从队尾插入数据，从队头提取出数据，对应的，在存储区域包含消息队列的条件下，播放音频数据存储至消息队列中，队头的播放音频数据的存储时间早于队尾的播放音频数据的存储时间，因此，在得到采集音频数据之后，为了提取出存储时间较早的播放音频数据，电子设备可以从消息队列的队头开始依次提取播放音频数据，直至提取出的播放音频数据的播放时长与采集音频数据的播放时长相匹配，从而实现按照所对应的存储时间由早到晚的顺序，从存储区域中提取播放音频数据。
需要说明的是，图10所示的步骤S210-步骤S220、步骤S240的具体实现细节可以参照图2所示的步骤S210-步骤S220、步骤S240。
在图10所示的实施例中，将播放音频数据存储至消息队列，可以提升从消息队列中提取播放音频数据的效率。
在一示例性实施例中，参见图11，图11是在图2所示的基础上提出的一种音频数据处理方法的流程图。
如图11所示，在播放音频数据包括本端与外部通信端的语音通信过程中外部通信端发送的音频数据，采集音频数据包括语音通信过程中本端采集到的音频数据的条件下，在根据提取出的播放音频数据，对采集音频数据进行回声消除处理之后，音频数据处理方法还可以包括步骤S1110，详细介绍如下：
步骤S1110，将回声消除处理后的采集音频数据发送至外部通信端。
需要说明的是，外部通信端是指除作为本端的电子设备之外的通信端，其能够与本端进行语音通信，其类型包括但不限于终端设备、服务器等，其中，对于终端设备、服务器的具体介绍，请参见前述记载。在作为本端的电子设备与外部通信端进行语音通信的过程中，本端的音频采集装置在采集声音的过程中，本端的音频播放装置会播放来自外部通信端的声音，从而使得本端采集到的声音中包含来自外部通信端的声音，若直接将本端采集到的声音发送至外部通信端进行播放，则会导致外部通信端的用户听到自己的声音，即回声。其中，语音通信包括但不限于即时语音通信和非即时语音通信，其中，即时语音通信包括但不限于语音通话、视频通话、直播连麦、在线会议等。
本实施例中的音频数据处理方法可以应用于语音通信场景，从而消除语音通常场景中的回声。在语音通信场景下，播放音频数据包含语音通信过程中外部通信端发送的音频数据，该音频数据可以是外部通信端对应的音频采集设备进行音频采集得到的；采集音频数据包含语音通信过程中本端的音频采集设备采集到的音频数据，该音频数据需要发送至外部通信端，因此，在对采集音频数据进行回声消除处理后，可以将回声消除处理后的采集音频数据发送至外部通信端。
可选的，播放音频数据还可以包含在语音通信过程中，本端播放的、除来自外部通信端的音频数据之外的其他音频数据。
需要说明的是，图11所示的步骤S210-步骤S240的具体实现细节可以参照图2所示的步骤S210-步骤S240。除了应用于语音通信场景，音频数据处理方法还可以应用于其他需要进行回声消除处理的场景，例如，若用户在播放音乐的过程中对自己的声音进行录音，录音中不仅包含用户的声音，还包含播放的音乐，导致录音中用户的声音比较模糊，因此，可以将播放的音乐作为播放音频数据，录音作为采集音频数据，通过音频数据处理方法进行回声消除处理，从而可以去除录音中包含的音乐，提升录音中用户的声音的清晰度。又例如，多人在线会议场景中，若会议成员在播放会议主持人的声音的过程中进行发言，会议成员的终端采集的声音中，不仅包含会议成员的声音，还包含主持人的声音，导致会议成员的终端采集的声音中会议成员的声音比较模糊，因此，可以将主持人的声音作为播放音频数据，会议成员所使用的终端采集的声音作为采集音频数据，通过音频数据处理方法进行回声消除处理，从而可以去除声音中包含的会议主持人声音，提升会议成员的声音的清晰度。
在图11所示的实施例中，通过音频数据处理方法对语音通信过程中需要发送给外部通信端的音频数据进行回声消除处理，可以提升语音通信质量。
在一示例性实施例中，参见图12，图12是在图2所示的基础上提出的一种音频数据处理方法的流程图。
如图12所示，步骤S210可以包括步骤S1210-步骤S1230，详细介绍如下：
步骤S1210，获取待播放的播放音频数据。
其中，播放音频数据的具体获取方式可以参见前述实施例的记载。
步骤S1220，若存储区域中不包含播放音频数据，则根据等待时长以及获取到的播放音频数据所对应的获取时间，计算获取到的播放音频数据所对应的存储时间；其中，存储时间晚于获取时间。
在获取到待播放的播放音频数据后，若存储区域中不包含播放音频数据，若将该播放音频数据立即添加至存储区域，则该播放音频数据会被用于当前获取到的采集音频数据的回声消除处理，由于存在第一时延、第二时延，当前获取到的采集音频数据中包含该播放音频数据的概率较小，例如，假设第一时延为1秒，第二时延为2秒，存储区域中不包含播放音频数据，且在11:30:20获取到播放音频数据，则该播放音频数据会在11:30:21被播放，而11:30:20需要进行回声消除处理的采集音频数据是11:30:18采集到的，若将该播放音频数据立即添加至存储区域，则会导致11:30:21播放的播放音频数据被用于11:30:18采集到的采集音频数据的回声消除，使得播放音频数据与采集音频数据没有对齐，降低了回声消除效果。因此，为了避免这种情况，电子设备可以设置等待时长，在获取到待播放的播放音频数据之后，可以先检测存储区域中是否包含播放音频数据，若不包含，则可以根据等待时长以及该播放音频数据的获取时间，计算该播放音频数据的存储时间，其中，存储时间＝获取时间+等待时长。
其中，等待时长可以根据第一时延、第二时延中的至少一个确定，等待时长可以与第一时延正相关、与第二时延正相关、或，与第一时延和第二时延的总时延正相关，例如，可以将第一时延、第二时延、总时延(第一时延与第二时延的和)中的任意一个作为等待时长。
步骤S1230，根据存储时间，将获取到的播放音频数据存储至存储区域。
电子设备根据计算出的存储时间，将获取到的播放音频数据存储至存储区域，也就是说，在获取到待播放的播放音频数据后，若存储区域中不包含播放音频数据，则电子设备可以在等待一定的时长后，将播放音频数据添加至存储区域中。
可选的，在获取到待播放的播放音频数据后，若存储区域中包含播放音频数据，则电子设备可以直接将该播放音频数据添加至存储区域中。
需要说明的是，图12所示的步骤S220-步骤S240的具体实现细节可以参照图2所示的步骤S220-步骤S240。
在图12所示的实施例中，在获取到待播放的播放音频数据后，若存储区域中不包含播放音频数据，则等待一定的时长后，再将播放音频数据添加至存储区域，使得播放音频数据与采集音频数据进行对齐，提升回声消除处理的准确性。
为了更好的理解本申请，此处以音频数据处理方法应用于即时语音通信场景为例进行说明。其中，参见图13，图13是本申请涉及的一种实施环境的示意图，其包含第一终端设备以及第二终端设备，音频数据处理方法由第一终端设备执行，第二终端设备为与第一终端设备进行即时语音通信的外部终端设备，第一终端设备中包含扬声器、麦克风、数据投喂模块、回声消除模块。在与第二终端设备进行即时语音通信的过程中，第一终端设备会接收到来自第二终端设备的外部语音数据，其中，外部语音数据是指即时语音通信的过程中，第二终端设备采集到的语音数据。在接收到外部语音数据之后，第一终端设备可以通过自身包含的扬声器对外部语音数据进行播放，也就是说，外部语音数据为第一终端设备的播放音频数据，并且，第一终端设备可以将外部语音数据发送至数据投喂模块，同时，在即时语音通信的过程中第一终端设备的麦克风也会采集用户声音，得到本地语音数据，由于采集用户声音的过程中，扬声器在播放外部语音数据，因此，本地语音数据中可能会包含外部语音数据，可以将本地语音数据传输至数据投喂模块，数据投喂模块将本地语音数据和外部语音数据投喂至回声消除模块，回声消除模块根据投喂的外部语音数据对本地语音数据进行回声消除处理，得到外发语音数据，并将外发语音数据发送至第二终端设备。
参见图14所示，数据投喂模块包含消息队列以及数据投喂决策模块，其中，消息队列用于缓存外部语音数据，数据投喂决策模块用于确定如何向回声消除模块投喂外部语音数据和本地语音数据。
数据投喂模块在获取到外部语音数据后，参见图15所示，可以先判断回声消除功能是否开启，若回声消除功能并未开启，则表明外部语音数据无需用于回声消除，因此，删除该外部语音数据。若回声消除功能开启，则表明需要缓存外部语音数据，作为后续回声消除的参考数据，因此，将外部语音数据添加至消息队列中。由于消息队列中包含的外部语音数据的时长太长，则表明其中的部分外部语音数据的播放时间太早，后续需要进行回声消息处理的本地语音数据中包含该外部语音数据的概率较低，因此，数据投喂模块还可以获取总时延，并判断消息队列中包含的多帧外部语音数据的总播放时长是否大于总时延，若总播放时长大于总时延，则删除消息队列中位于队头的一帧外部语音数据，即，删除消息队列中存储时间最早的一帧外部语音数据。若总播放时长小于或等于总时延，则重新判断消息队列中包含的多帧外部语音数据的总播放时长是否大于总时延。
其中，总时延是指播放链路时延与采集链路时延的和，其中，播放链路是指从接收到外部语音数据至通过扬声器播放外部语音的过程中，外部语音数据的传输链路，该链路中包含扬声器，该链路的时延即播放链路时延，也就是说，播放链路时延是指从接收到外部语音数据至播放外部语音数据的时延，采集链路是指从采集本地语音数据至对本地语音数据进行回声消除处理的过程中，本地语音数据的传输链路，该链路中包含麦克风，可选的，采集链路可以包含从麦克风的输入端至数据投喂模块的输入端这一段本地语音数据的传输链路，该链路的时延即采集链路时延，也就是说，采集链路时延是指从采集本地语音数据至对本地语音数据进行回声消除处理的时延。
数据投喂模块从麦克风获取到一帧本地语音数据后，参见图16所示，数据投喂决策模块可以获取一帧参考语音数据，并将一帧参考语音数据以及一帧本地语音数据投喂至回声消除模块进行处理，其中，参考语音数据可以静音语音数据，也可以是来自消息队列。具体的，参见图17所示，数据投喂决策模块获取到一帧本地语音数据后，可以判断消息队列中是否包含外部语音数据，若是，则从消息队列的队头提取一帧外部语音数据，即，提取存储时间最早的一帧外部语音数据，并将外部语音数据作为参考语音数据。若否，即，消息队列中不包含外部语音数据，则可以生成一帧静音语音数据，将静音语音数据作为参考语音数据，并投喂参考语音数据、本地语音数据至回声消息模块，回声消除模块通过参考语音数据对本地语音数据进行回声消除处理，得到回声消除模块输出的外发语音数据，这样，外发语音数据中不会包含来自第二终端设备的声音，将外发语音数据发送至第二终端设备进行播放，第二终端设备的用户不会听到自己的回声。
需要说明的是，本实施例中的各步骤的具体实现细节已在前述实施例中详细介绍。
本实施例中，能够保障在向回声消除模块投喂的过程中，是严格的按照一帧播放音频数据、一帧采集音频数据进行投喂，实现了采集音频数据与播放音频数据的对齐，提升了回声消除的效果，并且，利用采集链路时延和播放链路时延来管理消息队列中包含的播放音频数据，可以放心删除太老的播放音频数据，而不用担心出现漏回声问题，节约了存储资源，进一步提升了回声消除的效果。
参见图18，图18是本申请的一示例性实施例示出的音频数据处理装置的框图。如图18所示，该装置包括：
获取模块1801，配置为获取待播放的播放音频数据，将获取到的播放音频数据存储至设定的存储区域；
处理模块1802，配置为播放获取到的播放音频数据，并通过音频采集装置采集音频数据，得到采集音频数据；
提取模块1803，配置为按照所对应的存储时间由早到晚的顺序，从存储区域中提取所对应的播放时长与采集音频数据的播放时长相匹配的播放音频数据；
消除模块1804，配置为根据提取出的播放音频数据，对采集音频数据进行回声消除处理。
在一示例性实施例中，基于前述方案，装置还包括第一删除模块，配置为：
检测存储区域中包含的播放音频数据所对应的总播放时长；
若总播放时长大于或等于播放时长阈值，则按照所对应的存储时间由早到晚的顺序，从存储区域中提取出待删除的目标播放音频数据；
从存储区域中删除目标播放音频数据。
在一示例性实施例中，基于前述方案，装置还包括计算模块，配置为：
计算播放音频数据的获取时间与播放时间之间的时差，得到第一时延；
根据第一时延，计算播放时长阈值；其中，播放时长阈值与第一时延正相关。
在一示例性实施例中，基于前述方案，计算模块具体配置为：
计算采集音频数据的采集时间与进行回声消除处理的时间之间的时差，得到第二时延；
根据第一时延与第二时延的总时延，计算播放时长阈值；其中，播放时长阈值与总时延正相关。
在一示例性实施例中，基于前述方案，装置还包括第二删除模块，配置为：
从存储区域包含的多帧播放音频数据中，查找存储时间最早的播放音频数据；
若存储时间最早的播放音频数据在存储区域中的存储时长大于或等于存储时长阈值，则从存储区域中删除存储时间最早的播放音频数据。
在一示例性实施例中，基于前述方案，获取模块1801具体配置为：
检测音频采集装置的工作状态，得到状态检测结果；
若状态检测结果表征音频采集装置正在采集音频数据，则将获取到的播放音频数据存储至存储区域。
在一示例性实施例中，基于前述方案，装置还包括生成模块，配置为：
若存储区域中不包含播放音频数据，则生成所对应的播放时长与采集音频数据的播放时长相匹配的静音音频数据；
根据静音音频数据对采集音频数据进行回声消除处理。
在一示例性实施例中，基于前述方案，提取模块1803具体配置为：
若采集到设定数量帧采集音频数据，则按照所对应的存储时间由早到晚的顺序，从存储区域中提取设定数量帧播放音频数据；
根据提取出的播放音频数据，对采集音频数据进行回声消除处理，包括：
根据提取出的设定数量帧播放音频数据，对采集到的设定数量帧采集音频数据进行回声消除处理。
在一示例性实施例中，基于前述方案，在存储区域包含消息队列的条件下，提取模块1803具体配置为：
从消息队列的队头提取所对应的播放时长与采集音频数据的播放时长相匹配的播放音频数据。
在一示例性实施例中，基于前述方案，在播放音频数据包括与外部通信端的语音通信过程中外部通信端发送的音频数据，采集音频数据包括语音通信过程中采集到的音频数据的条件下，装置还包括，发送模块，配置为：
将回声消除处理后的采集音频数据发送至外部通信端。
在一示例性实施例中，基于前述方案，获取模块1801具体配置为：
若存储区域中不包含播放音频数据，则根据等待时长以及获取到的播放音频数据所对应的获取时间，计算获取到的播放音频数据所对应的存储时间；其中，存储时间晚于获取时间；
根据存储时间，将获取到的播放音频数据存储至存储区域。
需要说明的是，图18所提供的音频数据处理装置与上述实施例所提供的音频数据处理方法属于同一构思，其中各个模块和单元执行操作的具体方式、具体方式的技术效果已经在方法实施例中进行了详细描述。
本申请的实施例还提供了一种电子设备，包括：一个或多个处理器；存储装置，用于存储一个或多个计算机可读指令，当一个或多个计算机可读指令被一个或多个处理器执行时，使得电子设备实现上述各个实施例中提供的音频数据处理方法。
图19示出了适于用来实现本申请实施例的电子设备的计算机系统的结构示意图，该电子设备可以是图1所示的终端设备110或服务器120。
需要说明的是，图19示出的电子设备的计算机系统1900仅是一个示例，不应对本申请实施例的功能和使用范围带来任何限制。
如图19所示，计算机系统1900包括中央处理单元(Central Processing Unit，CPU)1901，其可以根据存储在只读存储器(Read-Only Memory，ROM)1902中的计算机可读指令或者从储存部分1908加载到随机访问存储器(Random Access Memory，RAM)1903中的计算机可读指令而执行各种适当的动作和处理，例如执行上述实施例中的音频数据处理方法。在RAM 1903中，还存储有系统操作所需的各种计算机可读指令和数据。CPU 1901、ROM 1902以及RAM 1903通过总线1904彼此相连。输入/输出(Input/Output，I/O)接口1905也连接至总线1904。
在一些实施例中，以下部件连接至I/O接口1905：包括键盘、鼠标等的输入部分1906；包括诸如阴极射线管(Cathode Ray Tube，CRT)、液晶显示器(Liquid Crystal Display，LCD)等以及扬声器等的输出部分1907；包括硬盘等的储存部分1908；以及包括诸如LAN(Local Area Network，局域网)卡、调制解调器等的网络接口卡的通信部分1909。通信部分1909经由诸如因特网的网络执行通信处理。驱动器1910也根据需要连接至I/O接口1905。可拆卸介质1911，诸如磁盘、光盘、磁光盘、半导体存储器等等，根据需要安装在驱动器1910上，以便于从其上读出的计算机可读指令根据需要被安装入储存部分1908。
在一些实施例中，计算机系统1900包括音频播放装置和音频采集装置，音频播放装置例如可以是扬声器、喇叭等，音频采集装置例如可以是麦克风等。
特别地，根据本申请的实施例，实现音频数据处理方法的计算机可读指令可以承载在计算机可读介质上，该计算机可读指令可以通过通信部分1909从网络上被下载和安装，和/或从可拆卸介质1911被安装。
需要说明的是，本申请实施例所示的计算机可读介质可以是计算机可读信号介质或者计算机可读存储介质或者是上述两者的任意组合。计算机可读存储介质例如可以是电、磁、光、电磁、红外线、或半导体的系统、装置或器件，或者任意以上的组合。计算机可读存储介质的更具体的例子可以包括但不限于：具有一个或多个导线的电连接、便携式计算机磁盘、硬盘、随机访问存储器(RAM)、只读存储器(ROM)、可擦式可编程只读存储器(Erasable Programmable Read Only Memory，EPROM)、闪存、光纤、便携式紧凑磁盘只读存储器(Compact Disc Read-Only Memory，CD-ROM)、光存储器件、磁存储器件、或者上述的任意合适的组合。在本申请中，计算机可读存储介质可以是任何包含或存储计算机可读指令的有形介质，该计算机可读指令可以被指令执行系统、装置或者器件使用或者与其结合使用。计算机可读信号介质可以包括在基带中或者作为载波一部分传播的数据信号，其中承载了计算机可读的计算机可读指令，这种传播的数据信号可以采用多种形式，包括但不限于电磁信号、光信号或上述的任意合适的组合。计算机可读介质包含的计算机可读指令可以用任何适当的介质传输，包括但不限于：无线、有线等等，或者上述的任意合适的组合。
附图中的流程图和框图，图示了按照本申请各种实施例的系统、方法和计算机程序产品的可能实现的体系架构、功能和操作。其中，流程图或框图中的每个方框可以代表一个模块、程序段、或代码的一部分，上述模块、程序段、或代码的一部分包含一个或多个用于实现规定的逻辑功能的可执行指令。也应当注意，在有些作为替换的实现中，方框中所标注的功能也可以以不同于附图中所标注的顺序发生。例如，两个接连地表示的方框实际上可以基本并行地执行，它们有时也可以按相反的顺序执行，这依所涉及的功能而定。也要注意的是，框图或流程图中的每个方框、以及框图或流程图中的方框的组合，可以用执行规定的功能或操作的专用的基于硬件的系统来实现，或者可以用专用硬件与计算机可读指令的组合来实现。
描述于本申请实施例中所涉及到的单元可以通过软件的方式实现，也可以通过硬件的方式来实现，所描述的单元也可以设置在处理器中。其中，这些单元的名称在某种情况下并不构成对该单元本身的限定。
本申请的另一方面还提供了一种非易失性的计算机可读存储介质，其上存储有计算机可读指令，该计算机可读指令被电子设备的处理器执行时，使电子设备实现如前的音频数据处理方法。该计算机可读存储介质可以是上述实施例中描述的电子设备中所包含的，也可以是单独存在，而未装配入该电子设备中。
本申请的另一方面还提供了一种计算机程序产品，该计算机程序产品包括计算机可读指令，计算机可读指令被处理器执行时实现上述各个实施例中提供的音频数据处理方法。其中，该计算机可读指令可以存储在计算机可读存储介质中。该计算机程序产品可以是作为产品的计算机可读指令，例如，APP(Application，手机应用软件)、网页、小程序等；或者，该计算机程序产品也可以是包含计算机可读指令的存储介质、设备、终端、虚拟机等。
上述内容仅为本申请的较佳示例性实施例，并非用于限制本申请的实施方案，本领域普通技术人员根据本申请的主要构思和精神，可以十分方便地进行相应变通或修改，故本申请的保护范围应以权利要求书所要求的保护范围为准。

Claims (20)

  1. 一种音频数据处理方法，由电子设备执行，所述方法包括：
    获取待播放的播放音频数据，并将获取到的播放音频数据存储至设定的存储区域；
    播放所述获取到的播放音频数据，并通过音频采集装置采集音频数据，得到采集音频数据；
    按照所对应的存储时间由早到晚的顺序，从所述存储区域中提取所对应的播放时长与所述采集音频数据的播放时长相匹配的播放音频数据；
    根据提取出的播放音频数据，对所述采集音频数据进行回声消除处理。
  2. 如权利要求1所述的方法，其特征在于，所述方法还包括：
    检测所述存储区域中包含的播放音频数据所对应的总播放时长；
    当所述总播放时长大于或等于播放时长阈值时，则按照所对应的存储时间由早到晚的顺序，从所述存储区域中提取出待删除的目标播放音频数据；
    从所述存储区域中删除所述目标播放音频数据。
  3. 如权利要求2所述的方法，其特征在于，所述方法还包括：
    计算播放音频数据的获取时间与播放时间之间的时差，得到第一时延；
    根据所述第一时延，计算所述播放时长阈值；其中，所述播放时长阈值与所述第一时延正相关。
  4. 如权利要求3所述的方法，其特征在于，所述根据所述第一时延，计算所述播放时长阈值，包括：
    计算采集音频数据的采集时间与进行回声消除处理的时间之间的时差，得到第二时延；
    根据所述第一时延与所述第二时延的总时延，计算所述播放时长阈值；其中，所述播放时长阈值与所述总时延正相关。
  5. 如权利要求1所述的方法，其特征在于，所述方法还包括：
    从所述存储区域包含的多帧播放音频数据中，查找存储时间最早的播放音频数据；
    在所述存储时间最早的播放音频数据在所述存储区域中的存储时长大于或等于存储时长阈值的情况下，则从所述存储区域中删除所述存储时间最早的播放音频数据。
  6. 如权利要求1所述的方法，其特征在于，所述将获取到的播放音频数据存储至设定的存储区域，包括：
    检测所述音频采集装置的工作状态，得到状态检测结果；
    当所述状态检测结果表征所述音频采集装置正在采集音频数据时，则将获取到的播放音频数据存储至所述存储区域。
  7. 如权利要求1所述的方法，其特征在于，在所述通过音频采集装置采集音频数据，得到采集音频数据之后，所述方法还包括：
    在所述存储区域中不包含播放音频数据的情况下，则生成所对应的播放时长与所述采集音频数据的播放时长相匹配的静音音频数据；
    根据所述静音音频数据对所述采集音频数据进行回声消除处理。
  8. 如权利要求1所述的方法，其特征在于，所述按照所对应的存储时间由早到晚的顺序，从所述存储区域中提取所对应的播放时长与所述采集音频数据的播放时长相匹配的播放音频数据，包括：
    在采集到设定数量帧采集音频数据的情况下，则按照所对应的存储时间由早到晚的顺序，从所述存储区域中提取设定数量帧播放音频数据；
    所述根据提取出的播放音频数据，对所述采集音频数据进行回声消除处理，包括：
    根据提取出的设定数量帧播放音频数据，对采集到的设定数量帧采集音频数据进行回声消除处理。
  9. 如权利要求1所述的方法，其特征在于，所述存储区域包含消息队列；
    所述按照所对应的存储时间由早到晚的顺序，从所述存储区域中提取所对应的播放时长与所述采集音频数据的播放时长相匹配的播放音频数据，包括：
    从所述消息队列的队头提取所对应的播放时长与所述采集音频数据的播放时长相匹配的播放音频数据。
  10. 如权利要求1所述的方法，其特征在于，所述播放音频数据包括所述电子设备与外部通信端的语音通信过程中所述外部通信端发送的音频数据，所述采集音频数据包括所述语音通信过程中采集到的音频数据；
    在所述根据提取出的播放音频数据，对所述采集音频数据进行回声消除处理之后，所述方法还包括：
    将回声消除处理后的采集音频数据发送至所述外部通信端。
  11. 如权利要求1所述的方法，其特征在于，所述将获取到的播放音频数据存储至设定的存储区域，包括：
    在所述存储区域中不包含播放音频数据的情况下，则根据等待时长以及获取到的播放音频数据所对应的获取时间，计算获取到的播放音频数据所对应的存储时间；其中，所述存储时间晚于所述获取时间；
    根据所述存储时间，将获取到的播放音频数据存储至所述存储区域。
  12. 根据权利要求1至11任一项所述的方法，其特征在于，所述根据提取出的播放音频数据，对采集音频数据进行回声消除处理，包括：
    根据提取出的播放音频数据，预测所述采集音频数据中包含的回声信号的大小；
    将所述采集音频数据对应的音频信号减去预测出的回声信号，得到回声消除处理后的采集音频数据。
  13. 一种音频数据处理装置，所述装置包括：
    获取模块，配置为获取待播放的播放音频数据，并将获取到的播放音频数据存储至设定的存储区域；
    处理模块，配置为播放所述获取到的播放音频数据，并通过音频采集装置采集音频数据，得到采集音频数据；
    提取模块，配置为按照所对应的存储时间由早到晚的顺序，从所述存储区域中提取所对应的播放时长与所述采集音频数据的播放时长相匹配的播放音频数据；
    消除模块，配置为根据提取出的播放音频数据，对所述采集音频数据进行回声消除处理。
  14. 根据权利要求13所述的装置，其特征在于，所述装置还包括：
    第一删除模块，配置为检测所述存储区域中包含的播放音频数据所对应的总播放时长；若所述总播放时长大于或等于播放时长阈值，则按照所对应的存储时间由早到晚的顺序，从所述存储区域中提取出待删除的目标播放音频数据；从所述存储区域中删除所述目标播放音频数据。
  15. 根据权利要求14所述的装置，其特征在于，所述装置还包括：
    计算模块，配置为计算播放音频数据的获取时间与播放时间之间的时差，得到第一时延；根据所述第一时延，计算所述播放时长阈值；其中，所述播放时长阈值与所述第一时延正相关。
  16. 根据权利要求15所述的装置，其特征在于，所述计算模块具体配置为计算采集音频数据的采集时间与进行回声消除处理的时间之间的时差，得到第二时延；根据所述第一时延与所述第二时延的总时延，计算所述播放时长阈值；其中，所述播放时长阈值与所述总时延正相关。
  17. 根据权利要求13所述的装置，其特征在于，所述装置还包括：
    第二删除模块，配置为从所述存储区域包含的多帧播放音频数据中，查找存储时间最早的播放音频数据；若所述存储时间最早的播放音频数据在所述存储区域中的存储时长大于或等于存储时长阈值，则从所述存储区域中删除所述存储时间最早的播放音频数据。
  18. 一种电子设备，包括：
    一个或多个处理器；
    存储装置，用于存储一个或多个计算机可读指令，当所述一个或多个计算机可读指令被所述一个或多个处理器执行时，使得所述电子设备实现权利要求1-11中的任一项所述的方法。
  19. 一种计算机可读存储介质，其上存储有计算机可读指令，当所述计算机可读指令被电子设备的处理器执行时，使所述电子设备实现权利要求1-11中的任一项所述的方法。
  20. 一种计算机程序产品，包括计算机可读指令，所述计算机可读指令被处理器执行时实现权利要求1-11中的任一项所述的方法。
PCT/CN2025/080169 2024-03-12 2025-03-03 音频数据处理方法、装置、电子设备、介质和程序产品 Pending WO2025190098A1 (zh)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
CN202410294566.3 2024-03-12
CN202410294566.3A CN120631297A (zh) 2024-03-12 2024-03-12 音频数据处理方法及装置、电子设备、存储介质

Publications (1)

Publication Number Publication Date
WO2025190098A1 true WO2025190098A1 (zh) 2025-09-18

Family

ID=96966562

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2025/080169 Pending WO2025190098A1 (zh) 2024-03-12 2025-03-03 音频数据处理方法、装置、电子设备、介质和程序产品

Country Status (2)

Country Link
CN (1) CN120631297A (zh)
WO (1) WO2025190098A1 (zh)

Citations (7)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US9997151B1 (en) * 2016-01-20 2018-06-12 Amazon Technologies, Inc. Multichannel acoustic echo cancellation for wireless applications
CN109346098A (zh) * 2018-11-20 2019-02-15 网宿科技股份有限公司 一种回声消除方法及终端
US20200302948A1 (en) * 2019-03-22 2020-09-24 Clear Peaks LLC Systems, Devices, and Methods for Synchronizing Audio
CN113495714A (zh) * 2020-04-01 2021-10-12 阿里巴巴集团控股有限公司 数据处理装置、音频处理设备、回音消除系统和回音消除方法
CN113823306A (zh) * 2021-08-17 2021-12-21 北京佳讯飞鸿电气股份有限公司 语音的回声消除方法、装置、设备及存储介质
CN114760389A (zh) * 2022-06-16 2022-07-15 腾讯科技（深圳）有限公司 语音通话方法、装置、计算机存储介质及电子设备
CN115620740A (zh) * 2021-07-12 2023-01-17 腾讯科技（深圳）有限公司 对回声路径的语音延时估计方法、装置和存储介质

Patent Citations (7)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US9997151B1 (en) * 2016-01-20 2018-06-12 Amazon Technologies, Inc. Multichannel acoustic echo cancellation for wireless applications
CN109346098A (zh) * 2018-11-20 2019-02-15 网宿科技股份有限公司 一种回声消除方法及终端
US20200302948A1 (en) * 2019-03-22 2020-09-24 Clear Peaks LLC Systems, Devices, and Methods for Synchronizing Audio
CN113495714A (zh) * 2020-04-01 2021-10-12 阿里巴巴集团控股有限公司 数据处理装置、音频处理设备、回音消除系统和回音消除方法
CN115620740A (zh) * 2021-07-12 2023-01-17 腾讯科技（深圳）有限公司 对回声路径的语音延时估计方法、装置和存储介质
CN113823306A (zh) * 2021-08-17 2021-12-21 北京佳讯飞鸿电气股份有限公司 语音的回声消除方法、装置、设备及存储介质
CN114760389A (zh) * 2022-06-16 2022-07-15 腾讯科技（深圳）有限公司 语音通话方法、装置、计算机存储介质及电子设备

Also Published As

Publication number Publication date
CN120631297A (zh) 2025-09-12

Similar Documents

Publication Publication Date Title
CN113014854B (zh) 互动记录的生成方法、装置、设备及介质
CN107241689A (zh) 一种耳机语音交互方法及其装置、终端设备
CN113823306A (zh) 语音的回声消除方法、装置、设备及存储介质
CN109144992A (zh) 一种数据存储的方法和装置
CN109766508A (zh) 信息审核方法、装置及电子设备
WO2022041879A1 (zh) 通知触发消息的处理方法和装置
CN114647698A (zh) 数据同步方法、装置及计算机存储介质
WO2025129975A1 (zh) 音视频数据的传输方法、装置、设备及介质
WO2025092132A1 (zh) 一种数据处理方法、装置及存储介质
CN118138790A (zh) 一种数据流切换方法、装置、存储介质及电子设备
CN114900477B (zh) 报文处理方法、服务器、电子设备及存储介质
CN115190097A (zh) 一种消息推送方法、装置、设备及存储介质
CN112099736A (zh) 数据存储方法、装置、电子设备及存储介质
CN120631297A (zh) 音频数据处理方法及装置、电子设备、存储介质
CN119356864A (zh) 数据处理的方法、装置、电子设备和存储介质
CN115442786B (zh) 蓝牙同步方法、系统、蓝牙设备及计算机可读取存储介质
CN111131860A (zh) 一种音视频播放方法、装置、设备及介质
CN112732728A (zh) 一种数据同步方法和系统
CN112882992A (zh) 用于显示信息的方法和装置
CN106375363A (zh) 社交应用客户端中消除消息提醒的方法、系统和装置
CN110138991B (zh) 回音消除方法和装置
CN119889273B (zh) 语音合成方法、装置、存储介质及电子装置
US20190295564A1 (en) Artificial intelligence voice interaction method, computer program product, and near-end electronic device thereof
CN111726280A (zh) 一种即时消息的传输方法、装置、电子设备及存储介质
WO2020057627A1 (zh) 一种引导开启应用程序的方法及设备

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 25769270

Country of ref document: EP

Kind code of ref document: A1