WO2023125537A1

WO2023125537A1 - Sound signal processing method and apparatus, and device and storage medium

Info

Publication number: WO2023125537A1
Application number: PCT/CN2022/142338
Authority: WO
Inventors: 张磊; 陈健; 刘智辉
Original assignee: 华为技术有限公司
Priority date: 2021-12-31
Filing date: 2022-12-27
Publication date: 2023-07-06
Also published as: CN116417006A

Abstract

A sound signal processing method and apparatus, and a device and a storage medium. In the method, on the basis of the position of an interference source in a sound pickup space, a reference signal is determined from a sound signal in the sound pickup space, and on the basis of the reference signal, the sound of the interference source is then filtered out from the sound signal, such that a target sound signal is enhanced. In the method, sound signal processing is performed according to the position of an interference source, and the sound of the interference source is shielded in a targeted manner, such that a target sound signal is enhanced, and the sound quality is improved.

Description

Sound signal processing method, device, equipment and storage medium

technical field

The present application relates to the technical field of audio processing, and in particular to an audio signal processing method, device, equipment and storage medium.

Background technique

In a multi-person conference, when the speaker is speaking, there will inevitably be some disturbing sounds in the venue, such as private conversations between participants and sudden mobile phone ringtones. In the case that a participant needs to have a private conversation, the sound pickup device can be turned off so that the sound pickup device no longer picks up the sound signal in the area where the participant is located, thereby avoiding interference to the speaker's voice.

However, the sound signal will inevitably be picked up by multiple adjacent sound pickup devices at the same time, which will cause interference to the voice of the speaker and greatly affect the sound quality of the conference.

Contents of the invention

The present application provides a sound signal processing method, device, equipment and storage medium, which can effectively improve sound quality. The technical solution is as follows:

In a first aspect, a sound signal processing method is provided, the method comprising:

Pick up the sound signal in the sound pickup space through the sound pickup device;

determining the position of the interference source in the sound pickup space;

Determining a reference signal from the sound signal based on the location of the interference source, the reference signal being used to filter out the sound of the interference source;

Based on the reference signal, the target sound signal is enhanced.

Wherein, the interference source refers to a sound source that is considered to cause interference among multiple sound sources existing in the sound pickup space, for example, a participant having a private conversation in a conference. By acquiring the position of the interference source in the sound pickup space, corresponding sound signal processing can be performed on the interference source. Based on different system deployment conditions, the location of interference sources will be determined in different ways. For example, in the case of deploying multiple microphones, the location of the interference source is determined based on the number of the microphone corresponding to the interference source; for another example, in the case of deploying a microphone array, the interference source is determined based on the angle of the interference source relative to the microphone array source location.

Wherein, the target sound signal refers to: among the multiple sound sources existing in the sound pickup space, the sound signal corresponding to the focused sound source, for example, the sound signal corresponding to the speaker in the conference.

Wherein, enhancing the target sound signal refers to: suppressing the reference signal in the sound signal to enhance the target sound signal, for example, by reducing the proportion of the corresponding part of the reference signal in the sound signal to increase the target sound in the sound signal The proportion of the signal, so as to achieve the purpose of enhancing the target sound signal.

In the technical solution provided by the embodiment of the present application, based on the position of the interference source in the sound pickup space, the reference signal is determined from the sound signal in the sound pickup space, and then the sound of the interference source in the sound signal is filtered out based on the reference signal, so as to enhance the target sound signal. Through the above technical solution, the sound signal processing is performed according to the position of the interference source, and the sound of the interference source can be shielded in a targeted manner to enhance the target sound signal, thereby improving the sound quality.

In a possible implementation manner, the determining the position of the interference source in the sound pickup space includes:

The location selection instruction is received, and the location corresponding to the location selection instruction is determined as the location of the interference source in the sound pickup space.

Through the above technical solution, multiple ways are provided to determine the location of the interference source based on the location selection instruction, and the participants can set it according to their needs, which effectively improves the practicability of the sound signal processing method.

In a possible implementation manner, the location selection instruction is triggered based on a selection operation of the location of the interference source in the control device.

Wherein, the control device is used to select the location of the interference source, for example, the control device is integrated on a microphone, or the control device may be a conference touch panel.

Through the above technical solution, based on various control devices actually deployed in the conference scene, multiple methods are provided to determine the location of the interference source, which further improves the practicability of the sound signal processing method while ensuring the positioning accuracy.

In a possible implementation manner, the position selection instruction is triggered by an image acquisition device when the first body behavior is detected in the collected image, and the image acquisition device is used to perform image acquisition for the sound pickup space , the first body behavior is used to indicate to mute the location.

Wherein, the first body behavior indicates mute at its position, for example, the participant puts the index finger vertically close to the lip.

Through the above technical solution, participants do not need to manually select, and can automatically locate the interference source based on the image, and realize the intelligent shielding of the interference source in the conference scene, which improves the conference experience while ensuring the sound quality.

Further, through the above technical solution, the position of the interference source can be obtained directly from the position selection instruction, the amount of data involved in the calculation process is reduced, and the efficiency of sound signal processing is improved.

Detecting the target image collected by the image acquisition device, the image acquisition device is used for image acquisition for the sound pickup space;

In response to detecting a first body action in the target image, determining a position of the first body action in the sound pickup space as the position of the interference source, the first body action being used to indicate the The above position is muted.

Through the above technical solution, the position of the interference source is determined based on the real-time image, which ensures the accuracy of the position of the interference source and further improves the sound quality.

In a possible implementation manner, in response to detecting the first body action in the target image, determining the position of the first body action in the sound pickup space as the position of the interference source includes:

In response to detecting a first body action in the target image, acquiring a position of the first body action in the target image;

Based on the position of the first body behavior in the target image and the spatial position of the image acquisition device in the sound pickup space, calculating the spatial position of the first body behavior in the sound pickup space, Determine the location of the interference source.

In a possible implementation manner, the method also includes:

In response to detecting a second body action in the image of the target, determining a position of the second body action in the sound pickup space as the position of the target, the second body action being used to indicate the The above-mentioned target sound signal is enhanced.

Wherein, the second body behavior is used to instruct to enhance the target sound signal, for example, the participant puts the index finger horizontally close to the lip to indicate that the participant needs to speak.

Through the above technical solution, the position corresponding to the target sound signal is determined based on the second body behavior, so that the target sound signal can be enhanced in a targeted manner, thereby improving the sound quality.

In a possible implementation manner, the method also includes:

Tracking the location of the interference source;

The determining the reference signal from the sound signal based on the location of the interference source includes:

Based on the tracked change in the position of the interference source, a reference signal is re-determined from the sound signal.

Through the above technical solution, the interference source can be locked after the interference source is determined, so that the location of the interference source can be determined based on the real-time position change, and the accuracy of the location of the interference source can be ensured by capturing the change of the location of the interference source in time. Further, Ensure that in the changing actual conference scene, the sound signal can always be processed against the interference source to ensure the sound quality.

In a possible implementation manner, the sound pickup device includes multiple microphones, and the determining the reference signal from the sound signal based on the location of the interference source includes:

The sound signal originating from the microphone corresponding to the location of the interference source is determined as a reference signal.

Through the above technical solution, in the scenario of picking up sound based on multiple microphones, the representative reference signal for the interference source can be determined based on the microphone corresponding to the interference source, so that the interference can be better filtered out based on the reference signal The sound source of the source can effectively improve the sound quality.

In a possible implementation manner, the multiple microphones have a positioning function.

Through the above technical solution, multiple microphones can be randomly placed according to the needs, which greatly reduces the scene restrictions during equipment deployment. While improving the flexibility of equipment deployment in the sound processing system, the real-time positioning of the microphones can realize the detection of interference sources. Accurate positioning, so that the sound of the interference source can be filtered out from the sound signal more accurately, and the sound quality can be effectively guaranteed.

In a possible implementation manner, the sound pickup device is a microphone array, and the determining the reference signal from the sound signal based on the location of the interference source includes:

Based on the angle information of the position of the interference source, determine a beam angle range matching the angle information;

Based on the beam angle range, a reference signal is determined from the sound signals picked up by the microphone array.

Wherein, the angle information of the position of the interference source refers to the angle of the interference source relative to the microphone array.

Wherein, the beam angle range refers to the angle range covered by the beam formed by the microphone array. Based on the specified beam angle range, it is possible to determine the sound signal within the pickup range at the specified angle to the microphone array.

The method provided by the embodiment of the present application can adapt to the spatial arrangement characteristics of the microphone array in the scenario where the microphone array is used to pick up sound, and use the angle information of the interference source to obtain the angle within the specified angle range targeted to the interference source. The sound signal ensures the representativeness of the reference signal to the interference source, improves the accuracy of sound signal processing for the interference source, and effectively improves the sound quality.

In a possible implementation manner, after determining the reference signal from the sound signal based on the location of the interference source, the method further includes:

determining a noise threshold based on the reference signal;

determining a signal-to-noise ratio of a reference signal based on the noise threshold and the reference signal;

Set the reference signal whose signal-to-noise ratio is smaller than the target threshold to 0.

In the embodiment of the present application, through the above technical solution, it is possible to mute the non-human voice part of the reference signal, obtain a reference signal including a purer human voice, improve the efficiency of subsequent sound signal processing based on the reference signal, and further improve the sound quality. quality.

In a possible implementation manner, the enhancing the target sound signal based on the reference signal includes:

Based on the reference signal, determine a first sound signal from the sound signals in the sound pickup space, the signal energy of the first sound signal is less than the signal energy of the reference signal, and the first sound signal the correlation with the reference signal is greater than a correlation threshold;

Based on the reference signal, the target sound signal in the first sound signal is enhanced.

Wherein, the magnitude of the signal energy can represent the strength of the human voice in the sound signal to a certain extent.

Among them, the correlation between signals can reflect the degree of mutual influence between signals.

Through the above technical solution, it is possible to determine the first sound signal that is greatly affected by the interference source from the multi-channel sound signals, and then filter out the sound of the interference source in the first sound signal in a targeted manner, by improving the accuracy of filtering performance, effectively improving the sound quality. Considering that in the actual conference scene, due to the need for private conversation, the participants think that they are the source of interference, the above-mentioned technical solution can ensure the privacy of the conversation of the participants in the conference scene on the basis of improving the sound quality, effectively Improved user experience.

In a possible implementation manner, the enhancing the target sound signal in the first sound signal based on the reference signal includes:

Using the reference signal as one input of the filter, using the first sound signal as the other input of the filter, and filtering out the first sound signal related to the reference signal through the filter to enhance the target sound signal in the first sound signal, and output a filtering result.

Through the above technical solution, the first sound signal is filtered based on the reference signal, so that the influence of the sound of the interference source on the first sound signal can be reduced in a targeted manner.

In a possible implementation manner, the filter includes a first filter and a second filter,

Filtering out the part related to the reference signal in the first sound signal through the filter, so as to enhance the target sound signal in the first sound signal, and outputting the filtering result includes:

Obtain an estimated signal of the reference signal through the first filter, the parameters of the first filter are determined based on the parameters of the second filter, and the parameters of the second filter are determined based on results of multiple filtering Determine the difference between;

Based on the estimated signal, filter the estimated signal in the first sound signal, and output a filtering result.

In the above technical solution, the adaptive filter can adjust the parameters of the filter through an adaptive algorithm during the filtering process, so as to obtain a better filtering effect.

In a possible implementation manner, the method also includes:

Adjusting parameters of the second filter based on the difference between the nth filtering result of the second filter and the n-1th filtering result, where n is an integer greater than 1;

If the adjusted parameters of the second filter satisfy a convergence condition, configure the adjusted parameters of the second filter to the first filter.

Through the above technical solution, the speed of parameter convergence of the adaptive filter can be effectively improved, thereby improving the efficiency of filtering.

In a possible implementation manner, the method also includes:

In a case where the attenuation of the first sound signal before and after filtering is greater than an attenuation threshold, the filtered first sound signal is clipped.

Through the above technical solution, in the case that the filtered first sound signal is weakened and causes distortion, the quality of the sound signal can be guaranteed.

A second aspect provides an audio signal processing device, which includes a plurality of functional modules for executing corresponding steps in the audio signal processing method provided in the first aspect.

In a third aspect, a sound signal processing device is provided, the sound signal processing device includes a processor and a memory, the memory is used to store at least one piece of program code, the at least one piece of program code is loaded by the processor and executes the above-mentioned sound signal Approach.

In a fourth aspect, a computer-readable storage medium is provided, and the computer-readable storage medium is used to store at least one piece of program code, and the at least one piece of program code is used to execute the above-mentioned sound signal processing method.

In a fifth aspect, a computer program product is provided. When the computer program product is run on a sound signal processing device, the sound signal processing device is made to execute the above sound signal processing method.

Description of drawings

FIG. 1 is a schematic structural diagram of a sound processing system provided by an embodiment of the present application;

Fig. 2 is a schematic diagram of deployment of a sound processing system provided by an embodiment of the present application;

Fig. 3 is a schematic diagram of deployment of a sound processing system provided by an embodiment of the present application;

Fig. 4 is a schematic diagram of deployment of a sound processing system provided by an embodiment of the present application;

Fig. 5 is a schematic diagram of deployment of a sound processing system provided by an embodiment of the present application;

FIG. 6 is a flow chart of a sound signal processing method provided by an embodiment of the present application;

FIG. 7 is a schematic diagram of an adaptive filter provided by an embodiment of the present application;

FIG. 8 is a flow chart of a sound signal processing method provided in an embodiment of the present application;

FIG. 9 is a flow chart of a sound signal processing method provided by an embodiment of the present application;

FIG. 10 is a flow chart of a sound signal processing method provided by an embodiment of the present application;

FIG. 11 is a schematic diagram of a distributed microphone positioning process provided by an embodiment of the present application;

Fig. 12 is a schematic structural diagram of an audio signal processing device provided in an embodiment of the present application;

Fig. 13 is a schematic diagram of a hardware structure of an audio signal processing device provided by an embodiment of the present application.

Detailed ways

In order to make the purpose, technical solution and advantages of the present application clearer, the implementation manners of the present application will be further described in detail below in conjunction with the accompanying drawings.

Before introducing the technical solutions provided by the embodiments of the present application, key terms involved in the present application will be described below.

Root mean square (root mean square, RMS): By summing the squares of all discrete values of the signal, then taking the mean of the summed results, and finally taking the square root of the mean, the root mean square of the signal is obtained. In physics, the root mean square is the effective value of a signal (such as a current signal and a voltage signal), and is used to characterize the energy of the signal.

Signal noise ratio (SNR): SNR refers to the ratio of signal to noise in an electronic device or electronic system, for example, the ratio of signal energy to noise energy. Signal refers to the signal from the outside of the device that needs to be processed by this device. Noise refers to the irregular extra signal (or information) that does not exist in the original signal after passing through the device, and the signal does not follow the original signal. signal changes.

Sub-band: Sub-band coding technology is a technology that converts the original signal from the time domain to the frequency domain, then divides it into several sub-bands, and digitally encodes them respectively. It uses a band-pass filter bank to divide the original signal into several subbands, each subband corresponds to a specified frequency bandwidth, that is, each subband corresponds to a specified signal frequency.

Background noise suppression (automatic noise suppression, ANS) technology is used to detect the noise of background fixed frequency (such as: fan sound and air conditioner sound) and automatically filter out, so as to present the clear voice of the participants, widely used in video conferencing, voice In the sound signal processing technology in conferences and other scenarios.

Cross correlation (cross correlation, CC): The result of cross correlation operation reflects the measure of similarity between two signals.

Adaptive filter (adaptive filter, ADF): The adaptive filter can adaptively adjust the parameters of the filter according to the difference from the expected signal based on the characteristics of the input signal to ensure the filtering effect. Therefore, the adaptive filter is widely used. It is widely used in signal system identification, signal prediction and noise elimination.

Next, the technical solutions provided by the embodiments of the present application will be introduced.

The sound signal processing method provided in the embodiment of the present application is applied to a sound signal processing device. For example, in a conference scene such as a video conference or a voice conference, the sound signal processing device may be a conference terminal or a smart speaker. Wherein, the sound signal processing device is used for processing the sound signal picked up by the sound pickup device from the sound pickup space. For example, in a conference scenario, the conference terminal performs noise reduction on sound signals in the conference site.

Pickup equipment is used to pick up sound signals. The sound pickup device has various forms, for example, the sound pickup device may be a microphone or a microphone array, and the like. The microphone may be a fixed microphone, for example, a desktop embedded microphone; the microphone may also be a movable microphone. Among them, the microphone array refers to an array structure obtained by arranging multiple microphones according to a certain spatial structure. According to the spatial characteristics of the array structure, the microphone array can process sound signals in multiple directions to obtain sound signals in various angle ranges. . According to different usage scenarios, different forms of sound pickup devices can be selected to pick up sound signals, and the form of the sound pickup device is not limited in the embodiments of the present application.

The pickup space is a pre-configured three-dimensional pickup area. The sound pickup space may be a closed space, ie the size of the sound pickup space is limited. For example, the sound pickup space can be in the shape of a cuboid, and the size of the sound pickup space can be represented by length, width, and height. Alternatively, the sound pickup space may also be an open space, for example, the height of the sound pickup space is not limited. The size and shape of the sound pickup space can be set according to sound pickup requirements or sound pickup scenarios, and the embodiment of the present application does not limit the size and shape of the sound pickup space.

Fig. 1 is a schematic structural diagram of a sound processing system provided by an embodiment of the present application. As shown in FIG. 1 , the sound processing system includes a sound pickup device, a sound signal processing device, and a sound pickup control device. Wherein, the sound pickup control device is used to determine the position of the interference source in the sound pickup space. In some embodiments, the sound pickup control device includes a control device for selecting the location of the interference source, for example, the control device is integrated on a microphone, or the control device may be a conference touch panel. In some embodiments, the sound pickup control device includes an image collection device, and the image collection device is used for image collection of a sound pickup space, for example, a camera in a venue. Wherein, the sound signal processing device obtains the sound signal picked up by the sound pickup device, and determines the position of the interference source in the sound pickup space through the sound pickup control device, and then determines the reference signal based on the position of the interference source, so as to filter out the sound through the reference signal The sound of the interference source in the signal, to achieve the purpose of enhancing the target sound signal. It can be understood that the sound processing system shown in FIG. 1 is only used as an example for illustration, and is not used as a limitation to the sound processing system applied to the solution of the present application.

In the embodiment of the present application, since the composition of the sound processing system is different, the deployment method of the sound processing system may also be different. The embodiment of the present application is based on Fig. 2 to Fig. 5, for the four types of The deployment of the sound processing system is schematically illustrated. The technical solution of the present application will be described below by taking the audio signal processing device as a conference terminal as an example.

Fig. 2 is a schematic diagram of deployment of a sound processing system provided by an embodiment of the present application. The sound processing system is applied in a conference scene, and the sound pickup space is the meeting place. As shown in FIG. 2 , the sound processing system includes: a plurality of microphones 210 as sound pickup devices; a conference terminal 220 as a sound signal processing device; and a conference touch panel 230 as a sound pickup control device. Wherein, the conference terminal 220 is deployed on the wall of the venue, and the multiple microphones 210 are deployed at designated positions on the conference table. The conference terminal 220 can acquire sound signals picked up by the plurality of microphones 210 from the venue. Wherein, the multiple microphones 210 have physical buttons 211 . In response to a selection operation on the physical button 211 of any microphone, the microphone returns a selection instruction for the microphone to the conference terminal, and then the selected microphone is determined as the microphone corresponding to the interference source. In some embodiments, the microphone is corresponding to an indicator light, and the indicator light is used to indicate the selection state of the corresponding microphone. For example, if the indicator light is on to indicate that the microphone is selected, the microphone is determined to be the microphone corresponding to the interference source. Optionally, the conference touch panel 230 provides the function of selecting a microphone. In response to the selection operation of any microphone on the conference touch panel, the conference touch panel 230 returns a microphone selection instruction to the conference terminal 220, and the selection instruction indicates that the microphone is determined as the microphone corresponding to the interference source. In some embodiments, the conference touch panel 230 can control the indication status of the indicator light corresponding to the microphone to indicate the selection status of the microphone, for example, the conference touch panel controls the indicator light corresponding to the microphone to light up, indicating that the microphone is selected, Then the microphone is determined as the microphone corresponding to the interference source. It should be noted that the above is only an exemplary description, and the embodiment of the present application does not limit the deployment location of the conference terminal. For example, the conference terminal may be deployed on a movable stand in the conference venue.

Fig. 3 is a schematic diagram of deployment of another sound processing system provided by an embodiment of the present application. The sound processing system is applied in a meeting scene, and the sound pickup space is the meeting place. As shown in FIG. 3 , the sound processing system includes: a plurality of microphones 310 as sound pickup devices; a conference terminal 320 as a sound signal processing device; a conference touch panel 330 and a camera 340 as sound pickup control devices. Wherein, the multiple microphones 310 have physical buttons 311 . In the sound processing system corresponding to FIG. 3 , the configuration of the sound processing system other than the camera 340 is the same as that of the sound processing system corresponding to FIG. 2 , which will not be repeated here. Wherein, the camera 340 is deployed on the wall of the meeting place, and is used to collect images in the meeting place. In some embodiments, the camera is an external camera connected to the conference terminal. In some other embodiments, the camera is a built-in camera of the conference terminal. Optionally, the camera has data processing capability, can process the collected images, and send a microphone selection instruction to the conference terminal, instructing the conference terminal to process the sound signal from the microphone accordingly. It should be noted that the above is only an exemplary description, and the embodiment of the present application does not limit the deployment position of the camera. For example, the camera may also be hung on the ceiling in the venue.

Fig. 4 is a schematic diagram of deployment of another sound processing system provided by an embodiment of the present application. The sound processing system is applied in a meeting scene, and the sound pickup space is the meeting place. As shown in Figure 4, the sound processing system includes: a microphone array 410 as a sound pickup device; a conference terminal 420 as a sound signal processing device; a desktop physical button 430 as a sound pickup control device, a conference touch panel 440 and a camera 450 . In some embodiments, the microphone array and the conference terminal are physically integrated as one device, that is, the conference terminal has a built-in microphone array. In some other embodiments, the microphone array and the conference terminal are physically separated two devices. Optionally, based on the actual situation of the venue, you can choose where the device is deployed in the venue, so that the sound pickup range of the microphone array can cover the venue evenly. For example, deploy the conference terminal with the built-in microphone array in the middle of the wall of the venue . Wherein, the conference terminal 420 can obtain sound signals from various angle ranges in the venue from the microphone array 410 . Wherein, the desktop physical button 430 is used to select a position in the meeting place. In some embodiments, in response to a selection operation on any desktop physical button 430 , the desktop physical button 430 returns a selection instruction for the location of the desktop physical button in the sound pickup space to the conference terminal 420 . In some embodiments, the desktop physical button 430 is corresponding to an indicator light, and the indicator light is used to indicate the selection state of the corresponding position of the desktop physical button. For example, if the indicator light is on to indicate that the corresponding position is selected, the position is determined as interference The location corresponding to the source. Optionally, the meeting touch panel 440 provides a function of selecting a location in the meeting place. In response to a selection operation for any position on the conference touch panel, the conference touch panel 440 returns an instruction for selecting any position in the conference site to the conference terminal 420 . In some embodiments, the conference touch panel 440 can control the indication state of the indicator light to indicate the selection status of the location in the venue. For example, the conference touch panel control indicator light is on, and the position corresponding to the indicator light is selected. Then the position is determined as the position corresponding to the interference source. Wherein, for the camera 450, refer to the description of the camera 340 in the audio processing system corresponding to FIG. 3 above, which will not be repeated here. In some embodiments, the microphone array 410, the conference terminal 420 and the camera 450 are integrated together as one device, that is, the conference terminal has a built-in microphone array and camera.

Fig. 5 is a schematic diagram of deployment of another sound processing system provided by an embodiment of the present application. The sound processing system is applied in a conference scene, and the sound pickup space is the meeting place. As shown in Figure 5, the sound processing system includes: a plurality of distributed microphones 510 with positioning functions as sound pickup devices; a conference terminal 520 as a sound signal processing device; a desktop physical button 530 as a sound pickup control device, a meeting A touch panel 540 and a camera 550 . The distributed microphone 510 is randomly placed on the conference table in front of the conference terminal, and the position of the distributed microphone can be updated in real time in the conference terminal 520 . Optionally, the conference touch panel acquires the positions of distributed microphones to provide a function of selecting distributed microphones. In response to the selection operation of any distributed microphone on the conference touch panel, the conference touch panel returns a selection instruction for the distributed microphone to the conference terminal, indicating that the distributed microphone is selected, and the distributed microphone is selected. The microphone is determined as the distributed microphone corresponding to the interference source. Optionally, the physical button 530 on the desktop is used to select a distributed microphone in the venue. In some embodiments, in response to a selection operation for any desktop physical button 530 , the desktop physical button 530 returns a selection instruction for the location of the desktop physical button in the sound pickup space to the conference terminal 520 . The conference terminal selects the distributed microphone closest to the physical button on the desktop based on the location of the physical button on the desktop and the locations of the multiple distributed microphones. In some embodiments, the distributed microphone 510 corresponds to an indicator light, which is used to indicate the selection status of the corresponding distributed microphone. For example, if the indicator light is on to indicate that the corresponding distributed microphone is selected, the distributed microphone is determined to be Distributed microphones corresponding to interference sources. Wherein, for the camera 550, refer to the description of the camera 340 in the audio processing system corresponding to FIG. 3 above, which will not be repeated here.

It should be noted that, in the above-mentioned sound processing systems in FIGS. 1 to 5 , data transmission between devices can be performed through wireless communication, or data transmission can be performed through wired communication, which is not discussed in this embodiment of the present application. Do limited.

In some embodiments, in any of the above-mentioned sound processing systems, the sound signal processing device can obtain information such as the size and shape of the sound pickup space and the position information of each device in the sound pickup space, for example, the length of the sound pickup space, Width and height, position information of microphone (or microphone array), conference terminal, camera in sound pickup space, and numbers of multiple microphones, etc.

In some embodiments, the conference terminal in the above audio processing system is used as a local conference terminal, and can send the processed audio signal to the remote conference terminal. A remote conference terminal refers to a conference terminal that participates in the same conference as a local conference terminal and is deployed in a different area. Optionally, the local conference terminal and the remote conference terminal are connected through a multimedia control platform. The local conference terminal can send the enhanced sound signal to the multimedia control platform, and the multimedia control platform mixes and codes the received sound signal and sends it to the remote conference terminal. Of course, the conference terminal can also integrate part or all of the functions of the multimedia control platform, and the local conference terminal can mix and encode the enhanced audio signal and send it directly to the remote conference terminal.

In this embodiment of the application, the sound signal processing device obtains information such as the size and shape of the sound pickup space and the position information of each device in the sound pickup space in the sound pickup space based on the actual needs of the conference scene and the deployment of the device, so as to ensure The deployment of the sound processing system is adapted to the meeting scene, so that the sound signal processing equipment can perform sound signal processing based on the actual situation of the meeting scene, which improves the flexibility and compatibility of the sound signal processing method, and provides sound quality in different meeting scenes. Assure.

Through the above-mentioned Figures 1 to 5, the sound processing system provided by the embodiment of the present application is introduced from the perspective of system architecture and system deployment, and the flow of the sound signal processing method provided by the embodiment of the present application will be described below based on the above-mentioned sound processing system for example.

Fig. 6 is a flowchart of a sound signal processing method provided by an embodiment of the present application. This method is applied to the sound processing system corresponding to FIG. 2 above, and the sound processing system includes multiple microphones, a conference terminal, and a conference touch panel. The sound signal processing method is executed by the conference terminal. As shown in Figure 6, the method includes:

601. The conference terminal picks up sound signals in the sound pickup space through multiple microphones.

In the embodiment of the present application, the sound processing system includes a plurality of microphones, a conference terminal, and a conference touch panel. In some embodiments, the sound processing system runs based on the system control software, and the sound processing system needs to be configured based on the system control software before performing sound signal processing. For example, system control software is installed on the conference terminal, and the conference terminal can obtain configuration information of the sound processing system through the system control software. For example, the configuration information entered on the configuration interface of the system control software is acquired. In some embodiments, the configuration information includes: the length, width and height of the sound pickup space; the position information of the multiple microphones and the conference terminal in the sound pickup space, for example, the microphones in the spatial coordinate system corresponding to the sound pickup space The coordinates of the multiple microphones; the number of the multiple microphones and the corresponding pickup range of each microphone. Of course, the sound processing system can be reconfigured through the system control software. For example, if the scope of the pickup space needs to be adjusted, the length, width and height of the pickup space can be adjusted through the system control software. Based on the configuration information, the conference terminal can determine the positions, numbers and sound pickup ranges of the multiple microphones in the sound pickup space, so as to obtain multiple sound signals in the sound pickup space through the multiple microphones.

In some embodiments, the above-mentioned system control software is installed on the conference touch panel, and correspondingly, the configuration information of the sound processing system can be acquired through the conference touch panel.

Wherein, the conference terminal acquires multiple sound signals from the sound pickup space through the multiple microphones. In some embodiments, since each sound signal includes sound within a certain pickup range corresponding to the microphone, each sound signal may be composed of sound signals from multiple sound sources. For example, in a conference scene, multiple If two participants speak at the same time, the sound signal picked up by one microphone may include the voices of multiple participants within the pickup range. The proportion of the sound signals of the multiple sound sources in the sound signals picked up by one microphone is determined according to the relative position of each sound source and the microphone. For example, for a microphone, the closer the participant is to the microphone, the The larger the proportion of the corresponding sound signal in the sound signal picked up by the microphone, that is, the louder the volume of the participant's voice is in the sound signal picked up by the microphone.

602. The conference terminal receives the location selection instruction, and determines the location corresponding to the location selection instruction as the location of the interference source in the sound pickup space.

In the embodiment of the present application, there are multiple sound sources in the sound pickup space, and the interference source refers to the sound source that is considered to cause interference among the multiple sound sources in the sound pickup space, for example, the Participants in private conversations. Based on the received position selection instruction, the conference terminal can determine the sound source considered as the interference source based on the position, so that in the subsequent sound signal processing process, the sound of the interference source is processed, for example, the interference in the sound signal is filtered out source sound.

Wherein, the location selection instruction is triggered based on a selection operation of the location of the interference source on the control device. In some embodiments, the microphone corresponding to the selection operation is considered to be the microphone closest to the interference source, therefore, the microphone corresponding to the selection operation is taken as the microphone corresponding to the interference source. In some embodiments, the selection operation includes pressing a physical button corresponding to the microphone. Based on the pressing operation, a location selection instruction for the location of the microphone can be triggered, and the conference terminal will select the location according to the received location selection instruction. The location of the microphone is determined as the location of the interference source. In some other embodiments, the selection operation includes a selection operation of the microphone on the conference touch panel, and the conference touch panel sends a location selection instruction for the location of the microphone to the conference terminal in response to the selection operation. In some embodiments, in response to receiving the location selection instruction, the conference terminal acquires the microphone number carried in the location selection instruction, and determines the microphone location corresponding to the microphone number as the location of the interference source.

In some embodiments, the microphone is corresponding to an indicator light, and after the microphone is determined to be the microphone corresponding to the location of the interference source, the indication state of the indicator light corresponding to the microphone is switched to indicate that the microphone corresponds to the interference source.

In some other embodiments, after the microphone corresponding to the location of the interference source is determined, the microphone is displayed on the conference touch panel as the microphone corresponding to the interference source.

It should be noted that the above step of switching the state of the indicator light and the step displayed on the conference touch panel can be executed synchronously or sequentially, which is not limited in this embodiment of the present application.

603. The conference terminal determines the sound signal originating from the microphone corresponding to the position of the interference source as a reference signal.

In the embodiment of the present application, after determining the microphone corresponding to the location of the interference source, the conference terminal determines the sound signal picked up by the microphone as the reference signal. Among them, since the reference signal comes from the microphone closest to the interference source, the proportion of the sound signal of the interference source in the reference signal is greater than the proportion of the sound signal of the interference source in the sound signals of other microphones, that is, Yes, the reference signal is better representative of the sound signal of the interferer than the sound signal picked up by other microphones. Based on this, the reference signal can represent the sound signal of the interference source during sound signal processing, and is used to filter out the sound of the interference source.

604. The conference terminal performs denoising on the reference signal.

In the embodiment of the present application, since the reference signal originates from multiple sound sources in the sound pickup space, when there is noise in the reference signal, when the denoised reference signal is used to filter out the sound of the interference source, it can for better filtering.

In the embodiment of this application, the process of denoising the reference signal includes the following steps 6041 to 6042:

6041. The conference terminal determines the noise threshold based on the reference signal.

In some embodiments, the reference signal is divided into multiple signal frames of a specified time length (for example, 30 milliseconds), and the reference signal is denoised with the signal frame as the minimum processing unit.

In some embodiments, based on the principle of minimum global amplitude spectrum, it is considered that the sound corresponding to the signal frame with the smallest signal amplitude spectrum is not human voice, and the non-human voice is considered as noise. In some embodiments, since the signal energy is positively correlated with the amplitude spectrum of the signal, the magnitude of the amplitude spectrum can be compared based on the signal energy of the signal frame. Based on this, the minimum signal energy is determined as the noise threshold in the conference scene from the signal energy of 100 (or other values) signal frames of the reference signal, and the reference signal is denoised based on the noise threshold. Wherein, the noise threshold is used as a criterion for judging the human voice, and the signal frames whose signal energy is lower than the noise threshold are noise, that is, non-human voice. Wherein, the principle of calculating signal energy refers to formula (1).

In the formula (1), X is the signal amplitude set corresponding to the signal frame; N is the number of signal amplitudes of the signal frame X, and N is a positive integer; RMS _X is the signal energy of the signal frame X.

In some embodiments, the conference terminal uses the recursive average noise estimation algorithm to determine the long-term stationary noise energy in the conference scene according to the reference signal obtained in real time, and continuously updates the noise threshold in the conference scene with the long-term stationary noise energy .

In some embodiments, the determination process of determining the long-term stationary noise energy based on the recursive average noise estimation algorithm refers to formula (2) to formula (4). The smoothing coefficient is determined based on the speech existence probability of the current signal frame through a recursive average noise estimation algorithm. The closer the speech existence probability of the current signal frame is to 1, the smoother the coefficient tends to be 1, indicating that the signal energy of the previous signal frame is tended to be used as the noise energy estimation of the current signal frame; the closer the speech existence probability of the current signal frame is to 0 , then the smoothing coefficient tends to 0, indicating that the signal energy of the current signal frame is tended to be used as the noise energy estimate.

ρ'(k,l)=α _p ρ'(k,l-1)+(1-α _p )I(k,l) (2)

Based on formula (2), it can be determined that the kth signal frame of the reference signal is located at the speech existence probability ρ'(k, l) of the l subband, and α _p (0<α _p <1) is the first smoothing constant, where , when the signal energy at the l subband of the k signal frame is greater than the preset noise threshold, I(k,l) is 1; the signal energy at the l subband of the k signal frame is less than the preset In the case of the noise threshold, I(k,l) is 0.

Based on formula (3), it is possible to calculate the (time-varying) smoothing coefficient of the kth signal frame of the reference signal at the l subband

α _d (0<α _d <1) is the second smoothing constant.

After determining the speech existence probability of the signal frame and the smoothing coefficient corresponding to the signal frame, based on formula (4), the noise energy spectrum of the kth signal frame of the reference signal at the l+1 subband can be determined

Wherein, the Y(k, l) is a signal expression in which the kth signal frame of the reference signal is located at the l subband. based on

The long-term stationary noise energy can be updated.

6042. The conference terminal determines the signal-to-noise ratio of the reference signal based on the noise threshold and the reference signal, and sets the reference signal whose signal-to-noise ratio is smaller than the target threshold to 0.

Based on the noise threshold, the conference terminal calculates the ratio of the signal energy of each signal frame to the noise threshold, that is, the signal-to-noise ratio of the signal frame. If the signal-to-noise ratio of the signal frame is less than the target threshold, the signal frame is likely to be noise, and the signal amplitude of the signal frame is set to 0. Wherein, the principle of calculating the signal-to-noise ratio refers to the formula (5).

In the formula (5), X is the signal amplitude set corresponding to the signal frame; SNR _X is the signal-to-noise ratio of the signal frame X; RMS _X is the signal energy of the signal frame X; RMS _N is the noise energy (or long-term stationary noise energy ), that is, the noise threshold. The RMS _N may be the noise energy determined based on multiple local signal frames of the reference signal, or the long-term stationary noise energy determined based on the accumulation of the reference signal, which is not limited in this embodiment of the present application.

In the embodiment of the present application, through the above step 604, it is possible to mute the non-human voice part of the reference signal, obtain a reference signal including a purer human voice, improve the efficiency of subsequent sound signal processing based on the reference signal, and further improve the sound quality.

It should be noted that step 604 is an optional step, and in some embodiments, step 605 may be performed directly based on the reference signal determined in step 603 .

In some embodiments, after performing the above step 604, the conference terminal inputs the denoised reference signal and other multi-channel sound signals from the sound pickup space into the ANS module for processing, so as to filter out the reference signal The background noise in and the background noise in the other multi-channel sound signals, thereby improving the efficiency of subsequent sound signal processing and further improving the sound quality.

605. The conference terminal determines the first sound signal from the sound signals in the sound pickup space based on the denoised reference signal.

In the embodiment of the present application, the reference signal is used to filter out the sound of the interference source. Therefore, it is first necessary to determine the sound signal affected by the interference source from the sound signals in the pickup space, and then based on the reference signal, Filter out the sound of the interference source in a targeted manner.

In some embodiments, the first sound signal affected by the sound signal of the interference source is determined based on the magnitude of the signal energy and the correlation with the reference signal. Wherein, since the signal energy is positively correlated with the amplitude spectrum of the signal, the magnitude of the signal energy can represent the strength of the human voice in the sound signal to a certain extent. If the signal energy of one sound signal is greater than the signal energy of the other sound signal, it means that there is indeed a human voice in the sound signal, and the strength of the human voice can affect the other sound signal. Further, if a sound signal of one path is affected by the interference source, it means that the sound signal of the sound signal of the path will continue to be interleaved with the sound signal of the interference source, therefore, the correlation between the sound signal of the path affected and the sound signal of the interference source will be higher than Other audio signals that are not affected. Based on this, since the reference signal can well represent the sound signal of the interference source, when the signal energy of the first sound signal is less than the signal energy of the reference signal, and the correlation between the first sound signal and the reference signal is greater than The correlation threshold indicates that the reference signal affects the first sound signal, that is, the first sound signal is affected by the sound of the interference source. For example, if the source of interference is participant A who is having a private conversation at a certain volume, and there is a participant B next to the participant A, the sound signal picked up by the microphone in front of the participant B will continue to be interwoven with the The voice of participant A having a private conversation, therefore, the sound signal picked up by the microphone in front of participant B is the sound signal affected by the interference source, that is, the first sound signal. Wherein, the correlation threshold can be set based on the accuracy requirement of the sound signal processing, which is not limited in this embodiment of the present application.

In some embodiments, the conference terminal receives the reference signal and other multi-channel sound signals processed by the ANS module, and based on the signal energy of the reference signal, the signal energy of other multi-channel sound signals, and the relationship between the other multi-channel sound signals and the reference signal and determine the first sound signal from the other multi-channel sound signals. Wherein, the calculation principle of the signal energy refers to the above formula (1).

It should be noted that the embodiment of the present application uses the signal frame as the smallest unit to compare the signal energy. In some embodiments, the comparison of the signal energy can also be based on the average energy of multiple signal frames within a period of time, so as to improve the energy The accuracy of the comparison.

In some embodiments, the magnitude of the correlation between the signals can be reflected by the cross-correlation value between the signals, and the principle of calculating the cross-correlation value between the signals can be referred to formula (6).

In formula (6), f(t) and g(t) are two signals;

is the cross-correlation value between signal f(t) and signal g(t).

In some embodiments, for any sound signal, if the signal energy of the sound signal is greater than the signal energy of the reference signal, and the correlation with the reference signal is greater than the correlation threshold, it indicates that the reference signal is not In this case, the reference signal is set to zero, for example, the signal amplitudes of multiple signal frames in the reference signal are set to 0, so that in the subsequent processing, there is no need to consider The influence of the reference signal on the audio signal of the channel.

It should be noted that step 605 is an optional step, and in some embodiments, step 606 may be directly performed based on the reference signal determined in step 603 . In other embodiments, step 606 is performed based on the denoised reference signal in step 604 .

606. The conference terminal enhances the target sound signal in the first sound signal based on the reference signal.

In some embodiments, the first sound signal includes sound signals corresponding to multiple sound sources, wherein the target sound signal is a sound signal corresponding to a focused sound source, for example, a sound signal corresponding to a speaker in a meeting, Therefore, the purpose of processing the sound signal is usually to highlight the target sound signal. Since the reference signal can well represent the sound signal of the interference source, processing the first sound signal through the reference signal can specifically reduce the influence of the sound of the interference source on the first sound signal, thereby ensuring that the first sound Prominence of the target sound signal in the signal.

In this embodiment of the present application, enhancing the target sound signal in the first sound signal refers to suppressing the reference signal in the first sound signal to enhance the target sound signal in the first sound signal, for example, By reducing the proportion of the corresponding portion of the reference signal in the first sound signal, the proportion of the target sound signal in the first sound signal is increased, thereby achieving the purpose of enhancing the target sound signal.

In some embodiments, the reference signal is used as one input of the filter, and the first sound signal is used as the other input of the filter. Through the filter, the first sound signal related to the reference signal is filtered out. to enhance the target sound signal in the first sound signal, and output a filtering result.

In some embodiments, the filter includes a first filter and a second filter, by inputting the reference signal into the first filter, based on the parameters of the first filter, adjusting the signal components of different frequencies in the reference signal The weight value is used to reconstruct the reference signal to obtain an estimated signal of the reference signal, and the estimated signal is a result of estimating the sound signal of the interference source in the reference signal. Based on this, the difference signal between the first sound signal and the estimated signal is used as a filtering result, and the part related to the reference signal in the first sound signal is filtered out by filtering out the estimated signal in the first sound signal. In some embodiments, the parameters of the first filter are determined based on the parameters of the second filter, and the parameters of the second filter are determined based on the difference between multiple filtering results. Wherein, when the reference signal is input into the first filter, the reference signal is also input into the second filter, so as to obtain the nth filtering result of the second filter. The second filter adjusts the parameters of the second filter based on the difference between the nth filtering result of the second filter and the n-1th filtering result, so that the estimated signal obtained based on the adjusted parameters can be more accurate A sound signal that is close to the interference source in the first sound signal. When the adjusted parameters of the second filter meet the convergence condition, the adjusted parameters of the second filter are configured to the first filter, thereby improving the effect of filtering the first sound signal. Wherein, n is an integer greater than 1.

In some embodiments, the above-mentioned filter is an adaptive filter. During the filtering process, the adaptive filter uses an adaptive algorithm to adjust the parameters of the filter to obtain a better filtering effect. For example, the second filter Based on the difference between the nth filtering result of the second filter and the n-1th filtering result, the parameters of the second filter are adjusted through an adaptive algorithm, wherein the filter parameters include a filter step size , by adjusting the filter step size, the convergence speed of the filter parameters can be changed. Further, based on different requirements, adaptive algorithms under different optimization criteria can be selected, for example, recursive least square algorithm (recursive least square, RLS), least mean square error algorithm ((least mean square, LMS) and normalized Normalized least mean square (NLMS), etc., which are not limited in this embodiment of the present application.

The embodiment of the present application provides a schematic diagram of an adaptive filter, as shown in FIG. 7 , wherein the reference signal is the input signal x(n); the desired signal y(n) includes the first sound signal v(n ) and the system echo d(n) of the reference signal; x(n) is input to the first filter and the second filter at the same time after fast Fourier transform processing; the first filter output (frequency domain) Estimated signal X'(m); the y(n) is processed by fast Fourier transform to obtain Y(m), the Y(m) is subtracted from the X'(m) by the adder, and the difference signal E is output (m), the E(m) undergoes inverse Fourier transform to obtain the filtering result e(n); the (frequency domain) estimated signal output by the second filter and Y(m) are obtained by adding the The difference signal is returned to the second filter for updating the parameters of the filter. Among them, H(n) is the system function used to simulate the system echo.

In some embodiments, the above-mentioned adaptive filtering process can be performed based on a deep learning model, and the parameters of the adaptive filter can be trained through the deep learning model, which can effectively improve the convergence speed of the adaptive filter parameters, thereby improving the filtering efficiency. efficiency.

In some embodiments, when the attenuation of the first sound signal before and after filtering is greater than the attenuation threshold, the filtered first sound signal may be weakened and cause distortion. In this case, the filtered first sound signal needs to be Carry out corresponding processing, for example, enhance the human voice in the signal or cut the distorted segment in the signal, so as to further ensure the quality of the sound signal.

In some embodiments, the conference terminal sends the filtered first sound signal to the multimedia control platform, and the multimedia control platform encodes the received first sound signal and sends it to the remote conference terminal.

Fig. 8 is a flowchart of a sound signal processing method provided by an embodiment of the present application. This method is applied to the sound processing system corresponding to FIG. 3 above, and the sound processing system includes multiple microphones, a conference terminal, a conference touch panel, and a camera. The sound signal processing method is executed by the conference terminal. As shown in Figure 8, the method includes:

801. The conference terminal picks up sound signals in the sound pickup space through multiple microphones.

For this step, refer to step 601, and details are not repeated here. Wherein, the camera is used for image collection for the sound pickup space. When configuring the sound processing system, it is necessary to configure the position information of the camera in the sound pickup space and the angle range of the camera for image collection, so as to determine the difference between the image collected by the camera and The relationship between positions in pickup space. For example, in the case that the image and the actual sound pickup space are mirror-symmetrical, the left half of the image corresponds to the right half of the sound pickup space.

802. The conference terminal receives the location selection instruction, and determines the location corresponding to the location selection instruction as the location of the interference source in the sound pickup space.

In the embodiment of the present application, there are multiple sound sources in the sound pickup space, and the conference terminal can determine the sound source considered as the interference source based on the position based on the received position selection instruction, so that in the subsequent sound signal processing process In , the sound of the interference source is processed, for example, the sound of the interference source in the sound signal is filtered out.

In some embodiments, the camera has data processing capability and can detect the collected images, and if the camera detects the first body behavior from the collected images, the camera sends the position selection instruction to the conference terminal. Wherein, the first body behavior is used to indicate to mute the position, for example, the participant puts the index finger vertically close to the lip. Based on the relationship between the image captured by the pre-configured camera and the position in the sound pickup space, the camera can determine the position of the first body behavior in the sound pickup space according to the position of the first body behavior in the image, Therefore, the position of the first body behavior in the sound pickup space is indicated in the position selection instruction. Based on this, the conference terminal receives the location selection instruction from the camera, acquires the location indicated by the location selection instruction, and determines the microphone corresponding to the location of the interference source based on the location indicated by the location selection instruction.

In some other embodiments, the camera has data processing capabilities, and the camera detects the collected images, and in the case that the first limb behavior is detected from the collected images, based on the multiple microphones in the sound pickup space The position information determines the microphone closest to the position of the first body behavior, and determines a position selection instruction based on the number of the microphone to indicate that the position of the microphone is the position of the interference source. Based on this, the conference terminal receives the position selection command from the camera, acquires the microphone number carried in the position selection command, and determines the microphone corresponding to the microphone number in the position selection command as the microphone corresponding to the position of the interference source.

Through the above technical solution, the position of the interference source can be obtained directly from the position selection instruction, the amount of data involved in the calculation process is reduced, and the efficiency of sound signal processing is improved.

In some other embodiments, the location selection instruction is triggered based on a selection operation of the location of the interference source in the control device, and for a principle, refer to step 602 .

The above process is described by taking the conference terminal receiving the position selection instruction sent by the camera as an example. In some embodiments, the conference terminal receives the image collected by the camera and detects the image to determine the location of the interference source. As an example, the process of determining the location of the interference source includes the following steps 1 to 2:

Step 1. The conference terminal detects the image collected by the camera.

Step 2: In response to detecting the first body behavior in the image, the conference terminal determines the location of the first body behavior in the sound pickup space as the location of the interference source.

In some embodiments, based on the relationship between the image collected by the camera and the position in the sound pickup space, the conference terminal can determine the position of the first body behavior in the sound pickup space according to the position of the first body behavior in the image. position, and then based on the position information of the plurality of microphones in the sound pickup space, determine the microphone with the closest distance to the position of the first body behavior, and determine the microphone as the microphone corresponding to the position of the interference source.

In some embodiments, the first body behavior indicates to mute the location where it is located. Therefore, the location of the interference source can be determined based on the first body behavior, and then the target sound signal can be enhanced by filtering out the sound of the interference source. In some other embodiments, the target sound signal can be determined based on the second body behavior, so as to directly enhance the target sound signal, wherein the second body behavior is used to instruct the target sound signal to be enhanced, for example, the participant puts the index finger Hold it close to your lips to indicate that it needs to speak. In this example, in response to detecting the second body action in the image, the conference terminal determines the position of the second body action in the sound pickup space as the position of the target. Wherein, the target refers to a target sound source that needs to be focused on among the multiple sound sources existing in the sound pickup space, and therefore, the target sound signal corresponding to the target sound source needs to be enhanced.

803. The conference terminal determines the sound signal originating from the microphone corresponding to the location of the interference source as a reference signal.

For this step, refer to step 603, which will not be repeated here.

In some embodiments, after the conference terminal determines the location of the interference source based on the images collected by the camera, it can continuously track the location of the interference source. For example, according to the characteristics of the interference source, the location of the interference source is tracked and detected. If the position of the interference source is tracked to change, the conference terminal re-determines the reference signal from the sound signal based on the changed position of the interference source. In some embodiments, the conference terminal determines the object corresponding to the position of the first body behavior in the image as the object corresponding to the interference source, and tracks the position change of the object based on the image collected in real time, and based on the changed object Position, to determine the position of the interference source after the change. Of course, the tracking of the interference source can be manually released through the conference terminal or conference touch panel, or it can be set to be automatically released after a certain period of time.

804. The conference terminal performs denoising on the reference signal.

For this step, refer to step 604, which will not be repeated here.

805. The conference terminal determines the first sound signal from the sound signals in the sound pickup space based on the denoised reference signal.

For this step, refer to step 605, which will not be repeated here.

806. The conference terminal enhances the target sound signal in the first sound signal based on the reference signal.

For this step, refer to step 606, which will not be repeated here.

Furthermore, through the above technical solution, participants do not need to manually select, and can automatically locate the interference source based on the image, and realize intelligent shielding of the interference source in the conference scene, which improves the conference experience while ensuring the sound quality.

FIG. 9 is a flow chart of a sound signal processing method provided by an embodiment of the present application. This method is applied to the sound processing system corresponding to FIG. 4 above, and the sound processing system includes a microphone array, a conference terminal, physical keys on a desktop, a conference touch panel, and a camera. The sound signal processing method is executed by the conference terminal. As shown in Figure 9, the method includes:

901. The conference terminal picks up the sound signal in the sound pickup space through the microphone array.

For this step, refer to step 801, which will not be repeated here. Wherein, when configuring the sound processing system, it is necessary to configure the beam angle range corresponding to the sound signal picked up by the microphone array and the position information of the microphone array in the sound pickup space, so as to determine the corresponding beam angle range of the sound signal picked up by the microphone array. The relationship between the angular extent of the beam and the position in that pickup space. For example, the beam angle range corresponding to the sound signal A of the microphone array covers the left half of the sound pickup space. In some embodiments, when configuring the sound processing system, the sound signals corresponding to different beam angle ranges are numbered, so that in the subsequent sound signal processing process, the required sound signal can be selected based on the number.

Wherein, the conference terminal obtains the sound signal from the sound pickup space through the microphone array. In some embodiments, since the microphone array includes a plurality of microphones arranged in a certain spatial structure, the microphone array determines the relative sound source relative The angle of the microphone array is used to determine the position of the sound source relative to the microphone array.

902. The conference terminal receives the location selection instruction, and determines the location corresponding to the location selection instruction as the location of the interference source in the sound pickup space.

For this step, refer to step 802.

In some embodiments, the camera has data processing capability and can detect the collected images, and if the camera detects the first body behavior from the collected images, the camera sends the position selection instruction to the conference terminal. Based on the preconfigured relationship between the image captured by the camera and the position in the sound pickup space, the camera can determine the position of the first body behavior in the sound pickup space according to the position of the first body behavior in the image. Based on this, combined with the position information of the microphone array in the sound pickup space, the angle of the first body behavior relative to the microphone array can be determined. Therefore, the angle of the first body behavior relative to the microphone array is indicated in the position selection instruction. Based on this, the conference terminal receives the position selection instruction from the camera, and determines the angle indicated by the position selection instruction as the angle of the position of the interference source relative to the microphone array.

In some other embodiments, the location selection instruction is triggered based on the selection operation of the location of the interference source in the control device. For the principle, refer to step 602. In this example, the location selection instruction indicates that the first body behavior is relative to The angle of the microphone array.

Step 1. The conference terminal detects the image collected by the camera.

In some embodiments, based on the relationship between the image collected by the camera and the position in the sound pickup space, the conference terminal can determine the position of the first body behavior in the sound pickup space according to the position of the first body behavior in the image. position, and further based on the position information of the microphone array in the sound pickup space, the angle of the first body behavior relative to the microphone array is determined as the angle of the position of the interference source relative to the microphone array.

903. The conference terminal determines a beam angle range matching the angle information based on the angle information of the interference source location.

In some embodiments, the angle information of the location of the interference source refers to the angle of the location of the interference source relative to the microphone array. Based on the angle information, the conference terminal can determine the beam angle range of the microphone array corresponding to the location of the interference source.

904. The conference terminal determines a reference signal from the sound signals picked up by the microphone array based on the beam angle range.

In some embodiments, the conference terminal obtains the multi-path sound signal components corresponding to the beam angle range from the multi-path sound signals picked up by the microphone array, and based on the characteristics of each sound signal component, the multi-path sound signal components Combine them to get a reference signal.

In some other embodiments, the conference terminal pre-numbers the corresponding sound signals of different beam angle ranges, and based on this, the conference terminal acquires the corresponding sound signal based on the beam angle range matched with the angle information of the interference source position , so that the sound signal corresponding to the number is directly determined as the reference signal.

905. The conference terminal performs denoising on the reference signal.

For this step, refer to step 604, which will not be repeated here.

906. The conference terminal determines the first sound signal from the sound signals in the sound pickup space based on the denoised reference signal.

For this step, refer to step 605, which will not be repeated here.

907. The conference terminal enhances the target sound signal in the first sound signal based on the reference signal.

For this step, refer to step 606, which will not be repeated here.

Furthermore, the method provided by the embodiment of the present application can adapt to the spatial arrangement characteristics of the microphone array in the scenario where the microphone array is used to pick up sound, and use the angle information of the interference source to obtain a targeted designation for the interference source. The sound signal within the angular range ensures the representativeness of the reference signal to the interference source, improves the accuracy of sound signal processing for the interference source, and effectively improves the sound quality.

Fig. 10 is a flowchart of a sound signal processing method provided by an embodiment of the present application. This method is applied to the sound processing system corresponding to FIG. 5 above, and the sound processing system includes a plurality of distributed microphones with a positioning function, a conference terminal, a conference touch panel, and a camera. The sound signal processing method is executed by the conference terminal. As shown in Figure 10, the method includes:

1001. The conference terminal picks up the sound signal in the sound pickup space through the distributed microphone with positioning function.

For this step, refer to step 801, and details are not described here. Before the start of the meeting, the multiple distributed microphones with positioning functions perform signal interaction with the conference terminal, and the conference terminal determines the position information of each distributed microphone in the sound pickup space according to the signals received from the multiple distributed microphones . Wherein, when the position of the distributed microphone changes, the conference terminal can update the position information of the distributed microphone in real time based on the received signal. Optionally, the distributed microphone can perform signal interaction with the conference terminal through bluetooth, ultrasonic wave or wireless local area network. Optionally, multiple distributed microphones maintain time synchronization through continuous signal interaction.

The embodiment of the present application provides a schematic diagram of a distributed microphone positioning process, as shown in FIG. For signal interaction, the relative positions of each signal interaction device have been predetermined, see the coordinates of each signal interaction device in FIG. 11 . The conference terminal obtains the moments when the four signal interaction devices receive the signals sent by the distributed microphone 1105 respectively as t _i (i=1, 2, 3, 4), which is used to calculate the distance between the distributed microphone 1105 and the i-th signal interaction device. Distance r _i (i=1, 2, 3, 4). For the distance calculation process, refer to the following formula (7) to formula (14).

d _i,12 =r ₁ -r ₂ =(t ₁ -t ₂ )×c (7)

d _i,23 =r ₂ -r ₃ =(t ₂ -t ₃ )×c (8)

d _i,34 =r ₃ -r ₄ =(t ₃ -t ₄ )×c (9)

d _i,41 =r ₄ -r ₁ =(t ₄ -t ₁ )×c (10)

Wherein, the di _{, 12} _is the distance difference between the distributed microphone 1105 relative to the signal interaction device 1101 and the signal interaction device 1102; The distance difference between; the d _{i, 34} is the distance difference between the distributed microphone 1105 relative to the signal interaction device 1103 and the signal interaction device 1104; the d _{i, 41} is the distance difference between the distributed microphone 1105 and the signal interaction device 1104 and The distance difference between the signal interaction devices 1101; c is the speed of light.

Based on the above d _i,12 , d _i,23 , d _i,34 and d _i,41 , a system of hyperbolic equations can be established for determining the distance r _i (i= 1, 2, 3, 4), to realize the positioning of the distributed microphone 1105.

Among them, (x ₁ , y ₁ , z ₁ ) is the coordinates of the signal interaction device 1101; (x ₂ , y ₂ , z ₂ ) is the coordinates of the signal interaction device 1102; (x ₃ , y ₃ , z ₃ ) is the signal The coordinates of the interaction device 1103; (x ₄ , y ₄ , z ₄ ) are the coordinates of the signal interaction device 1104; wherein, i=1, 2, 3, 4.

1002. The conference terminal receives the location selection instruction, and determines the location corresponding to the location selection instruction as the location of the interference source in the sound pickup space.

For this step, refer to step 802, which will not be repeated here.

1003. The conference terminal determines the sound signal from the distributed microphone corresponding to the location of the interference source as a reference signal.

For this step, refer to step 803, and details are not repeated here.

1004. The conference terminal performs denoising on the reference signal.

For this step, refer to step 804, which will not be repeated here.

1005. The conference terminal determines the first sound signal from the sound signals in the sound pickup space based on the denoised reference signal.

For this step, refer to step 805, which will not be repeated here.

1006. The conference terminal enhances the target sound signal in the first sound signal based on the reference signal.

For this step, refer to step 806, which will not be repeated here.

Furthermore, through the above technical solution, multiple microphones can be randomly placed according to the requirements, which greatly reduces the scene restrictions during equipment deployment. While improving the flexibility of equipment deployment in the sound processing system, real-time positioning of the microphones is achieved. Accurate positioning of the interference source, thereby more accurately filtering out the sound of the interference source from the sound signal, effectively ensuring the sound quality.

Fig. 12 is a schematic structural diagram of an audio signal processing device provided by an embodiment of the present application. As shown in Figure 12, the sound signal processing device includes:

The sound pickup module 1201 is used to pick up the sound signal in the sound pickup space through the sound pickup device;

A position determining module 1202, configured to determine the position of the interference source in the sound pickup space;

A signal determination module 1203, configured to determine a reference signal from the sound signal based on the location of the interference source, and the reference signal is used to filter out the sound of the interference source;

An enhancement module 1204, configured to enhance the target sound signal based on the reference signal.

In a possible implementation manner, the location determining module 1202 includes:

The first determining unit is configured to receive a location selection instruction, and determine the location corresponding to the location selection instruction as the location of the interference source in the sound pickup space.

An image detection unit, configured to detect the target image collected by the image acquisition device, and the image acquisition device is used for image acquisition for the sound pickup space;

A second determination unit, configured to determine a position of the first body action in the sound pickup space as the position of the interference source in response to detecting a first body action in the target image, the first Physical behavior is used to indicate muting of said location.

In a possible implementation manner, the device further includes:

A third determining unit, configured to determine a position of the second body action in the sound pickup space as the position of the target in response to detecting a second body action in the target image, the second Physical behavior is used to indicate the enhancement of the target sound signal.

In a possible implementation manner, the device further includes:

a tracking unit, configured to track the position of the interference source;

The signal determination module is used for:

In a possible implementation manner, the sound pickup device includes multiple microphones, and the signal determination module is configured to:

In a possible implementation manner, the sound pickup device is a microphone array, and the signal determining module 1203 is configured to:

In a possible implementation manner, the enhancement module 1204 includes:

A signal determining unit, configured to determine a first sound signal from sound signals in the sound pickup space based on the reference signal, the signal energy of the first sound signal is less than the signal energy of the reference signal, and, the correlation between the first sound signal and the reference signal is greater than a correlation threshold;

An enhancement unit, configured to enhance a target sound signal in the first sound signal based on the reference signal.

In a possible implementation manner, the enhancing unit is used for:

It should be noted that: when the sound signal processing device provided in the above-mentioned embodiment performs sound signal processing, the division of the above-mentioned functional modules is used as an example for illustration. In practical applications, the above-mentioned functions can be assigned to different functional modules according to needs. To complete means to divide the internal structure of the device into different functional modules to complete all or part of the functions described above. In addition, the sound signal processing device provided in the above embodiment and the sound signal processing method embodiment belong to the same idea, and the specific implementation process thereof is detailed in the method embodiment, and will not be repeated here.

In the technical solution provided by the embodiment of the present application, based on the position of the interference source in the sound pickup space, the reference signal is determined from the sound signal in the sound pickup space, and then the sound of the interference source in the sound signal is filtered out based on the reference signal, so as to enhance Target sound signal. Through the above technical solution, the sound signal processing is performed according to the position of the interference source, and the sound of the interference source can be shielded in a targeted manner to enhance the target sound signal, thereby improving the sound quality.

An embodiment of the present application provides a sound signal processing device, which can be used as the sound signal processing device in the above sound processing system. Schematically, refer to FIG. 13 , which is a schematic diagram of a hardware structure of an audio signal processing device provided by an embodiment of the present application. As shown in FIG. 13 , the audio signal processing device 1300 includes a memory 1301 , a processor 1302 , a communication interface 1303 and a bus 1304 . Wherein, the memory 1301 , the processor 1302 , and the communication interface 1303 are connected to each other through a bus 1304 .

Memory 1301 may be read-only memory (read-only memory, ROM) or other types of static storage devices that can store static information and instructions, random access memory (random access memory, RAM) or other types that can store information and instructions It can also be an electrically erasable programmable read-only memory (EEPROM), compact disc read-only memory (CD-ROM) or other optical disc storage, optical disc storage (including compact discs, laser discs, optical discs, digital versatile discs, Blu-ray discs, etc.), magnetic disk storage media or other magnetic storage devices, or can be used to carry or store desired program code in the form of instructions or data structures and can be programmed by a computer Any other medium accessed, but not limited to. The memory 1301 can store at least one piece of program code, and when the program code stored in the memory 1301 is executed by the processor 1302, the sound signal processing device can implement the above sound signal processing method. The memory 1301 may also store various types of data, including but not limited to images and audio signals, which are not limited in this embodiment of the present application.

The processor 1302 may be a network processor (network processor, NP), a central processing unit (central processing unit, CPU), a specific application integrated circuit (application-specific integrated circuit, ASIC) or an integrated circuit for controlling the program execution of the application scheme. circuit. The processor 1302 may be a single-core (single-CPU) processor, or a multi-core (multi-CPU) processor. The number of the processor 1302 may be one or more. The communication interface 1303 uses a transceiver module such as a transceiver to implement communication between the sound signal processing device 1300 and other devices or communication networks. For example, sound signals can be acquired through the communication interface 1303 .

Wherein, the memory 1301 and the processor 1302 may be provided separately, or may be integrated together.

The bus 1304 may include a path for transferring information between various components of the sound signal processing device 1300 (eg, memory 1301 , processor 1302 , communication interface 1303 ).

In the present invention, the terms "first" and "second" are used to distinguish the same or similar items with basically the same function and function. It should be understood that "first", "second" and "nth" There are no logical or timing dependencies, nor are there restrictions on quantity or order of execution. It should also be understood that although the following description uses the terms first, second, etc. to describe various elements, these elements should not be limited by the terms. These terms are only used to distinguish one element from another. For example, a first microphone could be termed a second microphone, and, similarly, a second microphone could be termed a first microphone, without departing from the scope of the various described examples. Both the first microphone and the second microphone may be microphones, and in some cases may be separate and distinct microphones.

The meaning of the term "at least one" in the present invention refers to one or more, the meaning of the term "multiple" in the present invention refers to two or more, for example, a plurality of microphones refers to two or more microphone.

The above description is only a specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any person familiar with the technical field can easily think of various equivalent modifications within the technical scope disclosed in the present invention Or replacement, these modifications or replacements should be covered within the protection scope of the present invention. Therefore, the protection scope of the present invention should be based on the protection scope of the claims.

In the above embodiments, all or part of them may be implemented by software, hardware, firmware or any combination thereof. When implemented using software, it may be implemented in whole or in part in the form of a program product. The program product includes one or more program instructions. When the program instructions are loaded and executed on the sound signal processing device, the processes or functions according to the embodiments of the present invention will be generated in whole or in part.

Those of ordinary skill in the art can understand that all or part of the steps for implementing the above-mentioned embodiments can be completed by hardware, and can also be completed by instructing related hardware through a program. The program can be stored in a computer-readable storage medium. The above-mentioned The storage medium can be read-only memory, magnetic disk or optical disk and so on.

As mentioned above, the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that: it can still understand the foregoing The technical solutions recorded in each embodiment are modified, or some of the technical features are replaced equivalently; and these modifications or replacements do not make the essence of the corresponding technical solutions depart from the scope of the technical solutions of the embodiments of the present invention.

Claims

A sound signal processing method, characterized in that the method comprises:

Pick up the sound signal in the sound pickup space through the sound pickup device;

determining the position of the interference source in the sound pickup space;

Determining a reference signal from the sound signal based on the location of the interference source, the reference signal being used to filter out the sound of the interference source;

Based on the reference signal, the target sound signal is enhanced.
The method according to claim 1, wherein said determining the position of the interference source in the sound pickup space comprises:

The location selection instruction is received, and the location corresponding to the location selection instruction is determined as the location of the interference source in the sound pickup space.
The method according to claim 2, wherein the location selection instruction is triggered based on a selection operation of the location of the interference source in the control device.
The method according to claim 2, wherein the position selection instruction is triggered by an image acquisition device when a first limb behavior is detected in the captured image, and the image acquisition device is used for the picked-up Image acquisition is performed in a sound space, and the first body behavior is used to indicate to mute the location.
The method according to claim 1, wherein said determining the position of the interference source in the sound pickup space comprises:

Detecting the target image collected by the image acquisition device, the image acquisition device is used for image acquisition for the sound pickup space;

In response to detecting a first body action in the target image, determining a position of the first body action in the sound pickup space as the position of the interference source, the first body action being used to indicate the The above position is muted.
The method according to claim 5, wherein the method further comprises:

In response to detecting a second body action in the image of the target, determining a position of the second body action in the sound pickup space as the position of the target, the second body action being used to indicate the The above-mentioned target sound signal is enhanced.
The method according to any one of claims 1 to 6, wherein the method further comprises:

Tracking the location of the interference source;

The determining the reference signal from the sound signal based on the location of the interference source includes:

Based on the tracked change in the position of the interference source, a reference signal is re-determined from the sound signal.
The method according to any one of claims 1 to 7, wherein the sound pickup device includes a plurality of microphones, and determining a reference signal from the sound signal based on the position of the interference source comprises:

The sound signal originating from the microphone corresponding to the location of the interference source is determined as a reference signal.
The method according to claim 8, wherein the plurality of microphones have a positioning function.
The method according to any one of claims 1 to 7, wherein the sound pickup device is a microphone array, and determining a reference signal from the sound signal based on the position of the interference source comprises:

Based on the angle information of the position of the interference source, determine a beam angle range matching the angle information;

Based on the beam angle range, a reference signal is determined from the sound signals picked up by the microphone array.
The method according to any one of claims 1 to 10, wherein said enhancing the target sound signal based on the reference signal comprises:

Based on the reference signal, determine a first sound signal from the sound signals in the sound pickup space, the signal energy of the first sound signal is less than the signal energy of the reference signal, and the first sound signal the correlation with the reference signal is greater than a correlation threshold;

Based on the reference signal, the target sound signal in the first sound signal is enhanced.
The method according to claim 11, wherein said enhancing the target sound signal in the first sound signal based on the reference signal comprises:

Using the reference signal as one input of the filter, using the first sound signal as the other input of the filter, and filtering out the first sound signal related to the reference signal through the filter to enhance the target sound signal in the first sound signal, and output a filtering result.
A sound signal processing device, characterized in that the device comprises:

The pickup module is used to pick up the sound signal in the pickup space through the pickup device;

A position determining module, configured to determine the position of the interference source in the sound pickup space;

A signal determination module, configured to determine a reference signal from the sound signal based on the position of the interference source, and the reference signal is used to filter out the sound of the interference source;

An enhancement module, configured to enhance the target sound signal based on the reference signal.
The device according to claim 13, wherein the position determining module comprises:

The first determining unit is configured to receive a location selection instruction, and determine the location corresponding to the location selection instruction as the location of the interference source in the sound pickup space.
The apparatus according to claim 14, wherein the location selection instruction is triggered based on a selection operation of the location of the interference source in the control device.
The device according to claim 14, wherein the position selection instruction is triggered by an image acquisition device when a first body action is detected in the captured image, and the image acquisition device is used for the picked-up Image acquisition is performed in a sound space, and the first body behavior is used to indicate to mute the location.
The device according to claim 13, wherein the position determining module comprises:

An image detection unit, configured to detect the target image collected by the image acquisition device, and the image acquisition device is used for image acquisition for the sound pickup space;

A second determination unit, configured to determine a position of the first body action in the sound pickup space as the position of the interference source in response to detecting a first body action in the target image, the first Physical behavior is used to indicate muting of said location.
The device according to claim 17, further comprising:

A third determining unit, configured to determine a position of the second body action in the sound pickup space as the position of the target in response to detecting a second body action in the target image, the second Physical behavior is used to indicate the enhancement of the target sound signal.
The device according to any one of claims 13 to 18, wherein the device further comprises:

a tracking unit, configured to track the position of the interference source;

The signal determination module is used for:

Based on the tracked change in the position of the interference source, a reference signal is re-determined from the sound signal.
The device according to any one of claims 13 to 19, wherein the sound pickup device includes a plurality of microphones, and the signal determination module is used for:

The sound signal originating from the microphone corresponding to the location of the interference source is determined as a reference signal.
The device according to claim 20, wherein the plurality of microphones have a positioning function.
The device according to any one of claims 13 to 19, wherein the sound pickup device is a microphone array, and the signal determination module is used for:

Based on the angle information of the position of the interference source, determine a beam angle range matching the angle information;

Based on the beam angle range, a reference signal is determined from the sound signals picked up by the microphone array.
The device according to any one of claims 13 to 22, wherein the enhancement module comprises:

A signal determining unit, configured to determine a first sound signal from sound signals in the sound pickup space based on the reference signal, the signal energy of the first sound signal is less than the signal energy of the reference signal, and, the correlation between the first sound signal and the reference signal is greater than a correlation threshold;

An enhancement unit, configured to enhance a target sound signal in the first sound signal based on the reference signal.
The device according to claim 23, wherein the enhancing unit is used for:

Using the reference signal as one input of the filter, using the first sound signal as the other input of the filter, and filtering out the first sound signal related to the reference signal through the filter to enhance the target sound signal in the first sound signal, and output a filtering result.
A sound signal processing device, characterized in that the sound signal processing device includes a processor and a memory, the memory is used to store at least one piece of program code, and the at least one piece of program code is loaded and executed by the processor as claimed in the claims The sound signal processing method described in any one of claims 1 to 12.
A computer-readable storage medium, characterized in that the computer-readable storage medium is used to store at least one piece of program code, and the at least one piece of program code is used to execute any one of claims 1 to 12. sound signal processing method.
A computer program product, characterized in that, when the computer program product is run on a computer, the computer is made to execute the sound signal processing method according to any one of claims 1 to 12.