EA041269B1 - METHOD AND DEVICE FOR AUDIO SIGNAL DIARIZATION - Google Patents

Description

Technical field

The presented technical solution relates in general to measuring technology, in particular to a method and device for diarization of an audio signal, and is designed to divide the incoming audio stream into homogeneous segments in accordance with the belonging of the audio stream to one or another speaker (speaker).

State of the art

From a scientific point of view, the presented technical solution belongs to the category of solutions that use the spatial features of the signal source for diarization. Devices that use its spatial features to isolate the required signal can be both stationary (microphones fixed at certain points in the room) and portable (acoustic reconnaissance tools). A common feature of devices of this type is a large number of microphones (up to 40 or more) and, accordingly, the need to provide a wide bandwidth of the data transmission channel for transmitting the captured signal. From the point of view of computational complexity, the implementation of the beamforming algorithm (direction, the force coefficient of the sound coming from which will be maximum) is a complex task that requires a search for a solution in real time. However, with the growth of the computing power of portable devices and the increase in the bandwidth of data transmission channels, the implementation of this functionality has become possible and accessible to a wide range of people, and therefore simplified systems of this type are used in devices with voice control with a predetermined location of the operator.

Well-known examples of sensors of this type for home use are FMR (phased microphone array) Microsoft Kinnect, as well as FMR used in the so-called. smart speakers, such as Apple Home Pod, Amazon Echo, Yandex Station, etc. A feature of the FMR used in products of this type is the location of microphones in a horizontal plane (in the case of Kinnect, a linear array of microphones, since the search for a signal source is carried out only in one half-plane), which is due to the need to solve the problem of determining the sound source and separating it from the general sound stream (amplifying the signal in a given spatial direction). As a rule, in the process of voice control of devices, the implementation of the algorithm for extracting the words of the target speaker from the audio stream is performed in 3 steps:

1) in standby mode, the device constantly analyzes the incoming audio stream from one microphone for the presence of a turn-on trigger word, while buffering the signals of all array microphones;

2) at the moment of detecting the trigger word, the system performs calculations for beam formation: it searches for such values of the phase delay of the signals of each of the microphones, which, when added to the function of the received signal by each of the microphones and the addition of the microphone signals with certain coefficients, will provide the maximum amplification of the outgoing sound on the direction of the target speaker (detected trigger word), compared to background sounds, presumably coming from different directions;

3) processing of the audio stream received further from all microphones of the array is carried out taking into account the phase delay values obtained at step 2, due to which the resulting signal (of the beam) has a higher signal-to-noise ratio compared to the signal of a separate microphone (this allows the subsequent conversion of the speech signal into text with less chance of error).

The advantage of this implementation of the beamforming algorithm is the possibility of significantly amplifying the speech of the target speaker (compared to the acoustic background), but the lack of predetermination makes it necessary to position the array of microphones in the scanning plane. Segmentation of speech according to the criterion of its belonging to different speakers using this technology involves the solution of only 2 parts of this task, i.e. search for such a set of phase (time) delays for the signals of individual microphones, when the maximum amplification of the sounds of the speaker's speech is reached. This set of phase delays for individual microphone signals will determine the direction of the speech source relative to the receiver, which in turn will be a feature for segmenting the speech audio signal. The disadvantage of this method is the low resolution under the condition of a combination of small dimensions of the receiving device (and, accordingly, the distance between the microphones, for example, on the order of a few cm) and a low sampling rate (for example, 16 kHz, which is the standard for speech analytics systems). This is due to the fact that when working with digital signals, the phase (time) delays of the signal from the source received by different microphones can only be an integer number of samples. The solution of the diarization problem is also relevant for another class of devices that involve personal use: telephones, headsets, etc. To date, to ensure small dimensions and weight of devices, a typical solution is the use of microphones made using MEMS technology in such devices. The advantage of microphones of this type is a wide dynamic range, high sensitivity and a wide directivity pattern (DN) (up to spherical), which is important for providing

- 1 041269 the ability to operate the device, both in handset mode and in speakerphone mode. Due to the wide DN, the pickup of sound by such a device will be omnidirectional. In order to ensure the comfort of the interlocutor (to exclude extraneous noise from the transmitted signal during pauses), as well as to reduce the amount of transmitted data, it is advisable to record and transmit sound by a headset or telephone only at the moment the user pronounces a replica, which also requires solving the problem of determining the ownership of the speech signal to the user. The selection of the user's speech from the incoming sound signals can be carried out either by the sound pressure level, or by fixing the vibrations of the user's body surface at the moment of speaking. The latter can be achieved, for example, by embedding a MEMS accelerometer in the device, which generates a physical trigger signal that it is the user who speaks at a given time (see, for example, the solution disclosed in the application US 2014093093A1, publ. 03.04. 2014). The advantage of this principle of implementing the function of extracting the user's speech from the general sound stream coming to the microphone is the inherent high accuracy of determining the belonging of the speech signal to the user, as well as the absence of the need to perform complex calculations. This, in turn, ensures high performance of segmentation algorithms and potentially low power consumption by devices implementing them. A significant limitation of this method is the need to ensure mechanical contact of the device with the surfaces of the operator's body that are vibrating during speech pronunciation (lateral region of the skull, ear canal, larynx, chest) sufficient to transmit vibrations.

The essence of the technical solution

The technical problem or technical challenge posed in this technical solution is the creation of a new effective, simple and reliable method of diarization of an audio signal, which makes it possible to make a decision about whether an audio recording belongs to a specific speaker.

The technical result achieved when solving the above technical problem or technical problem is to provide the possibility of marking (segmentation) of an audio signal with a small error and low power consumption, based on data received from 2 microphones, including in real time.

The specified technical result is achieved due to the implementation of a method for diarization of a speech audio signal, which includes the steps of receiving digital audio signals containing voice data synchronously recorded by at least two microphones;

determining a difference signal for the signals of the two microphones based on the digital audio signal data received from said microphones;

determine the values of the envelope function of the difference signal;

determining the values of the envelope function of the original audio signal based on the data of the digital audio signal received from one of the microphones;

based on the value of the envelope function of the difference signal and the value of the envelope function of the original audio signal, the characteristic value of the audio signal is determined;

based on the characteristic value of the audio signal, digital audio signal data is tagged indicating to which audio signal source the corresponding digital audio data block belongs.

In one of the particular examples of the implementation of the method, two microphones are vertically spaced relative to each other.

In another particular embodiment of the method, the difference signal is determined by sample-by-sample subtraction (for synchronously received signals) of the signal magnitude value determined for the audio signal received from one of the microphones from the signal magnitude value determined for the audio signal received from the other microphone.

In another particular embodiment of the method, the characteristic value of the audio signal (sp) is determined by the formula sp= Env( 11-12)/Env( 11), where Env(11-12) is the value of the envelope function of the difference signal, a

Env(11) - the value of the envelope function of the original audio signal received from one of the microphones.

In another particular embodiment of the method, digital audio signal data is marked by comparing the characteristic value of the audio signal with a predetermined threshold value, and if the characteristic value of the audio signal is greater than the threshold value, then the corresponding block of digital audio signal data is marked as related to the first source of the audio signal, and if the characteristic value of the audio signal is less than the threshold, the corresponding digital audio data block is marked as belonging to the second audio source.

In another particular example of the implementation of the method, the marking of digital audio signal data is carried out by dividing the recorded audio stream of the speakers' speech into stereo channels.

- 2 041269

In another particular example of the implementation of the method, the labeling of digital audio signal data is carried out by creating an additional data block indicating timestamps characterizing the recording time of the replicas of at least one speaker.

In another preferred embodiment of the claimed solution, a speech audio signal diarization device is provided, comprising at least one computing device and at least one memory device containing machine-readable instructions, which, when executed by at least one computing device, perform the above method.

Brief description of the drawings

The features and advantages of the present technical solution will become apparent from the following detailed description of the invention and the accompanying drawings, in which in Fig. 1 shows the general layout of the microphones;

in fig. 2 shows an example of speech signal parameters;

in fig. 3 shows a diagram of an audio signal diarization device;

in fig. 4 is an example of a general view of a computing device.

Implementation of the invention

The concepts and terms necessary for understanding this technical solution will be described below.

In this technical solution, a system means, including a computer system, a computer (electronic computer), CNC (numerical control), PLC (programmable logic controller), computerized control systems and any other devices capable of performing a given, well-defined sequence of operations (actions, instructions).

A command processing device is an electronic unit, a computing device, or an integrated circuit (microprocessor) that executes machine instructions (programs).

An instruction processing device reads and executes machine instructions (programs) from one or more data storage devices. The role of a storage device can be, but not limited to, hard drives (HDD), flash memory, ROM (read only memory), solid state drives (SSD), optical drives.

Program - a sequence of instructions intended for execution by a computer control device or a command processing device.

A data block is a sequence of bits that has a fixed length and is used to represent data in memory or to transfer it. In FIG. 1 shows the layout of the microphones relative to the sound signal sources. In particular, the diagram shows the first source of the sound signal - the operator 1, the second source of the sound signal - the client 2, the device 10 for recording the sound signal, containing microphones 11 and 12. As the device 10 for recording the sound signal, any known computing device, modified in the software and hardware in such a way as to ensure the collection, processing and storage of audio signal data. For example, said device 10 can be made in the form of a digital badge/insignia, pre-placed in a certain way relative to the first source of the sound signal - the operator 1. For example, the sound signal recording device 10 can be placed on the chest of the operator on the left side. Since the location of the device 10 relative to the first source of the sound signal is known in advance, accordingly, the solution of the problem of determining the direction of the sound signal of the operator's voice is redundant and for diarization it is sufficient to check whether the direction of the sound source corresponds to the expected direction. Such a simplification of the algorithm makes it possible for a given class of devices (with a predetermined usage scenario) to carry out diarization of records using minimal computing resources. This feature, along with the use of low-power electronic components, makes it possible to reduce the overall power consumption of the device, which, accordingly, allows for a long operating time with a small weight and volume of the battery. The logic of the diarization algorithm is based on the fact that the sound signal sources (the mouth of the operator 1 and the mouth of the client 2) are located at different points in space and the mouth of the operator 1 is slightly displaced relative to the straight line on which the microphones 11 and 12 are located (presumably lying in a vertical plane) . In this regard, the time delay between the signals of two microphones 11 and 12 located one below the other at a given distance from the sounds of the speech of the operator 1 will be maximum, while the time delay between the signals of the same microphones from the sounds of the speech of the client 2 will be much less.

For example, if the distance between the microphones is about 50 mm (see Fig. 1), the badge will be on the chest of the operator at a distance of about 20 cm from his mouth and the distance between the operator and the client is approximately = 1 m, then the sound wave from client 2 to the lower microphone 12 will need to travel an additional 10 mm after the sound wave arrives at the microphone 11, while for the sound wave due to the operator's speech, the acoustic path difference between the microphones 11 and 12 will be approximately 50 mm. Thus, under the condition of equality of characteristics

- 3 041269 both microphones when subtracting the signal of one of the microphones from the signal of the other microphone, the amplitude of the resulting signal for low frequencies (up to 300 Hz) will be proportional to the phase difference due to the time delay in signal propagation.

In FIG. 2 shows an example of the dependence of the sound pressure value on time for recorded sound waves, which shows sin(pi*x) - the original signal (the signal recorded by microphone 11), sin(pi*(x-1*k)) - the signal received by the second microphone (12), for the case when the signal would come from the point where the client is located (small delay between the moments when the front of the sound wave reaches microphones 11 and 12), sin(pi*(x-5*k)) - the signal received by the second microphone (12), for the case when the signal would come from the point where the operator's mouth is located (large delay between the moments when the front of the sound wave reaches microphones 11 and 12), delta1 is the difference signal, if the signal source is the client's voice {sin(pi *(x)sin(pi*(x-1*k))}, delta2 - difference signal, if the source of the signal is the operator's voice {sin(pi*(x)sin(pi*(x-5*k) )}.

pi is the Pi number, x is a time variable to illustrate a graph of the magnitude of the sound pressure perceived by the microphones versus time;

k - the amount of delay due to the presence of a difference in the acoustic path for the sound wave from the source to the microphones (11) and (12), provided that its source is the client (2) (in the case when the delay is 1 * k) or the operator (1) (in the case when the delay is 5*k).

Next, a method for diarizing an audio signal will be described with reference to FIG. 3, which shows an exemplary circuit diagram of an audio diarizer 100. The audio diarization device 100 may be implemented on the basis of the audio signal recording device 10 and include a signal processing module 101, a signal characterization module 102, an audio signal marking module 103, and a data storage module 104. The listed modules can be implemented on the basis of the firmware of the audio signal diarization device 100, implemented in the software part in such a way as to perform the functions assigned to them below.

Accordingly, digital audio signals containing voice data of the operator 1 or customer 2 and synchronously recorded by the microphones 11 and 12 are buffered by the signal processing module 101 as a data stream. From the received digital audio signals, the module 101 forms a data array.

The generated digital audio data array is sent by the module 101 to the signal characterization module 102 .

Next, the signal characterization module 102 determines the difference signal for the signals of the two microphones by sample-by-sample subtraction (for synchronously obtained samples) of the signal magnitude value determined for the audio signal received from the microphone 12 from the signal magnitude value determined for the audio signal received from the microphone 11. After To this end, the signal processing unit 102 determines the values of the envelope function of the difference signal and the envelope function of the original audio signal received from the microphone 11 or 12 by methods known from the prior art, and determines the characteristic values of the audio signal based on the obtained values. For example, the characteristic values of an audio signal (sp) can be determined by the formula sp=Env( 11-12)/Env( 11), where Env(11-12) is the value of the envelope function of the difference signal, a

Env(11) - the value of the envelope function of the original audio signal received from microphone 11.

In order for the characteristic value of the audio signal to have a physical meaning, it is necessary that the microphones 11 and 12 of the pair under consideration be vertically spaced relative to each other, and the sound signal recording device 10 is located on the operator’s chest so that the operator’s mouth is on a straight line passing through the center of the apertures microphones 11 and 12 or with a slight deviation from it. The step of determining the characteristic value of the audio signal is necessary to normalize the signal to its own amplitude, so that the characteristic value of the audio signal is dependent only on the location of the sound sources, and not on the amplitude of the signal.

Next, the audio characteristic value data and the digital audio signal data array are sent by said module 102 to the audio signal tagging module 103, which, based on the characteristic value of the audio signal, tags the digital audio signal data indicating to which audio signal source the digital audio signal data blocks belong. For example, the characteristic values of the audio signal by said module 103 can be compared with a predetermined threshold value, and if the characteristic values of the audio signal are greater than the threshold value, then the digital audio data block is marked as related to the first source of the audio signal - operator (1). If the characteristic values of the audio signal are less than the threshold value, then the digital audio data block is marked as belonging to the second

- 4 041269 sound source - client (2). Audio data marking can be carried out either by dividing the recorded audio stream of the speakers' speech into stereo channels (the speech of one of the speakers into the right channel, the other into the left channel) with their subsequent saving as an audio file, or as an additional data block (a separate file or an additional track of the above audio file) with timestamps characterizing the recording time of the replicas of one or both speakers.

The resulting audio markup file and/or an additional markup file of audio data can be stored in the memory of the data storage module 104 for further transmission to external devices and data processing systems through appropriate data output interfaces, which will be described later in the text of the description.

Thus, due to the fact that the source of the audio signal is determined based on the characteristic value of the audio signal obtained on the basis of the envelope function value of the difference signal and the envelope function value of the original audio signal, the accuracy of its determination is increased and the error in labeling digital audio signal data based on the characteristic value of the audio signal is reduced. The power consumption of digital audio data tagging is also reduced by the fact that digital audio data tagging does not require additional processing of the digital audio signals received from the microphones. In general terms (see Fig. 3), the computing device (200) contains one or more processors (201), memory facilities such as RAM (202) and ROM (203), input / output interfaces (204) connected by a common information exchange bus ), input/output devices (205), and a device for networking (206).

The processor (201) (or multiple processors, multi-core processor, etc.) can be selected from a range of devices currently widely used, for example, manufacturers such as Intel™, AMD™, Apple™, Samsung Exynos™, MediaTEK™ , Qualcomm Snapdragon™, etc. Under the processor or one of the processors in use in the system (200), it is also necessary to take into account the graphics processor, for example, NVIDIA GPU with a CUDA-compatible software model, or Graphcore, the type of which is also suitable for full or partial execution of the method, and can also be used for training and application of machine learning models in various information systems.

RAM (202) is a random access memory and is designed to store machine-readable instructions executable by the processor (201) to perform the necessary operations for logical data processing. The RAM (202) typically contains the executable instructions of the operating system and associated software components (applications, program modules, etc.). In this case, the RAM (202) may be the available memory of the graphics card or graphics processor.

The ROM (203) is one or more persistent storage devices such as a hard disk drive (HDD), a solid state data drive (SSD), flash memory (EEPROM, NAND, etc.), optical storage media (CD-R/ RW, DVD-R/RW, BlueRay Disc, MD), etc.

Various types of I/O interfaces (204) are used to organize the operation of device components (200) and organize the operation of external connected devices. The choice of the appropriate interfaces depends on the specific implementation of the computing device, which can be, but not limited to, PCI, AGP, PS/2, IrDa, FireWire, LPT, COM, SATA, IDE, Lightning, USB (2.0, 3.0, 3.1, micro, mini, type C), TRS/Audio jack (2.5, 3.5, 6.35), HDMI, DVI, VGA, Display Port, RJ45, RS232, etc.

Various means (205) of I/O information are used to ensure user interaction with the device (200), for example, a keyboard, a display (monitor), a touch screen, a touchpad, a joystick, a mouse, a light pen, a stylus, a touchpad, a trackball, speakers, microphone, augmented reality tools, optical sensors, tablet, light indicators, projector, camera, biometric identification tools (retinal scanner, fingerprint scanner, voice recognition module), etc. The networking means (206) provides data transmission via an internal or external computer network, for example, an Intranet, Internet, LAN, etc. As one or more means (206), an Ethernet card, a GSM modem, a GPRS modem, an LTE modem, a 5G modem, a satellite communication module, an NFC module, a Bluetooth and/or BLE module, a Wi-Fi module, and others

Additionally, satellite navigation tools within the device (200) can also be used, such as GPS, GLONASS, BeiDou, Galileo.

The specific choice of elements of the device (200) for the implementation of various software and hardware architectural solutions may vary while maintaining the required functionality provided.

Modifications and improvements to the above described embodiments of the present technical solution will be clear to experts in this field of technology. The foregoing description is provided by way of example only and is not intended to be limiting in any way. Thus, the scope of the present technical solution is limited only by the scope of the appended claims.

Claims (8)

CLAIM 1. A method for diarizing an audio signal performed by at least one computing device, comprising the steps of receiving digital audio signals containing voice data synchronously recorded by at least two microphones; determining a difference signal for the signals of the two microphones based on the digital audio data received from said microphones; determine the values of the envelope function of the difference signal; determining the values of the envelope function of the original audio signal based on the data of the digital audio signal received from one of the microphones; based on the value of the envelope function of the difference signal and the value of the envelope function of the original audio signal, the characteristic value of the audio signal is determined; based on the characteristic value of the audio signal, digital audio signal data is tagged indicating to which audio signal source the corresponding digital audio data block belongs. 2. The method according to claim 1, characterized in that at least two microphones are vertically spaced apart from each other. 3. The method according to claim 1, characterized in that the difference signal is determined by sample-by-sample subtraction (for synchronously received signals) of the signal magnitude value determined for the audio signal received from one of the microphones from the signal magnitude value determined for the audio signal received from the other microphone. 4. The method according to claim 1, characterized in that the characteristic value of the audio signal (sp) is determined by the formula sp=Env(11-12)/Env(11), where Env(11-12) is the value of the envelope function of the difference signal, a Env(11) - the value of the envelope function of the original audio signal received from one of the microphones. 5. The method according to claim 1, characterized in that marking the data of the digital audio signal is carried out by comparing the characteristic value of the audio signal with a predetermined threshold value, and if the characteristic value of the audio signal is greater than the threshold value, then the corresponding block of data of the digital audio signal is marked as related to the first audio source. signal, and if the characteristic value of the audio signal is less than the threshold value, then the corresponding block of data of the digital audio signal is marked as belonging to the second source of the audio signal. 6. The method according to claim 1, characterized in that the marking of digital audio signal data is carried out by dividing the recorded audio stream of the speakers' speech into stereo channels. 7. The method according to claim 1, characterized in that the labeling of the digital audio signal data is carried out by creating an additional data block indicating timestamps characterizing the recording time of the replicas of at least one speaker. 8. A device for diarizing a speech audio signal, comprising at least one computing device and at least one memory device containing machine-readable instructions, which, when executed by at least one computing device, perform the method according to any one of claims 1-7.