JP7273078B2 - Speaker Diarization Method, System, and Computer Program Using Voice Activity Detection Based on Speaker Embedding - Google Patents

Description

The following description relates to speaker diarization techniques.

Speaker diarization refers to a technique for dividing speech segments for each speaker from a speech file in which contents uttered by a large number of speakers are recorded.

The speaker diarization technique detects speaker boundary sections from speech data, and is divided into distance-based and model-based techniques depending on whether prior knowledge of speakers is used.

For example, Patent Literature 1 (registered February 23, 2018) describes a speaker's speech recognition based on the speaker's speech without being affected by changes in the environment in which the speaker's speech is recognized and the speaker's utterance state. Techniques are disclosed for generating a speaker recognition model that can discriminate between .

Such speaker diarization technology is a variety of technologies that automatically record the contents of each speaker's speech in a situation where many speakers speak out of order, such as in meetings, interviews, transactions, and trials. , and is used for the automatic creation of meeting minutes.

Korean Patent No. 10-1833731

A method and system are provided for detecting speech intervals, which are speech activity regions, based on speaker embeddings.

A method and system are provided for performing voice activity detection and speaker embedding extraction using a single model, a speaker recognition model, rather than using a separate model for detecting voice activity.

A computer system implemented speaker diarization method, said computer system including at least one processor configured to execute computer readable instructions contained in a memory, said speaker diarization method comprising: extracting, by the at least one processor, speaker embeddings for each audio frame for a given audio file; and extracting, by the at least one processor, speech activity regions based on the speaker embeddings. ).

According to one aspect, the speaker diarization method may utilize a single model, a speaker recognition model, to perform the steps of extracting the speaker embeddings and detecting the speech intervals.

According to another aspect, the step of detecting speech intervals includes determining a norm value for a speaker embedding vector of each of the speech frames; The method may include determining the speech interval, and determining speech frames below the threshold as non-speech intervals.

According to yet another aspect, the speaker diarization method further includes adaptively setting, by the at least one processor, the threshold for classifying speech and non-speech according to a given speech file. OK.

According to yet another aspect, the method of speaker diarization further includes setting, by the at least one processor, the threshold estimated by a Gaussian mixture model for the audio file. OK.

According to yet another aspect, the method for speaker diarization further includes setting, by the at least one processor, the threshold for classifying speech and non-speech to an empirically determined fixed value. good.

According to yet another aspect, extracting the speaker embeddings may include extracting the speaker embeddings for each of the speech frames using a sliding window scheme.

According to yet another aspect, the step of extracting speaker embeddings includes using a speaker recognition model trained using a combination of classification loss and hard negative mining loss to: Extracting said speaker embeddings may be included.

According to yet another aspect, the step of extracting speaker embeddings includes projection It may include obtaining utterance-level embeddings by passing through projection layers.

According to yet another aspect, detecting the speech intervals includes obtaining speech activity labels as the output of the speaker recognition model is propagated through the projection layer without aggregation over time. may contain

A computer program recorded on a non-transitory computer-readable recording medium is provided for causing the computer system to execute the speaker diarization method.

A computer system comprising at least one processor configured to execute computer readable instructions contained in a memory, the at least one processor speaking for each audio frame for a given audio file. A computer system is provided, including a speaker embedder for extracting speaker embeddings, and a speech activity detector for detecting speech activity regions, which are speech activity regions, based on the speaker embeddings.

According to the embodiments of the present invention, by detecting speech segments, which are speech active regions, based on speaker embeddings, only segments with clear speaker recognition can be detected, enhancing the performance of speaker diarization. be able to.

According to embodiments of the present invention, it is possible to perform voice activity detection and speaker embedding extraction with a single model by utilizing the speaker recognition model used to extract speaker embeddings for voice activity detection. can simplify the speaker diarization pipeline.

1 is a diagram showing an example of a network environment in one embodiment of the present invention; FIG. It is a block diagram for explaining an example of an internal configuration of a computer system in one embodiment of the present invention. FIG. 2 illustrates an example of components that a processor of a computer system may include in one embodiment of the present invention; 1 is a flowchart illustrating an example of a speaker diarization method that may be performed by a computer system in accordance with one embodiment of the present invention; Figure 4 is a flow chart showing the overall process for speaker diarization in one embodiment of the present invention; FIG. 4 is a diagram showing an example of a model for extracting speaker embeddings in one embodiment of the present invention; FIG. 4 shows experimental results of speaker diarization performance using a speech activity detection method based on speaker embedding in an embodiment of the present invention;

BEST MODE FOR CARRYING OUT THE INVENTION Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings.

Embodiments of the present invention relate to a speaker diarization technique for detecting speaker boundary intervals from speech data.

Embodiments, including those specifically disclosed herein, can improve the performance of speaker diarization by detecting speech activity regions, speech intervals, based on speaker embeddings, The diarization pipeline can be simplified.

FIG. 1 is a diagram showing an example of a network environment in one embodiment of the present invention. The network environment of FIG. 1 illustrates an example including a plurality of electronic devices 110 , 120 , 130 , 140 , server 150 and network 160 . Such FIG. 1 is merely an example for explaining the invention, and the number of electronic devices and the number of servers are not limited as in FIG.

The plurality of electronic devices 110, 120, 130, 140 may be fixed terminals or mobile terminals implemented by a computer system. Examples of the plurality of electronic devices 110, 120, 130, and 140 include smartphones, mobile phones, navigation systems, PCs (personal computers), notebook PCs, digital broadcasting terminals, PDAs (Personal Digital Assistants), and PMPs (Portable Multimedia Players). ), tablets, game consoles, wearable devices, IoT (Internet of things) devices, VR (virtual reality) devices, AR (augmented reality) devices, and the like. As an example, FIG. 1 shows a smart phone as an example of the electronic device 110 , but in embodiments of the present invention, the electronic device 110 substantially utilizes wireless or wired communication schemes and communicates with other devices via the network 160 . may refer to one of a variety of physical computer systems capable of communicating with the electronic devices 120 , 130 , 140 and/or the server 150 .

The communication method is not limited, and not only the communication method using the communication network (eg, mobile communication network, wired Internet, wireless Internet, broadcast network, satellite network, etc.) that can be included in the network 160, but also the device It may also include short-range wireless communication between For example, the network 160 includes a PAN (personal area network), a LAN (local area network), a CAN (campus area network), a MAN (metropolitan area network), a WAN (wide area network), a BBN (broadband network). k), such as the Internet Any one or more of the networks may be included. Additionally, network 160 may include any one or more of network topologies including, but not limited to, bus networks, star networks, ring networks, mesh networks, star-bus networks, tree or hierarchical networks, and the like. will not be

Server 150 may be implemented by one or more computing devices in communication with a plurality of electronic devices 110, 120, 130, 140 over network 160 to provide instructions, code, files, content, services, and the like. For example, the server 150 may be a system that provides intended services to a plurality of electronic devices 110 , 120 , 130 , 140 connected via the network 160 . As a more specific example, the server 150 provides services (for example, voice recognition (such as an artificial intelligence minutes service based on the

FIG. 2 is a block diagram illustrating an example computer system, in accordance with one embodiment of the present invention. The server 150 described with reference to FIG. 1 may be implemented by the computer system 200 configured as shown in FIG.

As shown in FIG. 2, computer system 200 includes memory 210, processor 220, communication interface 230, and input/output interface 240 as components for performing the speaker diarization method according to an embodiment of the present invention. may contain.

The memory 210 is a computer-readable storage medium and may include random access memory (RAM), read only memory (ROM), and permanent mass storage devices such as disk drives. Here, a permanent mass storage device such as a ROM or disk drive may be included in computer system 200 as a separate permanent storage device separate from memory 210 . Also stored in memory 210 may be an operating system and at least one program code. Such software components may be loaded into memory 210 from a computer-readable medium separate from memory 210 . Such other computer-readable recording media may include computer-readable recording media such as floppy drives, disks, tapes, DVD/CD-ROM drives, memory cards, and the like. In other embodiments, software components may be loaded into memory 210 through communication interface 230 that is not a computer-readable medium. For example, it may be loaded into memory 210 of computer system 200 based on a computer program installed by files received over network 160 .

Processor 220 may be configured to process computer program instructions by performing basic arithmetic, logic, and input/output operations. Instructions may be provided to processor 220 by memory 210 or communication interface 230 . For example, processor 220 may be configured to execute received instructions according to program code stored in a storage device, such as memory 210 .

Communications module 230 may provide functionality for computer system 200 to communicate with other devices over network 160 . By way of example, processor 220 of computer system 200 may transmit requests, instructions, data, files, etc. generated according to program code stored in a storage device such as memory 210 to others via network 160 under the control of communication interface 230 . device. Conversely, signals, instructions, data, files, etc. from other devices may be received by computer system 200 through communication interface 230 of computer system 200 over network 160 . Signals, instructions, data, etc., received through communication interface 230 may be communicated to processor 220 and memory 210, and files, etc., may be stored on storage media (the permanent storage devices described above) that computer system 200 may further include. may be recorded.

The communication method is not limited, and not only the communication method using the communication network (for example, mobile communication network, wired Internet, wireless Internet, broadcasting network) that can be included in the network 160, but also the wired/communication method between devices. Wireless communication may be included. For example, the network 160 includes a PAN (personal area network), a LAN (local area network), a CAN (campus area network), a MAN (metropolitan area network), a WAN (wide area network), a BBN (broadband network). k), such as the Internet Any one or more of the networks may be included. Additionally, network 160 may include any one or more of network topologies including, but not limited to, bus networks, star networks, ring networks, mesh networks, star-bus networks, tree or hierarchical networks, and the like. will not be

Input/output interface 240 may be a means for interfacing with input/output device 250 . For example, input devices may include devices such as microphones, keyboards, cameras, or mice, and output devices may include devices such as displays, speakers, and the like. As another example, input/output interface 240 may be a means for interfacing with a device that integrates functionality for input and output, such as a touch screen. Input/output device 250 may be one device with computer system 200 .

Also, in other embodiments, computer system 200 may include fewer or more components than the components of FIG. However, most prior art components need not be explicitly shown in the figures. For example, computer system 200 may be implemented to include at least some of the input/output devices 250 described above, and may also include other components such as transceivers, cameras, various sensors, databases, and the like. It's okay.

Specific embodiments of speaker diarization methods and systems that detect speech activity based on speaker embeddings are described below.

FIG. 3 is a block diagram illustrating exemplary components that a processor of a server may include in accordance with one embodiment of the present invention, and FIG. 4 is a flow chart showing an example of a possible method;

The server 150 according to the present embodiment serves as a service platform that provides an artificial intelligence service that divides speech sections for each speaker from a speech file that records the content uttered by many speakers and organizes them as documents.

Server 150 may comprise a speaker diarization system implemented by computer system 200 . As an example, the server 150 targets a plurality of electronic devices 110 , 120 , 130 , 140 that are clients, and is associated with a dedicated application installed on the electronic devices 110 , 120 , 130 , 140 and the server 150 . A connection to a web/mobile site may provide an artificial intelligence minutes service based on speech recognition.

In particular, server 150 may detect speech intervals, which are speech active regions, based on speaker embeddings.

The processor 220 of the server 150 includes a speaker embedding unit 310, a voice interval detection unit 320, and a clustering execution unit 330 as shown in FIG. 3 as components for executing the speaker diarization method according to FIG. may contain.

Depending on the embodiment, components of processor 220 may be selectively included or excluded from processor 220 . Also, depending on the embodiment, the components of processor 220 may be separated or merged to represent the functionality of processor 220 .

Such processor 220 and components of processor 220 may control server 150 to perform steps 410-430 included in the speaker diarization method of FIG. For example, processor 220 and components of processor 220 may be implemented to execute instructions according to the code of an operating system and the code of at least one program contained in memory 210 .

Here, the components of processor 220 may represent different functions performed by processor 220 according to instructions provided by program code stored on server 150 . For example, speaker embedder 310 may be utilized as a functional representation of processor 220 controlling server 150 according to the instructions described above so that server 150 extracts speaker embeddings.

Processor 220 may read the necessary instructions from memory 210 loaded with instructions associated with the control of server 150 . In this case, the read instructions may include instructions for controlling processor 220 to perform steps 410-430 described below.

Steps 410-430 described below may be performed in a different order than shown in FIG. 4, and additional steps may be included even if some of steps 410-430 are omitted. good too.

Referring to FIG. 4, in step 410, when given a speech file containing recorded contents uttered by a number of speakers, the speaker embedding unit 310 uses a speaker recognition model to convert the speech file to the given speech file. Alternatively, speaker embeddings may be extracted for each audio frame. For example, the speaker embedding unit 310 may extract speaker embeddings for each speech frame using a sliding window scheme.

At step 420, the speech activity detector 320 may detect speech activity, which is the speech activity area, based on the speaker embeddings. A speaker recognition model (eg, SpeakerNet, etc.) for extracting speaker embeddings exhibits a high embedding Norm value for speech and a low embedding norm value for non-speech. As an example, the speech interval detection unit 320 may obtain a norm value for the speaker embedding vector of each speech frame, and may determine that a speech frame having an embedding norm value greater than or equal to a threshold is a speech interval. is less than the threshold may be determined as a non-speech segment.

At step 430, the clustering performer 330 may perform speaker diarization clustering based on the speech intervals detected at step 420 by grouping the speaker embeddings. After calculating an affinity matrix for speaker embeddings, the clustering execution unit 330 may determine the number of clusters based on the affinity matrix. At this time, the clustering execution unit 330 performs eigendecomposition on the similarity matrix to extract eigenvalues, arranges the extracted eigenvalues in order of size, and arranges the arranged eigenvalues adjacent to each other. Based on the eigenvalue difference, the number of eigenvalues corresponding to effective principal components may be determined as the number of clusters. A high eigenvalue means a high degree of influence in the similarity matrix. means high. In other words, the clustering execution unit 330 may select eigenvalues having sufficiently large values from the aligned eigenvalues, and determine the number of selected eigenvalues as the number of clusters indicating the number of speakers. The clustering execution unit 330 may perform speaker diarization clustering by mapping speech intervals based on the determined number of clusters.

As shown in FIG. 5, the overall process 50 for speaker diarization includes a speech region detection step 51, an Extract speaker embeddings step 52, and a speaker diarization clustering step 53. may contain.

Conventionally, the speech interval is detected by measuring the energy of each frame and distinguishing between speech and non-speech. You were using an independent model that was different from the model. When detecting speech segments based on energy, some of the detected speech segments may include segments where speaker recognition is difficult. Since it is a type that could not be done, the quality of speaker embedding is degraded. As a result, the quality of detected speech intervals dominates the performance of speaker diarization.

In this embodiment, processor 220 does not use a separate model for detecting voice activity, but utilizes a single model, the speaker recognition model, to perform voice activity detection and speaker embedding extraction. In other words, the present invention can perform speech activity detection step 51 and speaker embedding extraction step 52 with only an embedding model.

The core architecture applied to the speaker diarization system according to the present invention will be described as follows.

Obtaining speaker representations that are well recognized by speaker recognition models is at the heart of the problem of speaker diarization. A method of learning and extracting speaker embeddings using a deep neural network will be described below.

Input representations may be implemented linearly intervalwise on the Mel scale. Processor 220 extracts the spectrogram from each utterance in a window of constant size (eg, 25 ms wide and 10 ms stride). A 64-dimensional mel filter bank is used as input to the network. Mean and Variance Normalization (MVN) uses instance normalization and is performed on all frequency bins of the spectrum and filterbank at the utterance level.

The speaker embedding extraction model may use ResNet (Residual networks), which is one of speaker recognition models. For example, ResNet-34 without pre-activation residual units may be applied as the basic architecture. An example of the ResNet-34 architecture is shown in FIG.

The output of the speaker embedding extraction model is aggregated over time using a temporal average pooling layer and then passed through a linear projection layer to obtain utterance-level embeddings. can be obtained.

Processor 220 uses a combination of classification loss and hard negative mining loss as the objective function to learn the speaker embedding extraction model.

The classification loss L _CE is defined as Equation (1), and the hard negative mining loss L _H is defined as Equation (2).

値が大きい上位Ｈ話者ベースの集合を意味する。特定の話者に対する話者の基準は、話者に該当する出力層の加重値行列の一行ベクトルである。各サンプルに対するハード集合であるＨ_iは、サンプルｘ_iと学習セットのすべての話者ベースの間のコサイン類似性に基づき、すべてのミニバッチに対して選択される。範疇型交差エントロピー損失である分類ロスＬ_CEとハードネガティブマイニングロスＬ_Hは、同じ加重値で結合される。

プロセッサ２２０は、有名人の音声を抽出および検収することによって生成された学習データセット（例えば、ＶｏｘＣｅｌｅｂ２など）を利用して話者埋め込み抽出モデルを学習する。このとき、プロセッサ２２０は、各発言からランダムに抽出された固定の長さ（例えば、２秒）の時間セグメント（ｔｅｍｐｏｒａｌ ｓｅｇｍｅｎｔｓ）を利用して話者埋め込み抽出モデルを学習してよい。 Here, N is the batch size, x _i and W _yi are the embedding vector of the i-th utterance and the basis of the corresponding speaker. H _i is

We mean the set of the top H-speaker bases with large values. A speaker criterion for a particular speaker is a single row vector of the weight matrix of the output layer corresponding to the speaker. A hard set H _i for each sample is selected for every mini-batch based on the cosine similarity between sample x _i and all speaker bases in the training set. The categorical cross-entropy loss, the classification loss L _CE and the hard negative mining loss L _H are combined with the same weight.

Processor 220 trains the speaker embedding extraction model using a training data set (eg, VoxCeleb2, etc.) generated by extracting and accepting voices of celebrities. At this time, the processor 220 may learn the speaker embedding extraction model using temporal segments of fixed length (eg, 2 seconds) randomly extracted from each utterance.

The speaker recognition model used in the speaker embedding extraction step 52, which extracts speaker embeddings representing speaker information in the frames selected in the speech activity detection step 51, may also be utilized in the speech activity detection step 51. Speaker embeddings can distinguish between speech and non-speech, as they can distinguish one person's speech from another's.

In that the embedding norm value and confidence on the target task are correlated, if the embedding vector is classified by the same linear classifier as the output layer activated by the softmax function, the norm value is High means that there is a large margin between the embedding vector and the hyper plane, ie the confidence score of the model is high.

Since the speaker recognition model is trained only for human speech, it has a low embedding norm value and a very low confidence score for non-speech that is not the object of learning. Therefore, the speaker recognition model can be used in the speech activity detection step 51 without modification of the independent modules or models.

To obtain the refined speech activity labels, we import all the outputs by the speaker embedding extraction model and pass them through the projection layer without temporal aggregation. This is in contrast to using aggregated embeddings in windows of constant size (eg, 2 seconds) using temporal average pooling for speaker representation.

The processor 220 obtains a norm value for the speaker embedding vector of each speech frame, judges speech frames with an embedding norm value equal to or greater than a threshold to be speech segments, and speech frames with embedding norm values less than the threshold to be non-speech segments. to decide.

As an example, the processor 220 may set the threshold for classifying speech and non-speech at a fixed experimental value. The development set may be used to manually set the thresholds for the embedding norm values by experimenting and finding the best results within the threshold range. Processor 220 may set a single threshold for all data sets.

As another example, processor 220 may automatically set the optimal threshold for a given audio file. Processor 220 may then use a Gaussian Mixture Model (GMM) to estimate the optimal threshold for each utterance. For this, two mixture components are used to train a Gaussian mixture model, and as one utterance the distribution of norm values is learned. At this time, the mixed component indicates a speech cluster and a non-speech cluster. After training the Gaussian mixture model, the threshold may be estimated by equation (4).

where μ ₀ and μ ₁ are the respective mean values of the mixture components and α means the weighting factor of the two mean values.

By adaptively estimating thresholds for classifying speech and non-speech with speech data, processor 220 can set robust thresholds in diverse dataset domains.

Processor 220 may divide each session of audio data into a plurality of speech activity segments based on the results of speaker embedding-based speech activity detection step 51 .

The processor 220 performs a PD (end point detection) process to ensure that the result of voice activity detection does not change excessively. EPD is the process of finding only the beginning and end of utterances that distinguish between speech and non-speech. As an example, processor 220 finds the beginning and end by sliding a window of constant size. For example, starting points may be identified as points where the percentage of speech activity frames exceeds 70%, and ending points may be identified by the same rule for non-speech frames.

Processor 220 may utilize an Agglomerative Hierarchical Clustering (AHC) algorithm to group speaker embeddings. The AHC algorithm may cluster speaker representations by a distance threshold or number of clusters. The processor 220 automatically selects the optimal number of clusters for each session or audio file (or video with audio) based on the silhouette score (2≤C≤10) in multiple different domains. good.

Silhouette scores are an interpretation of consistency within data clusters and may be viewed as a measure of confidence. The number of silhouette points may be defined by the average distance within the cluster as shown in Equation (5).

A mean nearest-cluster distance may be defined as in Equation (6) for each sample.

In particular, the number of sample silhouette points s(i) may be defined as in Equation (7).

Clustering methods using silhouette scores do not require parameter optimization, unlike methods that manually adjust the threshold for each data set.

In this embodiment, the method of detecting speech active regions (ie, speech intervals) based on speaker embeddings provides a very simple and effective solution for enhancing the performance of speaker diarization.

FIG. 7 is a diagram showing experimental results of speaker diarization performance of the speech activity detection method based on speaker embedding in the present invention.

The experiment utilizes DIHARD as a challenge dataset for speaker diarization and implements a fully split-pipeline speaker diarization method with a model for detecting speech activity and a model for extracting speaker embeddings. Use as a baseline. SE (speech enhancement) includes a denoising process for speech.

MS (missed speech) is the ratio of speech not included in the result, FA (false alarm) is the ratio of non-speech included in the result, SC (speaker confusion) is the ratio of mapping error included in the result (speaker ID DER (diarsation error rate) means the sum of MS, FA, and SC. That is, the lower the DER, the better the performance of speaker diarization.

Comparing the performance of our speaker diarization, which performs voice activity detection and speaker embedding extraction with a single model, to the baseline, we find that the fixed-threshold method (Ours w/ We found that both the SpeakerNet SAD Fixed) and adaptively auto-configured methods (Ours w/SpeakerNet SAD GMM) show higher performance compared to baseline.

As described above, according to the embodiment of the present invention, by detecting the speech section, which is the speech activity area, based on the speaker embedding, it is possible to detect only the section where the speaker recognition is clear, and the speaker diarization can be performed. performance can be improved. Also, according to embodiments of the present invention, a single model performs voice activity detection and speaker embedding extraction by utilizing the speaker recognition model used to extract speaker embeddings to detect voice activity. can be used to simplify the speaker diarization pipeline.

The apparatus described above may be realized by hardware components, software components, and/or a combination of hardware and software components. For example, the devices and components described in the embodiments include processors, controllers, arithmetic logic units (ALUs), digital signal processors, microcomputers, field programmable gate arrays (FPGAs), programmable logic units (PLUs), microprocessors, Or may be implemented using one or more general purpose or special purpose computers, such as various devices capable of executing and responding to instructions. The processing unit may run an operating system (OS) and one or more software applications that run on the OS. The processor may also access, record, manipulate, process, and generate data in response to executing software. For convenience of understanding, one processing device may be described as being used, but those skilled in the art will appreciate that a processing device may include multiple processing elements and/or multiple types of processing elements. You can understand that. For example, a processing unit may include multiple processors or a processor and a controller. Other processing configurations are also possible, such as parallel processors.

Software may include computer programs, code, instructions, or a combination of one or more of these, to configure a processor to operate at its discretion or to independently or collectively instruct a processor. You can Software and/or data may be embodied in any kind of machine, component, physical device, computer storage medium, or device for interpretation by, or for providing instructions or data to, a processing device. good. The software may be stored and executed in a distributed fashion over computer systems linked by a network. Software and data may be recorded on one or more computer-readable recording media.

The method according to the embodiments may be embodied in the form of program instructions executable by various computer means and recorded on a computer-readable medium. Here, the medium may record the computer-executable program continuously or temporarily record it for execution or download. In addition, the medium may be various recording means or storage means in the form of a combination of single or multiple hardware, and is not limited to a medium that is directly connected to a computer system, but is distributed over a network. It may exist in Examples of media include magnetic media such as hard disks, floppy disks, and magnetic tapes, optical media such as CD-ROMs and DVDs, magneto-optical media such as floptical disks, and ROM, RAM, flash memory, etc., and may be configured to store program instructions. Other examples of media include recording media or storage media managed by application stores that distribute applications, sites that supply or distribute various software, and servers.

As described above, the embodiments have been described based on the limited embodiments and drawings, but those skilled in the art will be able to make various modifications and variations based on the above description. For example, the techniques described may be performed in a different order than in the manner described and/or components such as systems, structures, devices, circuits, etc. described may be performed in a manner different from the manner described. Appropriate results may be achieved when combined or combined, opposed or substituted by other elements or equivalents.

Accordingly, different embodiments that are equivalent to the claims should still fall within the scope of the appended claims.

220: Processor 310: Speaker embedding unit 320: Speech interval detection unit 330: Clustering execution unit

Claims (18)

A computer system implemented speaker diarization method comprising:
The computer system includes at least one processor configured to execute computer readable instructions contained in memory;
The speaker diarization method comprises:
The at least one processor generates speech frames for a given speech file with a speaker recognition model trained using a combination of classification loss and hard negative mining loss. and detecting, by the at least one processor, speech intervals that are speech activity regions based on the speaker embeddings. The speaker diarization method comprises:
2. The speaker diarization of claim 1, wherein extracting the speaker embeddings and detecting the speech intervals are performed using a single model, a speaker recognition model. Method. The step of detecting the speech interval includes:
determining a norm value for a speaker embedding vector of each speech frame; determining speech frames with an embedding norm value greater than or equal to a threshold as the speech period, and speech frames with an embedding norm value less than the threshold as the non-speech period; 2. The speaker diarization method of claim 1, comprising the step of determining. The speaker diarization method comprises:
4. The method of speaker diarization of claim 3, further comprising: adaptively setting, by the at least one processor, the threshold for classifying speech and non-speech according to a given speech file. The speaker diarization method comprises:
4. The speaker diarization method of claim 3, further comprising: setting, by the at least one processor, the threshold estimated by a Gaussian mixture model for the audio file. The speaker diarization method comprises:
4. The method of speaker diarization of claim 3, further comprising: setting, by the at least one processor, the threshold for classifying speech and non-speech to an empirically determined fixed value. Extracting the speaker embeddings includes:
2. The speaker diarization method of claim 1, comprising extracting the speaker embeddings for each speech frame using a sliding window scheme . A computer system implemented speaker diarization method comprising:
The computer system includes at least one processor configured to execute computer readable instructions contained in memory;
The speaker diarization method comprises:
extracting, by the at least one processor, speaker embeddings for each audio frame for a given audio file, comprising:
The output of the speaker recognition model is aggregated over time using a temporal average pooling layer and then passed through a projection layer to determine the utterance-level and detecting, by the at least one processor, speech intervals that are speech activity regions based on the speaker embeddings.
Speaker diarization method. The step of detecting the speech interval includes:
9. The speaker diarization method of claim 8, comprising: obtaining speech activity labels by transmitting the output of the speaker recognition model through the projection layer without aggregation over time. A computer program that causes the computer system to perform the speaker diarization method according to any one of claims 1-9. a computer system,
at least one processor configured to execute computer readable instructions contained in memory;
The at least one processor
a speaker embedding unit that extracts speaker embeddings for each audio frame for a given audio file using a speaker recognition model trained using a combination of classification loss and hard negative mining loss; and said speaker. A computer system comprising: a speech interval detector that detects speech intervals that are speech activity regions based on embeddings. The at least one processor
12. The computer system of claim 11, wherein a single model, a speaker recognition model, is utilized to perform the steps of extracting speaker embeddings and detecting speech intervals. The voice interval detection unit is
obtaining a norm value for the speaker embedding vector of each of the speech frames;
12. The computer system according to claim 11, wherein a speech frame with an embedding norm value equal to or greater than a threshold value is determined to be the speech segment, and a speech frame with an embedding norm value less than the threshold value is determined to be the non-speech segment. The at least one processor
14. The computer system of claim 13, wherein the threshold for classifying speech and non-speech is adaptively set according to a given speech file. The at least one processor
14. The computer system of claim 13, wherein the threshold estimated by a Gaussian mixture model is set for the audio file. The speaker embedding unit includes:
12. The computer system of claim 11, wherein a sliding window scheme is used to extract the speaker embeddings for each speech frame. a computer system,
at least one processor configured to execute computer readable instructions contained in memory;
The at least one processor
A speaker embedding unit for extracting speaker embeddings for each audio frame for a given audio file, the speaker embedding unit comprising:
a speaker embedding unit configured to obtain utterance level embeddings by passing the output of the speaker recognition model aggregated over time using a temporal average pooling layer and then through a projection layer; and a speech interval detector that detects a speech interval, which is a speech activity area, based on the speaker embeddings.
computer system. The voice interval detection unit is
18. The computer system of claim 17, wherein the output of the speaker recognition model is propagated through the projection layer without aggregation over time to obtain voice activity labels.

Images