JP7453733B2 - Method and system for improving multi-device speaker diarization performance - Google Patents

Description

The following description relates to speaker diarization techniques.

Speaker diarization is a technology that divides utterance sections for each speaker from a recorded audio file of utterances by multiple speakers.

Speaker diarization techniques detect speaker boundary sections from audio data, and are divided into distance-based methods and model-based methods depending on whether or not prior knowledge of speakers can be used.

For example, in Patent Document 1 (registration date: February 23, 2018), the speaker is recognized based on the speaker's voice without being affected by changes in the environment in which the speaker's voice is recognized or the speaker's speaking state. A technique is disclosed for generating a speaker recognition model that can be differentiated.

This type of speaker diarization technology automatically records the utterances divided by each speaker in situations where multiple speakers speak in a random order, such as in meetings, interviews, transactions, and court cases. This technology is used for various purposes such as automatically creating minutes of meetings.

Korean Registered Patent No. 10-1833731

A method and system are provided that can improve multi-device speaker diarization performance.

A method and system are provided that can perform speaker diarization in a multi-device environment that utilizes personal equipment owned by each user.

A method and system are provided that can estimate the number of speakers (number of clusters) based on reliability.

A speaker diarization method performed by a computer system, the computer system including at least one processor configured to execute computer readable instructions contained in a memory, the speaker diarization method comprising: , receiving, by the at least one processor, an audio file recorded by each electronic device from a plurality of electronic devices; an embedding matrix calculated for the audio file of each of the electronic devices by the at least one processor; estimating the number of candidate clusters based on the at least one processor, determining a final number of clusters using the number of candidate clusters of each electronic device; A speaker diarization method is provided, the method comprising performing speaker diarization clustering using a method.

According to one aspect, the receiving step includes performing end point detection (EPD) on the audio files of the respective electronic devices, and integrating the EPD results of the respective electronic devices. The method may include generating an EPD union.

According to another aspect, the estimating step includes calculating an affinity matrix by embedding and extracting from the EPD result of the audio file of each of the electronic devices, and calculating the similarity of each of the electronic devices. The method may include calculating the number of candidate clusters and a reliability value of the similarity matrix using a matrix.

According to another aspect, calculating the number of candidate clusters and the reliability value of the similarity matrix includes performing eigen decomposition on the similarity matrix to extract eigenvalues. and, after arranging the extracted eigenvalues, calculating the number of candidate clusters and a reliability value of the similarity matrix based on a difference between adjacent eigenvalues.

According to another aspect, the step of calculating the number of candidate clusters and the reliability value of the similarity matrix includes the step of performing eigenvalue decomposition on the similarity matrix to extract eigenvalues; After arranging the eigenvalues, the number of eigenvalues selected based on the difference between adjacent eigenvalues is determined as the number of candidate clusters, and the eigenvalues remaining unselected in the process of determining the number of candidate clusters are used to It may include calculating a confidence value.

According to another aspect, the step of calculating the reliability value using the remaining eigenvalues may determine the largest eigenvalue among the remaining eigenvalues as the reliability value of the similarity matrix.

According to another aspect, the step of calculating the reliability value using the remaining eigenvalues may include determining an average value obtained by calculating the average of the remaining eigenvalues as the reliability value of the similarity matrix. .

According to another aspect, the estimating step may further include applying a weighted sum to the similarity matrix based on weight values learned for the EPD result of the audio file. .

According to another aspect, the determining step may determine the number of candidate clusters estimated from the similarity matrix having the largest reliability value as the final number of clusters.

According to still another aspect, the step of performing includes calculating a similarity matrix by performing embedding extraction from the EPD results of the audio files of each of the electronic devices, and averaging the similarity matrix of each of the electronic devices. and performing the speaker diarization clustering based on the average similarity matrix and the final number of clusters.

A computer program is provided that is recorded on a non-transitory computer-readable recording medium to cause the computer system to execute the speaker diarization method.

A non-transitory computer-readable recording medium is provided, on which a program for causing a computer to execute the speaker diarization method is recorded.

A computer system comprising at least one processor configured to execute computer-readable instructions contained in a memory, the at least one processor configured to execute audio recordings at each electronic device from a plurality of electronic devices. a process of receiving a file; a process of estimating the number of candidate clusters based on the embedding matrix calculated for the audio file of each of the electronic devices; and determining a final number of clusters using the number of candidate clusters of each of the electronic devices. and performing speaker diarization clustering using the final cluster number.

According to an embodiment of the present invention, it is possible to improve the performance of multi-device speaker diarization.

According to embodiments of the present invention, speaker diarization can be performed in a multi-device environment that requires no additional equipment and utilizes personal equipment owned by each user.

According to the embodiment of the present invention, the number of speakers (number of clusters) can be estimated more accurately based on reliability.

1 is a diagram illustrating an example of a network environment in an embodiment of the present invention. FIG. FIG. 1 is a block diagram showing an example of the internal configuration of a computer system in an embodiment of the present invention. 1 is a diagram illustrating an example of components that a processor of a computer system may include in an embodiment of the present invention. FIG. 1 is a flowchart illustrating an example of a speaker diarization method that can be performed by a computer system in an embodiment of the invention. FIG. 3 is a diagram illustrating an example of the overall process for speaker diarization in an embodiment of the present invention. FIG. 3 is an exemplary diagram illustrating a process of merging audio regions recognized in individual audio files in an embodiment of the present invention. FIG. 3 is an exemplary diagram illustrating a process of merging audio regions recognized in individual audio files in an embodiment of the present invention. FIG. 3 is an exemplary diagram for explaining a process of determining the number of clusters in an embodiment of the present invention. FIG. 3 is an exemplary diagram for explaining a process of performing speaker diarization clustering in an embodiment of the present invention.

Embodiments of the present invention will be described in detail below with reference to the accompanying drawings.

Embodiments of the present invention relate to speaker diarization techniques for detecting speaker boundary sections from audio data.

Embodiments including matters specifically disclosed herein can improve speaker diarization performance by performing multi-device speaker diarization, and can improve speaker diarization performance by performing multi-device speaker diarization. By utilizing it, system construction costs can be reduced.

FIG. 1 is a diagram showing an example of a network environment in an embodiment of the present invention. The network environment of FIG. 1 shows an example including a plurality of electronic devices 110, 120, 130, 140, a server 150, and a network 160. Such FIG. 1 is only an example for explaining the invention, and the number of electronic devices and the number of servers are not limited as shown in FIG. 1.

The plurality of electronic devices 110, 120, 130, and 140 may be fixed terminals or mobile terminals realized by a computer system. Examples of the plurality of electronic devices 110, 120, 130, and 140 include smartphones, mobile phones, navigation systems, PCs (personal computers), notebook PCs, digital broadcast terminals, PDAs (personal digital assistants), and PMPs (portable multimedia players). er ), tablets, game consoles, wearable devices, IoT (Internet of Things) devices, VR (Virtual Reality) devices, AR (Augmented Reality) devices, etc. As an example, although FIG. 1 shows a smartphone as an example of the electronic device 110, in the embodiment of the present invention, the electronic device 110 may utilize a substantially wireless or wired communication method to communicate with others via the network 170. may refer to one of a variety of physical computer systems capable of communicating with electronic devices 120, 130, 140 and/or server 150.

The communication method is not limited, and can include not only the communication method using the communication network that the network 160 can include (for example, a mobile communication network, wired Internet, wireless Internet, broadcasting network, satellite network, etc.), but also the equipment. It may also include short-range wireless communication between. For example, the network 160 includes a PAN (personal area network), a LAN (local area network), a CAN (campus area network), a MAN (metropolitan area network), and a WAN (wide area network). area network), BBN (broadband network), the Internet, etc. may include any one or more of the networks. Further, network 160 may include any one or more of network topologies including, but not limited to, a bus network, a star network, a ring network, a mesh network, a star-bus network, a tree or a hierarchical network, and the like. It will not be done.

Server 150 may be implemented by one or more computing devices that communicate with multiple electronic devices 110, 120, 130, 140 via network 160 to provide instructions, code, files, content, services, etc. For example, the server 150 may be a system that provides targeted services to a plurality of electronic devices 110, 120, 130, and 140 connected via the network 160. As a more specific example, the server 150 provides services (for example, voice recognition may be provided to a plurality of electronic devices 110, 120, 130, 140.

FIG. 2 is a block diagram illustrating an example of a computer system in an embodiment of the present invention. The server 150 described in FIG. 1 may be realized by a computer system 200 configured as shown in FIG.

As shown in FIG. 2, a computer system 200 includes a memory 210, a processor 220, a communication interface 230, and an input/output interface 240 as components for performing a speaker diarization method according to an embodiment of the present invention. may be included.

Memory 210 is a computer readable storage medium and may include permanent mass storage devices such as random access memory (RAM), read only memory (ROM), and disk drives. Here, a permanent large capacity storage device such as a ROM or a disk drive may be included in the computer system 200 as a separate permanent storage device separate from the memory 210. Additionally, an operating system and at least one program code may be recorded in the memory 210. Such software components may be loaded into memory 210 from a computer-readable storage medium separate from memory 210. Such other computer-readable recording media may include computer-readable recording media such as floppy drives, disks, tapes, DVD/CD-ROM drives, memory cards, and the like. In other embodiments, software components may be loaded into memory 210 through communication interface 230 that is not a computer-readable storage medium. For example, software components may be loaded into memory 210 of computer system 200 based on a computer program installed by a file received over network 160.

Processor 220 may be configured to process instructions of a computer program by performing basic arithmetic, logic, and input/output operations. Instructions may be provided to processor 220 by memory 210 or communication interface 230. For example, processor 220 may be configured to execute instructions received according to program code recorded on a storage device, such as memory 210.

Communication interface 230 may provide functionality for computer system 200 to communicate with other devices and each other via network 160. As an example, requests, instructions, data, files, etc. generated by the processor 220 of the computer system 200 according to program codes recorded in a storage device such as the memory 210 may be transmitted to others via the network 160 under the control of the communication interface 230. may be transmitted to the device. Conversely, signals, instructions, data, files, etc. from other devices may be received by computer system 200 through network 160 and through communication interface 230 of computer system 200. Signals, instructions, data, etc. received through communication interface 230 may be communicated to processor 220 and memory 210, files, etc. may be transferred to a storage medium (such as a persistent storage device as described above) that computer system 200 may further include. May be recorded.

The communication method is not limited, and is not limited to communication methods that utilize communication networks that can be included in the network 160 (for example, mobile communication networks, wired Internet, wireless Internet, and broadcasting networks), as well as communication methods that utilize short distances between devices. Wired/wireless communications may also be included. For example, the network 160 may be a PAN (personal area network), a LAN (local area network), a CAN (campus area network), a MAN (metropolitan area network), or a WAN (wide area network). area network), BBN (broadband network), the Internet, etc. may include any one or more of the networks. Further, network 160 may include any one or more of network topologies including, but not limited to, a bus network, a star network, a ring network, a mesh network, a star-bus network, a tree or a hierarchical network, and the like. It will not be done.

Input/output interface 240 may be a means for interfacing with input/output device 250. For example, input devices may include devices such as a microphone, keyboard, camera, mouse, etc., and output devices may include devices such as a display, speakers, etc. As another example, input/output interface 240 may be a means for interfacing with a device that has integrated input and output functionality, such as a touch screen. Input/output device 250 may be a single device with computer system 200.

Also, in other embodiments, computer system 200 may include fewer or more components than those of FIG. 2. However, most prior art components need not be clearly illustrated. For example, computer system 200 may be implemented to include at least some of the input/output devices 250 described above, and may further include other components such as transceivers, cameras, various sensors, databases, etc. But that's fine.

Specific embodiments of methods and systems for improving speaker diarization performance with multiple devices are described below.

FIG. 3 is a block diagram illustrating an example of components that a processor of a server may include, in an embodiment of the invention, and FIG. 2 is a flowchart showing an example of a possible method.

The server 150 according to the present embodiment serves as a service platform that provides an artificial intelligence service that can organize minutes recorded voice files as documents through speaker diarization.

Server 150 may be configured with a speaker diarization system implemented by computer system 200 . The server 150 targets a plurality of electronic devices 110 , 120 , 130 , and 140 , which are clients, and includes dedicated applications installed on the electronic devices 110 , 120 , 130 , and 140 , and the server 150 . A voice recognition-based artificial intelligence transcription service may be provided by connecting to an associated web/mobile site.

In particular, the server 150 can improve speaker diarization performance through multiple devices using personal equipment owned by each user.

The processor 220 of the server 150 includes a speech integration section 310, a cluster determination section 320, and a clustering execution section 330, as shown in FIG. 3, as components for executing the speaker diarization method according to FIG. good.

Depending on the embodiment, components of processor 220 may be selectively included or excluded from processor 220. Also, depending on the embodiment, components of processor 220 may be separated or combined to express the functionality of processor 220.

Such processor 220 and components of processor 220 may control server 150 to perform steps 410-430 included in the speaker diarization method of FIG. For example, processor 220 and components of processor 220 may be implemented to execute instructions in accordance with operating system code and at least one program code contained in memory 210.

Here, the components of processor 220 may be representations of different functions performed by processor 220 according to instructions provided by program code stored on server 150. For example, the audio integration unit 310 may be used as a functional representation of the processor 220 that controls the server 150 according to the above-described instructions so that the server 150 integrates the audio regions recognized for each device.

Processor 220 may read the necessary instructions from memory 210 loaded with instructions related to controlling server 150. In this case, the read instructions may include instructions for controlling the processor 220 to perform steps 410-430 described below.

The steps 410-430 described below may be performed in a different order than shown in FIG. 4, and some of the steps 410-430 may be omitted or additional steps may be further included. good.

Referring to FIG. 4, in step 410, the audio integration unit 310 targets the plurality of electronic devices 110, 120, 130, and 140 and processes the audio files (hereinafter referred to as "individual audio files") recorded from each electronic device with the corresponding device. '') and may integrate the recognized audio regions from the individual audio files.

This embodiment executes speaker diarization in a multi-device based environment, and for example, a plurality of electronic devices 110, 120, 130, 140 consisting of personal devices owned by users participating in a conference. You can take advantage of it.

A dedicated application or web/mobile site associated with the server 150 may include a start button for initiating participation in a meeting and an exit button for ending participation in a meeting, such that once the start button is entered, At the same time, a function of transmitting audio recorded by the device to the server 150 in real time may be included.

This embodiment does not require any additional equipment for recording conference audio and transmitting it to the server 150, and utilizes personal devices such as smartphones and tablets that conference participants have during the conference. You may do so. In particular, in order to improve speaker diarization performance, the equipment for recording conference audio and transmitting it to the server 150 is not a single equipment, but a multi-device consisting of the personal devices of multiple participants. good.

The audio integration unit 310 receives individual audio files from each of the electronic devices 110, 120, 130, and 140, and then integrates audio segments extracted from the individual audio files. Since the detected audio region may differ depending on the device, the audio regions of each device are integrated in order to eliminate missing sections by adding the audio region that is not detected from a specific device.

In step 420, the cluster determining unit 320 estimates the number of candidate clusters for each individual audio file based on the embedding matrix calculated for the individual audio file (hereinafter referred to as the "individual embedding matrix"), and then estimates the number of candidate clusters for each individual audio file. The final number of clusters may be determined based on the reliability of the embedding matrix.

The cluster determining unit 320 may independently estimate the number of clusters for each individual audio file, and then determine the final number of clusters from among the estimated cluster numbers.

In particular, in the process of estimating the number of candidate clusters for each individual audio file in order to determine the final number of clusters, the cluster determining unit 320 may also calculate the reliability, and the cluster determining unit 320 may also calculate the reliability of the individual audio file estimated using the individual audio file with the highest reliability. The number of candidate clusters may be determined as the final number of clusters.

The specific process of determining the number of clusters will be described in more detail below.

In step 430, the clustering execution unit 330 performs clustering for speaker diarization using the embedding matrix calculated for the speech region integrated in step 410 and the final number of clusters determined in step 420. may be executed.

The clustering execution unit 330 may obtain an average embedding matrix by averaging the individual embedding matrices for the audio files of each device, and may perform speaker diarization clustering based on the average embedding matrix and the final number of clusters.

Therefore, in this embodiment, the estimation of the number of clusters and the speaker diarization clustering can be performed based on different embedding matrices rather than the same embedding matrix, and the estimation of the number of clusters is performed using individual embedding matrices. Diarization clustering can utilize the average embedding matrix.

FIG. 5 is a diagram illustrating an example of the overall process of speaker diarization in an embodiment of the present invention.

Referring to FIG. 5, the speaker diarization process is performed by integrating an independent process performed for each individual audio file received from each electronic device 110, 120, 130, 140, and an individual audio file. It may consist of an integration process.

The audio integration unit 310 collects audio files (individual audio files) recorded at the locations of conference participants from electronic devices 110, 120, 130, and 140, which are personal devices of multiple participants participating in the conference, during the conference. Receive (S51).

The audio integration unit 310 independently performs an EPD (end point detection) process on each individual audio file (S52). EPD means searching for the beginning and end of utterances that are classified as speech/silence by removing acoustic features from frames corresponding to silent periods and then measuring the energy of each frame. In other words, the audio integrator 310 performs EPD to search for a region of audio in the individual audio files.

For example, as shown in FIG. 6, the audio integration unit 310 may acquire the audio area 601 detected as the EPD result from each device of the conference participants. Since the positions of each participant in the conference are different, the detected audio areas 601 are also different.

Referring again to FIG. 5, the audio integration unit 310 may integrate the EPD results of each device of the conference participants to generate an EPD union (S53).

As shown in FIG. 7, the audio regions 601 detected from each device of the conference participants are all different, so to avoid omission of sections, the EPD results of each device are integrated to generate an EPD union 702. good.

In other words, the audio integration unit 310 integrates each individual EPD result of each individual audio file received from each device of a conference participant as one EPD result.

Referring again to FIG. 5, the cluster determining unit 320 independently performs the embedding extraction process on the EPD results of each device (S54).

The cluster determination unit 320 calculates an individual similarity matrix (affinity matrix) by performing embedding extraction from the EPD results of each device, and then calculates the number of clusters using the individual similarity matrix of each device (S55). .

At this time, the cluster determining unit 320 may calculate the reliability of the individual similarity matrix as well as the number of clusters.

Referring to FIG. 8, the cluster determining unit 320 performs eigen decomposition on the individual similarity matrix 803 calculated for each individual audio file of each device to determine the eigenvalue and eigenvector. may be extracted.

At this time, the cluster determining unit 320 may arrange the eigenvalues extracted from the individual similarity matrix 803 in order of the size of the eigenvalues, and determine the number of clusters 804 and the reliability value 805 based on the arranged eigenvalues.

The cluster determining unit 320 may determine the number of eigenvalues corresponding to valid principal components as the number of clusters 804 based on the difference between the eigenvalues adjacent to the aligned eigenvalues. A high eigenvalue means that it has a large influence on the individual similarity matrix 803. In other words, when constructing the individual similarity matrix 803 for the audio region in the individual audio file, the This means that there is a high emphasis on vocalization.

In other words, the cluster determining unit 320 may select eigenvalues having a sufficiently large value from among the arranged eigenvalues, and determine the number of selected eigenvalues as the number of clusters 804 indicating the number of speakers. .

Eigenvalues that are not selected in the process of determining the number of clusters 804 may be regarded as noise included in the individual similarity matrix 803, and the smaller the eigenvalues that are not selected, the more accurate the calculation of the individual similarity matrix 803 is. As a result, it may be determined that the reliability of the individual similarity matrix 803 is high.

The cluster determining unit 320 may calculate the reliability value 805 using the eigenvalues that were not selected in the process of determining the number of clusters 804 and remained as noise among the sorted eigenvalues.

As an example, the cluster determining unit 320 may use the largest eigenvalue among the eigenvalues not selected in the process of determining the number of clusters 804 as the reliability value 805. For example, when the four highest eigenvalues among the sorted eigenvalues are determined as the number of effective principal components, that is, the number of clusters 804, the fifth eigenvalue may be used as the reliability value 805.

As another example, the cluster determining unit 320 may use, as the reliability value 805, an average eigenvalue obtained by calculating the average of all eigenvalues that were not selected in the process of determining the number of clusters 804.

In that the audio regions 601 detected from each device of a conference participant are different, the individual similarity matrices 803 calculated from this may also be different, and the results of the number of clusters 804 indicating the number of speakers may also be different. .

If 4 speakers are estimated for the individual audio file of device 1 and 5 speakers are estimated for the individual audio file of device 2, we will use confidence to integrate these different results. be.

The cluster determining unit 320 can also determine the number of clusters 804 using an average similarity matrix obtained by averaging the individual similarity matrices 803 of each device. However, when using the average similarity matrix, an error may occur in which the number of clusters 804 is incorrectly estimated.

Since the number of clusters 804 is estimated by analogizing the number of effective principal components from among the eigenvalues calculated from the similarity matrix, performance may deteriorate if the sharpness of the similarity matrix decreases.

Therefore, in determining the number of clusters 804, it may be better to use the sharpening results (individual similarity matrix for each device) than the results of smoothing the audio files (average similarity matrix). Accurate results may be obtained.

In some embodiments, a weighted sum of individual similarity matrices 803 may be applied.

Considering that the reliability may differ for each section of the individual similarity matrix 803, instead of performing eigenvalue decomposition by applying the same weight to all the sections of the individual similarity matrix 803, A weight value may be learned and applied by decreasing the weight value of a region that is not detected as an EPD.

For example, after integrating the matrices by randomly applying weight values to each section of the individual similarity matrix 803, the weight values may be learned in a direction that increases reliability by calculating eigenvalues.

Referring again to FIG. 5, the cluster determining unit 320 integrates the estimated number of clusters and reliability values for each individual audio file of each device, and then finally determines the number of clusters based on the reliability. Yes (S56).

The cluster determining unit 320 determines the number of clusters calculated as the individual similarity matrix with the highest reliability value among the individual similarity matrices calculated for each individual audio file of each device as the final number of clusters. It's fine.

The clustering execution unit 330 may calculate an individual similarity matrix for each device by independently performing embedding and extraction using the EPD union that is the result of integrating the EPD results of each device (S57).

The clustering execution unit 330 calculates an average similarity matrix by averaging the individual similarity matrices calculated independently for each device, and then calculates the average similarity matrix along with the average similarity matrix determined based on the reliability in step S56. Speaker diarization clustering may be performed using the number of clusters obtained (S58).

As shown in FIG. 9, the clustering execution unit 330 may calculate an average similarity matrix 902 by averaging individual similarity matrices 901 calculated independently for each device.

For example, the clustering execution unit 330 may calculate the average similarity matrix 902 by performing element-wise matrix arithmetic operations on the individual similarity matrix 901 calculated for each device.

Next, the clustering execution unit 330 may perform eigenvalue decomposition on the average similarity matrix 902 and perform clustering based on the eigenvectors arranged in order of eigenvalues.

When m voice sections are extracted from one individual voice file, a matrix containing m×m elements is generated, and in this case, v _{i, j} indicating each element is the i-th voice section. It means the distance from to the j-th voice section.

At this time, the clustering execution unit 330 may perform speaker diarization clustering by selecting eigenvectors by the number of clusters determined based on reliability.

The overall process for speaker diarization is to receive audio files recorded simultaneously on multiple personal devices during a conference, perform EPD on the audio files of each device, and identify the segments on which EPD was performed (audio After extracting embeddings for each area (region) and estimating the number of clusters (number of speakers), clustering is performed based on the estimated number of clusters.

In this embodiment, the process for improving the speaker diarization performance includes generating an EPD union using the individual audio files of each device, and calculating individual embeddings for the individual audio files of each device. This includes estimating the number of clusters using a matrix and then determining the final number of clusters based on confidence, and performing speaker diarization clustering using the number of clusters based on confidence and an average similarity matrix. It's fine.

Thus, embodiments of the present invention allow multi-device speaker diarization to be performed without the need for additional equipment and while leveraging personal equipment carried by multiple conference participants. .

According to an embodiment of the present invention, after estimating the number of clusters from the audio file of each device, the final number of clusters is determined based on the confidence level, thereby improving speaker diarization performance based on the accurately estimated number of clusters. can be improved.

In this way, in this embodiment, a new task of multi-device speaker diarization can be defined, and system construction costs can be reduced by utilizing personal devices owned by each conference participant. This allows for a wider range of meeting spaces to be efficiently covered.

Although the most common approach is to train a model to suit a new task, learning a new model requires consideration of data collection, the actual environment in which it will be applied, generalization performance, etc. There is a need. On the other hand, in this embodiment, the conventional speaker diarization model can be used as is, and even in the case of a speaker diarization system that is already in service, there is no need to retrain the model and multi-device Speaker diarization performance can be improved simply by adding the ability to receive conference audio from speakers.

The apparatus described above may be realized by hardware components, software components, and/or a combination of hardware and software components. For example, the devices and components described in the embodiments include a processor, a controller, an arithmetic logic unit (ALU), a digital signal processor, a microcomputer, a field programmable gate array (FPGA), a programmable logic unit (PLU), a microprocessor, or may be implemented using one or more general purpose or special purpose computers, such as various devices capable of executing and responding to instructions. A processing device may execute an operating system (OS) and one or more software applications that execute on the OS. The processing device may also be responsive to execution of the software to access, record, manipulate, process, and generate data. Although for convenience of understanding, one processing device may be described as being used, those skilled in the art will appreciate that a processing device may include multiple processing elements and/or multiple types of processing elements. You will understand. For example, a processing device may include multiple processors or a processor and a controller. Other processing configurations are also possible, such as parallel processors.

Software may include computer programs, code, instructions, or a combination of one or more of these that configure a processing device or instruct a processing device, independently or collectively, to perform operations as desired. You may do so. The software and/or data may be embodied in a machine, component, physical device, computer storage medium or device of any kind for being interpreted by or providing instructions or data to a processing device. good. The software may be distributed on computer systems connected by a network, and may be recorded or executed in a distributed manner. The software and data may be recorded on one or more computer readable storage media.

Methods according to embodiments may be implemented in the form of program instructions executable by various computer means and recorded on computer-readable media. Here, the medium may be one that continuously records a computer-executable program, or one that temporarily records it for execution or download. Also, the medium may be a variety of recording or storage means in the form of a single or multiple hardware combinations, and is not limited to a medium directly connected to a computer system, but may be distributed over a network. It may also exist. Examples of media include magnetic media such as hard disks, floppy disks, and magnetic tape; optical media such as CD-ROMs and DVDs; magneto-optical media such as floptical disks; It may also include ROM, RAM, flash memory, etc., and may be configured to record program instructions. Further, other examples of the medium include an application store that distributes applications, a site that supplies or distributes various other software, and a recording medium or storage medium managed by a server.

As mentioned above, although the embodiments have been described based on limited embodiments and drawings, those skilled in the art will be able to make various modifications and variations based on the above description. For example, the techniques described may be performed in a different order than in the manner described, and/or components of the systems, structures, devices, circuits, etc. described may be performed in a different form than in the manner described. Even when combined or combined, opposed or replaced by other components or equivalents, suitable results can be achieved.

Therefore, even if the embodiments are different, if they are equivalent to the scope of the claims, they fall within the scope of the appended claims.

Please note the following additional notes.
(Additional Note 1) A speaker diarization method executed by a computer system, comprising:
The computer system includes at least one processor configured to execute computer-readable instructions contained in memory;
The speaker diarization method includes:
receiving, by the at least one processor, audio files recorded by each electronic device from a plurality of electronic devices;
estimating a number of candidate clusters based on an embedding matrix calculated for the audio file of each electronic device by the at least one processor;
determining a final number of clusters by the at least one processor using the number of candidate clusters of each electronic device; and performing speaker diarization clustering by the at least one processor using the final number of clusters. A speaker diarization method comprising the steps of:
(Additional Note 2) The receiving step includes:
The speaker according to appendix 1, comprising: performing endpoint detection (EPD) on the audio file of each of the electronic devices; and integrating EPD results of each of the electronic devices to generate an EPD union. Diarization method.
(Additional Note 3) The step of estimating is
calculating a similarity matrix by performing embedding extraction from the EPD results of the audio files of each of the electronic devices; and calculating the number of candidate clusters and the similarity matrix using the similarity matrix of each of the electronic devices. The speaker diarization method according to appendix 1, comprising the step of calculating a confidence value.
(Additional Note 4) The step of calculating the number of candidate clusters and the reliability value of the similarity matrix includes:
extracting eigenvalues by performing eigenvalue decomposition on the similarity matrix; and after arranging the extracted eigenvalues, the number of candidate clusters and the reliability of the similarity matrix are determined based on the difference between adjacent eigenvalues. The speaker diarization method according to appendix 3, comprising the step of calculating a value.
(Additional Note 5) The step of calculating the number of candidate clusters and the reliability value of the similarity matrix includes:
performing eigenvalue decomposition on the similarity matrix to extract eigenvalues;
After arranging the extracted eigenvalues, determining the number of eigenvalues selected based on the difference between adjacent eigenvalues as the number of candidate clusters; and eigenvalues remaining unselected in the process of determining the number of candidate clusters. The speaker diarization method according to appendix 3, comprising the step of calculating the confidence value using .
(Additional Note 6) The step of calculating the reliability value using the remaining eigenvalues includes:
The speaker diarization method according to appendix 5, characterized in that the largest eigenvalue among the remaining eigenvalues is determined as the reliability value of the similarity matrix.
(Additional Note 7) The step of calculating the reliability value using the remaining eigenvalues includes:
The speaker diarization method according to appendix 5, characterized in that an average value obtained by calculating the average of the remaining eigenvalues is determined as a reliability value of the similarity matrix.
(Additional note 8) The step of estimating is
The speaker diarization method according to claim 3, further comprising: applying a weighted sum to the similarity matrix based on weights learned for the EPD results of the audio file.
(Additional note 9) The step of determining the
The speaker diarization method according to appendix 3, characterized in that the number of candidate clusters estimated by the similarity matrix with the largest reliability value is determined as the final number of clusters.
(Additional Note 10) The step of performing is
calculating a similarity matrix by performing embedding extraction from the EPD results of the audio files of each of the electronic devices; and averaging the similarity matrices of each of the electronic devices, and calculating the similarity matrix based on the average similarity matrix and the final number of clusters. The speaker diarization method according to appendix 1, further comprising the step of performing the speaker diarization clustering using the following steps.
(Appendix 11) A computer program that causes the computer system to execute the speaker diarization method according to any one of Appendices 1 to 10.
(Additional Note 12) A non-transitory computer-readable recording medium on which a program for causing a computer to execute the speaker diarization method according to any one of Appendices 1 to 10 is recorded.
(Additional note 13) A computer system,
at least one processor configured to execute computer-readable instructions contained in the memory;
The at least one processor includes:
a process of receiving audio files recorded by each electronic device from multiple electronic devices;
estimating the number of candidate clusters based on the embedding matrix calculated for the audio file of each electronic device;
A computer system that processes: determining a final number of clusters using the number of candidate clusters of each electronic device; and performing speaker diarization clustering using the final number of clusters.
(Additional Note 14) The receiving process includes:
14. The computer system according to appendix 13, comprising: executing EPD on the audio files of each of the electronic devices; and generating an EPD union by integrating EPD results of each of the electronic devices.
(Additional note 15) The estimation process is
calculating a similarity matrix by performing embedding extraction from the EPD results of the audio files of each of the electronic devices; and calculating the number of candidate clusters and the similarity matrix using the similarity matrix of each of the electronic devices. 14. The computer system according to claim 13, comprising: calculating a reliability value.
(Additional Note 16) The process of calculating the number of candidate clusters and the reliability value of the similarity matrix is as follows:
Extracting eigenvalues by performing eigenvalue decomposition on the similarity matrix;
After arranging the extracted eigenvalues, the number of eigenvalues selected based on the difference between adjacent eigenvalues is determined as the number of candidate clusters; and eigenvalues remaining unselected in the process of determining the number of candidate clusters. 16. The computer system according to appendix 15, comprising: calculating the reliability value using .
(Additional Note 17) The process of calculating the reliability value using the remaining eigenvalues is as follows:
17. The computer system according to appendix 16, wherein the largest eigenvalue among the remaining eigenvalues is determined as the reliability value of the similarity matrix.
(Additional note 18) The estimation process is
16. The computer system of claim 15, further comprising: applying a weighted sum to the similarity matrix based on weights learned for the EPD result of the audio file.
(Additional note 19) The process of determining the
16. The computer system according to appendix 15, wherein the number of candidate clusters estimated using the similarity matrix with the largest reliability value is determined as the final number of clusters.
(Additional Note 20) The process to be executed is:
calculating a similarity matrix by performing embedding extraction from the EPD results of the audio files of each of the electronic devices; and averaging the similarity matrices of the respective electronic devices and calculating the similarity matrix based on the average similarity matrix and the final number of clusters. 14. The computer system according to appendix 13, comprising: performing the speaker diarization clustering using the method.

220: Processor 310: Audio integration unit 320: Cluster determination unit 330: Clustering execution unit

Claims (14)

A speaker diarization method performed by a computer system, the method comprising:
The computer system includes at least one processor configured to execute computer-readable instructions contained in memory;
The speaker diarization method includes:
receiving, by the at least one processor, individual audio files recorded by each electronic device from a plurality of electronic devices;
performing endpoint detection (EPD) on the individual audio files of each electronic device;
integrating the individual EPD results of each electronic device to generate an EPD union;
estimating the number of candidate clusters based on the individual embedding matrices calculated for the individual audio files of each electronic device by the at least one processor, the step of: performing embedding extraction using the individual EPD results; calculating an individual embedding matrix for each electronic device; and calculating a reliability value of the number of candidate clusters and the individual embedding matrix using the individual embedding matrix of each electronic device. a step of estimating;
determining, by the at least one processor, a final number of clusters using the number of candidate clusters of each electronic device based on the confidence value; and, by the at least one processor, using the EPD union. A speaker diarization method comprising: performing speaker diarization clustering based on an average similarity matrix obtained by averaging individual similarity matrices of each electronic device calculated by embedding extraction and the final number of clusters. . Calculating the number of candidate clusters and the confidence value of the individual embedding matrix comprises:
performing eigenvalue decomposition on the individual embedding matrix to extract eigenvalues; and after aligning the extracted eigenvalues, the number of candidate clusters and the reliability of the individual embedding matrix are calculated based on the difference between adjacent eigenvalues; The speaker diarization method according to claim 1, comprising the step of: calculating a degree value. Calculating the number of candidate clusters and the confidence value of the individual embedding matrix comprises:
performing eigenvalue decomposition on the individual embedding matrix to extract eigenvalues;
After arranging the extracted eigenvalues, determining the number of eigenvalues selected based on the difference between adjacent eigenvalues as the number of candidate clusters; and eigenvalues remaining unselected in the process of determining the number of candidate clusters. The speaker diarization method according to claim 1, comprising the step of: calculating the confidence value using . Calculating the reliability value using the remaining eigenvalues includes:
The speaker diarization method according to claim 3, characterized in that the largest eigenvalue among the remaining eigenvalues is determined as the reliability value of the individual embedding matrix. Calculating the reliability value using the remaining eigenvalues includes:
The speaker diarization method according to claim 3, characterized in that an average value obtained by calculating the average of the remaining eigenvalues is determined as the reliability value of the individual embedding matrix. The estimating step includes:
The speaker diarization method according to any one of claims 1 to 5, further comprising: applying a weighted sum to the individual embedding matrix based on weight values learned for the individual EPD results. . The determining step includes:
The speaker diarization method according to any one of claims 1 to 6, characterized in that the number of candidate clusters estimated by the embedding matrix with the largest reliability value is determined as the final number of clusters. . A computer program that causes the computer system to execute the speaker diarization method according to any one of claims 1 to 7. A non-transitory computer-readable recording medium on which a program for causing a computer to execute the speaker diarization method according to any one of claims 1 to 7 is recorded. A computer system,
at least one processor configured to execute computer-readable instructions contained in the memory;
The at least one processor includes:
a process of receiving individual audio files recorded by each electronic device from multiple electronic devices;
performing endpoint detection (EPD) on the individual audio files of each electronic device;
a step of integrating the individual EPD results of each electronic device to generate an EPD union;
a process of estimating the number of candidate clusters based on an individual embedding matrix calculated for the individual audio file of each of the electronic devices, the number of candidate clusters being estimated by performing embedding extraction using the individual EPD results; estimating the number of candidate clusters and the reliability value of the individual embedding matrix using the individual embedding matrix of each electronic device;
determining a final number of clusters using the number of candidate clusters of each electronic device based on the reliability value; and a step of determining a final number of clusters of each electronic device calculated by performing embedding extraction using the EPD union. A computer system for performing speaker diarization clustering based on an average similarity matrix obtained by averaging individual similarity matrices and the final number of clusters. The step of calculating the number of candidate clusters and the reliability value of the individual embedding matrix includes:
Extracting eigenvalues by performing eigenvalue decomposition on the individual embedding matrix;
After arranging the extracted eigenvalues, the number of eigenvalues selected based on the difference between adjacent eigenvalues is determined as the number of candidate clusters; and eigenvalues remaining unselected in the process of determining the number of candidate clusters. 11. The computer system of claim 10, further comprising: calculating the reliability value using: The step of calculating the reliability value using the remaining eigenvalues includes:
The computer system according to claim 11, wherein the largest eigenvalue among the remaining eigenvalues is determined as the reliability value of the individual embedding matrix. The estimating process is
A computer system according to any one of claims 10 to 12, further comprising: applying a weighted sum to the individual embedding matrix based on weight values learned for the individual EPD results. The determining process includes:
The computer system according to any one of claims 10 to 13, wherein the number of candidate clusters estimated by the embedding matrix with the largest reliability value is determined as the final number of clusters.