KR20210149336A - Method and system for improving speaker diarization performance based-on multi-device - Google Patents

Abstract

Disclosed are a method and system for improving a speaker segmentation performance based on a multi-device. The method for speaker segmentation comprises: a step of receiving a voice file recorded by each electronic device from a plurality of electronic devices; a step of estimating the number of candidate clusters based on an embedding matrix calculated for the voice file for each electronic device; a step of determining the final number of clusters using the number of candidate clusters for each electronic device; and a step of performing speaker segmentation clustering using the final number of clusters.

Description

METHOD AND SYSTEM FOR IMPROVING SPEAKER DIARIZATION PERFORMANCE BASED-ON MULTI-DEVICE

The description below relates to the speaker diarization technique.

Speaker segmentation is a technology for dividing speech sections for each speaker from a voice file in which the contents uttered by multiple speakers are recorded.

The speaker segmentation technology detects a speaker boundary section from audio data. The speaker segmentation method can be divided into a distance-based method and a model-based method depending on whether prior knowledge about the speaker is used.

For example, in Korea Patent Registration No. 10-1833731 (registration date of February 23, 2018), a speaker recognition model that can distinguish a speaker through the speaker's voice without affecting changes in the environment for recognizing the speaker's voice and the speaker's utterance state A technique for generating a

This speaker division technology is a general technology that divides and automatically records the contents of speeches by different speakers in a situation in which several speakers speak out of a certain order, such as in a meeting, interview, transaction, or trial, and can be used for automatic meeting minutes.

A method and system for improving speaker segmentation performance based on multi-device are provided.

A method and system for performing speaker segmentation in a multi-device environment using a personal device owned by each user are provided.

A method and system for estimating the number of speakers (number of clusters) based on reliability are provided.

A speaker division method executed in a computer system, the computer system comprising at least one processor configured to execute computer readable instructions contained in a memory, the speaker division method comprising: by the at least one processor, a plurality of Receiving a voice file recorded in each electronic device from the electronic device of the; estimating, by the at least one processor, the number of candidate clusters based on the embedding matrix calculated for the voice file for each electronic device; determining, by the at least one processor, a final number of clusters using the number of candidate clusters for each electronic device; and performing, by the at least one processor, speaker division clustering using the final number of clusters.

According to an aspect, the receiving may include: performing end point detection (EPD) on the voice file for each electronic device; and generating an EPD union by integrating the EPD results for each electronic device.

According to another aspect, the estimating may include: calculating an affinity matrix through embedding extraction from the EPD result of the voice file for each electronic device; and calculating the number of candidate clusters and a reliability value for the similarity matrix by using the similarity matrix for each electronic device.

According to another aspect, calculating the number of candidate clusters and the reliability value for the similarity matrix may include: performing eigen decomposition on the similarity matrix to extract eigenvalues; and calculating the number of candidate clusters and a reliability value for the similarity matrix based on a difference between adjacent eigenvalues after aligning the extracted eigenvalues.

According to another aspect, calculating the number of candidate clusters and a reliability value for the similarity matrix may include: performing eigenvalue decomposition on the similarity matrix to extract eigenvalues; determining the number of selected eigenvalues as the number of candidate clusters based on a difference between adjacent eigenvalues after aligning the extracted eigenvalues; and calculating the reliability value using eigenvalues that are not selected in the process of determining the number of candidate clusters.

According to another aspect, calculating the reliability value using the remaining eigenvalues may include determining a largest eigenvalue among the remaining eigenvalues as a reliability value for the similarity matrix.

According to another aspect, the calculating of the reliability value using the remaining eigenvalues may include calculating an average of the remaining eigenvalues and determining the average value as the reliability value for the similarity matrix.

According to another aspect, the estimating may further include applying a weighted sum to the similarity matrix based on a weight learned with respect to the EPD result of the voice file.

According to another aspect, in the determining, the number of candidate clusters estimated from the similarity matrix having the largest reliability value may be determined as the final number of clusters.

According to another aspect, the performing may include: calculating a similarity matrix through embedding extraction from the EPD result of the voice file for each electronic device; and averaging the similarity matrix for each electronic device, and performing the speaker division clustering based on the average similarity matrix and the final number of clusters.

Provided is a computer program stored in a non-transitory computer-readable recording medium for executing the speaker division method in the computer system.

There is provided a non-transitory computer-readable recording medium in which a program for executing the speaker division method on a computer is recorded.

A computer system comprising: at least one processor configured to execute computer readable instructions contained in a memory, the at least one processor comprising: receiving a voice file recorded in each electronic device from a plurality of electronic devices; estimating the number of candidate clusters based on an embedding matrix calculated for the voice file for each electronic device; determining a final number of clusters using the number of candidate clusters for each electronic device; and a computer system that processes a process of performing speaker division clustering using the final number of clusters.

According to embodiments of the present invention, speaker segmentation performance can be improved based on multi-device.

According to embodiments of the present invention, speaker segmentation can be performed in a multi-device environment using a personal device owned by each user without requiring additional equipment.

According to embodiments of the present invention, it is possible to more accurately estimate the number of speakers (number of clusters) based on reliability.

1 is a diagram illustrating an example of a network environment according to an embodiment of the present invention.
2 is a block diagram for explaining an example of an internal configuration of a computer system according to an embodiment of the present invention.
3 is a diagram illustrating an example of components that a processor of a computer system according to an embodiment of the present invention may include.
4 is a flowchart illustrating an example of a speaker division method that can be performed by a computer system according to an embodiment of the present invention.
5 shows an example of an overall process for speaker segmentation according to an embodiment of the present invention.
6 and 7 are exemplary views for explaining a process of merging recognized voice regions in individual voice files according to an embodiment of the present invention.
8 is an exemplary diagram for explaining a process of determining the number of clusters according to an embodiment of the present invention.
9 is an exemplary diagram for explaining a process of performing speaker division clustering according to an embodiment of the present invention.

Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings.

Embodiments of the present invention relate to a speaker segmentation technique for detecting a speaker boundary section from audio data.

In embodiments including those specifically disclosed in this specification, speaker division performance may be improved by performing speaker division based on multi-device, and system construction costs may be reduced by utilizing a personal device owned by each user.

1 is a diagram illustrating an example of a network environment according to an embodiment of the present invention. The network environment of FIG. 1 shows an example including a plurality of electronic devices 110 , 120 , 130 , 140 , a server 150 , and a network 160 . 1 is an example for explaining the invention, and the number of electronic devices or the number of servers is not limited as in FIG. 1 .

The plurality of electronic devices 110 , 120 , 130 , and 140 may be a fixed terminal implemented as a computer system or a mobile terminal. Examples of the plurality of electronic devices 110 , 120 , 130 , 140 include a smart phone, a mobile phone, a navigation device, a computer, a notebook computer, a digital broadcasting terminal, a personal digital assistant (PDA), and a portable multimedia player (PMP). ), tablet PCs, game consoles, wearable devices, Internet of things (IoT) devices, virtual reality (VR) devices, augmented reality (AR) devices, and the like. As an example, in FIG. 1 , the shape of a smartphone is shown as an example of the electronic device 110 , but in the embodiments of the present invention, the electronic device 110 is substantially different from another through the network 160 using a wireless or wired communication method. It may refer to one of various physical computer systems capable of communicating with the electronic devices 120 , 130 , 140 and/or the server 150 .

The communication method is not limited, and a communication method using a communication network (eg, a mobile communication network, a wired Internet, a wireless Internet, a broadcasting network, a satellite network, etc.) that the network 160 may include, as well as a short-distance wireless communication between devices may be included. can For example, the network 160 may include a personal area network (PAN), a local area network (LAN), a campus area network (CAN), a metropolitan area network (MAN), a wide area network (WAN), and a broadband network (BBN). , the Internet, and the like. In addition, the network 160 may include any one or more of a network topology including a bus network, a star network, a ring network, a mesh network, a star-bus network, a tree or a hierarchical network, etc. not limited

The server 150 is a computer device or a plurality of computer devices that communicates with a plurality of electronic devices 110 , 120 , 130 , 140 and the network 160 to provide commands, codes, files, contents, services, etc. can be implemented. For example, the server 150 may be a system that provides a desired service to a plurality of electronic devices 110 , 120 , 130 , and 140 connected through the network 160 . As a more specific example, the server 150 is installed in the plurality of electronic devices 110 , 120 , 130 , 140 through an application as a computer program that is driven, and provides a service (eg, voice recognition-based artificial intelligence) for the application. meeting minutes service, etc.) may be provided to the plurality of electronic devices 110 , 120 , 130 , and 140 .

2 is a block diagram illustrating an example of a computer system according to an embodiment of the present invention. The server 150 described with reference to FIG. 1 may be implemented by the computer system 200 configured as shown in FIG. 2 .

As shown in FIG. 2 , the computer system 200 is a component for executing the speaker division method according to embodiments of the present invention, and includes a memory 210 , a processor 220 , a communication interface 230 , and an input/output interface. (240).

The memory 210 is a computer-readable recording medium and may include a random access memory (RAM), a read only memory (ROM), and a permanent mass storage device such as a disk drive. Here, a non-volatile mass storage device such as a ROM and a disk drive may be included in the computer system 200 as a separate permanent storage device distinct from the memory 210 . Also, an operating system and at least one program code may be stored in the memory 210 . These software components may be loaded into the memory 210 from a computer-readable recording medium separate from the memory 210 . The separate computer-readable recording medium may include a computer-readable recording medium such as a floppy drive, a disk, a tape, a DVD/CD-ROM drive, and a memory card. In another embodiment, the software components may be loaded into the memory 210 through the communication interface 230 instead of a computer-readable recording medium. For example, the software components may be loaded into the memory 210 of the computer system 200 based on a computer program installed by files received over the network 160 .

The processor 220 may be configured to process instructions of a computer program by performing basic arithmetic, logic, and input/output operations. The instructions may be provided to the processor 220 by the memory 210 or the communication interface 230 . For example, the processor 220 may be configured to execute a received instruction according to a program code stored in a recording device such as the memory 210 .

The communication interface 230 may provide a function for the computer system 200 to communicate with other devices via the network 160 . For example, a request, command, data, file, etc. generated by the processor 220 of the computer system 200 according to a program code stored in a recording device such as the memory 210 is transmitted to the network ( 160) to other devices. Conversely, signals, commands, data, files, etc. from other devices may be received by the computer system 200 through the communication interface 230 of the computer system 200 via the network 160 . A signal, command, or data received through the communication interface 230 may be transmitted to the processor 220 or the memory 210 , and the file may be a storage medium (described above) that the computer system 200 may further include. persistent storage).

The communication method is not limited, and short-distance wired/wireless communication between devices as well as a communication method using a communication network (eg, mobile communication network, wired Internet, wireless Internet, broadcasting network) that the network 160 may include may also be included. have. For example, the network 160 may include a personal area network (PAN), a local area network (LAN), a campus area network (CAN), a metropolitan area network (MAN), a wide area network (WAN), and a broadband network (BBN). , the Internet, and the like. In addition, the network 160 may include any one or more of a network topology including a bus network, a star network, a ring network, a mesh network, a star-bus network, a tree or a hierarchical network, etc. not limited

The input/output interface 240 may be a means for an interface with the input/output device 250 . For example, the input device may include a device such as a microphone, keyboard, camera, or mouse, and the output device may include a device such as a display or a speaker. As another example, the input/output interface 240 may be a means for an interface with a device in which functions for input and output are integrated into one, such as a touch screen. The input/output device 250 may be configured as one device with the computer system 200 .

Also, in other embodiments, computer system 200 may include fewer or more components than those of FIG. 2 . However, there is no need to clearly show most of the prior art components. For example, the computer system 200 may be implemented to include at least a portion of the above-described input/output device 250 or may further include other components such as a transceiver, a camera, various sensors, and a database.

Hereinafter, specific embodiments of a method and system for improving multi-device-based speaker segmentation performance will be described.

3 is a block diagram illustrating an example of components that a processor of a server may include according to an embodiment of the present invention, and FIG. 4 is an example of a method that the server may perform according to an embodiment of the present invention is a flowchart showing

The server 150 according to the present embodiment serves as a service platform that provides an artificial intelligence service that can organize the meeting minutes audio files into documents through speaker segmentation.

The server 150 may include a speaker division system implemented as the computer system 200 . The server 150 targets a plurality of electronic devices 110 , 120 , 130 , 140 that are clients, and a dedicated application or server ( 150), it is possible to provide a voice recognition-based AI meeting minutes service through access to the related web/mobile site.

In particular, the server 150 may improve the speaker segmentation performance based on a multi-device using a personal device owned by each user.

The processor 220 of the server 150 is a component for performing the speaker segmentation method according to FIG. 4 , and as shown in FIG. 3 , a voice integrator 310 , a cluster determiner 320 , and a clustering performer 330 may be included.

Depending on the embodiment, components of the processor 220 may be selectively included or excluded from the processor 220 . Also, according to an embodiment, the components of the processor 220 may be separated or merged to express a function of the processor 220 .

The processor 220 and the components of the processor 220 may control the server 150 to perform the steps S410 to S430 included in the speaker division method of FIG. 4 . For example, the processor 220 and components of the processor 220 may be implemented to execute instructions according to the code of the operating system included in the memory 210 and the code of at least one program.

Here, the components of the processor 220 may be expressions of different functions performed by the processor 220 according to an instruction provided by the program code stored in the server 150 . For example, the voice integrator 310 may be used as a functional representation of the processor 220 that controls the server 150 according to the above-described command so that the server 150 integrates the recognized voice region for each device.

The processor 220 may read a necessary command from the memory 210 in which the command related to the control of the server 150 is loaded. In this case, the read command may include a command for controlling the processor 220 to execute steps S410 to S430 to be described later.

Steps S410 to S430 to be described later may be performed in an order different from that shown in FIG. 4 , and some of the steps S410 to S430 may be omitted or additional processes may be further included.

Referring to FIG. 4 , in step S410 , the voice integrator 310 targets a plurality of electronic devices 110 , 120 , 130 , 140 from each electronic device to a voice file recorded in the corresponding device (hereinafter, ' voice regions recognized in the individual voice files by receiving 'individual voice files').

The present embodiments perform speaker division in a multi-device-based environment. For example, a plurality of electronic devices 110 , 120 , 130 , and 140 including personal devices owned by each user participating in a conference may be utilized.

In a dedicated application or web/mobile site related to the server 150, a start button for starting the meeting and an end button for ending the meeting may be included, and as soon as the start button is inputted, the voice recorded from the device is transmitted to the server 150. It may include a function to transmit in real time.

In the present embodiments, a personal device such as a smart phone or tablet possessed by a meeting participant during a meeting may be used instead of additional additional equipment as equipment for recording meeting voice and transmitting it to the server 150 . In particular, in order to improve the speaker segmentation performance, a multi-device composed of individual devices of multiple participants rather than a single device may be used as equipment for recording conference voice and transmitting it to the server 150 .

The voice integrator 310 receives individual voice files from each of the electronic devices 110 , 120 , 130 , and 140 , and then serves to integrate voice sections extracted from each individual voice file. Since the detected voice region may be different for each device, the voice region that is not detected in a specific device is added, and the voice sections for each device are combined so that there is no missing section.

In step S420, the cluster determiner 320 estimates the number of candidate clusters based on the embedding matrix calculated for the individual voice files for each individual voice file (hereinafter, referred to as 'individual embedding matrix'), and then the individual embedding matrix. The final number of clusters can be determined based on the reliability of .

The cluster determiner 320 may independently estimate the number of clusters for each individual voice file and determine the final number of clusters from among the estimated number of clusters.

In particular, the cluster determiner 320 may calculate the reliability in the process of estimating the number of candidate clusters for individual voice files in order to determine the final number of clusters, and calculate the number of candidate clusters estimated in the individual voice file with the highest reliability. It can be determined by the final number of clusters.

A detailed process for determining the number of clusters will be described later.

In step S430, the clustering performing unit 330 performs clustering for speaker segmentation using the embedding matrix calculated for the integrated speech region in step S410 and the final number of clusters determined in step S420. .

The clustering performing unit 330 may obtain an average embedding matrix obtained by averaging individual embedding matrices for voice files for each device, and perform speaker division clustering based on the average embedding matrix and the final number of clusters.

Therefore, in this embodiment, estimation of the number of clusters and speaker division clustering can be performed based on a separate embedding matrix instead of the same embedding matrix. can

5 shows an example of the overall process of speaker division according to an embodiment of the present invention.

Referring to FIG. 5 , the speaker segmentation process may include an independent process independently performed for each individual voice file received from each electronic device 110 , 120 , 130 , 140 and an integration process performed by integrating the individual voice files. .

The voice integrator 310 receives a voice file (individual voice file) recorded at the location of the meeting participant from the electronic devices 110 , 120 , 130 , 140 that are personal devices of multiple participants participating in the meeting during the meeting. do (S51).

The voice integrator 310 independently performs an EPD (end point detection) process for each individual voice file ( S52 ). EPD removes the acoustic characteristics of the frame corresponding to the silent section and measures the energy of each frame to find only the start and end of the speech, which is divided into speech/silence. In other words, the voice integrator 310 performs EPD to find a region with a voice in an individual voice file.

For example, as shown in FIG. 6 , the voice integrator 310 may acquire a voice region 601 detected as an EPD result for each device of a meeting participant. Since each participant participating in the conference has a different location, the detected voice region 601 is also different.

Referring back to FIG. 5 , the voice integrator 310 may generate an EPD union by collecting the EPD results for each device of the conference participant ( S53 ).

As shown in FIG. 7 , the EPD union 702 can be formed by combining the EPD results for each device so that no missing section occurs because the voice region 601 detected by each device of the meeting participant is different.

In other words, the voice integrator 310 may combine the individual EPD results for each individual voice file received from each device of the meeting participant into one EPD result.

Referring back to FIG. 5 , the cluster determiner 320 independently performs an embedding extraction process on the EPD results for each device ( S54 ).

The cluster determiner 320 calculates an individual affinity matrix through embedding extraction from the EPD result for each device, and then calculates the number of clusters using the individual similarity matrix for each device (S55).

In this case, the cluster determiner 320 may calculate the reliability of the individual similarity matrix together with the number of clusters.

Referring to FIG. 8 , the cluster determiner 320 performs eigen decomposition on an individual similarity matrix 803 calculated for each individual voice file of each device to obtain eigenvalues and eigenvectors. ) can be extracted.

In this case, the cluster determiner 320 may determine the number of clusters 804 and the reliability value 805 based on the sorted eigenvalues by aligning the eigenvalues extracted from the individual similarity matrix 803 according to the eigenvalue size. .

The cluster determiner 320 may determine the number of eigenvalues corresponding to valid principal components as the number of clusters 804 based on the difference between adjacent eigenvalues in the sorted eigenvalues. A high eigenvalue means that the individual similarity matrix 803 has a large influence, that is, when an individual similarity matrix 803 is constructed for a voice region within an individual voice file, it means that the proportion of utterance among speakers with utterance is high. do.

In other words, the cluster determiner 320 may select an eigenvalue having a sufficiently large value from among the sorted eigenvalues and determine the number of selected eigenvalues as the number of clusters 804 indicating the number of speakers.

Since eigenvalues that are not selected in the process of determining the number of clusters 804 can be regarded as noise included in the individual similarity matrix 803, the smaller the number of unselected eigenvalues, the more accurate the calculation of the individual similarity matrix 803 is. It can be determined that as a result, the reliability of the individual similarity matrix 803 is high.

The cluster determiner 320 may calculate the reliability value 805 by using the eigenvalues remaining as noise that were not selected in the process of determining the number of clusters 804 among the sorted eigenvalues.

For example, the cluster determiner 320 may use the largest eigenvalue among eigenvalues not selected in the process of determining the number of clusters 804 as the reliability value 805 . For example, when four of the sorted eigenvalues in the order of the highest eigenvalue are determined as the number of valid principal components, that is, the number of clusters 804 , the fifth eigenvalue may be used as the reliability value 805 .

As another example, the cluster determiner 320 may calculate an average of all eigenvalues not selected in the process of determining the number of clusters 804 , and use the average eigenvalue as the reliability value 805 .

Since the speech region 601 detected for each device of the conference participant is different, the individual similarity matrices 803 calculated therefrom may all be different, and the result for the number of clusters 804 indicating the number of speakers may also be different.

In the case where 4 speakers are estimated from the individual voice files of device 1 and 5 speakers are estimated from the individual voice files of device 2, reliability is used to integrate these different results.

It is also possible for the cluster determiner 320 to determine the number of clusters 804 using an average similarity matrix averaging the individual similarity matrices 803 for each device. However, when the average similarity matrix is used, an error of erroneously estimating the number of clusters 804 may occur.

Since the number of clusters 804 is estimated by inferring the number of valid principal components among the eigenvalues calculated from the similarity matrix, performance may deteriorate when the sharpness of the similarity matrix decreases.

Therefore, in determining the number of clusters 804, in some cases, it is possible to obtain a more accurate result by using a sharp result (individual similarity matrix for each device) rather than a smoothing result (average similarity matrix) for an audio file. There is this.

According to an embodiment, a weighted sum of the individual similarity matrices 803 may be applied.

Considering that the reliability may be different for each section of the individual similarity matrix 803, eigenvalue decomposition is not performed by applying the same weight to all sections of the individual similarity matrix 803, but weights of regions not detected by EPD Weights can be learned and applied in the direction of lowering .

For example, by applying a random weight to each section of the individual similarity matrix 803, combining the matrices, calculating an eigenvalue, and learning the weight in a direction to increase reliability.

Referring back to FIG. 5 , the cluster determiner 320 may finally determine the reliability-based number of clusters after collecting the estimated number of clusters and reliability values for each individual voice file of each device ( S56 ).

The cluster determiner 320 may determine the number of clusters calculated from the individual similarity matrix having the highest reliability value among the individual similarity matrices calculated for each individual voice file of each device as the final number of clusters.

The clustering performer 330 may independently calculate an individual similarity matrix for each device through embedding extraction using an EPD union that is a result of collecting the EPD results for each device (S57).

The clustering performer 330 calculates an average similarity matrix by averaging individual similarity matrices independently calculated for each device, and then divides the speakers using the number of clusters determined based on the reliability in step S56 together with the average similarity matrix. Clustering may be performed (S58).

As shown in FIG. 9 , the clustering performer 330 may calculate an average similarity matrix 902 by averaging individual similarity matrices 901 independently calculated for each device.

For example, the clustering performer 330 may calculate the average similarity matrix 902 by performing a matrix arithmetic operation (element-wise) on the individual similarity matrix 901 calculated for each device.

Next, the clustering performer 330 may perform eigenvalue decomposition on the average similarity matrix 902 to perform clustering based on eigenvectors arranged according to eigenvalues.

When m speech sections are extracted from one individual speech file, a matrix including m × m elements is created, where v _i,j representing each element means the distance between the i-th speech section and the j-th speech section do.

In this case, the clustering performing unit 330 may perform speaker division clustering by selecting an eigenvector as many as the number of clusters determined based on reliability.

The whole process for speaker segmentation involves receiving voice files simultaneously recorded with multiple personal devices during a meeting, performing EPD on each device’s voice files, and extracting embeddings in units of segments (speech areas) where EPD is performed to determine the number of clusters. After estimating (number of speakers), clustering is performed based on the estimated number of clusters.

In this embodiment, as a process for improving speaker segmentation performance, an EPD union is generated using an individual voice file for each device, and the number of clusters is estimated using an individual embedding matrix calculated from an individual voice file for each device. Determining the final number of clusters based on , and performing speaker division clustering using the reliability-based number of clusters and an average similarity matrix may be included.

As described above, according to embodiments of the present invention, multi-device-based speaker segmentation can be performed by using personal devices possessed by a plurality of conference participants without requiring additional equipment.

According to embodiments of the present invention, speaker segmentation performance can be improved through a more accurately estimated number of clusters by estimating the number of clusters from a voice file of each device and determining the final number of clusters based on their reliability. .

As described above, in the present embodiments, a new task called multi-device-based speaker splitting can be defined, and by utilizing the personal devices owned by each meeting participant, the system construction cost can be reduced, and a wider space for the meeting can be effectively covered. can do.

Training a model for a new task is the most common approach, and in order to train a new model, it is necessary to consider data collection, actual application environment, and generalization performance. On the other hand, in the present embodiments, the existing speaker division model can be used as it is, and in the case of the speaker division system that is already being serviced, the speaker division performance can be improved by adding only the function of receiving conference voices through multi-devices without re-learning the model. have.

The device described above may be implemented as a hardware component, a software component, and/or a combination of the hardware component and the software component. For example, the apparatus and components described in the embodiments may include a processor, a controller, an arithmetic logic unit (ALU), a digital signal processor, a microcomputer, a field programmable gate array (FPGA), and a programmable logic unit (PLU). It may be implemented using one or more general purpose or special purpose computers, such as a logic unit, microprocessor, or any other device capable of executing and responding to instructions. The processing device may execute an operating system (OS) and one or more software applications executed on the operating system. A processing device may also access, store, manipulate, process, and generate data in response to execution of the software. For convenience of understanding, although one processing device is sometimes described as being used, one of ordinary skill in the art will recognize that the processing device includes a plurality of processing elements and/or a plurality of types of processing elements. It can be seen that can include For example, the processing device may include a plurality of processors or one processor and one controller. Other processing configurations are also possible, such as parallel processors.

Software may comprise a computer program, code, instructions, or a combination of one or more thereof, which configures a processing device to operate as desired or is independently or collectively processed You can command the device. The software and/or data may be embodied in any tangible machine, component, physical device, computer storage medium or device for interpretation by or providing instructions or data to the processing device. have. The software may be distributed over networked computer systems and stored or executed in a distributed manner. Software and data may be stored in one or more computer-readable recording media.

The method according to the embodiment may be implemented in the form of program instructions that can be executed through various computer means and recorded in a computer-readable medium. In this case, the medium may be to continuously store a program executable by a computer, or to temporarily store it for execution or download. In addition, the medium may be various recording means or storage means in the form of a single or several hardware combined, it is not limited to a medium directly connected to any computer system, and may exist distributed over a network. Examples of the medium include a hard disk, a magnetic medium such as a floppy disk and a magnetic tape, an optical recording medium such as CD-ROM and DVD, a magneto-optical medium such as a floppy disk, and those configured to store program instructions, including ROM, RAM, flash memory, and the like. In addition, examples of other media may include recording media or storage media managed by an app store that distributes applications, sites that supply or distribute other various software, and servers.

As described above, although the embodiments have been described with reference to the limited embodiments and drawings, various modifications and variations are possible from the above description by those skilled in the art. For example, the described techniques are performed in an order different from the described method, and/or the described components of the system, structure, apparatus, circuit, etc. are combined or combined in a different form than the described method, or other components Or substituted or substituted by equivalents may achieve an appropriate result.

Therefore, other implementations, other embodiments, and equivalents to the claims are also within the scope of the following claims.

Claims (20)

A method for dividing a speaker executed in a computer system, the method comprising:
the computer system comprising at least one processor configured to execute computer readable instructions contained in a memory;
The speaker division method is
receiving, by the at least one processor, a voice file recorded in each electronic device from a plurality of electronic devices;
estimating, by the at least one processor, the number of candidate clusters based on the embedding matrix calculated for the voice file for each electronic device;
determining, by the at least one processor, a final number of clusters using the number of candidate clusters for each electronic device; and
performing, by the at least one processor, speaker division clustering using the final number of clusters;
A speaker splitting method comprising According to claim 1,
The receiving step is
performing end point detection (EPD) on the voice file for each electronic device; and
Generating an EPD union by integrating the EPD results for each electronic device
A speaker splitting method comprising According to claim 1,
The estimating step is
calculating an affinity matrix through embedding extraction from the EPD result of the voice file for each electronic device; and
calculating the number of candidate clusters and a reliability value for the similarity matrix by using the similarity matrix for each electronic device
A speaker splitting method comprising 4. The method of claim 3,
Calculating the number of candidate clusters and a reliability value for the similarity matrix includes:
extracting eigenvalues by performing eigen decomposition on the similarity matrix; and
After aligning the extracted eigenvalues, calculating a reliability value for the number of candidate clusters and the similarity matrix based on a difference between adjacent eigenvalues
A speaker splitting method comprising 4. The method of claim 3,
Calculating the number of candidate clusters and a reliability value for the similarity matrix includes:
performing eigenvalue decomposition on the similarity matrix to extract eigenvalues;
determining the number of selected eigenvalues as the number of candidate clusters based on a difference between adjacent eigenvalues after aligning the extracted eigenvalues; and
calculating the reliability value using eigenvalues remaining unselected in the process of determining the number of candidate clusters
A speaker splitting method comprising 6. The method of claim 5,
Calculating the reliability value using the remaining eigenvalues includes:
Determining the largest eigenvalue among the remaining eigenvalues as a reliability value for the similarity matrix
A speaker segmentation method, characterized in that 6. The method of claim 5,
Calculating the reliability value using the remaining eigenvalues includes:
calculating the average of the remaining eigenvalues and determining the average value as a reliability value for the similarity matrix
A speaker segmentation method, characterized in that 4. The method of claim 3,
The estimating step is
applying a weighted sum to the similarity matrix based on a weight learned for the EPD result of the voice file
A speaker division method further comprising a. 4. The method of claim 3,
The determining step is
Determining the number of candidate clusters estimated in the similarity matrix having the largest confidence value as the final number of clusters
A speaker segmentation method, characterized in that According to claim 1,
The performing step is,
calculating a similarity matrix through embedding extraction from the EPD result of the voice file for each electronic device; and
performing the speaker division clustering based on the average similarity matrix and the final number of clusters by averaging the similarity matrix for each electronic device
A speaker splitting method comprising A computer program stored in a non-transitory computer-readable recording medium for executing the method of any one of claims 1 to 10 in the computer system. A non-transitory computer-readable recording medium in which a program for executing the method of any one of claims 1 to 10 on a computer is recorded. In a computer system,
at least one processor configured to execute computer readable instructions contained in memory
including,
the at least one processor,
receiving a voice file recorded by each electronic device from a plurality of electronic devices;
estimating the number of candidate clusters based on an embedding matrix calculated for the voice file for each electronic device;
determining a final number of clusters using the number of candidate clusters for each electronic device; and
A process of performing speaker division clustering using the final number of clusters
A computer system that processes them. 14. The method of claim 13,
The receiving process is
performing EPD on the voice file for each electronic device; and
The process of generating an EPD union by integrating the EPD results for each electronic device
A computer system comprising a. 14. The method of claim 13,
The estimation process is
calculating a similarity matrix through embedding extraction from the EPD result of the voice file for each electronic device; and
A process of calculating the number of candidate clusters and a reliability value for the similarity matrix by using the similarity matrix for each electronic device
A speaker splitting method comprising 16. The method of claim 15,
The process of calculating the number of candidate clusters and the reliability value for the similarity matrix includes:
performing eigenvalue decomposition on the similarity matrix to extract eigenvalues;
determining the number of selected eigenvalues as the number of candidate clusters based on a difference between adjacent eigenvalues after aligning the extracted eigenvalues; and
The process of calculating the reliability value using the eigenvalues remaining unselected in the process of determining the number of candidate clusters
A computer system comprising a. 17. The method of claim 16,
The process of calculating the reliability value using the remaining eigenvalues includes:
Determining the largest eigenvalue among the remaining eigenvalues as a reliability value for the similarity matrix
A computer system characterized by a. 16. The method of claim 15,
The estimation process is
The process of applying a weighted sum to the similarity matrix based on the weights learned for the EPD result of the voice file
A speaker division method further comprising a. 16. The method of claim 15,
The decision process is
Determining the number of candidate clusters estimated in the similarity matrix having the largest confidence value as the final number of clusters
A computer system characterized by a. 14. The method of claim 13,
The process to be performed is
calculating a similarity matrix through embedding extraction from the EPD result of the voice file for each electronic device; and
A process of averaging the similarity matrix for each electronic device and performing the speaker division clustering based on the average similarity matrix and the final number of clusters
A computer system comprising a.

Images