JP7377736B2 - Online speaker sequential discrimination method, online speaker sequential discrimination device, and online speaker sequential discrimination system - Google Patents

Description

The present invention relates to an online speaker sequential discrimination method, an online speaker sequential discrimination device, and an online speaker sequential discrimination system.

In recent years, utterance recordings of broadcasts, voice mails, call center calls, conferences, etc. have been increasing. In this situation, in order to efficiently and effectively automatically create indexes and perform search tasks, it is necessary to not only document the utterances but also be able to extract various types of non-verbal information. It is important to be prepared. Such non-linguistic information includes, for example, metadata. The metadata includes information such as the order of speakers, characteristics (gender, age), and changes in sound sources.

Speech sequential differentiation processing is used to identify sound sources within a spoken document and label them to define metadata. This sequential speech differentiation process finds homogeneous regions within speech segments and consistently labels them with respect to speaker, gender, music, noise, etc. The main part of the voice speaker sequential differentiation process is the sequential speaker differentiation, that is, speaker segmentation and clustering. In other words, this process is a task of finding out "who spoke and when."

Conventionally, most of the speech sequential discrimination means assume that the dialogue (meeting, telephone call, etc.) to be analyzed has already been completed and that the entire speech data can be used for analysis. Such a speech sequential discrimination system is referred to as an "offline speech sequential discrimination system." However, such an offline sequential speech discrimination system cannot analyze speech data in real time and provide speech discrimination results with low latency.

Therefore, in recent years, online speech sequential discrimination means that can analyze speech data in real time and provide speech discrimination results with low delay has been proposed. An example of this online voice sequential discrimination means is, for example, Patent Document 1.
Patent Document 1 states, ``The speaker discrimination system 30 includes a storage unit 42 that stores speaker GMMs 74-78, a voice activity detection unit 30 that segments voice data, and a system that identifies which speaker GMM 74-78 the current segment corresponds to. A novelty determination unit 34 determines whether the current segment belongs to any of the speaker GMMs 74-78, and generates a new speaker GMM, and converts the current segment into the new speaker GMM. and a speaker identification unit 44 that identifies the speaker and labels the current segment with that speaker when the current segment belongs to one of the speaker GMMs 74-78. , a training unit 48 that trains a speaker GMM using the current segment, and a merging unit 46 that merges segment labels according to the sequence of segments output by the voice activity detection unit 30.

Japanese Patent Application Publication No. 2009-109712

The above-mentioned Patent Document 1 discloses the following technology. First, a GMM (Gaussian Mixture Model) is generated and stored for an input audio data segment. Next, as the audio data segment is input, it is compared to the GMM in which it is stored. If the segment belongs to an already stored GMM (that is, the speaker is the same person), the speaker is identified. If the segment does not belong to an already stored GMM (that is, the speaker is not the same person), a new GMM is generated.
This makes it possible to analyze voice data in real time and provide voice discrimination results with low delay.

However, the method described in Patent Document 1 assumes that one voice data segment corresponds to one speaker, and it is not assumed that multiple speakers speak at the same time. Accurate speech sequential discrimination results cannot be provided for speech data segments in which utterances overlap.

Therefore, the present invention uses an utterance information buffer that stores important information (information useful for determining who spoke and when) extracted from previous audio data segments, so that the utterances of multiple different users can be It is an object of the present invention to provide an online speaker sequential discrimination method that can generate accurate speech sequential discrimination results in real time even for overlapping voice data.

In order to solve the above problems, one of the representative online speaker sequential discrimination methods of the present invention includes a first audio data segment corresponding to a user interaction, and a speaker indicating the number of speakers of the user interaction. receiving number data; and based on the first audio data segment and the speaker number data, a particular speaker is at a particular time for a first set of time positions in the first audio data segment. calculating a first set of probability values indicative of a probability of being speaking at a location; a first subset of time locations that is at least a portion of the first set of time locations; and the first probability. selecting at least a portion of the set of values and a first subset of probability values corresponding to the first subset of time positions based on a predetermined selection method, and storing the selected subset in an utterance information buffer; receiving a second audio data segment corresponding to user interaction; and extracting audio features from the first audio data segment corresponding to a subset of the first time locations stored in the speech information buffer; combining the second audio data segment to produce a combined audio data segment; and a second set of time positions in the combined audio data segment based on the combined audio data segment and the speaker count data. calculating a second set of probability values indicative of the probability that a particular speaker is speaking at a particular time location; and a subset of the first probability values and the second set of probability values; and generating a sequential speaker discrimination result that identifies each of the speakers of the user interaction based on the user interaction.

According to the present invention, by using an utterance information buffer that stores important information extracted from previous audio data segments (information useful for determining who spoke and when), the utterances of multiple different users can be overlapped. It is also possible to provide an online speaker sequential discrimination method that can generate accurate speech sequential discrimination results in real time even for voice data.

FIG. 1 is a diagram illustrating a computer system for implementing an embodiment of the invention. FIG. 2 is a diagram showing an example of the configuration of an online speaker sequential discrimination system according to an embodiment of the present invention. FIG. 3 is a diagram illustrating an overview of an online speaker sequential discrimination method according to an embodiment of the present invention. FIG. 4 is a flowchart showing the flow of the online speaker sequential discrimination method. FIG. 5 is a diagram illustrating a first audio data segment according to an embodiment of the invention. FIG. 6 is a diagram illustrating a combined audio data segment according to an embodiment of the invention. FIG. 7 is a diagram illustrating an example of the flow of an online speaker sequential discrimination method according to an embodiment of the present invention. FIG. 8 is a diagram illustrating an example of the flow of the online speaker sequential discrimination method according to the embodiment of the present invention, following FIG. 7 . FIG. 9 is a diagram illustrating an example of a method for updating a speech information buffer according to an embodiment of the present invention.

DESCRIPTION OF THE PREFERRED EMBODIMENTS The background on which the present invention is based and the embodiments of the present invention will be described below with reference to the drawings. Note that the present invention is not limited to this embodiment. In addition, in the description of the drawings, the same parts are denoted by the same reference numerals.

(background)
Most conventional sequential speaker discrimination systems perform several key subtasks, including utterance detection, speaker change detection, gender classification, and speaker clustering. Recombining and reseparating clusters is also used in some cases to improve performance.

The purpose of speech detection is to find a region of speech consisting only of speech. The most common technique for performing this task is maximum likelihood classification using acoustic Gaussian mixture models (GMM). The model is typically trained with some labeled data, and in the simplest case has two models: spoken data and non-spoken data. Some systems use several models depending on the gender of the speaker and the type of channel. Another method may be to perform single-pass or multi-pass Viterbi segmentation of the audio stream. For news broadcast data, the typical error rate for speech detection is 2% to 3%.

After speech segments are identified, speaker change detection can be used to find any speaker changes that may occur in each segment. If this speaker change is detected, the segment is further divided into smaller segments, each belonging to one speaker.

There are two main techniques for change detection. The first one uses the Bayesian information criterion (BIC) to evaluate potential changes within the window by determining whether two distributions are better modeled than one. Discover the points. The second is configured to measure the distance between two fixed length windows, most often represented by a single Gaussian function, the Gaussian divergence or the generalized likelihood ratio. In this case, the peak of the distance exceeding a certain threshold is considered to be the point of change.

Gender classification is used to divide the segments into two groups (male and female), thereby reducing the burden of subsequent clustering and giving more information about the speakers. Typically, two GMMs are pre-trained, one for each gender, and maximum likelihood is used as the decision criterion. The reported error rate for gender classification is 1% to 2%.

The final subtask, speaker clustering, is to assign each segment its correct speaker label. This is done by clustering the segments into sets corresponding to speakers. The most widely used strategy is hierarchical agglomerative clustering using BIC termination criteria.

Each of the clusters is represented by a single Gaussian function, and a generalized likelihood ratio (GLR) is commonly used to measure intercluster distances. Variations of this method have been proposed, but these are still based on similar bottom-up clustering techniques.

Embodiments of the invention may use an End to End Neural Diarization Network (EEND). This end-to-end speaker sequential discrimination network is configured to determine the probability that a particular speaker is speaking at a particular time position (who spoke and when) without using speaker embedding. It is a neural network created by

This end-to-end sequential speaker discrimination network can produce good sequential speaker discrimination results even when speech data segments with overlapping utterances of speakers occur. Given a speech data segment x _t as input, the end-to-end speaker sequential discrimination network calculates, for each time location t, the probability y that the speaker spoke for a given number of speakers. For example, suppose that when there are two speakers c1 and c2, this end-to-end speaker sequential discrimination network outputs y _t,c1 =1, y _t,c2 =0 as the speaker sequential discrimination results. . This means that at time position t, one speaker c1 is speaking, and the other speaker c2 is not speaking at time position t. Furthermore, if y _t,c1 = 0 and y _t,c2 = 0 are output as the speaker sequential discrimination results, this means that neither speaker c1 nor c2 is speaking at time position t. means. Furthermore, if y _t,c1 = 1 and y _t,c2 = 1 are output as the speaker sequential discrimination results, this means that both speakers c1 and c2 are speaking at time position t. means.
In this way, the end-to-end speaker discrimination network can generate speaker discrimination results that identify each of the speakers of a user interaction.

However, in order to generate a speaker sequential discrimination result regarding a specific voice data segment, the end-to-end speaker sequential discrimination network described above needs to analyze not only the voice data segment but also the voice data segments before and after the voice data segment. This method is only used for offline speech sequential discrimination systems in which the user interaction has already been completed and the entire speech data is available from the beginning, and is not intended for online (real-time) use.
Therefore, in the present invention, by using an utterance information buffer to store information useful for sequential speaker discrimination determination, an end-to-end sequential speaker discrimination network can be applied to online situations, and good speaker discrimination can be performed in real time. It becomes possible to generate sequential discrimination results.

(Hardware configuration)
Next, with reference to FIG. 1, a computer system 300 for implementing an embodiment of the present disclosure will be described. The mechanisms and apparatus of the various embodiments disclosed herein may be applied to any suitable computing system. The main components of computer system 300 include one or more processors 302 , memory 304 , terminal interface 312 , storage interface 314 , I/O (input/output) device interface 316 , and network interface 318 . These components may be interconnected via memory bus 306, I/O bus 308, bus interface unit 309, and I/O bus interface unit 310.

Computer system 300 may include one or more general purpose programmable central processing units (CPUs) 302A and 302B, collectively referred to as processors 302. In some embodiments, computer system 300 may include multiple processors, and in other embodiments, computer system 300 may be a single CPU system. Each processor 302 executes instructions stored in memory 304 and may include onboard cache.

In some embodiments, memory 304 may include random access semiconductor memory, storage devices, or storage media (either volatile or nonvolatile) for storing data and programs. Memory 304 may store all or a portion of the programs, modules, and data structures that perform the functions described herein. For example, memory 304 may store a sequential speaker differentiation application 350. In some embodiments, speaker sequential differentiation application 350 may include instructions or writing that performs the functions described below on processor 302.

In some embodiments, speaker sequential discrimination application 350 may be implemented on semiconductor devices, chips, logic gates, circuits, circuit cards, and/or other physical hardware instead of or in addition to processor-based systems. It may also be implemented in hardware via a hardware device. In some embodiments, speaker sequential differentiation application 350 may include data other than instructions or descriptions. In some embodiments, cameras, sensors, or other data input devices (not shown) may be provided to communicate directly with bus interface unit 309, processor 302, or other hardware of computer system 300. .

Computer system 300 may include a bus interface unit 309 that provides communication between processor 302 , memory 304 , display system 324 , and I/O bus interface unit 310 . I/O bus interface unit 310 may be coupled to I/O bus 308 for transferring data to and from various I/O units. The I/O bus interface unit 310 connects a plurality of I/O interface units 312, 314, 316, also known as I/O processors (IOPs) or I/O adapters (IOAs), via the I/O bus 308. and 318.

Display system 324 may include a display controller, display memory, or both. A display controller may provide video, audio, or both data to display device 326. Computer system 300 may also include devices, such as one or more sensors, configured to collect data and provide the data to processor 302.

For example, the computer system 300 may include a biometric sensor that collects heart rate data, stress level data, etc., an environmental sensor that collects humidity data, temperature data, pressure data, etc., and a motion sensor that collects acceleration data, exercise data, etc. May include. Other types of sensors can also be used. Display system 324 may be connected to a display device 326, such as a standalone display screen, a television, a tablet, or a handheld device.

The I/O interface unit has the ability to communicate with various storage or I/O devices. For example, the terminal interface unit 312 may include a user output device such as a video display device, a speaker television, or a user input device such as a keyboard, mouse, keypad, touchpad, trackball, buttons, light pen, or other pointing device. It is possible to attach user I/O devices 320 such as: Using the user interface, a user operates a user input device to input input data and instructions to user I/O device 320 and computer system 300, and to receive output data from computer system 300. Good too. The user interface may be displayed on a display device, played through a speaker, or printed through a printer, for example, via the user I/O device 320.

Storage interface 314 may include one or more disk drives or direct access storage devices 322 (typically magnetic disk drive storage devices, but also an array of disk drives or other storage devices configured to appear as a single disk drive). ) can be installed. In some embodiments, storage device 322 may be implemented as any secondary storage device. The contents of memory 304 may be stored in storage device 322 and read from storage device 322 as needed. I/O device interface 316 may provide an interface to other I/O devices such as printers, fax machines, etc. Network interface 318 may provide a communication pathway so that computer system 300 and other devices can communicate with each other. This communication path may be, for example, network 330.

In some embodiments, computer system 300 is a device that receives requests from other computer systems (clients) that do not have a direct user interface, such as a multi-user mainframe computer system, a single-user system, or a server computer. There may be. In other embodiments, computer system 300 may be a desktop computer, portable computer, laptop, tablet computer, pocket computer, telephone, smart phone, or any other suitable electronic device.

Next, an example of the configuration of the online speaker sequential discrimination system according to the embodiment of the present invention will be described with reference to FIG.

FIG. 2 is a diagram illustrating an example of the configuration of an online speaker sequential discrimination system 360 according to an embodiment of the present invention. As shown in FIG. 2, the online speaker sequential discrimination system 360 mainly includes a client terminal 375, a communication network 370, an audio data acquisition device 365, and a speaker sequential discrimination device 380. The client terminal 375, the audio data acquisition device 365, and the speaker sequential discrimination device 380 are connected to each other via the communication network 370.
Communication network 370 may include, for example, a local area network (LAN), wide area network (WAN), satellite network, cable network, WiFi network, or any combination thereof. Further, the connection between the client terminal 375, the audio data acquisition device 365, and the speaker sequential discrimination device may be wired or wireless.

The audio data acquisition device 365 is a device for acquiring audio data segments to be processed by the speaker sequential discrimination method described later. This audio data acquisition device 365 may be, for example, a computing device equipped with a microphone, such as a smartphone or a personal computer, or a recorder.

The client terminal 375 is a terminal that transmits the voice data segments acquired by the voice data acquisition device 365 described above and the number of speakers data indicating the number of speakers in the user dialogue to the speaker sequential discrimination device via the communication network 370. It is. This client terminal 375 may be a terminal used by an individual, or may be a terminal shared by an organization such as a private company. Further, this client terminal 375 may be any device such as a desktop computer, a notebook computer, a tablet, a smartphone, or the like.

The data storage unit 372 is a storage unit for storing voice data segments and number of speakers data that are transmitted from the client terminal 375 via the communication network 370 and are subject to processing in the sequential speaker discrimination method. This data storage unit may be, for example, a local storage such as an HDD (Hard Disk Drive) or an SSD (Solid State Drive), or may be a cloud-type storage area that is accessible to the speaker sequential discrimination device 380. .

The speaker sequential discrimination device 380 is a device for implementing processing in the speaker sequential discrimination method according to the embodiment of the present invention. The sequential speaker discrimination device 380 processes the audio data segments and the number of speakers data stored in the data storage unit 372 to generate a sequential speaker discrimination result that identifies each of the speakers of the user dialogue. This is the device.

Further, as shown in FIG. 2, the speaker sequential discrimination device 380 includes a data input section 382, a speaker probability determination section 384, an utterance information buffer, and a data input section 382, a speaker probability determination section 384, an utterance information buffer, etc., in order to implement the speaker sequential discrimination method according to the embodiment of the present invention. It includes a management section 386, an audio data combination section 388, and a speaker sequential discrimination section 390.

The data input unit 382 inputs the audio data segments (for example, the first audio data segment, the second audio data segment, etc.) input from the client terminal 375 (or the audio data acquisition device 365) and the number of speakers in the user dialogue. This is a functional unit that receives data on the number of speakers shown.

The speaker probability determination unit 384 determines whether a specific speaker is at a specific time position with respect to the time position (first set of time positions) in the audio data segment based on the input audio data segment and the number of speakers data. This is a functional unit that calculates the probability of being uttered (for example, a first set of probability values, a second set of probability values). This speaker probability determination unit may be, for example, an end-to-end neural diarization network (EEND) using the Self Attention method.

The utterance information buffer management unit 386 stores information (a subset of first time positions and a subset of first probability values corresponding to the subset of first time positions) to be stored in the utterance information buffer according to the embodiment of the present invention. This is a functional unit that determines and stores it in the speech information buffer. Here, the information to be stored in the speech information buffer may be determined based on a predetermined selection method.
Note that details of these selection methods will be described later.

The audio data combining unit 388 is a functional unit that generates a combined audio data segment by combining the audio features extracted from the first audio data segment and the second audio data segment. Here, the audio features extracted from the first audio data segment may be extracted based on the temporal position stored in the speech information buffer.

The speaker sequential discriminator distinguishes each of the speakers of the user interaction based on the probability values calculated by the speaker probability determiner 384 (e.g., a subset of the first probability values and a set of second probability values). This is a functional unit that generates successive speaker discrimination results.

Note that each functional unit included in the online speaker sequential discrimination system 360 may be a software module that constitutes the speaker sequential discrimination application 350 shown in FIG. 1, or may be an independent dedicated hardware device. good. Additionally, the functional units described above may be implemented in the same computing environment or in distributed computing environments. For example, the speaker probability determination unit 384 may be installed in a remote server or client terminal 375, and the other functional units may be installed in the speaker sequential discrimination device 380.

Furthermore, embodiments of the present invention are not limited to the configuration of the online speaker sequential discrimination system 360 described with reference to FIG. For example, a configuration in which the audio data acquisition device 365 directly transmits audio data segments to the speaker sequential discrimination device 380 without going through the client terminal 375, a configuration in which the audio data acquisition device 365 and the client terminal 375 are integrated, etc. Suitably modified configurations are also possible.

Next, an overview of the online speaker sequential discrimination method according to the embodiment of the present invention will be described with reference to FIG.

FIG. 3 is a diagram illustrating an overview of an online speaker sequential discrimination method according to an embodiment of the present invention. As shown in FIG. 3, a plurality of speakers 391 and 392 are having a dialogue, and this dialogue includes utterances 391a and 392b of the respective speakers 391 and 392. Generally, an interaction according to embodiments of the present invention includes at least one utterance uttered by one or more speakers, such as a telephone call, a conference room discussion, or a broadcast. However, in FIG. 3, a dialogue including two speakers 391 and 392 will be described as an example.
As mentioned above, the online speaker sequential discrimination method according to embodiments of the present invention performs well even for overlapping audio data segments (i.e., data containing audio from each speaker 391, 392 speaking at the same time and with overlapping voices). Since speaker distinctions can be generated sequentially, the utterances 391a and 392b of the speakers 391 and 392 may overlap in this dialogue.

The utterances 391a, 392b of the speakers making up the dialogue are recorded (for example, by the audio data acquisition device 365 shown in FIG. The information is transmitted to the serial discrimination device 380. This speaker sequential discrimination device 380 includes the data input section 382, speaker probability determination section 384, utterance information buffer management section 386, audio data combination section 388, and speaker sequential discrimination section 390 described with reference to FIG. By processing the input audio data segments, it is possible to generate sequential speaker discrimination results that identify each of the speakers of the dialogue with low latency.

Next, with reference to FIG. 4, the flow of the online speaker sequential discrimination method according to the embodiment of the present invention will be described.

FIG. 4 is a flowchart illustrating a method 400 for sequentially distinguishing online speakers according to an embodiment of the present invention. The online speaker sequential discrimination method 400 shown in FIG. This is a method that can generate accurate speech sequential discrimination results in real time even when voice data segments in which user utterances overlap occur in a dialogue. The method may be performed by the discrimination device 380.

First, in step S410, a first audio data segment corresponding to a user interaction and speaker count data indicating the number of speakers in the user interaction are received. An audio data segment here refers to a portion of a user's interaction that is recorded over a predetermined period of time. For example, this first audio data segment may be a 1 second (or 10 milliseconds, 2 seconds, 10 seconds, 1 minute, etc.) recording of an interaction occurring in real time. Moreover, the number of speakers data here is information indicating the number of speakers participating in the user dialogue (hereinafter referred to as "number of speakers").

The first audio data segment is recorded, for example, by the audio data acquisition device 365 described with reference to FIG. May be entered. Further, the number of speakers data may be attached as metadata to the first audio data segment when the first audio data segment is recorded, for example, and may be created by preprocessing performed by the client terminal 375. It may also be transmitted to the speaker sequential discrimination device 380.

Next, in step S420, based on the first audio data segment and the number of speakers input in step S410, a specific speaker is identified for the first set of time positions in the first audio data segment. A first set of probability values is calculated indicating the probability that the utterance is in progress at a particular time location.
The set of time positions here refers to the subsegment at a specific time (time) when an audio data segment is divided into subsegments of arbitrary time units (1 millisecond, 10 milliseconds, 1 second, etc.) It's a label. For example, if you divide a 1 second audio data segment into 100 ms subsegments, you will get 10 100 ms subsegments, each at a different time position (1, 2, 3...10). corresponds to

Next, for each sub-segment corresponding to a time position making up the first audio data segment, a first probability value indicating the probability that a particular speaker is speaking (i.e., speaking or not) is determined. The set is calculated. The calculation of this probability may be performed, for example, by the EEND network described above based on the input first audio data segment and the number of speakers data. The EEND network used here may be trained.
The first set of probability values is an arbitrary number as many as the number of speakers specified in the number of speakers data mentioned above for each sub-segment corresponding to each time position constituting the first audio data segment. For a speaker, it represents the probability that the speaker is speaking. This probability may be expressed as a value between 0 and 1, for example.
FIG. 5 is a diagram illustrating a first audio data segment 500 according to an embodiment of the invention. As shown in FIG. 5, the first audio data segment 500 is divided into ten sub-segments, each sub-segment being designated to a different time position constituting a first set of time positions 510. There is. For each time position of the first set of time positions 510, a first set of probability values 530 has been calculated indicating the probability that the two speakers (c1, c2) 520 have spoken. In addition, as shown in FIG. 5, for each of the 10 time positions included in the first set of time positions 510, the probability that each speaker (c1, c2) 520 spoke is recorded as a value between 0 and 1. has been done.

Next, in step S430, a first subset of time positions is at least part of the first set of time positions, and a first subset of time positions is at least part of the first set of probability values. The first subset of probability values corresponding to is selected based on a predetermined selection method and stored in the utterance information buffer. The utterance information buffer here is a storage area that temporarily stores information included in previous (past) audio data segments and useful for sequential speaker discrimination processing. More specifically, the information stored in the utterance information buffer is information about the time position regarding the audio data segment (eg, the first audio data segment) and information about the probability calculated for the time position.
However, since the size of the speech information buffer is limited, in order to reduce the amount of information to be stored in the buffer, rather than storing all the time position and probability information regarding the audio data segment, it is necessary to some information (i.e., a first subset of time locations that is at least part of a first set of time locations; and a subset of first time locations that is at least part of a first set of probability values; It is desirable to store only a subset of the first probability values corresponding to . Thereby, it is possible to generate good successive speaker discrimination results while suppressing the size of the utterance information buffer.

The predetermined selection method here refers to a part of the information to be stored in the utterance information buffer (for example, useful for the sequential speaker discrimination process by the EEND network described above) from among the time position and probability information about the audio data segment. This is a method for selecting specific information. Although the selection method here is not particularly limited, some examples of the selection method will be described below.

ここでは、ｙ_｛ｔ、ｃ｝は、任意の話者ｃが特定の時間位置ｔに話した確率を表し、ｙ_｛ｔ、ｃ１｝は、異なる話者ｃ２が同じ時間位置ｔに話した確率を表す。 One selection method for selecting information to be stored in the utterance information buffer includes selecting from a first set of probability values probability values (a subset of the first probability values) that satisfy a predetermined probability criterion; Time positions (a subset of the first probability values) corresponding to probability values that satisfy a predetermined probability criterion are selected as information to be stored in the utterance information buffer. This predetermined criterion may be, for example, "a probability value of 0.8 or more" or "the highest three probability values".
In one embodiment, for each of the first set of time positions, the absolute value of the probability difference between the speakers at that time position is calculated, and the n times having the probability value with the highest absolute value are A configuration is also possible in which the position and the probability value corresponding to the time position are stored in the speech information buffer.
Note that the absolute value a of the difference in probability between speakers can be obtained from Equation 1 below.

Here, y_{t, c} represents the probability that any speaker c spoke at a particular time position t, and y_{t, c1} represents the probability that a different speaker c2 spoke at the same time position t. represents.

Another selection method for selecting information to be stored in the utterance information buffer is to select probability values that satisfy a predetermined Attention Weight criterion (a subset of the first probability values) from a first set of probability values, and Time positions (subset of first probability values) corresponding to probability values that satisfy the Weight criterion are selected as information to be stored in the utterance information buffer. Attention Weight here quantitatively represents the influence that a specific input has on the final output in a neural network.
For each of the first set of probability values, the Attention Weight given in the neural network is calculated, and a probability value that satisfies a predetermined Attention Weight criterion (upper 10%, upper 20%, etc.) and corresponds to the corresponding probability value are calculated. By selecting the time position to be stored as information to be stored in the utterance information buffer and storing it in the utterance information buffer, it is possible to more efficiently generate good successive speaker discrimination results.
Note that the method for calculating Attention Weight here is not particularly limited, and any appropriate calculation method may be used depending on the structure of the neural network.

In another selection method for selecting information to be stored in the utterance information buffer, a random time position and a probability value corresponding to the time position are selected from the first set of time positions and stored in the utterance information buffer. may be done. For example, it is possible to determine which time position (and the probability value corresponding to the time position) to enter into the speech information buffer from among the first set of time positions based on pseudo-random numbers generated using an arbitrary method. good.

また、この場合、発話情報バッファーは、格納される情報に応じて、異なる種類に分けられてもよい。例えば、発話情報バッファーは、それぞれの話者の情報を格納する話者バッファー、重複している音声データセグメントに関する情報を格納するための重複バッファー、又は話者が話していない時間位置に該当する沈黙バッファー等を含んでもよい。 Note that the illustrated speech information buffer according to the embodiment of the present invention may be divided into a plurality of buffers or may be a single buffer. For example, in one embodiment of the invention, the speech information buffer may be divided into multiple buffers depending on the number of speakers n. In this case, the number of buffers is obtained from Equation 2 below.

Further, in this case, the speech information buffer may be divided into different types depending on the information stored. For example, the utterance information buffer may be a speaker buffer for storing information for each speaker, a redundant buffer for storing information about overlapping audio data segments, or a silence corresponding to a time position when a speaker is not speaking. It may also contain a buffer and the like.

Next, in step S440, a second audio data segment corresponding to user interaction is received. Like the first audio data segment described above, the second audio data segment here is recorded by the audio data acquisition device 365 described with reference to FIG. This is audio data in user dialogue that is sequentially transmitted to the discrimination device 380 and input to the data input section 382. The second audio data segment may also be a subsequent audio data segment following the first audio data segment input in step S410, for example, in a user interaction.

Next, in step S450, audio features corresponding to the first subset of temporal positions stored in the speech information buffer are extracted from the first audio data segment and combined with the second audio data segment. A combined audio data segment is generated. This combined audio data segment includes the audio information corresponding to the first subset of time positions stored in the speech information buffer (i.e., the time positions useful for the sequential speaker differentiation process) and the audio information corresponding to the first subset of time positions stored in the utterance information buffer and the 2 audio data segments are combined into one audio data file. Further, the audio feature here may be, for example, a Mel-frequency cepstral coefficient (MFCC), but is not particularly limited.

Mel frequency cepstral coefficients are a representation of the power spectrum of a sound based on a linear cosine transformation of the logarithmic power spectrum on a nonlinear mel scale of frequency.
Note that the extraction of the audio feature and the combination of the audio feature and the second audio data segment may be performed by any existing means.

Next, in step S460, based on the combined audio data segment generated in step S450 and the number of speakers data input in step S410, a specific talk is determined for a second set of time positions in the combined audio data segment. A second set of probability values is calculated indicating the probability that the person is speaking at a particular time location. The calculation of this probability, similar to the first set of probability values described above, may be performed by the EEND network described above.
FIG. 6 is a diagram illustrating a combined audio data segment 600 according to an embodiment of the invention. As shown in FIG. 6, the combined audio data segment 600 is divided into 15 sub-segments, each sub-segment designated to a different time location forming a second set of time locations 610. For each time position in the second set of time positions 610, a second set of probability values 630 has been calculated indicating the probability that the two speakers (c3, c4) 620 have spoken. Also, as shown in FIG. 6, for each of the 15 time positions included in the second set of time positions 610, the probability that each speaker (c3, c4) 620 spoke is recorded as a value between 0 and 1. has been done.

The second set of probability values 630 generated in step S460 includes the audio features extracted from the first audio data segment and the second audio based on the subset of first temporal locations stored in the utterance information buffer. A first group of probability values 622, which are probability values corresponding to the audio features of this first audio data segment, and a second audio data segment. and a second group of probability values 624, which are probability values corresponding to the segments.
Note that both the subset of first probability values stored in the utterance information buffer and the group of first probability values 622 calculated in step S460 are calculated by the same EEND network based on the same speech. It should be noted that the values are substantially similar (high similarity) because of the

Next, in step S470, the user selects a A sequential speaker discrimination result is generated that identifies each of the speakers of the dialogue. This sequential speaker differentiation result indicates which speaker spoke for each of the second set of time locations in the combined audio data segment. Furthermore, by repeating the contents of S450 to S470 described above each time a new voice data segment is received, it is possible to sequentially generate speaker discrimination results for all voice data segments that constitute a user dialogue.
Note that the details of the process for generating the successive speaker discrimination results will be described later.

By the online speaker sequential discrimination method according to the embodiment of the present invention described above, the important information extracted from the previous audio data segment (the time position selected by the above-mentioned predetermined selection method and the information corresponding to the time position) By using an utterance information buffer that stores utterances (probability values), it is possible to generate accurate speech sequential discrimination results in real time, even when speech data segments that overlap the utterances of multiple different users occur in a dialogue. can.

Next, an example of the flow of the online speaker sequential discrimination method according to the embodiment of the present invention will be described with reference to FIGS. 7 and 8.

FIG. 7 is a diagram illustrating an example of the flow of an online speaker sequential discrimination method according to an embodiment of the present invention.
As described above, when the first voice data segment 500 is input to the speaker probability determination section, the first voice data is determined based on the first voice data segment 500 and the number of speakers data described above. For a first set of time positions in segment 500, a first set of probability values 530 is calculated that indicates the probability that a particular speaker 520 is speaking at the particular time position.

After the first set of probability values 530 is calculated, the first set of probability values 530 and the first time position are selected using the selection techniques described above (absolute value criterion, attention weight criterion, random selection, etc.). A first subset of time positions 752 and a first subset of probability values 754 are selected from the set 510 and stored in the utterance information buffer 750 .

and then indicating which speaker spoke for each of the first set of time positions in the first audio data segment 500 based on the first set of probability values calculated by the speaker probability determiner. A first speaker sequential discrimination result 770 is generated.
Note that this first speaker sequential discrimination result 700 is calculated based on the first set of probability values corresponding to the first audio data segment, so here the information stored in the utterance information buffer is Although not used, the information stored in the utterance information buffer is used in generating a second sequential speaker differentiation result for a second audio data segment following the first audio data segment.

FIG. 8 is a diagram illustrating an example of the flow of the online speaker sequential discrimination method according to the embodiment of the present invention, following FIG. 7 .

First, as described above, the audio features extracted from the first audio data segment 500 based on the first time position subset 752 stored in the utterance information buffer 750 shown in FIG. A combined audio data segment 600 is generated by combining the segments. When this combined voice data segment 600 is input to the speaker probability determining section, a second time position in the combined voice data segment 600 is set based on the combined voice data segment 600 and the number of speakers data described above. For 610, a second set of probability values 630 is calculated that indicates the probability that a particular speaker 620 is speaking at a particular time location.

As discussed above, the second set of probability values 630 computed here are extracted from the first audio data segment 500 based on the first subset of time positions 750 stored in the utterance information buffer 750. Since the calculation is based on the combined audio data segment 600 that combines the audio feature and the second audio data segment, the first probability value that is the probability value corresponding to the audio feature of the first audio data segment 500 is A group 622 and a second group of probability values 624 are probability values corresponding to a second segment of audio data.
However, since the second set of probability values 630 here is generated independently of the first set of probability values 530 described above, the explanation will be made with reference to the speaker 620 shown in FIG. 8 and FIG. The correspondence with the speaker 520 who has spoken is unknown. In other words, it is unclear whether speaker c3 of the combined audio data segment 600 corresponds to c1 or c2 of the first audio data segment 500. Similarly, it is unclear whether speaker c4 of combined audio data segment 600 corresponds to speaker c1 or c2 of first audio data segment 500.
This is known as the so-called "permutation problem" in neural networks.

Therefore, in order to generate a sequential speaker classification result that is consistent with the first sequential speaker classification result 770 described above, the speakers of the combined audio data segment 600 are mapped to the speakers of the first audio data segment 500. There is a need. Therefore, in order to solve the permutation problem described above by associating the speaker of the combined audio data segment 600 with the speaker of the first audio data segment 500, the utterance information buffer according to the embodiment of the present invention is used. It will be done. a probability value calculated for a previous audio data segment stored in the utterance information buffer; a probability value calculated for an audio data segment that is a combination of a portion of the previous audio data segment and the new audio data segment; By comparing, it is possible to associate speakers.

More specifically, after the second set of probability values 630 is calculated, the second set of probability values calculated for each speaker c3, c4 of the combined audio data segment 600 and the utterance information buffer 750 are A permutation of the subset of first probability values calculated for each speaker c1, c2 of the first audio data segment 500, stored in (c1×c3, c1×c4, c2×c3) is generated. , c2×c4).
Then, for each generated permutation, a correlation score is calculated that indicates the similarity between the first subset of probability values stored in the utterance information buffer 750 and the first group of probability values 622. The correlation score calculation here may be performed by, for example, an existing correlation coefficient or similarity calculation method, and is not particularly limited.

After correlation scores for all permutations are calculated, speakers of permutations that meet a predetermined correlation score criterion (eg, highest correlation score, etc.) are considered to be the same person. For example, as an example, a first probability value group 622 (see FIG. 8) calculated for speaker c3 is stored in the utterance information buffer 750. For speaker c2, a first group of probability values 622 (see FIG. 8) matching the subset of values (see FIG. 7) and calculated for speaker c4 is stored in the utterance information buffer 750. Since they match the calculated first probability value subset (see FIG. 7), speakers c1 and c3 are considered to be the same person, and speakers c2 and c4 are considered to be the same person.
In this way, by associating the speaker of the combined audio data segment 600 with the speaker of the first audio data segment 500, it is consistent with the first speaker sequential discrimination result 770 described above, and the speaker of the combined audio data segment 600 is For each of the second set of time locations, a second sequential speaker differentiation result 870 may be generated indicating which speaker has spoken.

Further, after the second set of probability values 630 is calculated, the second set of probability values 630 and the second time A second subset of time locations 852 and a second subset of probability values 854 are selected from the set of locations 610 and stored in the utterance information buffer 850 .
Note that by repeating the process described with reference to FIG. 8 every time a new audio data segment is received, it is possible to generate successive speaker discrimination results for all audio data segments that constitute a user interaction. .
Furthermore, the speech information buffer 750 shown in FIG. 7 and the speech information buffer 850 shown in FIG. 8 may be in the same storage area (that is, a common buffer), or may be in different storage areas (that is, independent buffers). There may be. The configuration of the buffer may be determined as appropriate based on, for example, available computational resources.

Next, with reference to FIG. 9, a method for updating the speech information buffer according to the embodiment of the present invention will be described.

FIG. 9 is a diagram illustrating an example of a method for updating a speech information buffer according to an embodiment of the present invention.
As described above, the utterance information buffer according to the embodiment of the present invention is a storage area that temporarily stores information useful for sequential speaker discrimination processing, and has a limited size. Therefore, in order to store new information in the buffer, it is necessary to delete the information already stored in the buffer, and this process is called buffer update.

As one method for updating the buffer, a so-called first in, first out (FIFO) method may be used. More specifically, for example, if there is a request to write new data into the speech information buffer, the data already stored in the speech information buffer will be replaced with the new data in the order of the oldest writing time. will be overwritten. This causes information regarding older audio data segments to be deleted and information regarding newer audio data segments to be stored.

As one method for updating the buffer, a buffer updating method based on attention weight is also possible. As described above, attention weight quantitatively represents the influence that a specific input has on the final output in a neural network. Attention that quantitatively indicates the influence that each sub-segment has on the output of the neural network by calculating the Attention Weight for each sub-segment (that is, input value) specified at each time position in a certain audio data segment. Weight can be obtained.
For example, as shown in FIG. 9, by generating an Attention Weight Matrix 900 consisting of an input value 910 and an output value 920, and performing a predetermined Attention Weight calculation, the influence of each input value on each output value 920 is calculated. Attention Weight 930 representing force is obtained. This Attention Weight 930 is represented by the value of each element of the Attention Weight Matrix 900. The higher the Attention Weight, the more influence the input value of the Attention Weight has on the corresponding output value.

After calculating the Attention Weight for each input value, the data already stored in the utterance information buffer that does not satisfy a predetermined Attention Weight standard (for example, 0.7, etc.) is deleted, and the data that does not meet the predetermined Attention Weight standard (for example, 0.7, etc.) is deleted. New data that satisfies the above conditions may be stored.
In this way, by preferentially storing information having a higher attention weight in the utterance information buffer, it is possible to obtain a better successive speaker discrimination result.

Although the embodiments of the present invention have been described above, the present invention is not limited to the embodiments described above, and various changes can be made without departing from the gist of the present invention.

360 Online speaker sequential discrimination system 365 Voice data acquisition device 370 Communication network 372 Data storage section 375 Client terminal 380 Speaker sequential discrimination device 382 Data input section 384 Speaker probability determination section 386 Speech information buffer management section 388 Speech data combination section 390 Speaker sequential discrimination unit

Claims (9)

An online speaker sequential discrimination method, comprising:
receiving a first audio data segment corresponding to a user interaction and speaker count data indicating the number of speakers in the user interaction;
a probability that a particular speaker is speaking at a particular time position for a first set of time positions in the first audio data segment based on the first audio data segment and the speaker count data; calculating a first set of probability values indicating
a first subset of time locations that is at least part of the first set of time locations; and a first subset of time locations that is at least part of the first set of probability values and that corresponds to the first subset of time locations. 1 based on a predetermined selection method, and storing the subset in an utterance information buffer;
receiving a second audio data segment corresponding to the user interaction;
extracting audio features corresponding to the first subset of time positions stored in the speech information buffer from the first audio data segment and combining them with the second audio data segment to create a combined audio data segment; The process of generating;
a second set of time positions in the combined audio data segment indicating, based on the combined audio data segment and the speaker count data, a probability that a particular speaker is speaking at a particular time position; calculating a set of probability values for
generating a sequential speaker discrimination result that identifies each of the speakers of the user interaction based on the first subset of probability values and the second set of probability values;
An online speaker sequential discrimination method characterized by comprising: The second set of probability values includes a group of first probability values corresponding to the audio features extracted from the first audio data segment and a second probability value corresponding to the second audio data segment. including a group of
The step of generating the speaker sequential discrimination result includes:
generating permutations of the first subset of probability values and the second set of probability values stored in the utterance information buffer; and for each of the permutations, generating the first subset of probability values and the second set of probability values; Calculate the correlation score for the group with a probability value of 1,
generating the speaker sequential discrimination result based on permutations that satisfy a predetermined correlation score criterion;
The online speaker sequential discrimination method according to claim 1, characterized in that: The predetermined selection method is
For each time position in the first set of time positions, calculate the absolute value of the difference in probabilities between speakers, and set the probability values for which the absolute value satisfies a predetermined criterion as a subset of the first probability values. including selecting
The online speaker sequential discrimination method according to claim 1, characterized in that: The predetermined selection method is
calculating an Attention Weight value for each of the time locations in the first set of time locations, and selecting time locations that meet a predetermined Attention Weight criterion as a subset of the first time locations. and
The online speaker sequential discrimination method according to claim 1. The predetermined selection method is
2. The online speaker sequential differentiation method of claim 1, comprising selecting random time locations as a subset of the first time locations from among the set of first time locations. When there is a request to write new data to the utterance information buffer,
The data already stored in the utterance information buffer is overwritten with the new data in the order of the oldest writing time.
The online speaker sequential discrimination method according to claim 1, characterized in that the method comprises: When there is a request to write new data to the utterance information buffer,
The data already stored in the utterance information buffer is overwritten with the new data in order of decreasing Attention Weight.
The online speaker sequential discrimination method according to claim 1, characterized in that the method comprises: An online speaker sequential discrimination device, comprising:
a first data input for receiving a first audio data segment corresponding to a user interaction and number of speakers data indicating the number of speakers in the user interaction;
a probability that a particular speaker is speaking at a particular time position for a first set of time positions in the first audio data segment based on the first audio data segment and the speaker count data; a first speaker probability determination unit that calculates a first set of probability values indicating
a first subset of time locations that is at least part of the first set of time locations; and a first subset of time locations that is at least part of the first set of probability values and that corresponds to the first subset of time locations. 1 based on a predetermined selection method, and storing the selected subset in the utterance information buffer;
a second data input for receiving a second audio data segment corresponding to the user interaction;
extracting audio features corresponding to the first subset of time positions stored in the speech information buffer from the first audio data segment and combining them with the second audio data segment to create a combined audio data segment; an audio data combination unit that generates;
a second set of time positions in the combined audio data segment indicating, based on the combined audio data segment and the speaker count data, a probability that a particular speaker is speaking at a particular time position; a second speaker probability determination unit that calculates a set of probability values;
a speaker sequential discrimination unit that generates a sequential speaker discrimination result that identifies each of the speakers of the user interaction based on the first subset of probability values and the second set of probability values;
An online speaker sequential discrimination device characterized by comprising: An online speaker sequential discrimination system, comprising:
In the online speaker sequential discrimination system,
A client terminal for acquiring an audio data segment corresponding to a user interaction, data on the number of speakers indicating the number of speakers in the user interaction, and generating sequential speaker discrimination results for identifying each of the speakers in the user interaction. An online speaker sequential discrimination device is connected via a communication network to
The online speaker sequential discrimination device includes:
a first data input unit that receives, from the client terminal, a first audio data segment corresponding to the user interaction and number of speakers data indicating the number of speakers in the user interaction ;
a probability that a particular speaker is speaking at a particular time position for a first set of time positions in the first audio data segment based on the first audio data segment and the speaker count data; a first speaker probability determination unit that calculates a first set of probability values indicating
a first subset of time locations that is at least part of the first set of time locations; and a first subset of time locations that is at least part of the first set of probability values and that corresponds to the first subset of time locations. 1 based on a predetermined selection method, and storing the selected subset in the utterance information buffer;
a second data input for receiving a second audio data segment corresponding to the user interaction;
extracting audio features corresponding to the first subset of time positions stored in the speech information buffer from the first audio data segment and combining them with the second audio data segment to create a combined audio data segment; an audio data combination unit that generates;
a second set of time positions in the combined audio data segment indicating, based on the combined audio data segment and the speaker count data, a probability that a particular speaker is speaking at a particular time position; a second speaker probability determination unit that calculates a set of probability values;
generating and transmitting to the client terminal a speaker sequential discrimination result that identifies each of the speakers of the user interaction based on the first subset of probability values and the second set of probability values; a sequential discrimination section;
An online speaker sequential discrimination system characterized by comprising:

Images