JP2020187346A - Speech dialization method and apparatus based on audio visual data - Google Patents

Abstract

To provide a speech dialization method and an apparatus therefor based on audio visual data.SOLUTION: There is provided a speaker dialization method having the steps of: calculating correlation between shapes of mouths of speakers included in video data in which a plurality of speakers are captured and a speech segment from a speaker included in audio data; structuring a speaker model for each speaker based on the calculated correlation; and specifying a speaker producing a speech of sound included in the audio data based on the structured speaker model.SELECTED DRAWING: Figure 1

Description

The following description relates to a technique for separating speakers by using video (video) data and audio data of a plurality of speakers, and more specifically, from the shape of the speaker's mouth from the video data and audio data. It relates to a technique for identifying a speaker (that is, speaker dataation) based on a speaker model constructed by correlation with a speech segment.

In recent years, there has been an increasing demand for recording and retrieving human communications (eg, meetings) while utilizing machine-readable formats. As the availability of large data sets and the accessibility of deep learning frameworks have increased, automatic speech recognition for recording such human communications has evolved significantly. As a result, it is important to add information about "when" and "who" uttered the relevant sentence to the transcript, rather than simply listing the sentence words.

For example, Patent Document 1 (publication date: May 26, 2010) measures the degree of contribution of the speaker identification result of each frame with respect to the method of measuring the reliability of each speaker identification result, and talks about each frame. By measuring the reliability of the speaker identification result based on the speaker identification contribution and using this to judge the authenticity of the speaker, the authenticity of the speaker presented at the time of speaker verification can be accurately determined. There are disclosed techniques that can improve the accuracy of speaker identification in a multi-channel environment.

The information described above is merely to aid understanding and may include content that does not form part of the prior art, or may not include content that the prior art can present to a normal engineer. ..

Korean Publication No. 10-2010-0055168 Gazette

The correlation between the shape of each speaker's mouth included in the video data of multiple speakers and each speech segment included in the audio data is calculated, and the speaker for each speaker is calculated based on the calculated correlation. It provides a speaker dialylation method that builds a model and, based on the built speaker model, identifies the speaker who speaks the voice contained in the audio data.

Communication between speakers is recorded by extracting the voice spoken by the specified speaker from the audio data, converting it into text, and recording the converted text in association with the time information when the corresponding voice was spoken. Provide a way to enable it.

In one aspect, it is a method executed by a computer system that separates speakers by using video (video) data and audio data of a plurality of speakers, and is a method of separating speakers by using the video data of each speaker included in the video data. The stage of calculating the correlation between the shape and each of the speech segments from the speaker included in the audio data, the stage of constructing a speaker model for each of the speakers based on the calculated correlation, and the above. Based on the constructed speaker model, a speaker dialylation method is provided that includes a step of identifying the speaker who speaks the voice included in the audio data among the speakers.

At the stage of constructing the speaker model, for each speaker of the speaker, the top N speech segments having a high correlation with the shape of the mouth of each speaker among the speech segments are selected for each speech. Used to build a speaker model for a person, N may be a natural number.

In constructing the speaker model for each speaker, only the speech segments whose correlation with the mouth shape of each speaker is equal to or greater than a predetermined threshold value may be used among the N speech segments.

The voices overlapped by the remarks of two or more of the speakers included in the audio data are stories in which the overlapped voices are uttered based on the shape of the speaker's mouth when the overlapped voices are spoken. By extracting the pronunciation of the person, generating a mask for the part of the audio data corresponding to the extracted pronunciation, and filtering the overlapping voices, the overlapping voices are separated into the voices of the speakers who uttered the overlapping voices. You can do it.

The shape of each speaker's mouth may be identified by recognizing each speaker's face from the video data and tracking the recognized face.

In the step of identifying the speaker, the face of each speaker is detected from the video data, and the detected face is tracked, so that the shape of the mouth of each speaker and the utterance included in the audio data are included. Including the step of calculating the correlation with the spoken voice and the step of identifying the speaker who spoke the spoken voice by using the calculated correlation and the constructed speaker model. Good.

The step of identifying the speaker who uttered the voice is that the specific speaker is blocked by hiding the face of the specific speaker or the mouth of the specific speaker in the video data. When the correlation between the audio data and the spoken voice cannot be calculated, the correlation between the specific speaker and the spoken voice may be regarded as 0.

The step of identifying the speaker is a step of determining information on the position of the speaker who uttered the spoken voice included in the audio data, and a step of determining the information on the position of the determined speaker and the constructed story. By using the person model, a step of identifying the speaker who spoke the spoken voice may be included.

The information regarding the position of the speaker who spoke the voice may include information regarding the direction of the spoken voice, and the information regarding the direction may include azimuth information related to the spoken voice.

Of the plurality of speakers, the speaker whose face is recognized from the video data but is recognized as not speaking at all may be ignored.

At the stage of constructing the speaker model, among the plurality of speakers, the shape of the face or mouth is not recognized from the video data, or the correlation between the shape of the mouth for constructing the speaker model and the speech segment. For a specific speaker for whom the speaker model is not constructed without any provision, embedding corresponding to the speech segment in which the correlation calculated at the stage of calculating the correlation is less than a predetermined value. ) May be built to build a speaker model for the particular speaker.

The video data and audio data may be data included in a video of the speaker captured in real time.

In the speaker dialylation method, a step of extracting the voice uttered by the specified speaker from the audio data, a step of converting the extracted voice into text, and the step of converting the converted text are extracted. It may further include the step of recording the voice in association with the time information spoken.

The speaker dialylation method may further include a step of filtering noise beyond the human voice range from the audio data by using a bandpass filter.

In another aspect, a computer system that uses video (video) data and audio data of a plurality of speakers to separate speakers, which is a memory and a computer readable connected to the memory and contained in the memory. The at least one processor includes at least one processor configured to execute an instruction, from the shape of each speaker's mouth included in the video data and from the speaker included in the audio data. The correlation with each of the speech segments of is calculated, a speaker model for each of the speakers is constructed based on the calculated correlation, and the speaker is used from the audio data based on the constructed speaker model. Provide a computer system that identifies the speaker who speaks among them.

A story that speaks a speaker model for each speaker constructed based on the correlation between the shape of each speaker's mouth included in the video data and each speech segment from the speaker included in the audio data. By using it to identify a person, even when a specific speaker is hidden in the video data, the utterance by the specific speaker can be accurately specified.

In addition to the speaker model for each speaker constructed, the correlation between each speaker's mouth shape and the spoken voice and / or the corresponding voice is spoken to identify the speaker who speaks the voice. Further use of location information can more accurately identify the speaker who speaks the voice.

By extracting the voice spoken by the identified speaker from the audio data, converting it into text, and recording the converted text in association with the time information in which the voice was spoken, for example, a story such as a conference. Communication between people can be recorded automatically.

It is a figure which showed the pipeline of the speaker dialyization system in one Embodiment briefly. As an example, it is a figure which showed the still image of the video data used for a test. As an example, as the video data used for the test, it is a figure which showed the still image of the public AMI conference data. As an example, as the video data used for the test, it is a figure which showed the still image of the public AMI conference data. It is a figure which showed the speaker dialyization which utters a voice and the method of recording the utterance from the separated (identified) speaker in one embodiment. It is a block diagram for demonstrating an example of the internal structure of the computer system in one Embodiment. As an example, it is a figure which showed the minutes which recorded the meeting between speakers. It is a flowchart which showed the method of constructing a speaker model and the method of separating the speaker who speaks in one Embodiment. As an example, it is a flowchart showing a method of identifying a speaker who speaks. As an example, it is a flowchart showing a method of identifying a speaker who speaks.

Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings.

The embodiments described below relate to techniques for determining "who spoke" (ie, the speaker who spoke the voice) at the actual meeting. The method of the embodiment uses video (eg, surround view video taken with a 360 degree camera) and single or multi-channel audio as inputs, based on which a reliable Speaker dialysis output is generated. To do.

To achieve this, the present disclosure first registers a speaker model using audiovisual relevance, and then uses the registered model and visual information to activate it. We propose a new repetitive approach technique that determines the speaker (ie, the speaker who speaks).

The method of the embodiment exhibits excellent quantitative and qualitative performance with respect to the actual conference dataset. The methods of the embodiment outperformed all other comparable methods when evaluated against public datasets (see test results below). Also, when multi-channel audio can be used, beamforming is used with the video to extract the position and / or direction of the audio.

Hereinafter, the background and outline of the method of the embodiment will be described.

Speaker dialysis, the task of breaking down multiple-speaker audio into single-speaker segments, has been an area of active research for several years. The speaker's voice is treated as a single-modality problem in which only audio is used, but it may also be treated with additional modality such as video. Speaker dialing techniques for both audio and audiovisual can be divided into the following two types.

The first is based on Speaker Modeling (SM), which assumes that each individual has different voice characteristics.

As an example, the speaker model may consist of mixed Gaussian models (GMMs) and i-vectors. In addition, based on the proof that deep learning is effective for speaker modeling, the speaker model by speaker modeling may be constructed through deep learning.

Speaker models in many systems are generally pre-trained for the target speaker and may not be applicable to unknown participants. Other algorithms use general models and clustering to adapt to unknown speakers. In addition, there are many audiovisual domain tasks based on feature clustering.

The second is to use the Sound Source Localization (SSL) method. It achieves better performance than SM-based approach methods, for example, with powerful beamforming methods such as SRP-PHAT. However, the SSL-based method is only effective if the speaker's position is fixed or known. Therefore, SSL is used as part of audiovisual methods, such as when the position of a speaker can be tracked using visual information. Such approaches are highly dependent on the ability to effectively track participants. SSL can be combined with a visual analysis module that measures movement and mouth movement in the embodiments of the present disclosure.

SM and SSL approaches may be combined using independent models for each type of observation, and this information may be fused with a stochastic framework based on the Viterbi algorithm or Bayesian filtering.

The present disclosure presents a system that uses audiovisual data to process speaker movement and occlusions and realizes a reliable speaker dialiation system. For such a system, a state-of-the-art deep audiovisual synchronization network is used to detect each participant's words when mouth or lip movements (ie, mouth shape) are clearly visible. May be done.

Such information may be used to register each participant's speaker model, and based on the registered speaker model, it is possible to determine who will speak even if the participant is hidden. become able to. Rework the unsupervised learning (clustering) problem with a supervised classification problem that estimates the probabilities of voice segments belonging to all participants by generating a speaker model for each participant. Can be configured. Unlike techniques for calculating the likelihood for each type of observation prior to multimodal fusion, the audiovisual synchronization of the present disclosure may be used in the speaker registration process.

As an additional explanation, when multi-channel microphones are highly available, beamforming may be applied to estimate the position of the sound source, and spatial cues from both modality are used to improve the performance of the system. May be done.

Hereinafter, a multimodal speaker dialylation system (audiovisual system) according to an embodiment of the present invention will be described.

The audio processing portion of the audiovisual system may be configured to include well-known audio processing system methods. For example, as a speech enhancement system, a long short-term memory (LSTM: Long Short-Term Memory) -based noise reduction model trained on simulated training data may be used. Also, a pre-trained x-vector model may be used to extract the speaker embedding (speaker model). The x-vector extractor and PLDA parameters may be trained for a dataset with data enhancement (additive noise).

According to embodiments of the present invention, at least three types of information (audio to video correlation, speaker model, audio orientation, etc.) may be used to determine the current speaker from within the video. ..

FIG. 1 is a diagram briefly showing the pipeline of the speaker dialylation system in one embodiment.

In the preprocessing stage, a face portion may be detected from the video data (Face detection), the corresponding face portion may be tracked (Face tracking), and a face portion image for the face portion may be acquired. In addition, by performing face recognition using a profile image, it may be possible to detect who the relevant face portion image is.

In face detection and face tracking, for example, SSD (Single Shot MultiBox Detector) -based CNN face detectors may be used to detect face shapes from all frames of video. Such a detector may track the face based on a variety of poses and lighting conditions. A position-based face tracker may be used to group individual face detections as facial partial images.

Face recognition requires each participant's face image, which allows the face to be identified and tracked in the conference room regardless of their location. It may consist of user input or profile image. Face images of all participants may be represented and recorded with commonly known face recognition features, such as embedding using the VGGFace2 network.

As shown in FIG. 2, the face part is detected from the video in which a plurality of speakers exchange conversations (displayed in a square area), and the corresponding face part video is sequentially displayed as Face track 1, Face track 2, ... Face track n. Name it. Of these, Face track 1 may be compared with one or more pre-registered profile images to identify who is the speaker corresponding to Face track 1 and confirm the identity of the specific speaker.

A method of preprocessing audio data (not shown) may include passing through a bandpass filter covering 200-7000 Hz in order to reduce noise beyond the human voice range contained in the audio data. Also, for example, a voice activity detector may be used to identify whether there is a speech in the audio.

In step 1 (Phase 1) of FIG. 1, a speaker model is registered for audio and facial partial video data by using an audio-to-video correlation (AV correlation).

In stage 2 (Phase 2) of FIG. 1, the registered speaker model, audio, and facial partial video data may be used to confirm who is the speaker of the current utterance. Specifically, 1) speaker verification is performed using audio and a speaker model, 2) the utterance direction is calculated using audio, and 3) AV is performed using audio and facial image. The correlation may be calculated, and the speaker may be finally determined using the results of 1) to 3).

One embodiment of the invention uses a method of performing step 1 on the entire audio and video data to register a speaker model prior to performing step 2, and also based on such embodiments below. However, for a new speaker who was not registered in step 2, it is also possible to realize the embodiment of the method of executing the speaker dialyration while registering the speaker model.

Hereinafter, the audio-to-video correlation (AV correlation) of the audiovisual system of the embodiment will be described. Cross-modal embedding of audio and mouth movements may be used to indicate the respective signals. Strategies for training such joint embeddings are, for example, as described below.

The network may consist of two streams (an audio stream that encodes the MFCC (Mel-Freequency Ceptral Cofficients) input into a 512-dimensional vector and a video stream that encodes the cropped face image into a 512-dimensional vector). The network may be trained by a multi-way matching task of one video clip and N audio clips. The Euclidean distances of audio and video features are calculated, and N distances may be calculated as a result. After passing through the softmax layer, the network may be trained by the cross-entropy error for the reciprocal of such distances, so the similarity of matching pairs is greater than that of non-matching ones.

The cosine distance between the two implants may be used to measure the correspondence between the two inputs. Therefore, if the facial image corresponds to the current speaker (speaking speaker), a small distance between features is expected, otherwise in-sync and long distance are expected. To. Since the video is based on one continuous source, it can be assumed that the AV offset is fixed throughout the session. The embedding distance becomes smooth over time by using a median filter to remove outliers.

The speaker model of the audiovisual system of the embodiment and the method of registering the speaker model will be described below. Since AV correlations can only be used when mouth movements are clearly visible, audiovisual synchronization is not possible in this disclosure due to speaker models for each speaker, such as occlusion. Allows the active speaker (ie, the speaker to speak) to be determined at any time (or even when the availability of such signals is low).

In the embodiment of the present invention, a speaker model (speaker embedding) for the corresponding speaker may be registered by using the audio data in which the speaker of the corresponding audio section is identified by the AV correlation.

In one embodiment, of the speech segments, which are sections of the audio data divided into fixed time intervals (eg, 1.5 seconds or 2 seconds, respectively), a reliable speech to each speaker that is pre-executed throughout the video. You may find the segment and use it to get the speaker model.

In the embodiment, as an example, N = 10 (or 3) may be used, and if there are less than N conviction segments that exceed the AV correlation threshold, only the segments whose correlation exceeds the threshold speak. May be used to create a person model.

In order to extract the speaker model using the segment, a conventional model such as x-vector or ResNet may be used, but the present invention is not limited to this. It is preferable to use a trained deep ResNet-50 model, as deeper models can adequately generalize more difficult datasets than smaller x-vector models.

For example, the speaker model calculates a feature value (feature) based on a 1.5 second window and moves 0.75 seconds at a time (or moves one frame at a time) according to the reference system. It may be extracted. By comparing the speaker model at each time step with the registered speaker model, the likelihood (likelihood) of the speech segment belonging to any speaker may be estimated. Even if there is no visual information in the inference time, this can be a generally more reliable supervised classification problem compared to unsupervised clustering. That is, it becomes a matter of classification or pairing confirmation and can generally produce very strong performance compared to clustering.

In the following, sound source position determination (SSL) used by the audiovisual system of the embodiment will be described. The direction of the sound source, as well as the speaker model, is a useful clue as to who speaks. To determine the orientation of the sound source, for example, the orientation of the audio source of the recording from the 4-channel microphone to the camera is estimated for each audio sample, and the orientation for all video frames is an interval size of 10 degrees ( It may be determined by having a bin size) and generating a histogram of all azimuth θ values for ± 0.5 seconds (period).

The likelihood of audio belonging to any speaker at a given time may correlate with the estimated audio source and the angle of face detection in the video for that speaker.

Hereinafter, the fusion of multimodal in the audiovisual system according to the embodiment will be described.

The three types of information (AV correlation, speaker model, audio direction) may provide confidence scores for each speaker and time step. Such scores may be combined into a single Coverall for all speakers and time steps using a simple weighted fusion, as described below. .. Here, C _sm is the reliability score from the speaker model, C _avc is the score from the AV correspondence, and θ ^* and φ are the estimated DoA (Directional of) of the face angle (position) and audio, respectively. Arrival, the direction in which audio is heard). α and β each indicate a predetermined weighted value, and the value may be adjusted according to the importance of each score. In general, use the value that gives the best performance of the training data.

In the following formula (1), the second and third terms may be set to 0 when the camera cannot see a particular speaker.

_Overall = C _sm + α * C _avc + β * cosine (φ−θ ^* ) ・ ・ ・ (1)

Based on the calculated reliability score ( _Overall ), the speaker who uttered the voice uttered in the corresponding time step may be identified.

In the following, a comparative test of the performance of a general audio system and an audiovisual system of an embodiment will be described. In this disclosure, two independent datasets, the internal dataset of the conference recorded at 360 degrees and the publicly available AMI conference corpus, were evaluated. Each will be described in more detail below.

The internal meeting dataset consists of audiovisual recordings of regular meetings without any special instructions from participants regarding video recording. The conference formed part of a day's discussion in the workspace and was not set up with the task of speaker dialiation in mind. A significant portion of the dataset consists of very short utterances that change the speaker frequently, which is a very difficult condition for speaker dialiation. The video shall be recorded with a GoPro® Fusion camera that has two fisheye lenses and captures 360 degree video of the conference. The video is 25 frames per second and may be combined together with a single surround view video with a resolution of 5228 x 2624. The audio may be recorded by a 4-channel microphone at 48 kHz. A still image of such a dataset is shown in FIG. FIG. 2 shows a still image of the video data used for the test according to one embodiment.

The dataset may include an approximately 3 hour validation set and a 40 minute carefully annotated test set. There are nine speakers in the test video. When the remarks overlap, only the ID of the main (maximum sound) speaker was annotated. Embedded extractors and AV synchronization networks are trained against external datasets, and validation sets may only be used to tune AHC thresholds in the reference system and fusion weights in the system of embodiments.

The AMI corpus consists of 100 hours of video recorded from multiple locations, and the system of the embodiment includes ES and 17 hours of video, respectively, of the 100 hours of video. Evaluated for IS category meetings. The image quality is relatively low, with a video resolution of 288 x 352 pixels. The audio was recorded from an 8-element circular equispaced microphone array with a diameter of 20 cm. However, in most of the tests, only one array microphone may be used. The video is recorded by four cameras that provide a close-up view of each participant in the conference, and unlike the internal dataset described above, the images are not concatenated together. The ES video may be used as a validation set for tuning the threshold. Figures 3a and 3b are video data used for the example test and show a still image of public AMI conference data. FIG. 3a is a still image of the video data showing a close-up view of each of the participants in the conference, and FIG. 3b is a still image of the video data corresponding to the distant view image of the participants of the conference and the whiteboard. ..

For each face part image detected, face embedding was extracted using VGGFace2 and compared to each of the N recorded Face recognition for face recognition, thus these are N stories. It may be classified as one of the persons. At any point in time, a constraint was applied that simultaneous facial footage could not indicate the same speaker.

Table 1 below shows the results of speaker dialiation by a comparative test of the performance of the reference system and the audiovisual system of the embodiment. The low numbers indicate that the recording performance is better. Except for the last four lines, the results of the AMI dataset are shown. WB is a whiteboard; NWB is a whiteboard; Xch + V is X channel audio + video; SM is speaker modeling; AVC is audiovisual compatible; SSL is sound source position search; MS is a missed speech; FA is an error alarm SPKE is the speaker error; DER is: the error rate of the speaker dialiation.

Regarding the evaluation index, DER was used as the performance index. DER is a missed speech (MS, reference speaker but not hypothetical speaker), error alarm (FA, hypothetical speaker but not reference speaker), And speaker error (SPKE, speaker ID is assigned to another speaker) may be decomposed into three components.

The tool used to evaluate the system was developed by NIST to evaluate RT-speaker dialiation and includes a 250 ms permissible margin to compensate for human error in reference annotations.

See Table 1 above as a result for the AMI corpus. The numbers for meetings where the whiteboard is used are provided separately and the results are compared.

Since the same VAD system was used in all tests, the percentage of missed speeches and error alarms is the same for each dataset in different models. Therefore, the speaker error rate (SPKE) is the only indicator affected by the speaker dialiation system.

The speaker model-only system (SM) may use visual information only to retrieve the registration timing of the speaker model and use only audio during use inference. When experimenting with a common audio processing pipeline and embedded extractor, the performance gain comes from turning the clustering problem into a dialification problem. This alone resulted in a relative improvement in speaker error of 48% and 26%, respectively, in the ES and IS sets.

From the results in Table 1, it can be confirmed that the performance is clearly improved by adding the AV correlation (AVC) and the sound source position determination (SSL) at the time of inference. It can be confirmed that the contribution of such modality to the overall relative performance is 20-40% and 19-39%, respectively, depending on the test set. Such results show that under all test conditions, they far exceed the results of the prior art.

Speaker error rates become significantly worse with internal conferencing datasets, which is caused by the difficult characteristics of the dataset (challenge nature) and the large number of speakers. From the results in Table 1, it is confirmed that the reference system is not generalized in such a dataset, but the multimodal system of the embodiment achieves relatively good performance even in such "real data". be able to.

From the results described above, the embodiments of the present disclosure generally incorporate a multimodal system that takes advantage of audiovisual correlation to register a speaker model for speaker dialiation. It can be confirmed that a considerable advantage is achieved compared to the clustering method used.

In addition, the following describes how to handle speakers who are not registered as speaker models. Unregistered speakers may attend a session for a variety of reasons. Such cases include, for example, (1) some people do not speak at all, (2) occlusion does not provide any AV correlation, or (3) some people join the meeting by telephone. , Can be mentioned.

In this regard, we propose two possible solutions. First, in the case of (1), it may be assumed that a person who does not speak at all is ignored. Further, in the cases of (2) and (3), after performing step 2 (Phase 2) in FIG. 1, one of the active models with low reliability is clustered for both AV correlation and speaker recognition. Then, the speakers corresponding to (2) and (3) may be registered (that is, a speaker model may be constructed).

Hereinafter, a more specific structure and a method for realizing the system of the embodiment will be described with reference to FIGS. 4 to 9.

FIG. 4 shows, in one embodiment, a speaker dialylation that speaks a voice and a method of recording utterances from a separated (identified) speaker.

With reference to FIG. 4, a method of separating a speaker who utters a voice by using the audiovisual system (multimodal system) of the above-described embodiment and recording the voice of the separated speaker will be described.

In the example shown in the figure, it is shown that the video (video) data and the audio data for the speakers (speakers A to D) are acquired by the 360-degree camera and the microphone (not shown), respectively. The audiovisual system of the embodiment is a speech segment from each of the mouth shapes (for example, changes in the shape of the mouth or lips) of the speakers A to D included in the video data and the speakers A to D included in the audio data. A speaker model for each of speakers A to D may be constructed (registered) based on the calculation of the correlation with. The audiovisual system of the embodiment may identify the speaker who utters the sound included in the audio data from the speakers A to D based on the constructed speaker model. That is, the audiovisual system may identify the utterances of the speakers A to D from the audio data.

For example, speakers A to D may be members of the conference and the audiovisual system may identify the speaker to speak during the conference.

The audiovisual system may extract the voice spoken by the identified speaker from the audio data and convert the extracted voice into text, and the converted text and the time information in which the extracted voice is spoken are input. It may be associated and recorded. For example, the audiovisual system may record the text and time information in association with each other as the minutes of a meeting in which speakers A to D participate.

FIG. 6 is a diagram showing minutes recording a meeting between speakers according to an example.

As shown in the figure, for each of the speakers A to D specified by the audiovisual system, the voice spoken by each speaker is converted into text, and the corresponding voice is recorded as minutes 600 together with the time information spoken. It may be recorded. Although not shown in the figure, Minute 600 may further include an image associated with each speaker (eg, a face photo or thumbnail of each speaker).

A more specific speaker identification method and the configuration and operation of the audiovisual system will be described in more detail with reference to FIGS. 5 and 7 to 9.

Since the description of the technical features described above with reference to FIGS. 1 to 3 is applied to FIGS. 4 and 6 as they are, duplicate description will be omitted.

FIG. 5 is a block diagram for explaining an example of the internal configuration of the computer system in one embodiment.

For example, the speaker dialylation device for identifying a speaker who speaks voice according to an embodiment of the present invention may be realized by the computer system 500 of FIG. The computer system 500 may correspond to the audiovisual system described above. As shown in FIG. 5, the computer system 500 includes a processor 510, a memory 520, a permanent recording device 530, a bus 540, as components for executing a speaker dialylation method for identifying a speaker who speaks a voice. An input / output interface 550 and a network interface 560 may be included. The computer system 500 may be composed of a plurality of computer systems, unlike those shown in the figure. The computer system 500 may be, for example, a system or a part thereof for recording communication such as a conference between a plurality of speakers. The computer system 500 is, or is part of, a computer or other server that is contained within a device that includes a camera that captures the speaker, or that communicates wired and / or wirelessly with the device that contains the camera. May be good.

Processor 510 includes, or is part of, any device capable of processing a sequence of instructions as a component for implementing a speaker dialylation method that identifies the speaker who speaks the voice. Good. Processor 510 may include, for example, a processor and / or a digital processor in a computer processor, mobile device, or other electronic device. The processor 510 may be included, for example, in a server computer device, a server computer, a set of server computers, a server farm, a cloud computer, a content platform, and the like. Processor 510 may be connected to memory 520 via bus 540.

Memory 520 may include volatile memory, persistent, virtual, or other memory used by or output by computer system 500. The memory 520 may include, for example, a RAM (random access memory) and / or a DRAM (dynamic RAM). Memory 520 may be used to record arbitrary information, such as state information for computer system 500. The memory 520 may be used, for example, to record the instructions of the computer system 500, including the instructions for performing a speaker dialing method that identifies the speaker who speaks the voice. The computer system 500 may include one or more processors 510 when necessary or appropriate.

Bus 540 may include a communication infrastructure structure that allows interaction between the various components of computer system 500. Bus 540 may carry data between components of computer system 500, for example, between processor 510 and memory 520. Bus 540 may include wireless and / or wired communication media between the components of computer system 500, and may include parallel, serial, or other topology arrays.

Persistent recording device 530, such as memory used by computer system 500 to record data over a predetermined extended period of time (eg, compared to memory 520), or other persistent recording device. Components may be included. Persistent recording device 530 may include non-volatile main memory such as that used by processor 510 in computer system 500. Persistent recording device 530 may include, for example, flash memory, a hard disk, an optical disk, or other computer-readable medium. Permanent recording device 530 may record, for example, the minutes 600 described above or data associated with the minutes 600.

The input / output interface 550 may include an interface to a keyboard, mouse, voice instruction input, display, or other input / output device. Instructions and / or inputs associated with a speaker dialylation method that identifies the speaker who speaks the voice may be received by the input / output interface 550.

Network interface 560 may include one or more interfaces to networks such as short-range networks or the Internet. The network interface 560 may include an interface for a wired or wireless connection. Instructions and / or inputs associated with a speaker dialing method that identifies the speaker who speaks the voice may be received by network interface 560.

Also, in other embodiments, the computer device 500 may include more components than the components of FIG. However, most prior art components need not be clearly illustrated. For example, the computer device 500 may be implemented to include at least a portion of the input / output devices linked to the input / output interface 550 described above, or a transceiver, GPS (Global Positioning System). Other components such as modules, cameras (eg, 360 degree cameras for capturing speakers), microphones (eg, at least one microphone for recording the speaker's voice), various sensors, databases, etc. Further may be included.

According to the embodiment realized by such a computer system 500, the correlation between the shape of each speaker's mouth included in the video data and each of the speech segments from the speaker included in the audio data is calculated and calculated. A speaker dialyization method is provided that builds a speaker model for each speaker based on the correlation, and identifies the speaker who speaks the voice contained in the audio data based on the constructed speaker model. Good.

As described above, the above-mentioned description of the technical features with reference to FIGS. 1 to 4 is applied to FIG. 5 as it is, and thus the duplicate description will be omitted.

FIG. 7 is a flowchart showing a method of constructing a speaker model and a method of separating speakers who speak in one embodiment.

With reference to FIG. 7, a speaker dialylation method for identifying a speaker who speaks a voice, which is executed by the computer system 500, will be described in detail.

Video and audio data of a plurality of speakers may be used to separate the speakers. The video data may be taken by a plurality of speakers with a camera, and the audio data may correspond to the corresponding video data. That is, the video data and the audio data may constitute one video (video). For example, the video data and the audio data may be data included in the video of the speaker captured in real time, or may be data included in the video of the speaker captured in advance.

In step 720, the processor 510 may calculate the correlation between each speaker's mouth shape (eg, a change in mouth shape) contained in the video data and each of the speech segments contained in the audio data. The speech segment is a part of the voice of each speaker included in the audio data, and may correspond to the voice within a predetermined time (for example, 1.5 seconds). The shape of each speaker's mouth may be identified by the processor 510 recognizing each speaker's face from the video data and tracking the recognized face.

At step 730, processor 510 may build and register a speaker model for each speaker based on the calculated correlation. In other words, stage 730 may be the stage of registering each speaker (speaker model) associated with communication (eg, attending a conference).

As an example, the processor 510 sets the top N speech segments, which have a high correlation with the shape of each speaker's mouth, among the speech segments included in the audio data for each speaker as a speaker model for each speaker. May be used to build. N may be a natural number, for example 10. Further, the processor 510 is for constructing (and registering) a speaker model for each speaker only for the speech segments whose correlation with the mouth shape of each speaker is equal to or more than a predetermined threshold value among the N speech segments. You may use it for.

At step 740, the processor 510 may identify the speaker who speaks the voice contained in the audio data from among the speakers, based on the constructed speaker model. For example, the processor 510 may specify by which speaker the sound uttered during reproduction of audio data (or video including the audio data) is uttered.

In another embodiment, even if the audio data is real-time audio data (that is, audio data included in a video that captures a conference between speakers in real time), the processor 510 also uses which speaker the spoken voice is. It may be identified in real time (or near real time) whether it was uttered by. At this time, the processor 510 builds (registers) a model for each speaker in real time (or almost in real time) for the video shot in real time, or creates a model for the speaker built (registered) in advance. You may update it.

On the other hand, when the audio data includes voices overlapped by the statements of two or more speakers, such overlapped voices may be processed separately. For example, the processor 510 may extract the pronunciation of the speaker who utters the overlapped voice based on the shape of the speaker's mouth (movement of the lips) when the overlapped voice is uttered, and corresponds to the extracted pronunciation. A mask for a portion may be generated from the audio data to be generated, and the overlapping audio may be filtered. As a result, the voices of a plurality of speakers uttering at the same time and overlapping may be recorded separately as the voices of the uttered speakers.

As shown in step 710, the video and / or audio data may be preprocessed prior to the execution of step 720. For example, the processor 510 may use a bandpass filter (for example, in the 200-7000 Hz range) to remove noise beyond the human voice range from the audio data. In addition, the processor 510 may execute preprocessing for recognizing the face of a specific person from the video data. The processor 510 may recognize the face of a specific person from the video data by detecting the face from the video data and tracking the detected face. At this time, the profile image (or others) of the speaker registered (recorded) in advance may be further used for identification.

At step 750, processor 510 may extract the voice spoken by the identified speaker from the audio data.

At step 760, processor 510 may convert the extracted voice into text. In order to convert the voice into text, the corresponding STT (Speech To Text) technology may be used. For example, speech may be converted to text by modules implemented in artificial intelligence, deep learning, or other neural networks.

At step 770, processor 510 may record the translated text in association with the time information at which the extracted speech was spoken. For example, the processor 510 may record the speech content of each speaker and the speech time in association with each other, as in the minutes 600 shown in FIG. As a result, the computer system 500 can automatically record the communication between speakers by classifying them by speaker.

In the embodiment, even if there is an utterance from the specific speaker with the mouth of the specific speaker hidden in the video data, the utterance is specified based on the constructed speaker model. It is possible to identify whether it is due to the speaker of.

On the other hand, among a plurality of speakers, a speaker whose face is recognized from the video data but is recognized as not speaking at all may be ignored. That is, a speaker model for such a speaker does not have to be constructed. Alternatively, the computer system 500 may record information indicating that the speaker exists as a participant of the conference but does not speak at all because the face is recognized from the video data.

In constructing the speaker model in stage 730, the shape of the face or mouth is not recognized from the video data among a plurality of speakers (for example, when attending a conference by telephone), or the speaker model is constructed. For a particular speaker for whom a speaker model is not built (ie, it is not possible to build a speaker model) because there is no correlation between the mouth shape and the speech segment. Thus, the processor 510 clusters the embeddings (or feature values) corresponding to the speech segments for which the correlation calculated in the step of calculating the correlation in step 720 is less than a predetermined value. You may build a speaker model for a person. As a result, a speaker model can be constructed even for a speaker whose speaker model has not been constructed by the correlation analysis using video data.

As described above, the above-mentioned description of the technical features with reference to FIGS. 1 to 6 is applied to FIG. 7 as it is, and thus duplicate description will be omitted.

8 and 9 are flowcharts showing a method of identifying a speaker who speaks, according to an example.

A method of identifying the speaker who speaks the voice of step 740 will be described in detail with reference to FIGS. 8 and 9.

In step 810, the processor 510 detects the face portion of each speaker from the video data, tracks the detected face, and calculates the correlation between the shape of each speaker's mouth and the sound contained in the audio data. You can do it. At step 820, processor 510 may use the calculated correlation and the constructed speaker model to identify the speaker who spoke the speech in that section. Not only when building a speaker model, but also when identifying a speaker, if the correlation between the shape of the speaker's mouth and the sound calculated using video data is used, the speaker dialification can be performed. The accuracy can be improved.

At this time, in the video data, the correlation between the specific speaker and the sound uttered in the audio data by hiding the face of the specific speaker or the mouth of the specific speaker (occluded). If cannot be calculated, the correlation between the particular speaker and the spoken voice may be considered to be zero. Therefore, when the mouth of a particular speaker is hidden, the processor 510 uses only the speaker model constructed to identify the speaker for the spoken voice for that particular speaker. You can do it.

At step 910, the processor 510 may determine information about the position of the speaker from which the voice contained in the audio data is spoken. Information about the position of the speaker in which the voice is spoken may include information about the direction of the voice being spoken. The directional information may include azimuth information and / or altitude information associated with the spoken voice. Alternatively, the position information may include data indicating the positional relationship between the microphone used for recording the audio data and the sound source corresponding to the speaker.

At step 920, processor 510 may identify the speaker who spoke the voice using information about the determined speaker position and the constructed speaker model. The accuracy of speaker dialification by further using information about the position of the speaker from which the voice is spoken (ie, information about the direction of the voice (voice signal)) as well as the constructed speaker model. Can be further enhanced.

As shown in the test results above, by further using information about the correlation between the speaker's mouth shape and speech and / or the position of the speaker, such as the speaker model, the speaker dialification The accuracy can be improved.

The devices described above may be implemented by hardware components, software components, and / or combinations of hardware components and software components. For example, the apparatus and components described in the embodiments include a processor, a controller, an ALU (arithmetic logic unit), a digital signal processor, a microcomputer, an FPGA (field program gate array), a PLU (programmable log unit), a microprocessor, and the like. Alternatively, it may be implemented using one or more general purpose computers or special purpose computers, such as various devices capable of executing and responding to instructions. The processing device may execute an operating system (OS) and one or more software applications running on the OS. The processing device may also respond to the execution of the software, access the data, and record, manipulate, process, and generate the data. For convenience of understanding, one processing device may be described as being used, but one of ordinary skill in the art may indicate that the processing device may include a plurality of processing elements and / or a plurality of types of processing elements. You can understand. For example, a processor may include multiple processors or one processor and one controller. Other processing configurations, such as parallel processors, are also possible.

The software may include computer programs, code, instructions, or a combination of one or more of these, configuring the processing equipment to operate at will, or instructing the processing equipment independently or collectively. You may do it. The software and / or data is embodied in any type of machine, component, physical device, computer recording medium or device to be interpreted based on the processing device or to provide instructions or data to the processing device. Good. The software is distributed on a computer system connected by a network and may be recorded or executed in a distributed state. The software and data may be recorded on one or more computer-readable recording media.

The method according to the embodiment may be implemented in the form of program instructions that can be executed by various computer means and recorded on a computer-readable medium. At this time, the medium may be a continuous recording of a computer-executable program, or a temporary recording for execution or download. In addition, the medium may be a variety of recording or storage means in the form of a combination of single or multiple hardware, and is not limited to a medium directly connected to a computer system, but is distributed over a network. It may exist. Examples of media include hard disks, floppy (registered trademark) disks, magnetic media such as magnetic tape, magneto-optical media such as CD-ROMs and DVDs, magneto-optical media such as flotropic disks, and the like. And ROM, RAM, flash memory, etc., and may be configured to record program instructions. In addition, other examples of media include recording media or storage media managed by application stores that distribute applications, sites that supply or distribute various other software, servers, and the like.

As described above, the embodiments have been described based on the limited embodiments and drawings, but those skilled in the art will be able to make various modifications and modifications from the above description. For example, the techniques described may be performed in a different order than the methods described, and / or components such as the systems, structures, devices, circuits described may be in a form different from the methods described. Appropriate results can be achieved even if they are combined or combined, or confronted or replaced by other components or equivalents.

Therefore, even different embodiments belong to the attached claims as long as they are equivalent to the claims.

500: Computer system 510: Processor 520: Memory 530: Persistent recording device 540: Bus 550: Input / output interface 560: Network interface

Claims (19)

A method performed by a computer system that uses video and audio data of multiple speakers to separate speakers.
A step of calculating the correlation between the mouth shape of each speaker included in the video data and each of the speech segments from the speaker included in the audio data.
A stage of constructing a speaker model for each of the speakers based on the calculated correlation, and a story of uttering a voice included in the audio data among the speakers based on the constructed speaker model. A speaker dialing method that includes the steps to identify a person. At the stage of building the speaker model,
For each speaker of the speaker, in order to construct a speaker model for each speaker, the top N speech segments having a high correlation with the shape of the mouth of each speaker among the speech segments are used. use,
N is a natural number,
The speaker dialylation method according to claim 1. In constructing the speaker model for each speaker, only the speech segments whose correlation with the mouth shape of each speaker is equal to or more than a predetermined threshold value are used among the N speech segments.
The speaker dialylation method according to claim 2. The voice in which two or more of the speakers included in the audio data overlap is the speaker who utters the overlapped voice based on the shape of the speaker's mouth when the overlapped voice is spoken. By extracting the pronunciation of the above, generating a mask for the part of the audio data corresponding to the extracted pronunciation, and filtering the overlapping voices, the overlapping voices are separated into the voices of the speakers who uttered the overlapping voices. Ru,
The speaker dialylation method according to claim 1. The shape of each speaker's mouth is identified by recognizing each speaker's face from the video data and tracking the recognized face.
The speaker dialylation method according to claim 1. The stage of identifying the speaker is
The face of each speaker of the speaker is detected from the video data, the detected face is tracked, and the correlation between the shape of the mouth of each speaker and the spoken voice included in the audio data is determined. Includes a step of calculating and a step of identifying the speaker who spoke the spoken voice using the calculated correlation and the constructed speaker model.
The speaker dialylation method according to claim 1. The stage of identifying the speaker who spoke the voice is
In the video data, the correlation between the specific speaker and the spoken voice of the audio data is calculated by hiding the face of the specific speaker or the mouth of the specific speaker among the speakers. If not,
The correlation between the particular speaker and the spoken voice is considered to be 0.
The speaker dialylation method according to claim 6. The stage of identifying the speaker is
The stage of determining the information regarding the position of the speaker who uttered the spoken voice included in the audio data, and the utterance using the information regarding the determined speaker position and the constructed speaker model. Including the step of identifying the speaker who spoke the voice
The speaker dialylation method according to claim 1. The information regarding the position of the speaker who uttered the voice includes information regarding the direction of the spoken voice.
The information about the direction includes the azimuth information associated with the spoken voice.
The speaker dialylation method according to claim 8. Of the plurality of speakers, the speaker whose face is recognized from the video data but is recognized as not speaking at all is ignored.
The speaker dialylation method according to claim 1. The stage of building the speaker model is
Among the plurality of speakers, the speaker model is due to the fact that the face or mouth shape is not recognized from the video data or the correlation between the mouth shape and the speech segment for constructing the speaker model is not prepared at all. For certain speakers who do not build
A speaker model for the particular speaker is constructed by clustering the embeddings corresponding to the speech segments for which the correlation calculated at the stage of calculating the correlation is less than a predetermined value.
The speaker dialylation method according to claim 1. The video data and audio data are data included in a video captured in real time of the speaker.
The speaker dialylation method according to claim 1. The stage of extracting the voice spoken by the specified speaker from the audio data,
The speaker dialyization according to claim 1, further comprising a step of converting the extracted voice into text and a step of recording the converted text in association with time information in which the extracted voice is spoken. Method. A computer program that causes a computer to execute the speaker dialing method according to any one of claims 1 to 13. A computer-readable recording medium in which a program for executing the speaker dialing method according to any one of claims 1 to 13 is recorded. A computer system that separates speakers by using video data and audio data of multiple speakers.
Includes memory and at least one processor concatenated with said memory and configured to execute computer-readable instructions contained in said memory.
The at least one processor
The correlation between the shape of each speaker's mouth included in the video data and each of the speech segments from the speaker included in the audio data is calculated, and the speaker is based on the calculated correlation. A speaker model for each is constructed, and a speaker who speaks among the speakers is identified from the audio data based on the constructed speaker model.
Computer system. The at least one processor
The face of each speaker of the speaker is detected from the video data, the detected face is tracked, and the correlation between the shape of the mouth of each speaker and the spoken voice included in the audio data is determined. Calculating and using the calculated correlation and the constructed speaker model to identify the speaker who spoke the spoken voice.
The computer system according to claim 16. The at least one processor
The information regarding the position of the speaker who spoke the spoken voice included in the audio data is determined, and the spoken voice is used by using the information regarding the determined speaker position and the constructed speaker model. Identify the speaker who spoke
The computer system according to claim 16. The at least one processor
Among the plurality of speakers, the face or mouth shape is not recognized from the video data, or the correlation between the mouth shape and the speech segment for constructing the speaker model is not prepared at all. For a specific speaker for whom a person model is not constructed, the specific story is described by clustering the embeddings corresponding to the speech segments whose correlation calculated at the stage of calculating the correlation is less than a predetermined value. Build a speaker model for a person,
The computer system according to claim 16.

Images