JP2007323318A - Speaker face image determination method, device, and program - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To specify a speaker without viewing a video even when the name of a speaker is not known. <P>SOLUTION: This speaker face image determination method includes recognizing the voice of a video input by a teacher or a speaker recognition technology, and applying a speaker ID to every speaker section configured of the start/end time of the video and every speaker corresponding to the speaker section, and storing it in a speaker recognition result storage means as the speaker recognition result, and detecting a face image included in the speaker section corresponding to the speaker ID by a method for detecting the position of the face from the input video, and storing the face image with the speaker ID in the face image storage means for every speaker ID, and extracting personal characteristics corresponding to each face image of each speaker ID stored in the image storage means for every speaker ID, determining the most suitable personal characteristics for every speaker ID from the personal characteristics corresponding to the face image, and storing the personal characteristics in the speaker personal characteristics decision result storage means. <P>COPYRIGHT: (C)2008,JPO&INPIT

Description

  The present invention relates to a speaker face image determination method, apparatus, and program, and more particularly, to a speaker face image determination method, apparatus, and program for associating a speaker with a face image from a video (with sound).

  One of the video indexing technologies is speaker recognition technology. Speaker recognition technology is roughly classified into supervised speaker recognition technology and unsupervised speaker recognition technology. Since supervised speaker recognition technology provides teacher data, it is necessary to create teacher data prior to recognition. On the other hand, in unsupervised speaker recognition, a video can be divided into sections for each speaker without preparing teacher data.

However, when unsupervised speaker recognition is performed, it can be divided into sections for each speaker, but it is unclear who the utterance is. Therefore, even when a label is assigned to a section,
"Speaker 1, Speaker 2, Speaker 1, ..."
It will be in the state. In order to confirm the case where the user wants to view only the utterance portion of the specific speaker (Mr. A), it is necessary to view the video once for which speaker label A corresponds to.

  The present invention has been made in view of the above points, and even when the speaker name is unknown, the speaker can be identified without viewing the video. It is an object of the present invention to provide a speaker face image determination method, apparatus, and program for associating face images.

  FIG. 1 is a diagram for explaining the principle of the present invention.

The present invention (Claim 1) is a face image determination method for determining a face image of a speaker shown in the video from the video including the input voice,
The acoustic analysis means recognizes the audio of the video input by the unsupervised speaker recognition technology, and sets the speaker ID including the start and end times of the video and the speaker ID for each speaker corresponding to the speaker interval. An acoustic analysis step (step 1) for providing and storing the result as a speaker recognition result in the speaker recognition result storage means;
The face detection means detects a face image included in the speaker section corresponding to the speaker ID by a method of detecting the position of the face from the input video, and stores it in the face image storage means for each speaker ID together with the speaker ID. Detecting face (step 2),
A personal feature extraction step (step 3) in which the personal feature extraction means extracts personal features for each face image of each speaker ID stored in the image storage means for each speaker ID;
A speaker individual feature determining unit determines a personal feature most suitable for each speaker ID from the individual features of the face image and stores it in the speaker individual feature determination result storing unit (step 4); I do.

Further, the present invention (Claim 2) provides a speaker individual feature determination step (Step 4).
Clustering is performed on the individual features, and the face image of the video in the speaker section is the most suitable personal feature.

In the present invention (Claim 3), in the speaker individual feature determination step (Step 4),
Among the individual features, the one with the highest appearance frequency among the individual features whose distance between the individual features is smaller than the threshold is set as the most suitable personal feature.

  FIG. 2 is a principle configuration diagram of the present invention.

The present invention (Claim 4) is a face image determination device for determining a face image of a speaker shown in the video from the video including the input voice,
Recognize the audio of the video input by the unsupervised speaker recognition technology, and assign a speaker ID to each speaker section consisting of the start and end times of the video and each speaker corresponding to the speaker section. Acoustic analysis means 1 for storing the recognition result in the speaker recognition result storage means 5;
Face detecting means for detecting a face image included in the speaker section corresponding to the speaker ID by a method of detecting the face position from the input video, and storing it in the face image storing means 6 for each speaker ID together with the speaker ID. 2,
Personal feature extraction means 3 for extracting personal features for each face image of each speaker ID stored in each speaker ID image storage means 6;
Speaker personal feature determination means 4 that determines the most suitable personal feature for each speaker ID from the personal features of the face image and stores the result in the speaker personal feature determination result storage means 7.

Further, the present invention (Claim 5) is provided in the speaker individual feature determining means 4,
Clustering is performed on the individual features, and the most frequently used facial image among the face images of the video in the speaker section is included as the most suitable personal feature.

Further, the present invention (Claim 6) is provided in the speaker individual feature determining means 4,
Among the individual features, a means for setting a personal feature having the highest appearance frequency from among the personal features whose distance between the personal features is smaller than a threshold is used.

  The present invention (Claim 7) is a face image determination program for causing a computer to execute each means of the face image determination apparatus according to Claims 4 to 6.

  As described above, according to the present invention, the personal feature is extracted from the detected face image, the speaker is identified from the speaker information and the personal feature, and displayed together with the face image. By displaying the face image, it is possible to obtain information about the speaker without playing it once.

  Hereinafter, embodiments of the present invention will be described with reference to the drawings.

  FIG. 3 shows the configuration of the speaker face image determination device according to the embodiment of the present invention.

  The speaker face image determination device shown in the figure is connected to a video storage device 8 in which video with sound is stored.

  The speaker face image determination device includes an unsupervised speaker recognition unit 1, a face detection unit 2, a personal feature extraction unit 3, a speaker personal feature determination unit 4, a speaker ID face image storage device 6, and a speaker personal feature determination. The result storage unit 7 is configured.

  The unsupervised speaker recognition unit 1 acquires a video with audio from the video storage device 8 and recognizes the speaker from the feature amount of the audio. As a speaker recognition method, for example, Japanese Patent Application Laid-Open Publication No. 2004-145161 (voice database registration processing method, voice generation source recognition method, voice generation section search method, voice database registration processing device, voice generation source recognition device, voice generation Section search device, its program, and its recording medium) can be used. Note that the method for speaker recognition is not limited to that shown in the document. Therefore, it does not mention the method of unsupervised speaker recognition itself. The speaker recognition result recognized by the unsupervised speaker recognition unit 1 is stored in the speaker recognition result storage unit 5.

  The speaker recognition result storage unit 5 stores the speaker recognition result. FIG. 4 shows an example of the speaker recognition result storage unit in the embodiment of the present invention. The speaker recognition result storage unit 5 shown in FIG. 4 includes a video section ID, a video section start time, an end time, and a speaker ID. The video section ID is a unique number given each time the speaker is switched. The display time of the video section may be any display method as long as the time can be specified. In the present embodiment, the second is described in the image. The speaker ID is a label given to each speaker, and is an unsupervised speaker recognition, so an individual name cannot be given. Therefore, the speaker ID is a symbol assigned to each speaker. Here, an integer is assigned, but an alphabet may be used.

  The face detection unit 2 refers to the speaker recognition result storage unit 5 and detects a face image included in the speaker section (the start time / end time of the video section corresponding to the video section ID). The face image detection method may be any method for detecting a face from an existing still image, and the method is not particularly limited. The detected face image is assigned the speaker ID of the speaker corresponding to the section ID, and the face image is stored in the speaker ID-specific face image storage unit 6 for each speaker ID.

  The personal feature extraction unit 3 extracts the personal feature of the speaker from the face image for each speaker ID stored in the face image storage unit 6 for each speaker ID. In the present invention, the personal feature extraction method is not particularly limited, but the face width, the distance between the centers of both eyes, the height of the eyes from the top of the head, the unique face, etc. proposed in the conventional face recognition method may be used. Conceivable. As a simpler method, it may be possible to perform simple personal authentication by obtaining a color from a region of clothes existing under the face.

  The speaker individual feature determination unit 4 performs clustering on the face image set corresponding to the speaker ID using the individual feature as a feature amount, and obtains the individual feature closest to the center of gravity of the individual feature included in the maximum cluster from the clustering result. Then, it is stored in the speaker individual feature determination result storage unit 7 as the individual feature for the speaker ID.

  Next, a series of operations in the above configuration will be described.

  FIG. 5 is a flowchart of the operation of the speaker face image determination device according to the embodiment of the present invention.

  Step 100) The unsupervised speaker recognition unit 1 recognizes the speaker from the input video with sound by acoustic analysis, and stores the speaker recognition result in the speaker recognition result storage unit 5. For example, clustering includes, for example, a method described in the document “Introduction to Cluster Analysis” written by Sadaaki Miyamoto, Morikita Publishing. By applying this to the feature quantity used in speaker recognition (the above-mentioned Japanese Patent Application Laid-Open No. 2004-145161), unsupervised speaker recognition is possible. Here, the method is not limited.

  Step 200) The face detection unit 2 obtains a face image based on the information in the speaker recognition result storage unit 5 and records it in the face image storage unit 6 for each speaker ID. Details will be described below.

  FIG. 6 is a flowchart of face image detection processing according to an embodiment of the present invention.

  Step 501) The section IDi is initialized to i = 0.

  Step 502) The end time is acquired from the start time of i from the speaker recognition result storage unit 5, the video in the time range is acquired from the video storage device 8, and the cut point is detected from the video.

  Step 503) It is determined whether or not there is a cut point. If there is a cut point, the process proceeds to Step 504. Otherwise, the process proceeds to Step 509. However, it is not necessary to be a cut point, and a predetermined time interval may be set in advance.

  Step 504) Face detection is performed for the cut point. There are various existing techniques for face detection from still images. Here, any method may be used as long as the position of the face can be detected. For example, Intel's Open Source Computer Vision Library also includes face detection.

  Step 505) It is determined whether a face is detected. If there is a face, the process proceeds to step 506, and if not, the process proceeds to step 508.

  Step 506) The speaker recognition result storage unit 5 is searched based on the section IDi, and the speaker ID of the speaker is obtained.

  Step 507) The detected face image is recorded in the face image storage unit 6 for each speaker ID as the face image of the speaker ID (S).

  Step 508) It is determined whether face detection processing has been performed for all the cuts detected in Step 502. If not, the process proceeds to Step 504. If face detection processing has been performed for all the cut points, Step 509 is performed. Transition.

  Step 509) It is determined whether the processing from Step 502 to Step 509 has been performed in all the sections. If not, the process proceeds to Step 510. Otherwise, the process is terminated.

  Step 510) The section ID i is incremented (i = i + 1), and the process proceeds to Step 502.

  An example of the face image storage unit 6 for each speaker ID stored by the processing from step 501 to step 510 is shown in FIG. As shown in the figure, the speaker ID and the face image obtained in the video section are a set. Here, the speaker ID face image storage unit 6 has a so-called one-to-many relationship in which a plurality of face images are associated with one speaker ID. Any information holding method may be used as long as the status can be held.

  Step 300) The personal feature extraction unit 3 extracts the personal feature from the face image for each speaker ID acquired in Step 200.

  Here, the personal feature extraction method is not limited, and it is assumed that F = P (X) is obtained as a result of the feature extraction process P (X) for the detected image X. Here, F is a feature vector in which the number of features obtained as a result of feature extraction is the number of dimensions.

  Step 400) The speaker individual feature determination unit 4 determines the speaker individual features by clustering based on the features extracted by the individual feature extraction unit 3.

Individual features are extracted from the face image obtained for each speaker ID in step 300, but the face image for each speaker ID includes not only the person of the speaker ID but also various persons. ing. This is because the person is not necessarily shown when a certain speaker is speaking. Therefore, it is necessary to truly determine the personal characteristics relating to the speaker k from the personal characteristics P (X _k ) obtained from the face image X _k obtained for a certain speaker ID (k).

  One method is clustering. This method is based on the premise that the most frequently appearing face image on the screen at the time of a certain speaker is the face of the speaker.

  An example of personal feature determination processing by clustering is shown below.

  FIG. 8 is a flowchart of speaker feature determination processing by clustering according to an embodiment of the present invention.

  Step 701) The speaker ID is initialized as q = 0.

  Step 702) Clustering is performed on the face image set X included in the speaker ID q with P (X), which is an individual feature of the face image set X, as a jump amount.

Step 703) A personal feature closest to the center of gravity of the personal feature P (X ′) included in the maximum cluster is obtained from the clustering result, and is set as P (X _m ).

Step 704) As a personal feature of the speaker ID q, the speaker personal feature determination result storage unit 7 stores a set of the speaker ID q and the personal feature P (X _m ).

  Step 705) It is determined whether or not the processing has been performed for all the speaker IDs. If the processing has been completed, the processing ends. If not, the processing proceeds to Step 706.

  Step 706) q is incremented (q = q + 1), and the process proceeds to Step 702.

  Next, as another implementation method of speaker individual feature determination processing, it is conceivable to use the most frequent individual feature. An example is shown below.

  FIG. 9 is a flowchart of the personal feature determination process using the most frequent personal feature according to the embodiment of the present invention.

  Step 801) The speaker ID is initialized as q = 0.

  Step 802) Initialize k = 0.

Step 803) Initialize n = 0 and counter C _k = 0.

Step 804) The distance | P (X _k ) −P (X _n ) | of P (X _k ) that is the personal feature of the face image Xk and P (X _n ) that is the personal feature of the face feature X _n
If the threshold value is less than or equal to the threshold th, the process proceeds to step 805, and if not, the process proceeds to step 806.

Step 805) If the distance obtained in Step 804 is less than or equal to the threshold, the frequency counter C _k is incremented (C _k = C _k +1).

  Step 806) If n <N, increment n (n = n + 1) and go to Step 804, otherwise go to Step 808.

  Step 807) If k <N, increment k (k = k + 1) and go to Step 803. Otherwise, the process proceeds to step 808.

Step 808) Find the maximum _{k of the} frequency counter C _k (k = 0,..., N−1) and substitute it into m.

Step 809) As a personal feature of the speaker ID q, the speaker personal feature determination result storage unit 7 stores a set of the speaker ID q and the personal feature P (X _m ).

  Step 810) It is determined whether or not the processing has been performed for all the speaker IDs. If the processing has been completed, the processing ends. If not, the processing proceeds to Step 811.

  Step 811) q is incremented (q = q + 1) and the routine goes to Step 802.

  FIG. 10 shows an example of the speaker individual feature determination result in the embodiment of the present invention.

  Here, a set of a speaker ID and personal characteristics is stored. Further, in this example, a face image that outputs the personal feature is held as a representative face image.

  Embodiments of the present invention will be described below with reference to the drawings.

  In the following, an example will be described in which a quiz is presented and a quiz program in which quiz respondents battle for battle in a panel composed of 25 squares is used as the subject. There are many quiz programs whose performers change every time they are viewers, and it can be said that this is an example of content that is difficult (no merit) to apply supervised speaker recognition.

  FIG. 11 is an example of an unsupervised speaker recognition result according to an embodiment of the present invention.

  For each speaker ID video, the face detection unit 2 performs cut point detection and face detection, and stores a face image for each speaker ID in the speaker ID face image storage unit 6. Individual features are extracted from these images.

In the quiz program which is the subject of this time, as shown in FIG. 12, since a color unique to the individual is arranged around the seat, the color information around the detected face is a personal feature. The personal feature extraction unit 3 sets X ₀ in FIG. 10 and acquires pixels in the shaded portion by P (X ₀ ). As described above, when there is information on how to be reflected in a program as well as a quiz program, it is appropriate to use a heuristic personal information extraction rule that matches the information. However, in a quiz program in which everyone wears the same color clothes, information on clothes colors cannot be used as personal features.

The speaker individual feature determination unit 4 performs clustering of the above-described individual features P (X _i ) for the image set X included in each speaker ID q, and obtains P (X _m ) closest to the center of gravity of the maximum cluster. As a result, the personal feature corresponding to the speaker ID q is obtained as P (X _m ). If the relationship of q⇔X _m is followed, it is possible to easily view only the utterance section of the corresponding speaker by clicking on the face image.

  FIG. 13 shows an example used as an indexing result display in one embodiment of the present invention. In the example of the figure, there is a video display unit at the top of the screen and a detected speaker display unit at the bottom. The lower face image is associated with the section shown in FIG. 11 where the speaker of the face in the video speaks. Then, by clicking on the image, the related video section is reproduced.

  In this way, by using the present invention, it is possible to list the speaker's face images and to reproduce them by associating the face images with the video section.

  In the present invention, each function of the apparatus shown in FIG. 3 can be constructed as a program and installed in a computer used as a speaker face image determination apparatus to be executed or distributed via a network. is there.

  Further, the constructed program can be stored in a portable storage medium such as a hard disk, a flexible disk, or a CD-ROM, and can be installed or distributed on a computer used as a speaker face image determination device. .

  The present invention is not limited to the above-described embodiments and examples, and various modifications and applications can be made within the scope of the claims.

  The present invention can be applied to video content processing technology, particularly multi-modal processing technology for speaker recognition and image processing.

It is a figure for demonstrating the principle of this invention. It is a principle block diagram of this invention. It is a block diagram of the speaker image determination apparatus in one embodiment of this invention. It is an example of the speaker recognition result memory | storage part in one embodiment of this invention. It is a flowchart of operation | movement of the speaker face image determination apparatus in one embodiment of this invention. It is a flowchart of the face image detection process in one embodiment of the present invention. It is an example of the face image memory | storage part for every speaker ID in one embodiment of this invention. It is a flowchart of the individual feature determination process by clustering in one embodiment of the present invention. It is a flowchart of the personal feature determination process using the most frequent personal feature in one embodiment of the present invention. It is an example of the speaker individual characteristic determination result in one embodiment of this invention. It is an example of an unsupervised speaker recognition result of one Example of this invention. It is an example of the individual characteristic of one Example of this invention. It is the example utilized as a display of the indexing result in one Example of this invention.

Explanation of symbols

DESCRIPTION OF SYMBOLS 1 Acoustic analysis means, Unsupervised speaker recognition part 2 Face detection means, Face detection part 3 Individual feature extraction means, Individual feature extraction part 4 Speaker individual feature determination means, Speaker individual feature determination part 5 Speaker recognition result storage means , Speaker recognition result storage unit 6 face image storage means for each speaker ID, face image storage unit for each speaker ID 7 speaker individual feature determination result storage means, speaker individual feature determination result storage unit

Claims (7)

A face image determining method for determining a face image of a speaker reflected in an image including an input voice,
The acoustic analysis means recognizes the audio of the video input by the unsupervised speaker recognition technology, and sets the speaker ID including the start and end times of the video and the speaker ID for each speaker corresponding to the speaker interval. And an acoustic analysis step for storing in the speaker recognition result storage means as a speaker recognition result,
A face detection unit detects a face image included in the speaker section corresponding to the speaker ID by a method of detecting a face position from the input video, and stores the face image for each speaker ID together with the speaker ID. A face detection step stored in the means;
A personal feature extracting step for extracting a personal feature for each face image of each speaker ID stored in the image storing unit for each speaker ID;
Speaker personal feature determination means determines a personal feature most suitable for each speaker ID from the personal characteristics of the face image, and stores the personal characteristics in the speaker personal feature determination result storage means;
A method for determining a face image. In the speaker individual feature determination step,
Clustering the individual features, among the face images of the video in the speaker section, the most frequent one is the most suitable personal feature,
2. The face image determination method according to claim 1. In the speaker individual feature determination step,
2. The face image determination method according to claim 1, wherein, among the individual features, a feature having the highest appearance frequency among the individual features whose distance between the individual features is smaller than a threshold is selected as the most suitable personal feature. A face image determination device for determining a face image of a speaker reflected in an image including an input voice,
Recognize the audio of the video input by the unsupervised speaker recognition technology, and assign a speaker ID to each speaker section consisting of the start and end times of the video and each speaker corresponding to the speaker section. Acoustic analysis means for storing the recognition result in the speaker recognition result storage means;
A face image included in the speaker section corresponding to the speaker ID is detected by a method of detecting a face position from the input video, and stored in the face image storage means for each speaker ID together with the speaker ID. Detection means;
Personal feature extraction means for extracting personal features for each face image of each speaker ID stored in the image storage means for each speaker ID;
Speaker personal feature determination means for determining a personal characteristic most suitable for each speaker ID from the personal characteristics for the face image and storing the speaker personal feature determination result storage means;
A face image determination device characterized by comprising: The speaker individual feature determining means includes:
Clustering the personal features, and including means for setting the most frequent facial images of the video in the speaker section as the most suitable personal features.
The face image determination apparatus according to claim 4. The speaker individual feature determining means includes:
5. The face image determination apparatus according to claim 4, further comprising means for determining a personal feature having the highest appearance frequency from among the personal features whose distance between the personal features is smaller than a threshold among the personal features. On the computer,
A face image determination program that causes each means of the face image determination device according to claim 4 to be executed.

Images