JP2021110996A - Utterer discrimination method, utterer discrimination program, and utterer discrimination apparatus - Google Patents

Abstract

To provide an utterer discrimination method which is a simple facility and can accurately discriminate an utterer from a plurality of object persons.SOLUTION: A method of discriminating an utterer based on video and voice data includes steps that obtains a plurality of time-series first lip behavior feature amounts based on a distance between an upper lip and a lower lip of an object person from the obtained video, obtains a plurality of time-series voice feature amounts based on the obtained voice data, and discriminates the utterer. The step that discriminates the utterer includes processes that obtains a plurality of time-series second lip behavior feature amounts from the plurality of voice feature amounts, obtains a discrimination difference which is a difference between the first lip behavior feature amount and the second lip behavior feature amount, and sets the object person with the smallest discrimination difference among the object persons included in the video as an utterer.SELECTED DRAWING: Figure 1

Description

The present invention relates to the determination of a speaker based on video and audio information.

In recent years, work efficiency has been improved and the working environment has been reviewed in order to realize work style reforms. One of them is to improve the efficiency of work and meetings as measures to improve labor in the workplace.
The minutes of the meeting are used to record the content and arrangements discussed and share the decisions and circumstances, and the minutes created are used to improve the quality of subsequent meetings and improve the efficiency of other operations. Contribute. According to the minutes automatic creation system constructed by applying the voice recognition technology, it is possible to reduce the human error in the minutes creation and the personnel and time required for the minutes creation. Furthermore, in such an automatic minutes creation system, a technology for automatically identifying the speaker for each remark and a technology for improving the voice recognition accuracy contribute to the reduction of man-hours for creating minutes and contribute to the efficiency of meetings and operations.

Patent Document 1 discloses a web conferencing system capable of improving the visibility of the lip portion of a participant who is speaking, and identifies the speaker by using voice and lip movement. Specifically, voice information is used to identify the client terminal in which the speaker is located, and the person with the most moving lips in the specified terminal is determined as the speaker. However, with this technique, when the lips of a person existing in the same terminal are moving at the same time, it becomes difficult to identify the speaker.

Patent Document 2 discloses a speaker estimation system capable of estimating at least one of the next speaker and the timing at which the next speaker speaks in a multi-person conversation by a plurality of participants. However, although this document describes the use of lip movements to discriminate the next speaker, there is no description of the technique for discriminating the current speaker.

Patent Document 3 discloses a terminal device capable of creating minutes of a meeting without requiring a special device including a large number of microphones and the like. However, in this technology, since it is necessary to register the voice information of the user in advance for each terminal, it is necessary to prepare in advance to use it.

Patent Document 4 discloses a system for recognizing the utterance state of a conference participant. In this technique, the area near the lips of each participant in the conference is set for the image acquired using the fisheye lens, and the utterance state is estimated using the feature amount indicating the brightness or color in the area near the lips. .. However, since this technique focuses only on the movement of the lips, it becomes difficult to identify the speaker when the lip regions of the conference participants are moving at the same time.

Patent Document 5 discloses a camera / microphone device for video conferencing that displays an image above (overhead) the face of a conference attendee. However, since this technique identifies the speaker using the direction of arrival of the voice, it becomes difficult to identify the speaker when the distance between the persons is short.

Patent Document 6 discloses a technique that makes it possible to synthesize emotional voice while taking into account the unique settings of each sounder. Here, it is described that a process of constructing a network for generating facial feature points of the corresponding frame by using the voice feature data extracted for each frame from the utterance voice of the speaker is implemented. However, there is a limit to the amount of features, and it is necessary to improve the accuracy.

JP-A-2019-117997 JP-A-2018-077791 Japanese Unexamined Patent Publication No. 2016-209468 Japanese Unexamined Patent Publication No. 2015-09162 Japanese Unexamined Patent Publication No. 2012-147420 Japanese Patent No. 6582157

In view of this point, it is an object of the present invention to provide a speaker discrimination method capable of accurately discriminating a speaker from a plurality of target persons as well as being a simple facility. It also provides a program and a device for that purpose.

One aspect of the present invention is a method of discriminating a speaker from video and audio data, in which the first lip behavior feature amount is time-series based on the distance between the upper lip and the lower lip of the subject from the acquired video. In the process of obtaining a plurality of voice features, the process of obtaining a plurality of voice features in time series based on the acquired voice data, and the process of discriminating the speaker, the process of discriminating the speaker has a plurality of voice features. The process of obtaining a plurality of second lip behavior features in chronological order, the process of obtaining a discriminant difference which is the difference between the first lip behavior features and the second lip behavior features, and the video It is a speaker discrimination method including a process in which a target person having the smallest discrimination difference among the target persons is used as a speaker.

The first lip behavior feature amount may be obtained by the ratio of the distance between the upper lip and the lower lip and the distance between the two points on the nose bridge of the subject.

In the process of determining the speaker, the process of creating a plurality of sections having different start points and having a predetermined time range, and the first lip behavior feature amount and the second lip behavior feature for each of the plurality of sections. The interval difference from the quantity may be obtained, and the average of the interval differences of each interval may be used as the discrimination difference.

In a plurality of sections, in adjacent sections, the starting time may be determined so that a part of the time overlaps.

The plurality of first lip behavior features and the plurality of voice features may be normalized and expressed in the range of 0.0 or more and 1.0 or less.

Another aspect of the present invention is a program for discriminating the speaker from video and audio data, in which the first lip behavior feature amount is time-series based on the distance between the upper lip and the lower lip of the subject from the acquired video. In the step of obtaining a plurality of voice features, a step of obtaining a plurality of voice features in time series based on the acquired voice data, and a step of determining the speaker, in the step of determining the speaker, a plurality of voice features are obtained. Included in the video is a step to obtain a plurality of second lip behavior features in chronological order, a step to obtain a discriminant difference which is a difference between the first lip behavior feature and the second lip behavior feature. It is a speaker discrimination program including a step of setting the target person having the smallest discrimination difference among the target persons as the speaker.

In the speaker discrimination program, the first lip behavior feature amount may be obtained by the ratio of the distance between the upper lip and the lower lip and the distance between two points on the nose bridge of the subject.

In the speaker discrimination program, in the step of discriminating the speaker, a step of creating a plurality of sections having different start points and having a predetermined time range, and a first lip behavior for each of the plurality of sections. The interval difference between the feature amount and the second lip behavior feature amount may be obtained, the average of the section differences in each section may be obtained, and this may be used as the discriminant difference.

In a plurality of sections of the speaker determination program, the time to be the starting point may be determined so that a part of the time overlaps in the adjacent sections.

In the speaker discrimination program, the plurality of first lip behavior features and the plurality of voice features may be normalized in the range of 0.0 or more and 1.0 or less.

Further, it is a device that discriminates a speaker from video and audio data, and is a camera that acquires video, a microphone that acquires audio data, a storage means in which the speaker discrimination program is stored, and a speaker discrimination program. The calculation means includes a calculation means that performs calculation based on the above, and the calculation means acquires video acquired by a camera and audio data acquired by a microphone, and uses the acquired video and audio data by a speaker discrimination program. Provided is a speaker discriminating device for performing calculations.

According to the present invention, the equipment is simple, and the speaker can be accurately discriminated from a plurality of target persons.

FIG. 1 is a diagram showing a flow of the speaker determination method S1. FIG. 2A is a diagram schematically illustrating a part of a video, and FIG. 2B is a diagram illustrating a part of audio data. FIG. 3 is a diagram showing the flow of the process S20 for calculating the first lip behavior feature amount. FIG. 4 is a diagram illustrating the arrangement of feature points. FIG. 5 is an enlarged view of the lip portion of FIG. FIG. 6 is a diagram for explaining the change in the vertical feature amount of the lips. FIG. 7 is an enlarged view of the nose portion of FIG. FIG. 8 is a diagram for explaining the first lip behavior feature amount. 9 (a) and 9 (b) are diagrams for explaining voice data. FIG. 10 is a diagram for explaining that the MFCC was obtained from the voice data. FIG. 11 is a diagram illustrating the flow of the speaker determination process S40. FIG. 12 is a diagram illustrating the speaker determination process S40. FIG. 13 is a diagram illustrating a configuration of a speaker discrimination device. 14 (a) and 14 (b) are diagrams illustrating a test procedure.

{Speaker identification method}
FIG. 1 is a diagram showing a flow of the speaker determination method S1 according to one form. As can be seen from FIG. 1, the speaker discrimination method S1 of the present embodiment includes a video / audio data acquisition process S10, a first lip behavior feature amount calculation process S20, a voice feature amount calculation process S30, and a speaker discrimination process S40. Includes. Each process will be described below.

[Video / audio data acquisition process S10]
In the video / audio data acquisition process S10, the video / audio data of the discrimination target person is acquired. The image can be acquired by a so-called camera, and the audio data can be acquired by a microphone. The speaker can be discriminated by using a microphone that can acquire the voice of the discriminating target person. That is, one video camera and one microphone may be used as long as the information of all the discrimination target persons can be acquired. A plurality of video cameras and microphones may be used, but the video and audio data of the discrimination target can be minimized as long as it can be acquired.
Further, the camera and the microphone may be separate devices or integrated. Therefore, the microphone provided in the camera can also be used.

By this video / audio data acquisition process S10, it is possible to acquire an image of the face portion of the discrimination target person, for example, as schematically shown in FIG. 2A. Further, as schematically shown in FIG. 2B, voice data can be acquired as a waveform with the horizontal axis as time.

[First lip behavior feature calculation process S20]
In the first lip behavior feature amount calculation process S20, the feature amount representing the lip behavior (first lip behavior feature amount) is calculated based on the image acquired in the video / audio data acquisition process S10. FIG. 3 shows the flow of the first lip behavior feature amount calculation process S20.
As can be seen from FIG. 3, the first lip behavior feature calculation process S20 includes the feature point placement process S21, the lip vertical feature calculation process S22, the nose feature calculation process S23, and the first. It has the calculation process S24 of the lip behavior feature amount of. Each process will be described below.

<Placement process of feature points S21>
In the feature point arrangement process S21, the feature points are arranged on the face portion of the discrimination target person with respect to the video acquired in the video / audio data acquisition process S10. An example is shown in FIG. In the example of FIG. 4, the feature points A indicated by “●” are arranged on the face portion of the image shown in FIG. 2 (a) (for easy viewing, the reference numeral A is assigned to only some feature points. Attached, and the others are omitted.) In this embodiment, a plurality of feature points A are provided along the contours of the lower half of the face (cheeks to chin), eyebrows, eyes, nose (bridge of nose, lower end), and lips (upper and lower lips). Have been placed.
The method of arranging the feature points is not particularly limited, but it is possible to determine a position having a brightness difference of a predetermined threshold value or more as the contour of each part by using the brightness difference of adjacent pixels. In addition, commercially available or publicly available software may be used, and examples thereof include Dlib.

In this embodiment, since the vertical position of the lips and the distance between the base of the nose and the tip of the nose are grasped in chronological order as feature points, the feature points A are arranged at least at the positions and numbers necessary for grasping them. It suffices if it is done. Therefore, in this embodiment, the feature points A are arranged at least on the lips and the nose.
However, the feature point A may be arranged in addition to this for other reasons. For example, the position and size of the face of the discrimination target person may be obtained by using the feature points A along the contour of the face, or a process of deleting information other than the face may be performed.

It should be noted that such arrangement of the feature points A is performed for each image in the moving image. That is, feature points A are arranged for each of a plurality of images that are continuous in time series for forming an image. FIG. 4 is an example explaining one image.

<Calculation process of vertical features of lips S22>
In the process of calculating the vertical feature amount of the lips S22, the vertical feature amount of the lip portion is calculated. FIG. 5 shows an enlarged view focusing on the lip portion of FIG.
Here, the "longitudinal feature amount of the lips" represents the feature amount in the direction in which the upper lip and the lower lip are lined up in the lip portion, and as specific examples, the feature point A ₁ belonging to the upper lip and the feature point A belonging to the lower lip. _{The distance from 2} in this direction can be mentioned. The selection of the feature point A ₁ and the feature point A ₂ is not particularly limited, and it is sufficient to understand that the vertical feature amount of the lips changes in time series according to the dictation of the discriminant. In the example shown in FIG. 5, among the feature points A closest to the median line of the face, the feature points at the positions farthest apart from each other were selected. This makes it easier to clarify the time-series changes in the vertical features of the lips.

Accordingly, in this process, we obtain a distance B between the feature point A ₁ and the feature point A ₂ in a direction lined with the upper lip and the lower lip. This distance B may be expressed in any unit such as coordinates, length, and the number of pixels. In this embodiment, the distance is expressed by the number of pixels. In this embodiment, this distance B is the "longitudinal feature amount of the lips".

Such a vertical feature amount of the lips is calculated for each of a plurality of consecutive images (frames) in chronological order. As a result, for example, as shown in FIG. 6, it is possible to obtain a time-series change in the vertical feature amount of the lips with the passage of time (frame number).

<Calculation process of nasal features S23>
In the process of calculating the feature amount of the nose S23, the feature amount in the vertical direction of the nose is calculated. FIG. 7 shows an enlarged view of FIG. 4 focusing on the nose.
Here, the "feature quantity of the nose", of the nose portion, represents a feature amount in the direction along the nasal bridge, Examples, two feature points arranged along the bridge of the nose (A _3, A ₄₎ The distance between them can be mentioned. Two characteristic points A _3, but are not limited particularly selected feature point A _4, in the example shown in FIG. 7, one of the characteristic point A ₃ closest to the root of the nose, closest to the nose tip and the other It was characterized by point _{a 4.} This makes it easier to clarify the time-series changes in the nasal features because they are separated from each other.

Accordingly, in this process, determine the distance C between the feature point A ₃ and the feature point A ₄ of the nose in the direction lined with the upper lip and the lower lip. This distance C may be expressed in any unit such as coordinates, length, number of pixels, etc., but is the same unit as the distance B related to the lips described above. Therefore, in this embodiment, the distance is expressed by the number of pixels.
In this embodiment, this distance C is the "feature amount of the nose".
Such a feature amount of the nose is calculated for each of a plurality of consecutive images (frames) in chronological order. As a result, although not shown, it is possible to obtain a change in the feature amount of the nose with the passage of time (frame number) in the same manner as in FIG. 6 described above.

<Calculation process of first lip behavior feature amount S24>
In the first calculation process of the lip behavior feature amount S24, the vertical feature amount of the lips obtained in the calculation process S22 of the vertical feature amount of the lips and the feature amount of the nose obtained in the calculation process S23 of the nose feature amount. From, the first lip behavior feature is calculated. Specifically, it can be obtained by the following equation.
First lip behavior feature = Longitudinal feature of lip / Nose feature In the case of B and C illustrated, the first lip behavior feature is calculated by the following formula.
First lip behavior feature = B / C

This first lip behavior feature is a dimensionless quantity that represents the ratio of the vertical feature of the lips to the feature of the nose, and thus the influence of the distance between the discriminant and the imaging means can be reduced. ..

Therefore, it is not always necessary to consider the feature amount of the nose, and the first lip behavior feature amount may be used as the vertical feature amount of the lips. The feature amount of the nose was used in this embodiment for the following reasons.
If the distance between the camera and the person to be discriminated changes during the acquisition of the image, the vertical width of the lips changes, so that the vertical feature amount of the lips changes even if the movement of the lips is the same. On the other hand, by taking the ratio of the nasal feature amount, which has little change with respect to the movement of the lips, and using this as an index, the influence of the change in the distance between the camera and the discriminant can be reduced.
Therefore, it does not necessarily have to be a nasal feature amount, and a feature amount that satisfies the following two conditions is extracted, and the ratio of this to the vertical feature amount of the lips is taken as the first lip behavior feature amount. good.
The first condition is that the distance between feature points is less likely to fluctuate with respect to changes in speech movements and facial expressions. In this regard, the distance between the base of the nose and the tip of the nose used for the nasal feature amount is a portion that is not easily affected by changes in speech movements and facial expressions, and is a distance between feature points with little movement.
The second condition is the distance between feature points where a change similar to the movement of the vertical width of the lips is observed when the angle of the face changes in the vertical direction and the horizontal direction. When the face faces the front of the camera, the ratio of the "distance between the feature points" to the "distance between the camera and the discrimination target" is constant regardless of which pair of feature points is used. However, when the angle of the face changes, this ratio changes. For example, even when the distance between the camera and the discrimination target does not fluctuate, the width of the face (the length of the straight line connecting the feature points at the left and right edges of the face) when the discrimination target turns sideways. Fluctuates. On the other hand, the vertical feature amount of the lips and the nasal feature amount as in this embodiment have a substantially parallel relationship and are present in the center of the face. Therefore, since the tendency of the change in the distance between the feature points is similar when the angle of the face changes, the influence on the change in the angle of the face can be reduced by using the nasal feature amount.

In this embodiment, the first lip behavior feature is calculated by the vertical feature of the lips and the feature of the nose in the images (frames) at the same time, and is calculated for each of a plurality of consecutive images (frames) in chronological order. Will be done. Therefore, as shown in FIG. 8, it is possible to obtain a change in the first lip behavior feature amount with the passage of time (frame number).
It is also possible to exclude the part without utterance and target only the part with utterance.

Further, the first lip behavior feature amount may be normalized in the range of 0.0 or more and 1.0 or less in order to reduce individual differences in the movement of the lips of the speaker.

[Calculation process of voice features S30]
In the audio feature calculation process S30, the audio feature is calculated from the audio data (for example, FIG. 2B) obtained in the video / audio data acquisition process S10. As a result, it is possible to extract the voice data necessary for determining the speaker from the voice data including a large amount of complicated information, and to facilitate the handling of the data while maintaining the accuracy.

The voice feature amount is not particularly limited as long as the voice data necessary for determining the speaker can be extracted from the voice data as described above and the data can be easily handled while maintaining the accuracy. It is preferable to use the frequency cepstrum coefficient (MFCC). It is more preferable to use the 0th dimension. This is because it has the characteristics of low-frequency components that are useful for the features of speech recognition, and the low-dimensional (0th-dimensional) components change due to the acoustic characteristics of the vocal tract and the shape of the oral cavity. by.

As a more specific example, the voice features are obtained as follows. 9 and 10 are diagrams for explanation.
_First, a portion (part E 1) having a predetermined time length D is extracted from the audio data obtained in the video / audio data acquisition process S10 shown in FIG. 9 (a) as shown in FIG. 9 (b). The size of D is not particularly limited, but this example is 20 ms.
Then obtain the MFCC data such as shown in Figure 10 obtains the MFCC for audio data of the portion _{E 1.} The method of obtaining the MFCC is as known, and examples thereof include the contents described in "Tatsuya Kawahara, Revised 2nd Edition of Voice Recognition System, Ohmsha, 2016".
For example, it is calculated as follows. First, the voice data (voice waveform) is Fourier transformed to obtain a frequency component, and the power spectrum (loudness at each frequency component) is calculated using this frequency component. Next, this power spectrum is multiplied by a mel filter bank. Since human hearing has a characteristic that the resolution decreases as the frequency becomes higher, it is possible to extract a feature amount according to the human auditory characteristic by applying a mel filter bank. Then, the cepstrum feature amount is calculated from this, and the harmonic component of the voiceprint wave (characteristic that changes depending on the person) and the envelope component due to the vocal tract (characteristic that changes depending on the difference in the utterance content) are separated. The low-dimensional components (0th to 14th dimensions) in the cepstrum feature amount are mainly used for speech recognition, but as described above, it is preferable to use the 0th dimension in this embodiment.
The cepstrum features extracted in this way are called MFCCs and are used as voice features.

Then, as shown in FIG. 9A, the voice feature amount is similarly obtained for _{the portion E 2} (time length D) delayed by the time d with respect _{to the portion E 1.} By repeating this sequentially, a plurality of voice features are obtained in chronological order. The magnitude of the delay time d is not particularly limited, but any of D> d, D = d, and D <d may be used, but D> d from the viewpoint of improving accuracy. preferable. In this example, D is 20 ms, whereas d is 10 ms.
It is also possible to exclude the part without utterance and target only the part with utterance.

From the above, it is possible to acquire the time-series change of the voice feature amount, which is the 0th-dimensional numerical value of MFCC.
In addition, this voice feature amount may be normalized in the range of 0.0 or more and 1.0 or less within the utterance section in order to reduce individual differences in the loudness of the speaker's voice.

[Speaker discrimination process S40]
In the speaker discrimination process S40, the speaker is specified from the first lip behavior feature amount obtained in the first lip behavior feature calculation process S20 and the voice feature amount obtained in the voice feature calculation process S30. The flow is shown in FIG. The speaker discrimination process S40 of the present embodiment includes a section setting process S41, a second lip behavior feature amount calculation process S42, a section difference calculation process S43, and a speaker discrimination process S44. Each process will be described below.

<Section setting process S41>
In the section setting process S41, the video and audio data obtained in the video / audio data acquisition process S10 are divided into a plurality of "sections". FIG. 12 shows a diagram for explanation.
This "section" is the minimum unit for calculating the section difference between the first lip behavior feature amount and the second lip behavior feature amount obtained thereafter. The interval can be set, for example, 1000 ms. Then, a plurality of sections are set so as to slightly shift the time as the starting point, and section 1 to section M are considered as shown in FIG.
By using the section in this way, it is possible to discriminate the speaker in consideration of the time-series change of the movement of the lips, not in the so-called frame unit.

The deviation of the start point time in the adjacent sections is not particularly limited, and may be the same as the length of the section, shorter than the length of the section, or longer than the length of the section. However, this deviation of the start point time is preferably shorter than the length of the section as shown in FIG. 12, and can be, for example, about 0.1 times the length of the section (100 ms in this example). That is, in a plurality of sections, in adjacent sections, the time to be the starting point can be determined so that a part of the time overlaps.
By setting the sections so that some of them overlap in time in the adjacent sections in this way, it is possible to acquire more features of the pattern as compared with the case where they do not overlap.

The length of the section is set so that each section includes a plurality of first lip behavior features and a plurality of voice features. For example, when the length of one section is 1000 ms and the voice feature amount is created every 10 ms as in the above example (d in FIG. 9A is 10 ms), the voice feature amount data included in this section The number is 100. On the other hand, regarding the first lip behavior feature amount, since the image is 30 frames per second (1000 ms) with a normal camera, one first lip behavior feature amount is obtained per frame, so the number of data is It is thirty.

<Second lip behavior feature calculation process S42>
In the second lip behavior feature amount calculation process S42, the voice feature amount obtained in the voice feature amount calculation process S30 is input to the pre-learned neural network for each section set in the section setting process S41, and the second lip behavior feature amount calculation process S42. Calculate the second lip behavior feature. Therefore, by this process, the lip behavior feature amount based on the voice data can be obtained.

It is preferable that the number of data of the second lip behavior feature amount is the same as the number of data of the first lip behavior feature amount. That is, when the number of data of the voice feature amount included in one section is 100 and the number of data of the first lip behavior feature amount is 30 as described above, 30 is based on the data of 100 of the voice feature amount. The second lip behavior feature amount of the data of is calculated. This makes it easier to obtain the section difference described later.

Then, such a second lip behavior feature amount is calculated for each section.

Here, the conditions and methods of prior learning to the neural network are not particularly limited as long as the voice features can be associated with the lip behavior features, but in this embodiment, the following is performed.

A neural network with a three-layer structure consisting of an input layer, an intermediate layer, and an output layer is provided under the conditions that match the length of the above-mentioned section, the number of voice feature data included therein, and the number of data of the first lip behavior feature. Use to learn. For example, using the above example, the section length is 1000 ms, the input layer is 100 dimensions according to the number of voice feature data, the intermediate layer is 50 dimensions, and the output layer is the data of the first lip behavior feature. It was set to 30 dimensions so that 30 data could be output according to the number. Adam can be used for the gradient method.
Then, learning proceeds by comparing the output from this output layer with the teacher data.

<Calculation process of section difference S43>
In the section difference calculation process S43, the first lip behavior feature amount obtained in the calculation process S20 of the first lip behavior feature amount belonging to the section and the second lip behavior feature amount belonging to the section are calculated for each section. The difference from the second lip behavior feature amount obtained in the amount calculation process S42 is taken to obtain the interval difference. More specifically, it is as follows.

As described above, in this embodiment, the number of data of the lip behavior feature amount based on the image (first lip behavior feature amount) and the lip behavior feature amount based on the voice (second lip behavior feature amount) are matched. Therefore, the difference may be taken in order from the earliest time. Therefore, for example, 30 difference data can be obtained in one section.
Then, in this process, each difference data obtained is represented by an absolute value, and this is averaged to obtain the interval difference δ in the relevant section. Therefore, as shown in FIG. 12, section differences δ ₁ , δ ₂ , δ ₃ , ..., δ _M are obtained for each section in this process.

<Speaker discrimination process S44>
_{In the speaker discrimination process S44, the discrimination difference δ ave} is calculated by averaging the plurality of section differences obtained in the section difference calculation process S43, and among the discrimination target persons, the person having the smallest _{discrimination difference δ ave is selected.} Be the speaker.
This makes it possible to identify the speaker.

According to the above method, even with one omnidirectional camera and microphone, the speaker can be identified by processing the acquired utterance video, so the number of devices is increased according to the number of people. There is no need and it is convenient.
In addition, since the lip behavior feature is calculated using the voice feature and the difference is taken by comparing this with the lip behavior feature based on the image, the lips are moving simultaneously by multiple people. It is possible to identify the speaker even in the case, and it is possible to appropriately identify the speaker even when the movements of the lips occur at the same time.

{Speaker discrimination program and speaker discrimination device}
FIG. 13 is a diagram conceptually showing the configuration of the speaker discrimination device 50 according to one form in which a specific calculation is performed according to the speaker discrimination method S1 described above. The speaker determination device 50 includes an input device 57, an arithmetic unit 51, and a display means 58. The arithmetic unit 51 includes an arithmetic unit 52, a RAM 53, a storage unit 54, a receiving unit 55, and an output unit 56.

The calculation means 52 is composed of a so-called CPU (central operator), is a means that can be connected to each of the above-mentioned constituent members and can control them. Further, it is also the calculation means 52 that executes various programs stored in the storage means 54 or the like that functions as a storage medium, and based on this, performs a data creation calculation for each process of the speaker determination method S1 described above. be.

The RAM 53 is a component that functions as a work area of the calculation means 52 and a temporary data storage means. The RAM 53 can be composed of an SRAM, a DRAM, a flash memory, or the like, and is the same as a known RAM.

The storage means 54 is a member that functions as a storage medium for storing programs and data that are the basis of various operations. Further, the storage means 54 may be able to store various intermediate and final results obtained by executing the program. More specifically, the program is stored (stored) in the storage means 54. In addition, other information may also be stored.

Here, the stored program includes a program that is a basis for calculating each process of the speaker determination method S1 described above. That is, the speaker discrimination program replaces each process with each step so as to correspond to each process of the speaker discrimination method S1 shown in FIG. 1 (including each process shown in FIGS. 3 and 11). Includes steps. The specific calculation contents of this program are as described in the speaker determination method S1 described above.
Further, the storage means 54 may store a database based on the learned result of the neural network, which is the basis for calculating the second lip behavior feature from the voice feature. In this case, the above program proceeds by sequentially referring to this database.

The receiving means 55 is a component having a function for appropriately incorporating information from the outside into the arithmetic unit 51, and the input device 57 is connected to the receiving means 55. This includes so-called input ports, input connectors, and the like.

The output means 56 is a component having a function of appropriately outputting information to be output to the outside among the obtained results, and a display means 58 such as a monitor and various devices are connected thereto. This includes so-called output ports, output connectors, and the like.

The input device 57 includes a device that acquires the video and audio of the speaker. A typical device is a microphone, a camera, or a video camera with a microphone. However, the device may be a device that acquires the video and audio of another type of speaker limited to this. The information input from here is taken into the arithmetic unit 51, and the above program is executed using this information.

In addition, information may be provided to the arithmetic unit 51 via the receiving means 55 via a network or communication. Similarly, information may be transmitted to an external device via the output means 56 via a network or communication.

According to such a speaker discrimination device 50, the speaker discrimination method S1 described above can be performed efficiently and accurately. As such a speaker discrimination device 50, a computer can be used, for example.

{Speaker discrimination test}
The inventor conducted a test to actually identify the speaker. The conditions and test method are shown below.
-Camera: Omni-directional camera, THITA V, manufactured by RICOH-Microphone: TA-1, manufactured by RICOH-Lighting: Fluorescent lamp, illuminance 700 x to 900 lpx
・ Discrimination target: 2 people (A, B)
-Arrangement of the person to be discriminated: A position 50 cm away from the camera, a posture facing the front toward the camera

Based on the above conditions, the test was conducted as follows.
-Two discrimination subjects (A and B) read the same sentence aloud separately and recorded it with the above camera and microphone.
・ The sentences read aloud by the discrimination target were 11 types extracted from news articles. Therefore, a total of 22 video and audio data were obtained.
-From these 22 data, 20 was used as the teacher data for learning the neural network, and the remaining 2 data were used as the test data for division. The combination of the teacher data and the test data was changed and the same was performed with different division patterns, resulting in a total of 231 patterns.

With the above preparations, the test was conducted as follows. FIG. 14 shows a diagram for explanation.
(1) For 22 data (discrimination target person A: 11 data, discrimination target person B: 11 data), arbitrary 2 data are selected as test data, and the remaining 20 data are selected as teacher data.
(2) The neural network is learned using the teacher data (20 data), and the trained neural network is constructed.
(3) Using this trained neural network, the speaker is discriminated against the test data. At this time, as shown in FIG. 14, the speaker is determined for each of the two prepared test data (test data 1 in FIG. 14A and test data 2 in FIG. 14B).
That is, for the test data 1, as shown in FIG. 14A, the second lip behavior feature amount is obtained using the voice data of the test data 1. This is compared with the first lip behavior feature amount obtained from the video of test data 1 and the first lip behavior feature amount obtained from the video of test data 2, and δ _ave is calculated for each. In this case, the discrimination is successful when _{the δ-ave of the test data 1 is smaller.}
Similarly, for the test data 2, as shown in FIG. 14 (b), a second lip behavior feature amount is obtained using the voice data of the test data 2. This is compared with the first lip behavior feature amount obtained from the video of test data 1 and the first lip behavior feature amount obtained from the video of test data 2, and δ _ave is calculated for each. In this case, the discrimination is successful when _{the δ-ave of the test data 2 is smaller.}
(4) The combination of the test data and the teacher data was changed, and the above steps (1) to (3) were repeated, and the entire pattern was performed 231 times.
(5) Using the discrimination results of 462 times (231 × 2) obtained in the above (1) to (4), the discrimination success rate and the average value of _{δ-ave were calculated.}
In this example, the first lip behavior feature amount and the voice feature amount were tested for each of the case where the normalization was performed in the range of 0.0 or more and 1.0 or less and the case where the normalization was not performed.

As a result of the above, as will be shown later, when the above test was performed by changing the number of learnings in the neural network, a discrimination success rate of 79.2% to 83.8% could be obtained when normalized. rice field. On the other hand, even when normalization was not performed, a discrimination success rate of 78.1% to 82.5% could be obtained. Therefore, a high discrimination success rate can be obtained regardless of the presence or absence of normalization. However, it is possible to increase the discrimination success rate by normalizing. The discrimination success rate is the ratio of the number of success discriminations to the total number of discriminations expressed as a percentage.

As described above, in this test, the inventor investigated the relationship between the number of learnings in the neural network and the discrimination success rate. That is, the number of times of learning was changed (the number of times of learning), and the relationship with the success rate of speaker discrimination was investigated. The results are shown in Table 1. This is the result of changing the number of learning times from 500 times to 10000 times by 500 times. The test method is the same as above.
Further, “δ _ave ” in Table 1 is a value of the discrimination difference.

As can be seen from Table 1, in each case, the speaker can be identified with a high probability.

50 Speaker discrimination device

Claims (11)

It is a method of identifying the speaker from the video and audio data.
The process of obtaining a plurality of first lip behavior features in chronological order based on the distance between the upper lip and the lower lip of the subject from the acquired video, and
The process of obtaining a plurality of voice features in chronological order based on the acquired voice data,
Has a process of determining the speaker,
In the process of identifying the speaker,
The process of obtaining a plurality of second lip behavior features from a plurality of the voice features in chronological order,
The process of obtaining a discriminant difference, which is the difference between the first lip behavior feature amount and the second lip behavior feature amount,
A speaker discrimination method comprising a process of using a target person having the smallest discrimination difference among the target persons included in the video as a speaker. The speaker discrimination method according to claim 1, wherein the first lip behavior feature amount is obtained by the ratio of the distance between the upper lip and the lower lip and the distance between two points on the nose bridge of the subject. In the process of identifying the speaker,
The process of creating multiple sections with different starting points and a predetermined time range,
Claim that the section difference between the first lip behavior feature amount and the second lip behavior feature amount is obtained for each of the plurality of sections, and the average of the section differences in each section is used as the discrimination difference. The speaker determination method according to 1 or 2. The speaker determination method according to claim 3, wherein a time as a starting point is determined so that a part of the time overlaps in the adjacent sections in the plurality of sections. The speaker according to any one of claims 1 to 4, wherein the plurality of first lip behavior features and the plurality of voice features are normalized in the range of 0.0 or more and 1.0 or less. Discrimination method. A program that identifies the speaker from video and audio data.
A step of obtaining a plurality of first lip behavior features in chronological order based on the distance between the upper lip and the lower lip of the subject from the acquired video.
A step of obtaining a plurality of voice features in chronological order based on the acquired voice data,
Has a step to determine the speaker, and
In the step of determining the speaker,
A step of obtaining a plurality of second lip behavior features from a plurality of the voice features in chronological order,
A step of obtaining a discriminant difference which is a difference between the first lip behavior feature amount and the second lip behavior feature amount.
A speaker discrimination program including a step of setting a target person having the smallest discrimination difference among the target persons included in the video as a speaker. The speaker discrimination program according to claim 6, wherein the first lip behavior feature amount is obtained by the ratio of the distance between the upper lip and the lower lip and the distance between two points on the nose bridge of the subject. In the step of determining the speaker,
A step to create multiple sections with different starting points and a predetermined time range,
For each of the plurality of sections, the section difference between the first lip behavior feature amount and the second lip behavior feature amount is obtained, the average of the section differences in each section is obtained, and this is referred to as the discrimination difference. The speaker determination program according to claim 6 or 7. The speaker determination program according to claim 8, wherein in the plurality of sections, the time serving as the starting point is determined so that a part of the time overlaps in the adjacent sections. The speaker discrimination program according to any one of claims 6 to 9, wherein the plurality of first lip behavior features and the plurality of voice features are normalized in the range of 0.0 or more and 1.0 or less. A device that identifies the speaker from video and audio data.
The camera that acquires the video and
The microphone that acquires the voice data and
It has a storage means in which the speaker discrimination program according to any one of claims 6 to 10 is stored, and a calculation means for performing a calculation based on the speaker discrimination program.
The calculation means acquires the video acquired by the camera and the voice data acquired by the microphone, and uses the acquired video and the voice data to perform a calculation by the speaker discrimination program to determine the speaker. Device.

Images