JP2000338987A - Utterance start monitor, speaker identification device, voice input system, speaker identification system and communication system - Google Patents

Abstract

PROBLEM TO BE SOLVED: To accurately, rapidly and easily decide whether an intended person is speaking or not, without imposing an unnecessary burden on a speaker, in surroundings wherein voice of others or sound from audio equipment or the like exists. SOLUTION: A deformation degree of lips is automatically acquired from an image of an object person 1 picked up by an object person imaging part 2, while an envelope curve of a voice is automatically acquired from the voice recorded by a sound recording part 3. A pattern matching part 8 calculates a similarity between time-series change patterns of the deformation degree of the lips and the envelope curve of the voice to identify a speaker. Thereby, the speaker can be identified without imposing an unnecessary burden on the object person 1, with a small information-processing load, at a high identification rate.

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention uses information about the lip movement of a subject whose speech is to be detected and voice information at the same time, so that the information processing load for identifying the speaker is small and the identification rate is low. The present invention proposes a speaker identification device with good sound quality. The speaker identification device can be used as a voice input system that uses voice input instead of key operation used for an in-vehicle navigation system, and video and audio used for a TV conference. The present invention relates to an utterance start monitoring device, a speaker identification device, a voice input system, a speaker identification system, and a communication system applied to a communication system that communicates with a communication device.

[0002]

2. Description of the Related Art For the purpose of speaker identification, there has been no method that simultaneously uses information about lip movement and voice information and uses the similarity of the time-series change for determination.
A conventional technique having similar components is disclosed in
Japanese Patent Application Publication No. 126126/1990 discloses a printing input device. With this device, the speaker is identified, and not only the sound from the microphone but also the opening and closing of the lips is determined, and it is distinguished by only the sound such as "M" and "N" or "P" and "T". This is to increase the recognition rate of sounds that are difficult to produce, and is applied in a state where the speaker has already been specified.

[0003] In addition, in the speech recognition apparatuses disclosed in Japanese Patent Application Laid-Open Nos. 60-196799 and 60-243700, sound waves are radiated to the lips of a speaker, and the sound waves enter the lips of the speaker. By using the reverberation sound generated in addition to the voice, the recognition rate of a sound that is difficult to be distinguished by the voice alone is raised, as in the case of the printing input device for printing. This is also applied when the speaker is already specified, and can be used only in limited cases such as when the speaker speaks into the microphone.

In a speaker identification device to which the present invention is applied, a switch indicating the fact is pressed when uttering.
In some cases, such as turning off the switch when the utterance is completed, the speaker is required to perform a secondary operation other than the utterance.

[0005]

Since the conventional speaker identification device is configured as described above, it is possible to generate a sound from a person other than the person who intends to operate, for example, a sound from another person or audio equipment. In an environment where there is a flowing sound, it is possible to determine whether the intended person is speaking or who among multiple candidates is speaking accurately, in a short time, easily, and to the speaker. There has been a problem that determination cannot be made without imposing an extra burden.

SUMMARY OF THE INVENTION The present invention has been made to solve the above-described problem, and it is possible to accurately, in a short time, simply and easily determine who is speaking among a plurality of candidates. It is an object of the present invention to obtain a speech start monitoring device, a speaker identification device, a voice input system, a speaker identification system, and a communication system that can make a determination without applying an extra burden.

[0007]

An utterance start monitoring apparatus according to the present invention comprises: a lip image photographing means for photographing an image of a lip portion of a subject whose utterance is to be detected; Lip movement time series data calculating means for obtaining series change data, voice extracting means for extracting voice from environmental sounds,
Speech time-series data calculation means for obtaining time-series change data of the extracted voice, and time-change similarity calculation means for obtaining a time-related similarity between the time-series change data of the lip movement and the time-series change data of the voice When the obtained time-series change data of the lip movement and the time-series change data of the voice match the stored words corresponding to the utterance start, and the obtained similarity is equal to or greater than a predetermined value, the utterance will be Utterance start recognition means for recognizing the start.

The utterance start monitoring apparatus according to the present invention comprises a lip image photographing means for photographing an image of a lip portion of a subject whose utterance is to be detected, and a lip for obtaining time series change data of lip movement from the photographed image. Exercise time series data calculation means, voice extraction means for extracting voice from environmental sounds, voice time series data calculation means for obtaining time series change data of the extracted voice, time series change data of lip movement and voice Time change similarity calculating means for calculating a similarity accompanying a time change with the time series change data, and a word corresponding to the operation content in which the obtained time series change data of the lip movement and the time series change data of the voice are stored And an operation content recognizing means for recognizing that the operation content is instructed when the similarity obtained is equal to or greater than a predetermined value.

A lip image photographing means for photographing an image of a lip portion of a subject whose speech is to be detected, and a lip for obtaining time-series change data of lip movement from the photographed image. Exercise time series data calculation means, voice extraction means for extracting voice from environmental sounds, voice time series data calculation means for obtaining time series change data of the extracted voice, time series change data of lip movement and voice Time change similarity calculating means for obtaining a similarity accompanying a time change with the time-series change data, and a speaker who determines that the sound is uttered by the subject when the obtained similarity is equal to or greater than a predetermined value. Judgment means.

In the speaker identification apparatus according to the present invention, the lip movement time series data calculation means quantifies the lip movement according to the difference between the maximum value and the small value of the vertical direction of the lip contour of the subject. It is something to do.

In the speaker identification device according to the present invention, the lip movement time series data calculating means quantifies the lip movement according to the difference between the horizontal maximum value and the minimum value of the lip contour of the subject. It is like that.

In the speaker identification apparatus according to the present invention, the lip movement time-series data calculating means calculates the difference between the maximum value and the small value in the vertical direction of the lip contour of the subject and the lip contour of the subject. The lip movement is quantified according to the difference between the maximum value and the minimum value in the horizontal direction.

[0013] In the speaker identification apparatus according to the present invention, the lip movement time series data calculation means quantifies the lip movement according to the curvature of at least one of the upper and lower lips of the subject. It is.

In the speaker identification apparatus according to the present invention, the speech time series data calculation means quantifies a time series change of speech according to an envelope of an output of a speech detection filter of a specific frequency. is there.

In the speaker identification apparatus according to the present invention, the time change similarity calculating means corrects the respective delay times of the time series change data of the lip movement and the time series change data of the voice to mutually correct the time series data. The correlation coefficient is obtained, and the delay time of the cross-correlation coefficient within a certain time around zero is determined by the speaker determination means as the similarity to be evaluated.

A speech input system according to the present invention includes a speech recognition unit that applies a speaker identification device and performs processing in accordance with a speech input when a speaker is determined to be a subject. .

A speaker identification system according to the present invention targets a plurality of persons and uses a plurality of windows from a lip image of each person photographed by a plurality of cameras or one photographed image. Using the clipped lip image of each person,
A speaker identification device is applied to identify who uttered from each person.

In the communication system according to the present invention, a speaker identification system is applied to a plurality of participating persons as subjects, a speaker among the plurality of persons is identified, and a video to be communicated is uttered. Communication data control means for converting an image from a camera zoomed in on the user.

The communication system according to the present invention has a three-dimensional sound localization function, and identifies a speaker among a plurality of persons by applying a speaker identification system to a plurality of participating persons. And a communication data control means for calculating a virtual sound generation position in accordance with the position of the speaker and the position of the listener on the receiving side, and presenting a three-dimensional sound of the speaker. .

[0020]

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS One embodiment of the present invention will be described below. Embodiment 1 FIG. FIG. 1 is a block diagram showing a speaker identification apparatus according to Embodiment 1 of the present invention. In the figure, 1 is a subject, 2 is a subject photographing section (lip image photographing means), and 3 is a sound recording section ( Voice extraction means). 4 is the subject photographing unit 2
Motion processing unit for performing a clipping process of the lips of the subject 1 from the image captured by the lip sensor 5 and a lip motion time series for obtaining time-series change data of lip movements from the clipped image It is a data calculation unit (lip motion time series data calculation means). Reference numeral 6 denotes a sound detection filter (sound extracting means) for passing only the frequency corresponding to the sound from the environmental sound recorded by the sound recording unit 3, and reference numeral 7 denotes a sound for obtaining time-series change data of the sound passing through the sound detection filter 6. It is a sequence data calculation unit (voice time-series data calculation means).

Reference numeral 8 denotes a similarity accompanied by a time change between the time series change data of the lip movement obtained by the lip movement time series data calculation unit 5 and the time series change data of the voice obtained by the voice time series data calculation unit 7. A time-series pattern matching unit (time-change similarity calculation means) 9 for calculating the degree is a sound uttered from the subject 1 when the similarity calculated by the time-series pattern matching unit 8 is equal to or more than a predetermined value. Speaker determination unit (speaker determination means, utterance start recognition means, operation content recognition means).

Next, the operation will be described. The subject photographing section 2 such as a camera photographs the subject 1 and outputs the photographed moving image to the cutout processing section 4. Cutout processing unit 4
Cuts out the lips of the subject 1 for each frame from the input moving image, and outputs the cut-out lip data to the lip movement time-series data calculation unit 5. A method using color information, a method using feature points of a face, and the like, which are conventionally used, are applied to the cutout processing unit 4. The lip movement time-series data calculation unit 5 obtains a lip-shaped characteristic parameter from the lip data and outputs the time-series change data. FIG. 2 is an explanatory diagram showing how to obtain the characteristic parameters of the lip shape.
More specifically, FIG.
As shown in FIG. 4, the distance X between the left and right ends of the lips and the distance Y between the upper and lower lips are used to determine the reference values (X ₀ , Y ₀ ) of these two parameters (X, Y) when not speaking. The norm is determined by the following equation (1) as the degree of deformation M of the lips. M (t) = ((X (t) −X ₀ ) ² + (Y (t) −Y ₀ ) ² ) ^1/2 (1) where M, X, and Y are sampling rates of an image frame rate. Where t = nΔt, where n is an integer and Δt is a sampling interval.

On the other hand, the sound recording section 3 such as a microphone records the environmental sound and outputs the recorded environmental sound to the sound detection filter 6. The voice detection filter 6 is a frequency band filter corresponding to voice, and when the gender or age of the subject is specified, a frequency band filter corresponding to the gender or age is provided. The voice detection filter 6 allows only the frequency corresponding to the voice to pass from the input environmental sound,
Output to the audio time-series data calculation unit 7. Then, the voice time-series data calculation unit 7 obtains an envelope S of the input voice, and sets the envelope S as time-series change data. The envelope S is obtained at the same sampling interval Δt as the lip deformation M by adjusting the time by adjusting the delay of the image processing system of the lip movement.

The time-series pattern matching unit 8 compares the degree of lip deformation M obtained by the lip movement time-series data calculation unit 5 with the speech envelope S obtained by the voice time-series data calculation unit 7 in the past. Using data for a fixed time T,
The cross-correlation coefficient R as the similarity is obtained by the following equation (2).

(Equation 1) Here, m is a maximum integer of m ≦ T / Δt, and T is an allowable time required for speaker identification. For example, from the utterance,
If it is sufficient to be identified after one second, T = 1 sec.

The speaker determination unit 9 calculates the evaluation value D for determination as the weighted sum of the coefficients within a fixed delay time around the delay time zero (τ = 0) of the cross-correlation coefficient R by the following equation (3). ).

(Equation 2) Here, F is a weighting function that takes the maximum value at i = 0, for example, F (0) = 0.5, F (± 1) = 0.25, F (i
<-1) = 0, F (i> 1) = 0 and the like. When the evaluation value D is equal to or greater than a preset threshold value D ₀ (D ≧ D ₀ )
Then, it is determined that the target person is a speaker. R (t, τ) is
It takes a value between -1 and 1, and becomes closer to 1 as the similarity between M (t) and S (t) is higher and there is no time delay.
Although the time delay of the processing system has been adjusted, it is considered that there is an essential fluctuation of the time relationship between the movement of the lips and the voice, so the weighting around the zero delay time (τ = O) is performed. Use the sum.

Although the first embodiment has been described as a speaker identification device for determining who is speaking among a plurality of candidates, an utterance start monitoring device for giving an utterance start instruction by utterance It can also be applied to In FIG. 1, when the present invention is applied to the utterance start monitoring device, the utterer determination unit (utterance start recognition means) 9 causes a word corresponding to the start of utterance in advance (for example, start, utterance start, operation start, operation instruction, etc.).
The lip movement time series change data and the voice time series change data are stored, and the lip movement time series change data and the voice time series data calculation section 7 obtained by the lip movement time series data calculation section 5 are stored. It is determined whether or not the time-series change data of the voice obtained by the above-described method matches the stored word corresponding to the start of the utterance, and the matching is recognized, and the similarity obtained by the time-series pattern matching unit 8 is determined by a predetermined value. When the value is equal to or larger than the value, the user recognizes that the utterance is about to start. With this configuration, in the conventional device, when the subject starts speaking in a noisy environment (eg, vehicle interior noise, speech of another person, audio sound, etc.), the user presses a switch indicating that fact. The utterance had to be started or the utterance had to be started after the other person's conversation had been interrupted, and the target person was required to perform a secondary operation other than the utterance. If the target person speaks words corresponding to the start of the utterance registered in advance, the apparatus can recognize that the utterance is about to start from the apparatus side. Troublesome operations can be eliminated.

Further, in the above utterance start monitoring device, the utterance start instruction is given by utterance. However, the instruction of the operation content is given by utterance so that the utterance start and the instruction of the operation content are given simultaneously. Can also. In FIG. 1, when this utterance start monitoring device is applied,
The utterer determination unit (operation content recognizing means) 9 provides time-series change data of lip movement and voice of words (eg, audio on, audio off, audio volume up, navigation system on, etc.) according to the operation content in advance. The time series change data is stored, and the time series change data of the lip movement obtained by the lip movement time series data calculation unit 5 and the time series change data of the voice obtained by the voice time series data calculation unit 7 are stored in the memory. When it is determined whether or not the word matches the word corresponding to the stored operation content, the match is recognized, and when the similarity obtained by the time-series pattern matching unit 8 is equal to or greater than a predetermined value, the utterance is started and the operation is started. Recognize that the contents have been indicated. With this configuration, in the utterance start monitoring device, if the subject speaks a word corresponding to the operation content registered in advance, the utterance is started on the device side and the operation content is instructed. Can be recognized,
Since the annoyance of secondary operations other than utterance to the target person can be eliminated, and the start of utterance and the instruction of the operation content can be simultaneously performed, the instruction of the operation content by utterance can be easily performed. it can. The utterer determination unit (utterance start recognition unit, operation content recognition unit) 9 obtains the lip movement time series change data obtained by the lip movement time series data calculation unit 5 and the voice time series data calculation unit 7. It is determined whether the time-series change data of the received voice matches the stored utterance or the word corresponding to the operation content. However, these determinations may be performed by the time-series pattern matching unit 8. The exercise time-series data calculation unit 5 and the voice time-series data calculation unit 7 may individually determine.

In the first embodiment, the contours of the subject's face and lips are almost bilaterally symmetric, but the axis of symmetry is the vertical direction, and the direction perpendicular to the vertical direction is the horizontal direction. In the case, the lip movement time series data calculation unit 5
In accordance with at least one of the difference between the maximum value and the minimum value of the contour of the lip of the subject in the vertical direction and the difference between the maximum value and the minimum value of the contour of the lip of the subject in the horizontal direction Lip movement may be quantified,
The movement of the lips may be quantified according to the curvature of at least one of the upper and lower lips of the subject. Further, the distance (opening distance) between the lower end of the upper lip and the upper end of the lower lip may be used. Further, the moving average value of the sound output from the sound detection filter 6 may be obtained by the sound time-series data calculation unit 7. Further, the frame rate of the image obtained by the subject photographing section 2 is used as a processing unit. However, if interpolation and sampling are performed on the series change data by the lip movement time series data calculation section 5, there is no need to be concerned with the frame rate. Further, although the delay time is adjusted in the audio data system, when calculating the cross-correlation, the difference between the delays in both systems may be considered by the weight function F of the equation (3) used in the speaker determination unit 9. good. Further, various operations are obtained by digital calculation, but may be performed by an analog circuit by combining a delay circuit, a multiplier, and an integrator.

As described above, according to the first embodiment, the degree of lip deformation of the subject 1 and the envelope of the voice are simultaneously used, and the speaker is identified based on the similarity of the time-series change pattern. By doing so, it is possible to obtain an effect that a speaker can be identified with a low information processing load and a high identification rate. In addition, the degree of lip deformation can be automatically obtained from the lip portion extracted from the image of the subject 1 captured by the subject imaging unit 2, and the audio envelope is recorded by the sound recording unit 3. Since the speech can be automatically obtained from the speech that has passed through the speech detection filter 6, the speaker can be identified without imposing an extra burden on the subject 1.

Embodiment 2 FIG. FIG. 3 is a block diagram showing a voice input system according to Embodiment 2 of the present invention.
In the figure, reference numeral 31 denotes an audio data buffer for temporarily storing audio data that has passed through the audio detection filter 6, and reference numeral 32 denotes a case where the speaker determination unit 9 determines that the audio data is uttered from the subject 1 An audio gate 33 that opens the gate at 33 recognizes the meaning of the audio data input from the audio data buffer 31 through the audio gate 32,
It is a speech recognition unit (speech recognition means) that is processed according to its meaning. Other configurations are the same as those in FIG.

Next, the operation will be described. The second embodiment relates to a voice input system using a voice input instead of a key operation used in an in-vehicle navigation system or the like.
Operations 9 to 9 are the same as those described in the first embodiment.
Audio data buffer 31, temporarily holds the voice data which has passed through the speech detection filter 6 always outputs the audio data obtained by delaying T _a time period obtained by adding the delay time of the processing system to the time T required for the determination of the utterance. The voice gate 32 opens the gate when the speaker 1 determines that the target person 1 is the speaker, sends the voice data output from the voice data buffer 31 to the voice recognition unit 33, and performs the voice recognition. The unit 33 processes the input as a voice operation input.

As described above, according to the second embodiment, a sound uttered by a person other than the person who intends to operate, for example, a sound uttered by another person or a sound flowing from an audio device or the like, is mistaken for an operation input. The effect that the processing can be prevented can be obtained.

Embodiment 3 FIG. 4 is a block diagram showing a speaker identification system according to Embodiment 3 of the present invention.
In the figure, reference numeral 41 denotes a candidate having the largest evaluation value among the evaluation values D output from the speaker determination units 9a to 9c as an utterance candidate, and the evaluation value D of the utterance candidate is set to a predetermined threshold. in the case of D ₁ or more, the utterance candidate is the determining speaker determination unit is speaker. In the third embodiment, the speaker identification devices of the a-system to the c-system are provided for the three subjects 1a to 1c. Is the same as that in FIG.

Next, the operation will be described. The operation in the speaker identification device of the a-system to the c-system is the same as that of the first embodiment, and the speaker identification devices of the a-system to the c-system are processed in parallel for three subjects 1a to 1c. And the speaker determination units 9a to 9
From c, each evaluation value D is output. The utterer determination unit 41 compares the evaluation values D output from the utterer determination units 9a to 9c, determines the target person having the largest evaluation value as the utterance candidate, and further, evaluates the evaluation value D of the utterance candidate. Is greater than or equal to a preset threshold D ₁ (D ≧ D ₁ ), it is determined that the utterance candidate is the utterer.

In the third embodiment, the subject photographing units 2a to 2c are provided for each speaker identification device. However, one subject photographing unit 2 is used for a plurality of subjects 1a to 1c. The cut-out processing unit 4 may cut out and process the lips of each of the subjects 1a to 1c. Also, the sound recording units 3a to
3c is not provided for each speaker identification device, and a plurality of subjects 1a to
One sound recording unit 3 may be used for 1c.

As described above, according to the third embodiment, it is possible to obtain an effect that a speaker can be identified from a plurality of speech candidates.

Embodiment 4 FIG. FIG. 5 is a block diagram showing a communication system according to Embodiment 4 of the present invention. In the figure, reference numeral 51 denotes a speaker photographing unit for speaker identification, 52 denotes a sound recording unit, 53 denotes a speaker identification unit, and 54 denotes a speaker identification unit. A transmission data control unit (communication data control means);
Is a reception data control unit (communication data control means), 57 is a video presentation unit, and 58 is a voice presentation unit. Subsystems 51 to 58 are provided with the same subsystem at two points having a communication system capable of mutual communication.

Next, the operation will be described. The fourth embodiment relates to a communication system for communicating video and audio used in a TV conference or the like, in which a speaker is determined from a plurality of subjects, and video and audio information to be communicated is controlled. This is a system that can increase the sense of presence. The utterer identification unit 53 performs the same operation as in the third embodiment using the video of the target person imaging unit 51 for utterer identification and the sound data of the sound recording unit 52 to determine the presence of the utterance and the utterer. judge.

In the transmission data control unit 54, when the speaker identification unit 53 determines the presence of an utterance, the transmission image photographing unit 55 is aimed at the determined speaker and zooms in. on the other hand,
When the speaker identification unit 53 does not determine the presence of the utterance, the transmission video imaging unit 55 is zoomed out or panned so that the entire video is captured. In the transmission data control unit 54, the video captured by the transmission video capturing unit 55
The sound data is transmitted according to the presence or absence of the speaker and the position of the speaker in the virtual space as shown in FIG.

On the other hand, the reception data control unit 56 that has received the transmission data presents the received video to the video presentation unit 57, and processes sound data according to the presence or absence of speech. Specifically, when there is no utterance, the received sound data is sent to the voice presenting unit 58 as it is. If there is an utterance, voice data is extracted by the voice detection filter 6 described in the first embodiment, and the other voice data is set as environmental sound. Regarding the voice data, a three-dimensional localized sound is created in consideration of the positional relationship between the speaker and the listener in the virtual space, and the environmental sound data is synthesized.
It is sent to the voice presentation unit 58. For example, in FIG. 6, when the speaker is identified as A1, it is presented to B1 as a sound that can be heard from the front, and to B2 it is presented so as to be heard from the right front.

As described above, according to the fourth embodiment, a speaker is identified from a plurality of utterance candidates, and is switched to a video of the utterer or zoomed in on the utterer. A communication system with a high feeling can be provided. Further, by controlling the direction in which the speaker's voice can be heard, an effect of realizing more realistic virtual presence communication can be obtained.

[0042]

As described above, according to the present invention, a lip image photographing means for photographing an image of a lip portion of a subject whose utterance is to be detected, and time-series change data of lip movement from the photographed image Lip movement time series data calculation means for obtaining the sound, speech extraction means for extracting sound from the environmental sound, voice time series data calculation means for obtaining the time series change data of the extracted sound, and lip movement time series change data Time-similarity calculating means for obtaining a time-based similarity between the lip movement and the time-series change data of the voice, and a utterance start in which the obtained time-series change data of the lip movement and the time-series change data of the voice are stored When the target word matches and the obtained similarity is equal to or greater than a predetermined value, the utterance start recognition means for recognizing that the utterance is to be started from now is provided. Utterance If the user speaks the word at the beginning, the device can recognize that the utterance is about to be started from the target person, so that the target person can be less troublesome by performing a secondary operation other than the utterance. The effect that can be obtained is obtained.

According to the present invention, a lip image photographing means for photographing an image of a lip portion of a subject whose speech is to be detected, and lip movement time series data for obtaining time series change data of lip movement from the photographed image Calculating means, voice extracting means for extracting voice from environmental sounds, voice time series data calculating means for obtaining time series change data of the extracted voice, time series change data of lip movement and time series change data of voice Time change similarity calculating means for calculating a similarity with time change with time, the obtained time series change data of the movement of the lips and the time series change data of the voice coincide with the words corresponding to the stored operation contents,
And when the obtained similarity is equal to or more than a predetermined value, the operation content recognizing means for recognizing that the operation content has been instructed. When the user speaks, the device can recognize that the speech has started and the operation content has been instructed, so that the troublesomeness of the subject person due to the secondary operation other than the speech can be eliminated, and Since the start of the utterance and the instruction of the operation content can be performed at the same time, the effect that the operation content can be easily instructed by the utterance can be obtained.

According to the present invention, lip image photographing means for photographing an image of a lip portion of a subject whose utterance is to be detected, and lip movement time series data for obtaining lip movement time series change data from the photographed image Calculating means, voice extracting means for extracting voice from environmental sounds, voice time series data calculating means for obtaining time series change data of the extracted voice, time series change data of lip movement and time series change data of voice Time-similarity calculating means for obtaining a similarity accompanied by a time change with, and speaker determining means for determining that the voice is uttered from the subject when the obtained similarity is a predetermined value or more. Since it is configured to use, the time series change data of the subject's lip movement and the time series change data of the voice are used at the same time, and the speaker is identified by their similarity, thereby reducing the information processing load. And effect can be obtained that can perform the identification of good speaker identification rate. In addition, the time-series change data of the lip movement can be automatically obtained from the lip portion extracted from the image of the subject taken by the lip image photographing means. Since the speech can be automatically obtained from the voice extracted by the extraction means, an effect is obtained that the speaker can be identified without imposing an extra burden on the subject.

According to the present invention, the lip movement time series data calculation means is configured to quantify the lip movement according to the difference between the maximum value and the minimum value in the vertical direction of the lip contour of the subject. As a result, an effect that the information processing load is small, and the speaker can be easily identified with a high identification rate can be obtained.

According to the present invention, the lip movement time series data calculation means is configured to quantify the lip movement according to the difference between the maximum value and the minimum value in the horizontal direction of the lip contour of the subject. Therefore, it is possible to obtain an effect that the speaker can be easily identified with a small information processing load and with a high identification rate.

According to the present invention, the distance between the upper and lower lips due to the difference between the maximum value and the minimum value in the vertical direction of the contour of the lip of the subject and the lip of the subject are calculated by the lip movement time series data calculating means. Since the lip movement is quantified according to the difference between the maximum and minimum values in the horizontal direction of the contour, it is possible to identify the speaker easily with a low information processing load and with a high identification rate. The effect that can be obtained is obtained.

According to the present invention, the lip movement time series data calculation means is configured to quantify the lip movement according to the curvature of at least one of the upper and lower lips of the subject. The effect that the load can be reduced and the speaker can be easily identified with a high identification rate can be obtained.

According to the present invention, the audio time-series data calculating means is configured to quantify the time-series change of the audio according to the envelope of the output of the audio detection filter of a specific frequency. A malfunction due to accidental synchronization with sound can be avoided, and the information processing load is reduced.
The effect that the speaker can be easily identified with a high identification rate can be obtained.

According to the present invention, the cross-correlation coefficient of the time-series data obtained by correcting the respective delay times of the time-series change data of the lip movement and the time-series change data of the voice by the time change similarity calculating means is obtained. The delay time of the cross-correlation coefficient is determined by the speaker determination means so that the cross-correlation coefficient within a certain time around zero is the similarity to be evaluated. Thus, an effect that the speaker can be identified with good identification rate can be obtained.

According to the present invention, the speaker identification device is applied, and the apparatus is provided with the voice recognition means for processing in response to the voice input when the speaker is determined to be the target person. Sounds other than those who intend
This provides an effect of preventing erroneous processing of an utterance of another person or a sound flowing from an audio device or the like as an operation input.

According to the present invention, a plurality of persons are targeted, and each person is cut out from a lip image of each person photographed by a plurality of cameras or a single photographed image using a plurality of windows. By using a lip image and applying a speaker identification device to identify who uttered, it was possible to identify the speaker from among multiple utterance candidates. The effect that can be obtained is obtained.

According to the present invention, a speaker identification system is applied to a plurality of participating persons as subjects, a speaker among the plurality of persons is identified, and an image to be communicated is given to the speaker. Since the communication data control means for providing the image from the zoomed-in camera is provided, an effect that a communication system having a high sense of reality can be obtained can be obtained.

According to the present invention, a speaker identification system is provided with a three-dimensional sound localization function, a plurality of participating persons as subjects, and a speaker among the plurality of persons is identified. According to the communication data control means for calculating a virtual sound generation position in accordance with the position of the speaker and the position of the listener on the receiving side and presenting a three-dimensional sound from the speaker, By controlling the direction in which the speaker's voice can be heard, an effect of realizing more realistic virtual presence communication can be obtained.

[Brief description of the drawings]

FIG. 1 is a block diagram showing a speaker identification device according to a first embodiment of the present invention.

FIG. 2 is an explanatory diagram showing how to obtain a lip-shaped feature parameter.

FIG. 3 is a block diagram showing a voice input system according to a second embodiment of the present invention.

FIG. 4 is a block diagram showing a speaker identification system according to Embodiment 3 of the present invention.

FIG. 5 is a block diagram showing a communication system according to a fourth embodiment of the present invention.

FIG. 6 is an explanatory diagram showing a positional relationship between a speaker and a listener in a virtual space.

[Explanation of symbols]

1 subject person, 2 subject person photographing unit (lip image photographing means),
3 sound recording unit (sound extraction means), 4 clipping processing unit (lip image photographing means), 5 lip movement time series data calculation unit (lip movement time series data calculation means), 6 sound detection filter (sound extraction means), 7 voice time series data calculation section (voice time series data calculation means), 8 time series pattern matching section (time change similarity calculation means), 9 speaker determination section (speaker determination means, speech start recognition means, operation content recognition) Means), 33 voice recognition unit (voice recognition unit), 54 transmission data control unit (communication data control unit), 56 reception data control unit (communication data control unit).

 ──────────────────────────────────────────────────続 き Continuing on the front page (72) Inventor Yamasaki Saki ▼ Hidenori 2-3-2 Marunouchi, Chiyoda-ku, Tokyo Inside Mitsubishi Electric Corporation (72) Inventor Akira Sawada 2-2-2 Marunouchi, Chiyoda-ku, Tokyo 3 No. 3 Mitsubishi Electric Corporation (72) Inventor Hiromi Terashita 2-3-2 Marunouchi, Chiyoda-ku, Tokyo F-term (reference) 5D015 AA03 DD03 HH01 HH04 LL07 9A001 GG05 HH16 HH21

Claims (13)

[Claims] 1. A lip image photographing means for photographing an image of a lip portion of a subject whose utterance is to be detected, and a lip movement for obtaining time-series change data of lip movement from an image photographed by the lip image photographing means. Time-series data calculation means, voice extraction means for extracting voice from environmental sounds, voice time-series data calculation means for obtaining time-series change data of the voice extracted by the voice extraction means, and lip movement time-series data calculation Time change similarity calculating means for calculating a similarity involving a time change between the time series change data of the lip movement obtained by the means and the time series change data of the voice obtained by the voice time series data calculating means; The time-series change data of the lip movement of the word and the time-series change data of the voice according to the start of the utterance are stored, and the lip movement obtained by the lip movement time-series data calculation means is calculated. The time-series change data of the speech and the time-series change data of the voice obtained by the voice time-series data calculation means coincide with the stored words corresponding to the start of the utterance, and are obtained by the time change similarity calculation means. An utterance start monitoring device for recognizing that utterance is to be started when the similarity is equal to or greater than a predetermined value. 2. A lip image photographing means for photographing an image of a lip portion of a subject whose utterance is to be detected, and a lip movement for obtaining time-series change data of lip movement from an image photographed by the lip image photographing means. Time-series data calculation means, voice extraction means for extracting voice from environmental sounds, voice time-series data calculation means for obtaining time-series change data of the voice extracted by the voice extraction means, and lip movement time-series data calculation Time change similarity calculating means for calculating a similarity involving a time change between the time series change data of the lip movement obtained by the means and the time series change data of the voice obtained by the voice time series data calculating means; The time-series change data of the lip movement and the time-series change data of the voice according to the operation content are stored, and the lip movement obtained by the lip movement time-series data calculating means is stored. And the time-series change data of the voice determined by the voice-time-series data calculating means coincide with the word corresponding to the stored operation content, and determined by the time-variation similarity calculating means. An utterance start monitoring device comprising: an operation content recognizing means for recognizing that the operation content has been instructed when the similarity is equal to or more than a predetermined value. 3. A lip image photographing means for photographing an image of a lip portion of a subject whose speech is to be detected, and a lip movement for obtaining time-series change data of lip movement from an image photographed by the lip image photographing means. Time-series data calculation means, voice extraction means for extracting voice from environmental sounds, voice time-series data calculation means for obtaining time-series change data of the voice extracted by the voice extraction means, and lip movement time-series data calculation Time change similarity calculating means for calculating a similarity with a time change between the time series change data of the lip movement obtained by the means and the time series change data of the voice obtained by the voice time series data calculating means; Speaker determining means for determining that the sound is from the subject when the similarity calculated by the time change similarity calculating means is equal to or more than a predetermined value. Identification device. 4. The lip movement time-series data calculation means quantifies the lip movement according to a difference between a maximum value and a small value in the vertical direction of the lip contour of the subject. 3. The speaker identification device according to 3. 5. The lip movement time-series data calculation means quantifies lip movement according to a difference between a maximum value and a minimum value in the horizontal direction of the lip contour of the subject. The described speaker identification device. 6. The lip movement time-series data calculating means calculates a difference between a maximum value and a minimum value of the contour of the subject's lip in the vertical direction and a maximum value and a minimum value of the horizontal direction of the contour of the subject's lip. 4. The speaker identification device according to claim 3, wherein the movement of the lips is quantified according to a difference from the value. 7. The speaker according to claim 3, wherein the lip movement time series data calculation means quantifies the lip movement according to the curvature of at least one of the upper and lower lips of the subject. Identification device. 8. The speaker identification according to claim 3, wherein the speech time series data calculation means quantifies a time series change of the speech according to an envelope of an output of the speech detection filter of a specific frequency. apparatus. 9. The time change similarity calculating means obtains a cross-correlation coefficient of time series data obtained by correcting the respective delay times of the time series change data of the lip movement and the time series change data of the voice. 4. The speaker identification device according to claim 3, wherein the determining means sets the cross-correlation coefficient within a predetermined time around zero as the delay time of the cross-correlation coefficient as the similarity to be evaluated. 10. A voice input system using voice input instead of key operation, wherein the speaker identification device according to any one of claims 3 to 9 is applied, and the target person is a speaker. A voice input system comprising: voice recognition means for determining whether there is a speaker, and when it is determined that the utterer is the target person, processing the voice in accordance with the voice input. 11. A lip image of each person photographed by a plurality of cameras, or a lip image of each person cut out from one photographed image by using a plurality of windows, with a plurality of persons as subjects. A speaker identification system for identifying who uttered each person by applying the speaker identification device according to any one of claims 3 to 9. 12. In a communication system for communicating video and audio, the speaker identification system according to claim 11 is applied to a plurality of participating persons, and a speaker among the plurality of persons is identified. A communication system comprising communication data control means for converting an image to be identified and communicated into an image from a camera zoomed in on the speaker. 13. A communication system for communicating video and audio, wherein a three-dimensional sound localization function is provided, and a plurality of participating persons are set as subjects, and the speaker identification system according to claim 11 is applied. Communication that identifies a speaker among the persons, calculates a virtual sound generation position according to the position of the speaker and the position of the listener on the receiving side, and presents a three-dimensional sound from the speaker. A communication system comprising data control means.