JP2001352530A - Communication conference system - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a communication conference system that can adaptively decide a delay time until the time when shift of a visual field of a camera and change in directivity of a microphone are started with alteration of talkers. SOLUTION: A position estimate section 10 estimates a position of a speaker, a position history management section 201 updates a position history database 202 according to the result, a visual field decision section 203 decides the visual field based on the position history, decides a delay time of delay section 204 based on the position history, the visual field and the visual field history, and a device control section 30 controls a visual field direction and a visual angle of a video camera with the lapse of the delay time from the time when the visual field is decided.

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a communication conference apparatus for determining the field of view of a video camera, the directivity of a microphone, and the like from the position history of past speaker positions and a new speaker position.
In particular, the present invention relates to a technique for adaptively controlling a delay time until the start of a change in a camera's field of view or a change in microphone directivity according to a change in a speaker position.

[0002]

2. Description of the Related Art FIG. 8 is a diagram for explaining a main part of a speaker automatic tracking system of a video camera used in a conventional communication conference apparatus. 10 is a position estimating unit for estimating the position of the speaker, and 20 'is a selectivity control unit. Selectivity control unit 2
At 0 ′, the position information of the speaker obtained by the position estimating unit 10 is received, the position history managing unit 201 updates the position history database 202 of the speaker, and the position history information of the position history database 202 is used as the field of view determining unit. A field of view is determined by inputting the information to a field of view 203, and the obtained field of view information is output to the device control unit 30 to control the field of view (zoom) and field of view (horizontal direction) of the video camera.

[0003]

Therefore, in the prior art, when the visual field is determined by the visual field determination unit 203, the video camera starts moving immediately or after a fixed delay time. Since the delay time until the video camera starts moving is fixed as described above, the following inconvenience may occur.

[0004] First, when there is a speaker change, if the delay time is long, the change of the speaker may end. On the other hand, when the speakers change frequently, if the delay time is shortened so that the video camera starts moving quickly in accordance with the change, there is a problem that the video is disturbed and the viewer is uncomfortable.

When the direction of the visual field of the video camera moves largely, it takes time to move in the direction of the visual field, so that a change of another speaker occurs during the moving time and the video camera strays. To prevent this,
The delay time needs to be set longer than when the movement is small. However, this causes a problem that the visual field movement start is always delayed.

[0006] On the other hand, it is not preferable that the timing of changing the directivity of the microphone accompanying the change of the speaker greatly deviates from the timing of changing the visual field of the video camera.

SUMMARY OF THE INVENTION An object of the present invention is to provide a communication conference apparatus which solves the above-mentioned problem by adaptively determining a delay time until the movement of the field of view of the video camera or the change in the directivity of the microphone is started by the change of speaker. That is.

[0008]

According to a first aspect of the present invention, there is provided a position estimating means for estimating a speaker position from a video signal or at least two or more audio signals;
A position history management unit (2) that records the speaker position obtained by the position estimation unit (10) and updates the position history (202).
01) and a visual field or directivity determining means (203) for determining the visual field of the imaging means for recording the video signal or the directivity of the sound collecting means for recording the audio signal based on the position history (202); Delay means (204) for delaying the determined field of view or directivity information;
4) a delay time determining means (205) for setting a variable delay time, and the visual field or the sound collecting means of the imaging means based on the information of the visual field or the directivity delayed by the delay means (204). And a device control means (30) for setting the directivity of the device to a new speaker position.

In a second aspect based on the first aspect, the speaker position history (202) comprises a set of a speaker position, a speech time, and a speech time, and the delay time determination means (205).
Is the utterance duration of the previous speaker obtained from the utterance time, the cumulative utterance time difference obtained by subtracting the cumulative utterance time of the previous speaker from the cumulative utterance time of the new speaker, and the new speaker from the previous speaker position. The delay time is set according to at least one selected from the amount of change to the position.

In a third aspect based on the second aspect, the history of speaker positions is recorded for each speaker, and the speaker is determined from a video signal or at least two or more audio signals. The delay time is set from the history of the user.

[0011]

FIG. 1 is a block diagram showing an embodiment of a communication conference apparatus according to the present invention. Reference numeral 10 denotes a position estimating unit (position estimating means) for estimating a speaker position from a video signal of a video camera or a sound signal of a microphone, and 20 denotes visual field information of the video camera based on the speaker position information obtained by the position estimating unit 10. A selectivity control unit 30, which is created and output, is a device control unit (device control means) that controls the viewing direction and the viewing angle of the video camera based on the view information from the selectivity control unit 20.

The position estimating unit 10 determines one or more moving objects from the captured image, and determines the person based on whether or not a head of a person whose characteristics are set in advance exists in the moving objects. Further, the position of one speaker is estimated based on whether the lips of the person are moving. Alternatively, the position of one speaker is estimated by processing audio signals input from a plurality of installed microphones.

In the selectivity control unit 20, reference numeral 201 denotes a position history management unit which receives the speaker position information obtained by the position estimation unit 10 and updates the position history database 202;
A field-of-view deciding unit (field-of-view deciding unit) 03 for creating and processing field information of the video camera so that the field angle and the field direction are matched with the speaker position by inputting and processing the position history information;
Reference numeral 204 denotes a delay unit (delay means) for delaying the visual field information obtained by the visual field determination unit 203; 205, a moving delay time determining unit (delay time determining means) for determining the moving delay time of the video camera; A visual field history management unit 207 that manages the history of information is a visual field history database.

The position history database 202 contains a speaker position (each speaker does not move from a specific position, and the position is registered in rectangular coordinates or polar coordinates), an utterance time, and an utterance time. Each data is stored. There is a speaker change, and the utterance time, utterance time, and utterer are updated each time the start and end of the utterance of the speaker are detected. The view history database 207 also stores view information (view angle, view direction) corresponding to the speaker.

The moving delay time determining unit 205 controls the delay unit 2
The following method is available as a method for setting the delay time 04. That is, as shown in FIG. 2, two amounts obtained from the newly detected speaker position and the past speaker position history,
In other words, the "utterance duration" of the previous speaker and the cumulative utterance time of the new speaker (cumulative utterance time from the start of operation of the present apparatus)
And the “accumulated utterance time difference” obtained by subtracting the accumulated utterance time of the previous speaker from.

Speakers with a long "speech continuation time" often give reports or lectures, and are likely to speak again after short speeches of other speakers, which causes discomfort. In order to prevent frequent video camera field-of-view shifts, when a speaker with a long "duration of utterance" changes from one speaker to another, a speaker with a short "duration of utterance" switches to another speaker. The delay time is set longer than when the speaker is changed. On the other hand, in the case of discussions by multiple people,
Differences in the cumulative speech time of the participants often occur, such as the short cumulative speech time of the moderator. As described above, when the speaker change occurs in a situation where the cumulative utterance time is different, when the “cumulative utterance time difference” is positive (the cumulative utterance time of the new speaker is longer than the cumulative utterance time of the previous speaker). When it is large, it indicates that there has been a speaker change from a speaker with a long "average utterance duration" to a speaker with a short "average utterance duration". Is more likely to continue to occur than when the speaker changes from a short speaker to a long speaker. Therefore, in order to prevent frequent movement of the visual field of the video camera, the movement delay time of the video camera is made longer when the "cumulative speech time difference" is positive than when it is negative. As described above, as shown in FIG. 2, the longer the “speech continuation time” and the larger the “cumulative utterance time difference”, the longer the movement delay time of the video camera.

Further, according to the present invention, the moving delay time determining unit 2
The visual field information is also taken into account as a determinant of the delay time at 05. That is, as shown in FIG. 3, when the "field of view movement amount" from the previous field of view to the new field of view is small, the change in the field of view of the video camera is small, so that the image disturbance is small, and the visual field is quicker than the image disturbance. Since the following can be given priority, the delay time is set short. Conversely, if the “field of view movement” is long,
Focusing on reducing the risk of image distortion,
Set a longer moving delay time for the video camera.

As a specific method for determining the delay time, a method is used in which a table is created and set so as to set the delay time from the above-mentioned "utterance continuation time", "cumulative utterance time difference", and "view movement amount" elements. There is. In this case, the contents of the table are updated each time there is a speaker change.

In addition, T is a delay time until the start of movement of the video camera, t1 is "speech continuation time", t2 is "cumulative speech time difference", w is "visual field movement amount", and a, b, c, d
Is a positive coefficient, it can also be obtained by an operation using an expression such as T = f (a · t1 + b · t2 + c · w + d). This function f () is a monotonically increasing function in which the minimum value of the range is equal to or greater than 0 and is bounded above. For example, tanh (x) + 1 = 2exp (x) / {exp (x) + exp (-x)} Alternatively, there is a step function s (x) = 1 (x> threshold) = 0 (x ≦ threshold).

As described above, the delay time from when the visual field of the video camera is determined to when the video camera starts moving is defined as at least one of “speech continuation time”, “cumulative utterance time difference”, and “visual field movement amount”. The delay time from the creation of the view information to the start of the movement of the video camera is adaptively adjusted. For this reason, compared to the case where the time until the video camera starts to move is fixed, the delay time is increased when the speakers change frequently to prevent the image from being disturbed, and when the visual field moves greatly, the delay time is increased. A quick change of the field of view can be realized in a short time. Note that if there is a speaker change before the delay time expires because the delay time is long, the previous delay time is reset and a delay time according to a new speaker is set.

FIG. 4 is a diagram showing a specific example of the position estimating unit 10 in which a speaker position is estimated by processing an image picked up by a video camera. Here, the motion area is detected by converting the luminance signal Y of the image into a digital signal by the A / D converter 101 and then calculating the difference between the frames by the motion area detection unit 102.
2 based on a threshold created from the output signal of converter 101
The value conversion unit 103 binarizes the difference information indicating the motion area. Next, the binarized difference information is taken into the horizontal moving area extracting unit 104, and the binarized difference information is accumulated and added temporally and spatially in the horizontal direction (horizontal direction of the image) to thereby obtain a moving object. Is obtained for each moving object, and is sent to the moving area selection unit 105. The moving area selection unit 105 selects the position coordinates of one moving object from the obtained horizontal position coordinates of the plurality of moving objects,
It is sent to the crown extraction unit 106 and the face width extraction unit 107. The head extraction unit 106 obtains the coordinates of the head of one moving object from the binary difference information obtained by the binary conversion unit 103 and the position coordinates of the moving object obtained by the moving area selection unit 106. Face width extraction unit 1
At 07, a moving object (person) is obtained from the binary difference information obtained by the binary conversion unit 103, the edge (contour) information of the image obtained by the edge detection unit 108, and the top coordinates obtained by the top extraction unit 106. The head information is created based on the left and right coordinates of the head and sent to the face feature extraction unit 109. The face feature extraction unit 109 determines whether or not a person's head is based on whether or not information corresponding to a vertical line on the cheek and a horizontal line such as eyebrows and eyes, which are one of the facial feature amounts of the person, is included. Then, when it is determined that it is the head, the information is sent to the speaker determination unit 110. This speaker determination unit 1
In 10, the region where the lips are located is determined from the left and right coordinates and the top coordinates of the head, the amount of change in the lip region at predetermined time intervals is detected, and whether the lips are moving up and down is determined by the person. It is determined whether or not the speaker. In this way, the speaker position is estimated from the image information (reference: Japanese Patent Laid-Open No. 7-22 / 1995).
5841).

On the other hand, FIG. 5 is a diagram showing another specific example of the position estimating unit 10 in which a plurality of audio signals obtained by a plurality of microphones are processed to estimate a speaker position. . Here, the cross-correlation function of the received audio signal is calculated for all combinations of microphones, and the obtained cross-correlation function is the maximum value of the cross-correlation function between one predetermined reference microphone and another microphone. Is obtained as a preliminary estimated time difference, a time difference that maximizes the delay sum power for all microphones is searched for in the vicinity of the preliminary estimated time difference, and this is used as an estimated time difference, and based on this estimated time difference, This is to calculate the sound source position (Reference: JP-A-Hei 1
1-304906). A method of estimating a speaker position by processing a plurality of signals obtained by other microphones is described in detail in the document "Acoustic system and signal processing", Oga et al., And Chapter 7 of IEICE. ing.

The device control unit 30 controls the visual field direction and the visual field angle of the video camera based on the visual field information from the selectivity control unit 20, and the visual field direction is directed to the estimated speaker position. The up, down, left, and right directions of the video camera may be controlled so as to be in the same direction.For the viewing angle, for example, a viewing angle corresponding to each speaker position is stored in a table in advance, and the viewing angle is determined according to the estimated speaker position. Then, the viewing angle may be read from the table and the zoom of the video camera may be controlled to the wide angle side or the telephoto side.

Note that the directivity of the microphone can be controlled together with the control of the viewing direction and viewing angle of the video camera based on the obtained viewing information. At this time,
The audio signals of the microphones are given a time delay so that the audio signals output from the plurality of microphones with respect to the audio from the visual field direction determined by the visual field information have the same phase. This allows
Except for the audio signal propagating from the determined viewing direction, the sound signals cancel each other out and attenuate, thereby realizing sharp directivity in the viewing direction.

FIG. 6 shows an embodiment in which the apparatus of the present invention is applied to a speaker automatic tracking system of a communication conference apparatus. The sound emitted by the speaker 60 is collected by the microphone array 50, the position of the speaker is estimated by the position estimating unit 10 from the collected sound signals of the plurality of channels, and the information of the estimated speaker position is selectedivity control. The information is input to the unit 20 to obtain the visual field information, and the device control unit 30 controls the visual field direction and the visual angle of the video camera 40 based on the visual field information.

FIG. 7 shows an embodiment in which the apparatus of the present invention is applied to a speaker automatic tracking directional sound collecting system of a communication conference apparatus. The sound emitted by the speaker 60 is collected by the microphone array 50, the position of the speaker is estimated by the position estimating unit 10 from the collected sound signals of a plurality of channels, and the selectivity control unit 20 collects the sound from the estimated speaker position. The sound area (that is, the field of view) is determined, the direction of the directivity and the width of the directivity are calculated by the device control unit 30, and the array signal processing device 70 outputs the audio signal of only the sound collection area from the audio signals of a plurality of channels. It is like that.

[0027]

As described above, according to the present invention, since the visual field history is used in addition to the position history of the speaker, the time from the determination of the visual field and the directivity to the start of the visual field change of the video camera and the start of the directivity change of the microphone are determined. Can be adaptively controlled, and there is an advantage that it is possible to appropriately cope with speaker change.

[Brief description of the drawings]

FIG. 1 is a block diagram of a main part of a communication conference device of the present invention.

FIG. 2 is an explanatory diagram of a moving delay time setting method of a video camera adapted to a speaker's utterance.

FIG. 3 is an explanatory diagram of a setting method of a moving delay time of the video camera adapted to a moving amount of a visual field of the video camera.

FIG. 4 is an explanatory diagram of a position estimating unit using an image.

FIG. 5 is an explanatory diagram of a position estimating unit using sound.

FIG. 6 is an explanatory diagram of a speaker automatic following video camera system.

FIG. 7 is an explanatory diagram of a speaker automatic tracking directional sound collection system.

FIG. 8 is a block diagram of a main part of a conventional communication conference device.

[Explanation of symbols]

10: position estimator, 101: A / D converter, 102: moving area detector, 103: binary converter, 104: horizontal moving area detector, 105: moving area selector, 106:
Top extraction section, 107: face width extraction section, 108: edge detection section, 109: face feature extraction determination section, 110: speaker determination section 20: selectivity control section, 201: position history management section, 20
2: position history database, 203: visual field determination unit, 20
4: delay unit, 205: moving delay time determination unit, 206: visual field history management unit, 207: visual field history database 30: device control unit 40: video camera 50: microphone array 60: speaker 70: array signal processing device

 ──────────────────────────────────────────────────続 き Continued on the front page F term (reference) 5C022 AA12 AB36 AB62 AB63 AB66 AC41 AC69 AC72 5C064 AA02 AB04 AC04 AC09 AC12 AC22 AD14 5D020 BB03 BB04

Claims (3)

[Claims] 1. A position estimating means for estimating a speaker position from a video signal or at least two or more audio signals, and a position history managing means for recording a speaker position obtained by the position estimating means and updating a position history. And a field of view or directivity determining means for determining the field of view of the image pickup means for recording the video signal or the directivity of the sound collecting means for recording the audio signal based on the position history, and the determined field of view or directivity Delay means for delaying the information of the above; delay time determining means for setting a variable delay time for the delay means; information on the visual field or directionality of the imaging means, A device control unit for setting the directivity of the sound collection unit to match a new speaker position. 2. The speaker according to claim 1, wherein the speaker position history includes a set of a speaker position, an utterance time, and an utterance time, and wherein the delay time determination unit determines the utterance of the speaker before the utterance obtained from the utterance time. A duration, a cumulative utterance time difference obtained by subtracting the cumulative utterance time of the previous speaker from the cumulative utterance time of the new speaker, and at least one or more selected from the amount of change from the previous speaker position to the new speaker position. And the delay time are set. 3. The speaker according to claim 2, wherein a history of speaker positions is recorded for each speaker, and said speaker is determined from a video signal or at least two or more audio signals. A communication conference device for setting a delay time.