US20200116820A1

US20200116820A1 - Audio based motion detection

Info

Publication number: US20200116820A1
Application number: US16/655,601
Authority: US
Inventors: Oystein BIRKENES
Original assignee: Cisco Technology Inc
Current assignee: Cisco Technology Inc
Priority date: 2017-04-25
Filing date: 2019-10-17
Publication date: 2020-04-16
Also published as: US10473751B2; US20180306900A1

Abstract

A controller of a collaboration endpoint generates a primary audio signal for an ultrasonic source audio signal produced by a source audio speaker, a reference audio signal for the ultrasonic source audio signal, and, based on the reference audio signal, a predicted signal that is predictive of the primary audio signal. The controller produces a prediction error of the predicted signal by comparing the primary audio signal with the predicted signal and determines whether the prediction error is indicative of a motion of one or more persons near the collaboration endpoint.

Description

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a continuation of U.S. patent application Ser. No. 15/496,362, filed Apr. 25, 2017, the entirety of which is incorporated herein by reference.

TECHNICAL FIELD

The present disclosure relates to motion detection based on at least one source audio signal.

BACKGROUND

Collaboration endpoints are generally used for multimedia meetings. These endpoints may include cameras, displays, microphones, speakers, and other equipment configured to facilitate multimedia meetings. Endpoint equipment often remains activated for long periods of time even when the equipment is only used intermittently for multimedia meetings.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a block diagram of a conference room endpoint configured for motion detection, according to an example embodiment.

FIG. 2 is a signal processing diagram for motion detection that includes one integrated primary microphone, a plurality of external source audio speakers, and a plurality of integrated reference microphones, according to an example embodiment.

FIG. 3 is a signal processing diagram for motion detection that includes one integrated primary microphone, one external source audio speaker, and one integrated reference microphone, according to an example embodiment.

FIG. 4 is a signal processing diagram for motion detection that includes one integrated primary microphone, two external source audio speakers, and two integrated reference microphones, according to an example embodiment.

FIG. 5 is a signal processing diagram for motion detection that includes one integrated primary microphone, a plurality of external source audio speakers, one integrated source audio speaker, and a plurality of integrated reference microphones, according to an example embodiment.

FIG. 6 is a signal processing diagram for motion detection that includes one integrated primary microphone and one integrated source audio speaker, according to an example embodiment.

FIG. 7 is a signal processing diagram for motion detection that includes one integrated primary microphone, one external source audio speaker, one integrated source audio speaker, and one integrated reference microphone, according to an example embodiment.

FIG. 8 is a signal processing diagram for motion detection that includes one integrated primary source audio speaker, a plurality of external audio speakers, and a plurality of integrated reference microphones, according to an example embodiment.

FIG. 9 is a signal processing diagram for motion detection that includes one integrated primary source audio speaker and one integrated reference microphone, according to an example embodiment.

FIG. 10 is signal processing diagram for nonlinear motion detection that includes one integrated primary microphone and one integrated source audio speaker, according to an example embodiment.

FIG. 11 is a block diagram of a collaboration endpoint controller configured to execute motion detecting techniques, according to an example embodiment.

FIG. 12 is a flowchart of a generalized method in accordance with examples presented herein.

DESCRIPTION OF EXAMPLE EMBODIMENTS

Overview

Example Embodiments

When endpoint equipment (e.g., cameras, displays, microphones, speakers, etc.) remains activated during periods of nonuse during a multimedia conference session, the equipment wastes electricity and experiences decreases in life expectancy. Therefore, as described in greater detail below, endpoint equipment may enter a standby mode when the conference room in which the endpoint is located is empty. When a user enters the room, the endpoint equipment may detect motion in the room and, in response, activate (enter an active mode from a standby mode) in anticipation of a multimedia meeting.
With reference made to FIG. 1, shown is a block diagram of a conference room 102 that includes an endpoint 104 configured for motion detection in accordance with examples presented herein. The endpoint 104 enables a user 106 to participate in a multimedia conference by communicating with users at other endpoints 104(a)-104(n) via network 108. The network interface unit 110 allows endpoint 102 to connect with network 108. The endpoint 102 is equipped with camera 112, display 114, microphones 116(a) and 116(b), and speaker 118. These components are connected to an endpoint controller 120, which serves as a control unit for the endpoint 104, and executes, among other software instructions, software instructions for motion detection logic 122. The conference room 102 may also include an external speaker 124 that is not connected to the endpoint controller 120.
When user 106 is participating in a multimedia conference, this equipment is active and facilitates user 106 participation in the multimedia conference. Camera 112 and microphones 116(a)/116(b) capture video and audio, respectively, for transmission to one or more of endpoints 104(a)-104(n). Display 114 and speaker 118 output video and audio, respectively, transmitted from one or more of endpoints 104(a) 104(n).
When the multimedia conference ends, the user 106 exits conference room 102. At this point, there are no people in conference room 102, and thus no movement in the conference room 102. In accordance with the motion detection logic 122, the endpoint controller 120 controls the components of the endpoint 104 to enter a standby mode in order to detect motion (e.g., the motion of one or more persons near the endpoint 102). As explained below, when motion is detected (e.g., when one or more persons are near the endpoint 102), the endpoint controller 120 controls the components of the endpoint 104 to activate (enter an active mode from the standby mode) in anticipation of a multimedia meeting.
During standby mode, camera 112 and display 114 may simply shut off/deactivate/power down/etc. However, speakers 118, 124 emit ultrasonic source audio signals. In an example, speaker 124 continuously emits its ultrasonic source audio signal, even when the components of the endpoint 104 are in standby mode, because the speaker 124 is not connected to the endpoint controller 120. Also during standby mode, microphones 116(a)/ 116(b) continuously sample the ultrasonic source audio signals from speakers 118 and 124.
In this example, the signals produced by microphones 116(a)/116(b) in response to the ultrasonic source audio signals serve as reference audio signals. Meanwhile, the ultrasonic source audio signal produced by speaker 118 is known to the endpoint controller 120 and serves as a primary audio signal. As explained in greater detail below, the endpoint controller executes the motion detection logic 122 to generate, based on the reference audio signals, a predicted signal that is predictive of the primary audio signal. The motion detection logic 122 compares the primary audio signal with the predicted signal to determine whether there is motion of one or more persons near the endpoint 104.
While in standby mode, the endpoint 104 may operate actively or passively. In active standby mode, the speaker 118 integrated with the endpoint 104 transmits an ultrasonic audio signal into an area/room/location (e.g., conference room). In passive standby mode, the endpoint 104 does not transmit an ultrasonic audio signal. FIGS. 2-4 illustrate example passive mode embodiments and FIGS. 5-10 illustrate example active mode embodiments. In both passive and active standby modes, one or more microphones integrated with the endpoint capture an audio signal from a speaker. The signal processing diagrams of FIGS. 2-10 may be implemented, for example, in a conference room. The endpoint controller 120 performs the signal processing depicted by the signal processing diagrams shown in FIGS. 2-10.
FIG. 2 shows a signal processing diagram for passive standby mode motion detection in accordance with examples presented herein. Speakers 224(1)-(I) are external to an endpoint 104. Endpoint 104 includes microphones 216(1)-(J), filter banks 226(1)-(J) that respectively map to microphones 216(1)-(J), and adaptive filters 228(2)-(J) that respectively map to microphones 216(2)-(J). There is a delay operator 230 and a detector function 232. The detector function 232 generates an output indicative of detected motion.
Speakers 224(1)-(I) produce respective source audio signals s₁(t), s₂(t), s₁(t). Source audio signals s₁(t), s₂(t), s₁(t) include ultrasonic frequencies (e.g., 20-24 kHz). Each microphone 216(1)-(J) captures the source audio signals s₁(t), s₂(t), s₁(t) after the source audio signals have traversed the room and, in response, produces a respective signal x₁, x₂, x₃, . . . , x_J. Each signal x₁, x₂, x₃, . . . , x_Jis a superposition of source audio signals convolved with impulse responses (e.g., h₁₁, h₁₂, etc.) and a noise signal (e.g., n₁, n₂, etc.). For purposes of viewability, only the impulse responses and the noise signals corresponding to the microphones 216(1) and 216(2) are shown. Signal x₁may be referred to as a primary audio signal and signals x₂, x₃, . . . , x_Jmay be referred to as reference audio signals.
The endpoint 104 detects motion by predicting the delayed time frequency representation of primary signal x₁of microphone 216(1) based on the reference signals x₂, x₃, . . . , x_Jof microphones 216(2)-(J). An endpoint controller of endpoint 104 (not shown) feeds each signal x₁, x₂, x₃, . . . , x_Jthrough a respective filter bank 226(1)-(J). These filter banks 226(1)-(J) separate the primary audio signal and reference audio signal into sub-bands. For instance, the filter banks 226(1)-(J) respectively partition the signals x₁, x₂, x₃, . . . , x_Jinto time-frequency domain signals X₁(m,k), X₂(m,k), X₃(m,k), . . . , X_J(m,k), where m and k denote a frame index and a frequency bin index, respectively. As explained in greater detail below, the delay operator 230 delays the time-frequency representation of the primary signal, X₁(m,k), by D frames, resulting in the time-frequency domain signal X₁(m−D, k). The endpoint controller of endpoint 104 respectively feeds the time-frequency domain representations of the reference signals, X₂(m,k), X₃(m,k), . . . , X_J(m,k), through adaptive filters 228(2)-(J), respectively. In an example, the adaptive filters 228(2)-(J) are normalized least mean square filters. The endpoint controller of endpoint 104 sums the contributions of the filtered time-frequency domain representations of the reference signals, X₂(m,k), X₃(m,k), . . . , X_J(m,k), to generate a predicted signal {circumflex over (X)}₁(m−D,k) that is predictive of the delayed time-frequency representation of the primary signal X₁(m−D, k). The endpoint controller of endpoint 104 generates prediction error E(m−D,k)=X₁(m−D,k)−{circumflex over (X)}₁(m−D,k) by comparing the delayed time-frequency representation of the primary audio signal with the predicted signal.
The endpoint controller of endpoint 104 determines whether the prediction error is indicative of the motion of one or more persons near the endpoint 104. The endpoint controller of endpoint 104 updates the adaptive filter coefficients based on the prediction error and sends the prediction error to the detector 232. A relatively large prediction error may indicate motion more strongly than a relatively small prediction error. As discussed above, if the endpoint controller of endpoint 104 determines that there is motion of one or more persons near the collaboration endpoint 104, the endpoint controller of endpoint 104 may activate the components of endpoint 104 in anticipation of a multimedia meeting.
The following is a mathematical derivation of primary signal x₁expressed as a function of the reference signals x₂, x₃, . . . , x_J.
Signal x₁is the superposition of the contributions from the audio signals s₁(t), s₂(t), . . . , s₁(t) and a noise signal n₁. This can be written mathematically as
x ₁(t)=(h ₁₁ *s ₁(t)+(h ₁₂ *s ₂)(t)+ . . . +(h _1I *s ₁)(t)+n ₁(t), (1)
where * is the convolution operator and h_1i, i ∈ {1, 2, . . . I}, denotes the impulse response from the ith audio speaker to the first microphone. Signal x₁is a digital signal (i.e., the sampled and quantized version of the analog signal captured at microphone 216(1)), and t denotes the sample index. The variables in (1) are functions of time. Assuming these signals are stationary, (1) may be re-written in the frequency domain as
X ₁(f)=H ₁₁(f)S ₁(f)S ₁(f)+H ₁₂(f)S ₂(f)+ . . . +H _1I(f)S ₁(f)+N ₁(f), (2)
where upper case variables are used to denote the discrete-time Fourier transform (DTFT) corresponding to the respective lower case variables, with f ∈ [0, 1] denoting digital frequency. Using vector notation, (2) can be written more compactly as
X ₁(f)=H ₁ ^T(f)S(f)+N ₁(f), (3)
where S(f)=[S₁(f), S₂(f), . . . , S_I(f)]^Tand H₁(f)=[H₁₁(f), H₁₂(f), . . . , H_1I(f)]^Tare I-dimensional vectors. Similar equations can be written for the other microphone signals:
$\begin{matrix} X_{2} (f) = H_{2}^{T} (f) S (f) + N_{2} (f), & (4) \\ X_{3} (f) = H_{3}^{T} (f) S (f) + N_{3} (f), & (5) \\ ⋮ \\ X_{J} (f) = H_{J}^{T} (f) S (f) + N_{J} (f) . & (6) \end{matrix}$
where H_j(f)=[H_j1(f), H_j2(f), . . . , H_jI(f)]^Tfor j ∈ [2, 3, . . . , J] are I-dimensional vectors. (4)-(6) can be written even more compactly using matrix notation as
X(f)=H(f)S(f)+N(f), (7)
where X(f)=[X₂(f), X₃(f), . . . , X_J(f)]^Tand N(f)=[N₂(f), N₃(f), . . . , N_J(f)]^Tare J−1-dimensional vectors and
$\begin{matrix} H (f) = [\begin{matrix} H_{21} (f) & H_{22} (f) & \dots & H_{2 I} (f) \\ H_{31} (f) & H_{43} (f) \\ ⋮ & ⋱ \\ H_{J 1} (f) & H_{JI} \end{matrix}] & (8) \end{matrix}$
is a J−1×I-dimensional matrix. From (7), if the columns of H(f) are linearly independent (this requires that J>1),
S(f)=(H ^T(f)H(f))⁻¹ H ^T(f)(X(f)−N(f)), (9)
Substituting S(f) into (3) yields
$\begin{matrix} X_{1} (f) = H_{1}^{T} (f) S (f) + N_{1} (f) & (10) \\ = H_{1}^{T} (f) {(H^{T} (f) H (f))}^{- 1} H^{T} (f) (X (f) - N (f)) + N_{1} (f) & (11) \\ = {\tilde{H}}_{1}^{T} (f) X (f) + {\tilde{N}}_{1} (f), & (12) \end{matrix}$
where Ĥ₁(f)=H₁ ^T(f)(H^t(f)H(f))⁻¹H^T(f)=[{tilde over (H)}₁₂(f), {tilde over (H)}₁₃(f), . . . , Ĥ_1.1(f)]^Tand Ñ₁(f)=N₁(f)−{tilde over (H)}₁ ^T(f)N(f).
Returning to scalar notation yields
X ₁(f)={tilde over (H)} ₁₂(f)X ₂(f)+{tilde over (H)} ₁₃(f)X ₃(f)+ . . . +{tilde over (H)}_1J(f)X _J(f)+Ñ ₁(f), (13)
and
x ₁(t)=(ti₁₂ *x ₂)(t)+(h₁₃ *x ₃)(t)+ . . . +(h_1,J *x _j)(t)+h₁(t), (14)
(14) is a formulation of the signal x₁of microphone 216(1) as a function of the signals x₂, x₃, . . . , x_Jof microphones 216(2)-(J).
However, the impulse h̆₁₂, h̆₁₃, . . . h̆_1,Jare not causal. For example, in a case involving J=2 microphones and I=1 source, x₁(t)=(h̆₁₂(x₂)(t)+ń₁(t). If the single source is located closer to microphone 1 than to microphone 2, microphone 1 receives the sound before microphone 2 and {tilde over (h)}₁₂is thus necessarily non-causal. Because it is desired for the impulse responses to be causal, one or more of the impulse responses h̆₁₂, h̆₁₃, . . . h̆_1,Jmay be delayed by T samples, where T may be equal to at least an amount of time between generating the primary audio signal and generating the reference audio signal. In the example of FIG. 2, the delay of D frames corresponds to at least T samples of delay in the time domain. In the above example involving J=2 microphones and I=1 source, T may be at least as large as the time between microphone 1 receiving the source audio signal and microphone 2 receiving the source audio signal. This implies that the impulse responses will be causal if the T-sample delayed microphone signal x₁(t−T) is predicted from non-delayed microphone signals x₂(t), . . . , x_J(t). This can be expressed mathematically as:
x ₁(t−T)=(g ₁₂ *x ₂)(t)+(g ₁₃ *x ₃(t)+ . . . +(g _1,J *x _J)(t)+ñ ₁(t−T), (15)
where g_1j, j ∈ {2, 3, . . . , J}, is a T-sample delayed version of {acute over (h)}_1j. When the impulse responses are causal, the T-sample delayed microphone signal x₁(t−T) may be predicted with multichannel adaptive filtering, as shown in (16) below:
{circumflex over (x)} ₁(t−T)=(w ₂ *x ₂(t)+(w ₃ *x ₃)(t)+ . . . +(w _J *x ₁)(t), (16)
where w₂, w₃, . . . , w_Jdenote the J−1 vectors of adaptive filter coefficients. As long as the impulse responses g₁₂, g₁₃, . . . , g_1Jare approximately constant and the noise term ñ₁(t) is approximately stationary and small compared to the reference signal terms in (15), the prediction error is expected to be small. This is typically the case when a person is not in the room. When one or more of the impulse responses changes or the noise term ñ₁(t) suddenly increases (e.g., due to a person entering the conference room in which the endpoint 104 is located), the prediction error increases. Thus, as shown in (16), the endpoint 104 detects motion by predicting the T-sample delayed primary signal x₁of microphone 216(1) based on the reference signals x₂, x₃, . . . , x_Jof microphones 216(2)-(J).
In an example, the detector 232 detects motion based on an estimate P(m,k) of the mean square error. The detector 232 may use exponential recursive weighting such as
P(m,k)=αP(m−1,k)+(1−α)|E(m,k)|², (17)
where α is a forgetting factor in the range [0,1). For example, for some threshold P,
P(m,k)<P⇒No motion, (18)
P(m,k)≥P⇒Motion. (19)
Thus, in this example, the detector detects motion from an increase in P(m,k). If P(m,k) is less than a threshold P, the endpoint controller of endpoint 104 may conclude that there is no motion of one or more persons near the endpoint 104. If the P(m,k) is equal to or greater than the threshold P, the endpoint controller of endpoint 104 may conclude that there is motion of one or more persons near the endpoint 104.
The above derivation involves multichannel adaptive filtering in the digital time domain. Alternatively, the endpoint controller of endpoint 104 may perform multichannel adaptive filtering in the time-frequency domain. Time-frequency domain adaptive filtering is computationally cheaper and provides faster convergence than time domain adaptive filtering. In addition, time-frequency domain adaptive filtering allows the endpoint 104 to predict only the ultrasonic portion of the signal x₁while ignoring other frequencies.
FIGS. 3 and 4 are specific examples of passive standby mode detection. With reference first to FIG. 3, a signal processing diagram is shown for predicting signal x₁based on another signal x₂with one source audio signal s₁in accordance with examples presented herein. Speaker 224(1) is external to endpoint 104 and emits audio signal s₁. Endpoint 104 includes microphones 216(1)-(2) (which produce signals x₁, x₂), filter banks 226(1)-(2), adaptive filter 228(2), delay operator 230, and detector 232.
In this example, signal x₁is the primary signal and signal x₂is the reference signal. As in FIG. 2, filter banks 226(1)-(2) respectively partition the signals x₁, x₂into time-frequency domain signals X₁(m,k), X₂(m,k). The delay operator 230 delays the time-frequency representation of the primary signal, X₁(m,k), by D frames, resulting in X₁(m−D, k). The endpoint controller of endpoint 104 feeds the time-frequency domain representations of the reference signal, X₂(m,k), through adaptive filter 228(2) and generates a predicted signal {circumflex over (X)}₁(m−D,k) that is predictive of the delayed time-frequency representation of the primary signal, X₁(m−D, k). The endpoint controller of endpoint 104 generates prediction error E(m−D,k)=X₁(m−D,k)−{tilde over (X)}₁(m−D,k) by comparing the delayed time-frequency representation of the primary audio signal with the predicted signal, and determines whether the prediction error is indicative of the motion of one or more persons near the collaboration endpoint 104.
The following is a mathematical derivation of the discrete time Fourier transform of the primary signal X₁(f) expressed as a function of the discrete time Fourier transform of the reference signal X₂(f) in the example of FIG. 3. In this case, the matrix at (8) reduces to the scalar
H(f)=H ₂₁(f), (20)
the channel vector in (12) reduces to the scalar
$\begin{matrix} {\tilde{H}}_{1} (f) = \frac{H_{11} (f)}{H_{21} (f)}, & (21) \\ and \\ {\tilde{N}}_{1} (f) = N_{1} (f) - \frac{H_{11} (f)}{H_{21} (f)} N_{2} (f) . & (22) \end{matrix}$
(13) thus simplifies to
$\begin{matrix} X_{1} (f) = \frac{H_{11} (f)}{H_{21} (f)} X_{2} (f) + N_{1} (f) - \frac{H_{11} (f)}{H_{21} (f)} N_{2} (f) . & (23) \end{matrix}$
Thus, filtering X₂(m,k) with linear adaptive filter 228(2) may enable the endpoint controller of endpoint 104 to generate predicted signal {circumflex over (X)}₁(m−D,k), and determine whether the prediction error is indicative of the motion of one or more persons near the endpoint 104.
Turning now to FIG. 4, a signal processing diagram is shown for predicting the delayed time-frequency representation of the primary signal x₁(t) from the time-frequency representations of the reference signals x₂(t) and x₃(t) in accordance with examples presented herein. In other words, X₁(m−D,k) is predicted from X₂(m,k) and X₃(m,k). Speakers 224(1)-(2) produce source audio signals s₁, s₂and are external to endpoint 104. Endpoint 104 includes microphones 216(1)-(3) (which produce signals x₁, x₂, x₃), filter banks 226(1)-(3), adaptive filters 228(2)-(3), delay operator 230, and detector 232.
In this example, signal x₁is the primary signal and signals x₂, x₃are reference signals. As in FIG. 2, filter banks 226(1)-(3) respectively partition the signals x₁, x₂, x₃into time-frequency domain signals X₁(m,k), X₂(m,k), X₃(m,k). The delay operator 230 delays the time-frequency representation of the primary signal, X₁(m,k), by D frames, resulting in X₁(m−D, k). The endpoint controller of endpoint 104 respectively feeds the time-frequency domain representations of the reference signals, X₂(m,k), X₃(m,k), through adaptive filters 228(2)-(3) and generates a predicted signal {circumflex over (X)}₁(m−D,k) that is predictive of the delayed time-frequency representation of the primary signal, X₁(m−D, k). The endpoint controller of endpoint 104 generates prediction error E(m−D,k)=X₁(m−D,k)−{circumflex over (X)}₁(m−D,k) by comparing the delayed time-frequency representation of the primary audio signal with the predicted signal, and determines whether the prediction error is indicative of the motion of one or more persons near the endpoint 104.
The following is a mathematical derivation of the discrete-time Fourier transform of the primary signal, X₁(f), expressed as a function of the discrete-time Fourier transform of the reference signals, X₂(1), X₃(1), in the example of FIG. 4. In this case, the matrix at (8) reduces to
$\begin{matrix} H (f) = [\begin{matrix} H_{21} (f) & H_{22} (f) \\ H_{31} (f) & H_{32} (f) \end{matrix}] . & (24) \end{matrix}$
This matrix has linearly independent columns, which means (13) can be re-written as
X ₁(f)={tilde over (H)} ₁₂(f)X ₂(f)+{tilde over (H)} ₁₃(f)X ₃(f)+Ñ ₁(f). (25)
Thus, respectively filtering X₂(m,k), X₃(m,k) with linear adaptive filters 228(2)-(3) may enable the endpoint controller of endpoint 104 to generate predicted signal {circumflex over (X)}₁(m−D,k), and determine whether the prediction error is indicative of the motion of one or more persons near the endpoint 104.
As mentioned above, FIGS. 5-10 illustrate example signal processing diagrams for active standby mode, in which the endpoint itself produces at least one source ultrasonic signal to be used for motion detection. FIGS. 5-7 illustrate a first category of active standby mode embodiments in which an integrated speaker serves as a reference audio device. FIGS. 8-9 illustrate a second category of active standby mode embodiments in which an integrated speaker serves as a primary audio device. FIG. 10 illustrates an example involving non-linear adaptive filtering according to the first category of active standby mode embodiments.
With reference first to FIG. 5, a generalized signal processing diagram is shown for predicting the delayed time-frequency representation of the primary signal x₁(t) from the time-frequency representations of the reference audio signals x₂(t), x₃(t), . . . , x_J(t) in accordance with examples presented herein. In other words, X₁(m−D,k) is predicted from X₂(m,k), X₃(m,k), . . . , X_J(m,k). Speakers 224(2)-(I) are external to endpoint 104, which includes microphone 216(1), speaker 224(1), and microphones 216(3)-(J). Endpoint 104 also includes filter banks 226(1)-(J), adaptive filters 228(2)-(J), delay operator 230, and detector 232.
In this example, signal x₁is the primary signal and signals x₂-x_Jare reference audio signals. Speakers 224(1)-(I) produce respective source audio signals s₁(t), s₂(t), . . . , s₁(t). Each microphone 216(1), 216(3)-(J) captures the source audio signals s₁(t), s₂(t), s₁(t) after the source audio signals have traversed the room and, in response, produces a respective signal x₁, x₃, . . . , x_J. In addition, endpoint controller generates reference signal x₂for source audio signal s₁since the speaker 224(1) is integrated into the endpoint 104 and, as such, the endpoint controller of endpoint 104 knows the source audio signal s₁. In other words, the speaker 224(1) serves as both a source audio device and a reference audio device. Each signal x₁, x₃, . . . , x_Jis a superposition of source audio signals convolved with impulse responses (e.g., h₁₁, h₁₂, etc.) and a noise signal (e.g., n₁, etc.). For purposes of viewability, only the impulse responses and noise signal corresponding to the primary microphone 216(1) are shown.
The endpoint 104 detects motion by predicting the delayed time-frequency representation of the primary signal x₁of microphone 216(1) based on the time-frequency representations of the reference signals x₂, x₃, . . . , x_J. In other words, X₁(m−D,k) is predicted from X₂(m,k), X₃(m,k), . . . , X_J(m,k). An endpoint controller of endpoint 104 feeds each of the audio signals x₁, x₂, x₃, . . . , x_Jthrough a respective filter bank 226(1)-(J). These filter banks 226(1)-(J) respectively partition the audio signals x₁, x₂, x₃, . . . , x_Jinto time-frequency domain signals X₁(m,k), X₂(m,k), X₃(m,k), . . . , X_J(m,k), where m and k denote a frame index and a frequency bin index, respectively. The delay operator 230 delays the time-frequency representation of the primary signal, X₁(m,k), by D frames, resulting in X₁(m−D, k). The endpoint controller of endpoint 104 respectively feeds the time-frequency domain representations of the reference audio signals, X₂(m,k), X₃(m,k), . . . , X_J(m,k), through adaptive filters 228(2)-(J). The endpoint controller of endpoint 104 sums the contributions of the filtered time-frequency domain representations of the reference audio signals to generate a predicted signal {circumflex over (X)}₁(m−D,k) that is predictive of the delayed time-frequency representation of the primary signal, X₁(m−D, k). The endpoint controller of endpoint 104 generates prediction error E(m−D,k)=X₁(m−D,k)−{circumflex over (X)}₂(m−D,k) by comparing the delayed time-frequency representation of the primary audio signal with the predicted signal. As mentioned above, the endpoint controller of endpoint 104 determines whether the prediction error is indicative of the motion of one or more persons near the endpoint 104 and, if the endpoint controller determines that there is motion of one or more persons near the collaboration endpoint 104, the endpoint controller of endpoint 104 may activate endpoint components in anticipation of a multimedia meeting.
Much of the mathematical framework for passive mode motion detection described above applies to active mode motion detection. This is because the active mode can be viewed as a special case of passive mode in which one of the source audio speakers has been moved very close to one of the microphones, such that the relative contributions from the other source audio signals and any noise are so small that they can be neglected. As such, in the example of FIG. 5, H₂(f)=[H₂₁(f), H₂₂(f), . . . , H_2I(f)]^T=[1, 0, . . . 0]^T. In other words, while in reality x₂is a reference audio signal that is generated by the endpoint controller of an endpoint and corresponds to source audio signal s₁, conceptually x₂may be considered a fictitious microphone signal equal to the source audio signal s₁, with no contributions from s₂-s_J. In addition, there is no noise added to x₂(i.e., n₂=0) because x₂is a loudspeaker signal known to endpoint 104 and not a microphone signal. Thus, the first row of H(f) is the unit vector H₂(f), so that (8) becomes
$\begin{matrix} H (f) = [\begin{matrix} 1 & 0 & \dots & 0 \\ H_{31} (f) & H_{32} (f) \\ ⋮ & ⋱ \\ H_{J 1} (f) & H_{JI} \end{matrix}], & (26) \end{matrix}$
and, since N₂(f)=0,
N(f)=[0 N ₃(f) . . . N _J(f)]^T. (27)
FIG. 6 is an example of the first category of active standby mode motion detection in which an endpoint controller uses a reference audio signal from a source audio speaker to predict the delayed time-frequency representation of a primary audio signal. As shown, endpoint 104 includes microphone 216(1), speaker 224(1), filter banks 226(1)-(2), adaptive filter 228(2), delay operator 230, and detector function 232. There are no external speakers in this example.
In this example, microphone signal x₁is the primary audio signal and loudspeaker signal x₂is the reference audio signal. As in FIG. 5, filter banks 226(1)-(2) respectively partition the audio signals x₁, x₂into time-frequency domain signals X₁(m,k), X₂(m,k). The delay operator 230 delays the time-frequency representation of the primary signal, X₁(m,k), by D frames, resulting in X₁(m−D, k). The endpoint controller feeds the time-frequency domain representations of the reference signal, X₂(m,k), through adaptive filter 228(2) and generates a predicted signal {circumflex over (X)}₁(m−D,k) that is predictive of the delayed time-frequency representation of the primary signal, X₁(m−D, k). The endpoint controller generates prediction error E(m−D,k)=X₁(m−D,k)−{circumflex over (X)}₁(m−D,k) by comparaing the delayed time-frequency representation of the primary audio signal with the predicted signal, and determines whether the prediction error is indicative of the motion of one or more persons near the endpoint 104.
In this case, the matrix in (8) reduces to the scalar
H(f)=1, (28)
the channel vector in (12) reduces to the scalar
Ĥ ₁(f)=H ₁₂(f), (29)
and
Ñ ₁(f)=N ₁(f), (30)
(13) thus simplifies to
X ₁(f)=H ₁₁(f)S ₁(f)+N ₁(f). (31)
Hence, the endpoint controller of the endpoint 104 may generate the predicted signal {acute over (X)}₁(m−D,k) by filtering X₂(m,k) through adaptive filter 228(2).
FIG. 7 is another example of the first category of active standby mode motion detection. In this example, the endpoint controller of the endpoint 104 predicts a delayed time-frequency representation of a primary audio signal from a reference audio signal from a source audio speaker, and a microphone signal. Endpoint 104 includes microphone 216(1), speaker 224(1), microphone 216(3), filter banks 226(1)-(3), adaptive filters 228(2)-(3), delay operator 230, and detector 232. Speaker 224(2) is external to the endpoint.
In this example, microphone signal x₁is the primary audio signal and audio signals x₂, x₃are reference audio signals. As in FIG. 5, filter banks 226(1)-(3) respectively partition the audio signals x₁, x₂, x₃into time-frequency domain signals X₁(m,k), X₂(m,k), X₃(m,k). The delay operator 230 delays the time-frequency representation of the primary signal, X₁(m,k), by D frames, resulting in X₁(m−D, k). The endpoint controller of the endpoint 104 respectively feeds the time-frequency domain representations of the reference audio signals, X₂(m,k), X₃(m,k), through adaptive filters 228(2)-(3) and generates a predicted signal {dot over (X)}₁(m−D,k) that is predictive of the delayed time-frequency representation of the primary signal, X₁(m−D, k). The endpoint controller generates prediction error E(m−D,k)=X₁(m−D,k)={circumflex over (X)}₁(m−D,k) by comparing the delayed time-frequency representation of the primary audio signal with the predicted signal, and determines whether the prediction error is indicative of the motion of one or more persons near the endpoint 104.
Mathematically the matrix in (8) reduces to
$\begin{matrix} H (f) = [\begin{matrix} 1 & 0 \\ H_{31} (f) & H_{32} (f) \end{matrix}] . & (32) \end{matrix}$
This matrix has linearly independent columns in general, which means that (13) can be rewritten as
X ₁(f)={tilde over (H)} ₁₂(f)X ₂(f)+{tilde over (H)} ₁₂(f)X ₃(f)+Ñ ₁(f), (33)
where {tilde over (H)}₁₂(f) and {tilde over (H)}₁₃(f) are functions of H₁₁(f), H₁₂(f), H₃₁(f), and H₃₂(f). The noise Ñ₁(f)=N₁(f)−{tilde over (H)}₁₃(f)N₃(f). Hence, the endpoint controller of the endpoint 104 may generate the predicted signal {tilde over (X)}₁(m−D,k) by filtering X₂(m,k), X₃(m,k) through adaptive filters 228(2)-(3).
As mentioned above, FIGS. 8-9 illustrate a second category of active standby mode embodiments in which an integrated speaker serves as a primary audio device. Turning first to FIG. 8, a generalized signal processing diagram is shown for predicting the delayed time-frequency representation of a primary audio signal x₁from a plurality of microphone signals x₂. . . x_Jin accordance with examples presented herein. Speakers 224(2)-(I) are external to endpoint 104, which includes speaker 224(1), and microphones 216(2)-(J). Endpoint 104 also includes filter banks 226(1)-(J), adaptive filters 228(2)-(J), delay operator 230, and detector function 232.
In this example, audio signal x₁is the primary audio signal and microphone signals x₂-x_Jare reference audio signals. Speakers 224(1)-(I) produce respective source audio signals s₁(t), s₂(t), . . . , s_I(t). Each microphone 216(1)-(J) captures the source audio signals s₁(t), s₂(t), . . . , s_I(t) after the source audio signals have traversed the room and, in response, produces a respective signal x₁, x₃, . . . , x_J. In addition, the endpoint controller of the endpoint generates primary audio signal x₁for source audio signal s₁since the speaker 224(1) is integrated into the endpoint 104 and, as such, the endpoint controller knows the source audio signal s₁. In other words, the speaker 224(1) serves as both an audio device and a primary audio device. Each signal x₂, x₃, . . . , x_Jis a superposition of source audio signals convolved with impulse responses (e.g., h₂₁, h₂₂, etc.) and a noise signal (e.g., n₂, etc.). For purposes of viewability, only the contributions and noise signal corresponding to the primary microphone 216(2) are shown.
The endpoint 104 detects motion by predicting the delayed time-frequency representation of the primary signal x₁(t) from the time-frequency representations of the reference signals x₂(t), x₃(t), . . . , x_J(t). In other words, X₁(m−D,k) is predicted from X₂(m,k), X₃(m,k), . . . , x_J(m,k).The endpoint controller of the endpoint 104 feeds each audio signal x₁, x₂, x₃, . . . , x_Jthrough a respective filter bank 226(1)-(J). These filter banks 226(1)-(J) respectively partition the audio signals x₁, x₂, x₃, . . . , x_Jinto time-frequency domain signals X₁(m,k), X₂(m,k), X₃(m,k), . . . , X_J(m,k), where m and k denote a frame index and a frequency bin index, respectively. The delay operator 230 delays the time-frequency representation of the primary signal, X₁(m,k), by D frames, resulting in X₁(m−D, k). The endpoint controller of the endpoint 104 respectively feeds the time-frequency domain representations of the reference audio signals, X₂(m,k), X₃(m,k), . . . , X_J(m,k), through adaptive filters 228(2)-(J). The endpoint controller sums the contributions of the filtered time-frequency domain representations of the reference audio signals to generate a predicted signal {circumflex over (X)}₁(m−D,k) that is predictive of the delayed time-frequency representation of the primary signal X₁(m−D, k). The endpoint controller generates prediction error E(m−D,k)=X₁(m−D,k)−{circumflex over (X)}₁(m−D,k) by comparing the delayed time-frequency representation of the primary audio signal with the predicted signal. As mentioned above, the endpoint controller determines whether the prediction error is indicative of the motion of one or more persons near the collaboration endpoint 104 and, if the endpoint controller determines that there is motion of one or more persons near the endpoint 104, the endpoint controller may activate endpoint components in anticipation of a multimedia meeting.
Like in the example of FIG. 5, much of the mathematical framework for passive mode motion detection described above applies here to active mode motion detection. In this case, H₁(f)=[H₁₁(f), H₁₂(f), . . . , H₁₁(f)]^T=[1, 0, . . . 0]^Tand N₁(f)=0.
FIG. 9 is an example of the second category of active standby mode motion detection in which an endpoint controller of the endpoint uses a reference signal from a microphone to predict the delayed time-frequency representation of a primary audio signal. As shown, endpoint 104 includes speaker 224(1), microphone 216(1), filter banks 226(1)-(2), adaptive filter 228(2), delay operator 230, and detector 232. There are no external speakers in this example.
In this example, audio signal x₁is the primary audio signal and audio signal x₂is the reference audio signal. As in FIG. 8, filter banks 226(1)-(2) respectively partition the audio signals x₁, x₂into time-frequency domain signals X₁(m,k), X₂(m,k). The delay operator 230 delays the time-frequency representation of the primary audio signal, X₁(m,k), by D frames, resulting in X₁(m−D, k). The endpoint controller of the endpoint 104 feeds the time-frequency domain representation of the reference signal, X₂(m,k), through adaptive filter 228(2) and generates a predicted signal {dot over (X)}₁(m−D,k) that is predictive of the delayed time-frequency representation of the primary signal, X₁(m−D, k). The endpoint controller generates prediction error E(m−D,k)=X₁(m−D,k)−{circumflex over (X)}₁(m−D,k) by comparing the delayed time-frequency representation of the primary audio signal with the predicted signal, and determines whether the prediction error is indicative of the motion of one or more persons near the endpoint 104.
In this case the matrix in (8) reduces to the scalar
H(f)=H ₂₁(f), (34)
the channel vector in (12) reduces to the scalar
$\begin{matrix} {\tilde{H}}_{1} (f) = \frac{1}{H_{21} (f)}, and & (35) \\ {\tilde{N}}_{1} (f) = - \frac{1}{H_{21} (f)} N_{2} (f) . & (36) \end{matrix}$
(13) then simplifies to
$\begin{matrix} X_{1} (f) = \frac{1}{H_{21} (f)} X_{2} (f) - \frac{1}{H_{21} (f)} N_{2} (f) . & (37) \end{matrix}$
Hence, the endpoint controller of the endpoint 104 may generate the predicted signal {circumflex over (X)}₁(m−D,k) by filtering X₂(m,k) through adaptive filter 228(2).
FIGS. 2-9 provide examples of linear prediction (i.e., a model that is linear in its prediction parameters but not necessarily its regressors, which are in this case the time-frequency representations of the reference audio signals). However, in certain situations non-linear prediction may be desirable. For example, the impulse responses in active mode may not always be sufficiently linear and time-invariant (e.g., when a third party amplifier card is used, when a codec is playing the audio source signal through a television with a poorly designed amplifier/speaker, etc.). As such, FIG. 10 provides an example signal diagram of active motion detection using non-linear prediction.
More specifically, FIG. 10 is an example signal diagram in which an endpoint controller of an endpoint uses a reference audio signal from a source audio speaker to predict the delayed time-frequency representation of a primary audio signal. As shown, endpoint 104 includes microphone 216(1), speaker 224(1), filter banks 226(1)-(2), non-linear adaptive filter 1028(2), delay operator 230, and detector 232. Endpoint 104 further includes a phase operator 1034, absolute value operator 1036, and a phase application operator 1038. There are no external speakers in this example.
In this example, the microphone signal x₁is the primary audio signal and audio signal x₂is the reference audio signal. As in FIG. 5, filter banks 226(1)-(2) respectively partition the audio signals x₁, x₂into time-frequency domain signals X₁(m,k), X₂(m,k). The delay operator 230 delays the time-frequency representation of the primary signal, X₁(m,k), by D frames, resulting in X₁(m−D, k). The endpoint controller of the endpoint 104 feeds the time-frequency domain representations of the reference signal, X₂(m,k), through adaptive filter 228(2) and generates a predicted signal {circumflex over (X)}₁(m−D,k) that is predictive of the delayed time-frequency representation of the primary signal, X₁(m−D, k). The endpoint controller generates prediction error E(m−D,k)=X₁(m−D,k)=X₁(m−D,k) by comparing the delayed time-frequency representation of the primary audio signal with the predicted signal, and determines whether the prediction error is indicative of the motion of one or more persons near the collaboration endpoint 104.
FIG. 11 is a block diagram of endpoint controller 120, which is configured to execute the operations described herein. In this example, the endpoint controller 120 includes one or more application-specific integrated circuits (ASICs) 1140, a memory 1142, which stores or is encoded with instructions for the motion detection logic 122, and one or more processors 1146. The one or more ASICs 1140 may be customized for one or more particular uses (e.g., audio/video signal processing). The one or more processors 1146 are configured to execute instructions stored in the memory 1142 (e.g., motion detection logic 122). When executed by the one or more processors 1146, the motion detection logic 122 enables the endpoint controller 120 to perform the operations described herein in connection with FIGS. 1-10 and 12.
The memory 1142 may include read only memory (ROM), random access memory (RAM), magnetic disk storage media devices, optical storage media devices, flash memory devices, electrical, optical, or other physical/tangible memory storage devices. Thus, in general, the memory 1142 may include one or more tangible (non-transitory) computer readable storage media (e.g., a memory device) encoded with software comprising computer executable instructions and when the software is executed (by the processor 1146) it is operable to perform the operations described herein.
FIG. 12 is a flowchart depicting a generalized method 1200 in accordance with examples presented herein. At 1210, an endpoint controller of a collaboration endpoint generates a primary audio signal for an ultrasonic source audio signal produced by a source audio speaker. At 1220, the endpoint controller generates a reference audio signal for the ultrasonic source audio signal. At 1230, the endpoint controller generates, based on the reference audio signal, a predicted signal that is predictive of the primary audio signal. At 1240, the endpoint controller produces a prediction error of the predicted signal by comparing the primary audio signal with the predicted signal. At 1250, the endpoint controller determines whether the prediction error is indicative of a motion of one or more persons near the collaboration endpoint.
Using the techniques presented herein, collaboration endpoints may accurately detect motion in a conference room in which the endpoint is located. Upon detecting motion, a collaboration endpoint may enter active mode and, for example, display a friendly and informative greeting to the user. Endpoint motion detection may also be used to trigger burglar alarms, control lights/blinds, record and/or analyze the usage of the conference room, provide up-to-date information regarding the usage of the conference room, etc.
These techniques are power efficient because the endpoint may constantly monitor for motion while in standby mode. Moreover, these techniques may be deployed on existing and/or future (to be developed) endpoints. In addition, extra/additional hardware may not be required, thereby minimizing cost and complexity. Further, impulse responses may be non-linear and/or time-variant due to, for example, systems having poor quality amplifier cards, or when an ultrasonic pairing signal is played through a television with poor quality amplifiers and/or speakers.
The techniques described herein may also be desirable in situations involving multiple collaboration endpoints in the same room that each produce ultrasonic pairing signals. This is often the case in office spaces and classrooms where each user has a personal endpoint in the same room. These techniques may also be compatible with ultrasonic noise from other equipment, such as third party motion detectors in the ceiling. In addition, they are operable in large rooms that are not properly covered with an optimal number of microphones.
The number of signals used for motion detection by the endpoint is at least one more than the number of audio source signals. Thus, since one signal is used as the primary signal, the number of reference audio signals is equal to at least the number of audio source signals. In addition, it may be desirable to locate microphones far from speakers. In an example, microphones may be located throughout a conference room so as to provide accurate motion detection coverage for the entire room.
In one form, a method is provided. The method comprises: at a controller of a collaboration endpoint, generating a primary audio signal for an ultrasonic source audio signal produced by a source audio speaker; generating a reference audio signal for the ultrasonic source audio signal; generating, based on the reference audio signal, a predicted signal that is predictive of the primary audio signal; producing a prediction error of the predicted signal by comparing the primary audio signal with the predicted signal; and determining whether the prediction error is indicative of motion of one or more persons near the collaboration endpoint.
In another form, an apparatus is provided. The apparatus comprises: a source audio speaker; at least one microphone of a collaboration endpoint; and one or more processors coupled to a memory, wherein the one or more processors are configured to: generate a primary audio signal for an ultrasonic source audio signal produced by the source audio speaker; generate a reference audio signal for the ultrasonic source audio signal; generate, based on the reference audio signal, a predicted signal that is predictive of the primary audio signal; produce a prediction error of the predicted signal by comparing the primary audio signal with the predicted signal; and determine whether the prediction error is indicative of a motion of one or more persons near the collaboration endpoint.
In another form, one or more non-transitory computer readable storage media are provided. The non-transitory computer readable storage media are encoded with instructions that, when executed by a processor of a collaboration endpoint, cause the processor to: generate a primary audio signal for an ultrasonic source audio signal produced by a source audio speaker; generate a reference audio signal for the ultrasonic source audio signal; generate, based on the reference audio signal, a predicted signal that is predictive of the primary audio signal; produce a prediction error of the predicted signal by comparing the primary audio signal with the predicted signal; and determine whether the prediction error is indicative of a motion of one or more persons near the collaboration endpoint.
The above description is intended by way of example only. Although the techniques are illustrated and described herein as embodied in one or more specific examples, it is nevertheless not intended to be limited to the details shown, since various modifications and structural changes may be made within the scope and range of equivalents of the claims.

Claims

What is claimed is:

1. A method comprising:

at a controller of a collaboration endpoint, generating a primary audio signal for an ultrasonic source audio signal produced by a source audio speaker;

generating a reference audio signal for the ultrasonic source audio signal;

generating, based on the reference audio signal, a predicted signal that is predictive of the primary audio signal;

producing a prediction error of the predicted signal by comparing the primary audio signal with the predicted signal; and

determining whether the prediction error is indicative of a motion of one or more persons near the collaboration endpoint.