JP2014191616A - Method and device for monitoring aged person living alone, and service provision system - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a method and a device for estimating the line of flow from footsteps for the purpose of not only detecting an abnormal state of an aged person living alone but also remotely monitoring the conditions of daily life while considering the privacy of the aged person, and to provide a service provision system which is integrated with other sensor information, thereby allowing a relative in a distant location or a caretaker to easily monitor daily activities of the aged person without causing the aged person to feel unrest or tension regarding the privacy.SOLUTION: As an application utilized by a general user for monitoring an aged person living alone, a method, a device and a service provision system are provided in which a sound recorder such as a microphone array is small-sized and easily installed and removed in a living environment, work such as complicated calibration is simplified as further as possible, additional expansion of the microphone array can be easily implemented when enlarging a monitoring range, and regarding interference of surrounding environmental noise, robustness is secured.

Description

  The present invention relates to a method for estimating a flow line from walking sound for the purpose of watching an elderly person living alone, an apparatus, and a service providing system.

  In Japan, which has reached a super-aged society, it is urgently necessary to develop information and communication technology and robot-based monitoring and life support technology so that elderly people living alone can live independently and safely.

  Conventionally, as disclosed in, for example, Patent Document 1, there is a technique of capturing a large number of elderly people with a small number of video cameras and storing video data for each elderly person in a nursing facility where a large number of elderly people live. However, such a method of directly storing video data may be difficult to operate from the viewpoint of privacy, particularly when applied to watching elderly people living alone.

  On the other hand, as disclosed in Patent Document 2, there is an action determination method for estimating a home state from an operation state of an electric device according to an attribute such as an age of a target person. By using information with a small amount of information about the subject, such as the operation status of electrical equipment, the subject's anxiety and tension related to privacy are alleviated, but the behavior of the subject is estimated from such information When it does, detailed discrimination | determination of action becomes difficult.

In addition, Non-Patent Document 1 attempts to realize life pattern classification and abnormality detection by performing advanced data mining processing on the information of pyroelectric sensors installed in each room.
However, in order to perform life pattern classification and abnormality detection, it is necessary to collect data in advance for a long period of time, and in order to extract necessary information from the data, advanced expertise is required. For this reason, the general operation of such a system is not easy.

  On the other hand, there are attempts to detect anomalies using sound. Compared to the operating status of electrical equipment and pyroelectric sensors, the amount of information that can be collected by using sound increases dramatically, and the privacy must be given to the subject as much as the indoor installation of the video camera. Be expected. For example, in the methods disclosed in Patent Document 3 and Patent Document 4, abnormal sounds are detected as deviations from everyday sounds. Further, Patent Document 4 realizes a function of determining an unusual sound when sound is generated from a direction in which sound is not generated on a daily basis by estimating the direction of arrival of the sound using a microphone array.

  Although these methods can detect the critical situation of the subject to some extent, they do not detect minor signs of abnormalities that cannot be said to be critical.

  For example, such conventional techniques cannot detect behavior that may be a sign of dementia, such as repeating the same daily behavior in the daily life of the elderly living alone. In addition, it is necessary to detect daily behavior such as how much you walk every day or how much time you spend in bed as a watch for maintaining the health of elderly people living alone, but it is difficult to estimate the amount of walking with conventional techniques.

  If it is possible to detect walking and estimate the flow line as daily activities, it is considered that the daily life of the elderly living alone can be captured to some extent. Conventionally, a method for estimating a walking position by collecting walking sounds with a microphone array and estimating an arrival time difference based on the PHAT method has been proposed in Non-Patent Document 2 and the like.

  This method can estimate the walking position well when there is only walking sound and no other environmental noise, but it can accurately estimate the time difference because it interferes with the walking sound when there is environmental noise. In addition, it is difficult to estimate the walking position. In general, since elderly people tend to be deaf, there is a tendency to set the volume of televisions, radios, etc. higher than the level heard by healthy people, and the influence of environmental noise cannot be ignored. For this reason, it cannot be said that it is realistic to use the method of nonpatent literature 2 for watching an elderly person living alone.

  Japanese Patent Application Laid-Open No. H10-228561 describes a method of installing a microphone array on a wall in a room and a mobile robot in order to improve the accuracy of sound source localization and sound source tracking in the room. However, fixing a wide microphone array to the wall of the entire room or using a mobile robot is large and expensive, and is not suitable for use in a general elderly living alone.

Patent application 2011-92636 "Monitoring video collection device" Patent application 2009-24765 “Behavior discrimination system and behavior discrimination method” Patent application 2009-249987 “Abnormality detection method and abnormality detection device” Patent application 2008-164395 “Unusual sound detection system” Patent application 2008-514510 “Sound source tracking system, method and robot”

Takeshi Mori, “Sensor Data Mining for Life Support (Development to“ Monitoring Engineering ”), IEICE Journal, Vol.94, No.4, pp.276-281,2011 Shoji, sound lecture (spring), 727-730, 2010.

  A method and device that realizes flow line estimation from walking sounds for the purpose of monitoring the daily life from a remote location, taking into account the privacy of the elderly as well as detecting the abnormal state of the elderly living alone, and others A service provision system that enables elderly relatives and caregivers to easily monitor the daily activities of the elderly without the feeling of anxiety and tension related to privacy by integrating with sensor information To do.

  However, the flow line estimation method, device and service provision system from walking sounds are considered to be used by general users in the application of watching elderly people living alone, and sound recording devices such as microphone arrays are compact in size. Easy installation and removal from the living environment, complicated calibration and other tasks are as simple as possible, and additional expansion of the microphone array can be easily realized when expanding the watch range, making it robust against ambient noise interference And a service providing system.

The present invention provides the following means (1) to (3).
(1) An apparatus for estimating the flow of a walking sound in an indoor space comprising a microphone array device and a sensor information integration device,
The sensor information integration device further includes a display unit for displaying the indoor space and a signal processing unit, and is connected to the microphone array device.
In the indoor space, two microphone arrays are grouped and arranged in a “C” shape,
The walking sound in the indoor space is recorded with one microphone array, the walking sound analog signal is AD converted, and the walking sound digital signal is generated,
The sound source position and direction of arrival are estimated from the walking sound digital signal using the MUSIC method (step 1),
The sound source is separated by a modified minimum dispersion beamformer (step 2),
Features are extracted from the separated sound sources, the likelihood of the acoustic model is calculated, and abnormal sounds are detected (step 3).
The number of people walking and the flow line are estimated by the particle filter (step 4),
An estimated walking flow line is displayed on a display unit of a sensor information integration device.
(2) The likelihood calculation of the acoustic model is performed using a walking sound / footstep acoustic model, the above-described apparatus for estimating the flow of walking sound in an indoor space.
(3) The weight estimation of the particle filter is performed by evaluating the sound source position observed in the past, and the flow line estimation of the walking sound in the indoor space according to any one of (1) to (2) Providing equipment.

Next, the present invention provides the following means (4) to (6).
(4) A method for estimating the flow of a walking sound in an indoor space, which is used in a device for estimating the flow of a walking sound in an indoor space comprising a microphone array device and a sensor information integration device,
The sensor information integration device further includes a display unit for displaying the indoor space and a signal processing unit, and is connected to the microphone array device.
In the indoor space, two microphone arrays are grouped and arranged in a “C” shape,
The walking sound in the indoor space is recorded with one microphone array, the walking sound analog signal is AD converted, and the walking sound digital signal is generated,
The sound source position and direction of arrival are estimated from the walking sound digital signal using the MUSIC method (step 1),
The sound source is separated by a modified minimum dispersion beamformer (step 2),
Features are extracted from the separated sound sources, the likelihood of the acoustic model is calculated, and abnormal sounds are detected (step 3).
The number of people walking and the flow line are estimated by the particle filter (step 4),
A method of estimating a flow line of walking sound in an indoor space.
(5) The likelihood calculation of the acoustic model is performed using an acoustic model of a walking sound / footstep, and the flow line estimation method for the walking sound in the indoor space as described above.
(6) The weight estimation of the particle filter is performed by evaluating a sound source position observed in the past, and the flow line estimation of the walking sound in the indoor space according to any one of (4) to (5) Provide a method.

Furthermore, the present invention provides the following means (7) to (8).
(7) The sensor information integration device of the flow estimation device for walking sound in the indoor space according to any one of (1) to (3) is further connected to an operation sensor of an electric device used in the indoor space,
The output signal of each sensor is transmitted to the sensor information integration device,
The sensor information integration device converts each output signal into operation information of the electrical device,
The sensor information integration device transmits the converted operation information of the electric device together with the estimated walking flow line information to a communication network,
The information terminal which has received the transmitted walking flow line information and operation information from the communication network has a display screen, and displays the walking flow line information in the indoor space at the time indicated by the time shift scroll bar and the operation information of the electric device. A remote indoor space monitoring service providing system that displays in real time on the screen.
(8) The electric room includes a bed pressure sensor, a wheelchair-mounted position / direction sensor, and a voice recognition device for a voice interactive robot. The remote indoor space monitoring system according to (7) is provided.

Finally, the present invention
(9) A program for executing the flow line estimation method for walking sound in an indoor space described in any one of (4) to (6) is provided.

  In addition to detecting the abnormal state of the elderly living alone, the purpose of the elderly is to monitor the daily life from a remote location while taking into account the privacy of the elderly. Remote relatives and caregivers can easily watch the daily behavior of the elderly.

It is a block diagram of a "C" shaped microphone array. It is a top view showing the example of installation of two "C" -shaped microphone arrays in indoor space. It is a flowchart showing the flow of the flow line estimation process from a walk sound. It is a figure showing the minimum structural example of the system which performs a walk flow line estimation. An installation example of four “C” -shaped microphone arrays when there is a shield. The connection example of a several microphone array apparatus and a sensor information integration apparatus. Connection example of various sensors and sensor information integration device. A configuration example of service provision using a cloud server. The example which reconfigure | reconstructed the elderly person's life condition on the screen of the information terminal by CG. Explanatory drawing of the sound reception function by a microphone array. Explanatory drawing of the flow line estimation by this invention. It is explanatory drawing of the Example by this invention, and is a figure showing 3 flow lines in case the mixing level of a television audio | voice is Clean, 1 dB, and -5 dB. In the explanatory diagram of the embodiment according to the present invention, the flow line when the mixing level of the television sound is Clean is shown. In the explanatory diagram of the embodiment according to the present invention, the flow line when the mixing level of the television sound is 1 dB is shown. In the explanatory diagram of the embodiment according to the present invention, the flow line when the mixing level of the television sound is −5 dB is shown.

The outline of the present invention is shown in FIG. 8 showing a configuration example of a service providing system using a cloud server, FIG. 6 showing a connection example of a plurality of microphone array devices and a sensor information integration device, and the life status of an elderly person on an information terminal. The screen is divided into three views, FIG. 9 and an example of reconfiguration by CG.
The present invention particularly relates to the microphone array device shown in FIG. 6 and its arrangement, and the walking flow line from the walking sound of the occupant in the indoor space collected in the sensor information integration device shown in FIG. 9 and FIG. Since the estimation method has characteristics, the embodiment will be described first, and the service providing system will be described later.

First, the microphone array apparatus shown in FIG. 6 and its arrangement will be described.
In the present invention, as shown in FIG. 1, two microphone array units in which four microphones are arranged on a straight line at intervals of 3 cm are spaced apart by, for example, a little less than 20 cm so as to form a letter “C”. ° Use two compact sized "C" shaped microphone arrays, characterized by a tilted arrangement (Figure 2).

  Regarding the microphone arrangement, if a single sound is generated at each position in the flow line detection area subject to sound source localization, the phase difference between the sound recorded by all microphones and the sound observed at the reference position It is necessary to consider an arrangement that uniquely determines the pattern. For a sound having a frequency within a certain range, the “C” shaped microphone array makes it difficult to estimate the position of a sound source located far away, but the direction of arrival can be identified.

  In order to make it possible to estimate the position even when walking far away from the microphone array to some extent, in the present invention, two “C” -shaped microphone arrays, for example, as shown in FIG. Fix to the floor with an interval of about 5m. With this configuration, the walking position can be estimated from the walking sound in a room of about a dozen square meters.

  Since the “C” shaped microphone array is compact, it can be easily attached to and removed from the indoor floor. Normally, the two microphone array units of the “C” -shaped microphone array are fixed with metal fittings or the like, so that the relative positional relationship of these eight microphones remains unchanged. Therefore, when two “C” -shaped microphone arrays are installed in an actual environment, it is necessary to determine the relative positions of all 16 microphones by measuring the distance between them and their respective directions. . In actual operation, the two microphone arrays are fixed on a straight line with the front facing both (Fig. 2). Thus, by measuring only the interval between the two units, the positional relationship of all 16 microphones can be clarified. The calibration operation is completed by inputting the measured distance to the apparatus of the present invention, and can be easily executed by a general user.

  Next, a walking flow line estimation method from the collected walking sounds of the occupants in the indoor space in the sensor information integration apparatus shown in FIGS. 9 and 6 will be described according to the flowchart of FIG.

  In order to robustly estimate the walking position even in the presence of environmental noise interference, in the present invention, as shown in FIG. 3, a MUSIC method capable of simultaneously estimating a plurality of sound source positions is used in step 1. Within the flow line detection area for estimating the walking position, the position of the sound source is localized on an equally spaced grid, and the direction of arrival is estimated for the sound source coming from outside the flow line detection area.

  In the subsequent step 2, a modified minimum dispersion beamformer is constructed based on the position information of these sound sources, and each sound source detected in the flow line estimation region is separated.

  In the present invention, in order to correctly distinguish only walking sounds and footsteps even in the presence of environmental noise, step 3 evaluates the likelihood of walking sound or footstepping of each separated sound source. In this method, feature quantities such as mel cepstrum (MFCC) are extracted in advance from learning samples for walking sounds and footsteps, and an acoustic model is learned using the probability quantities such as GMM and HMM. Then, by calculating the likelihood of the acoustic model for the feature amount obtained from each sound source separated, the likelihood of walking sound or footstep is numerically evaluated. In addition, based on the method of Patent Document 3, abnormal sounds such as falling sounds and screams are also detected.

  In step 4, based on the information on the position of each sound source and the likelihood of the acoustic model detected in the flow line estimation region for estimating the walking position, the number of simultaneous pedestrians and the number of each pedestrian are determined using a particle filter. Estimate the flow line.

  In this way, in the present invention, the likelihood of the acoustic model representing the likelihood of walking sound or footsteps of the separated sound source is evaluated, and further, based on the position information of the intermittently generated walking sound and footsteps and the likelihood of the acoustic model, By estimating the flow line by the particle filter, it is possible to realize the flow line estimation that is robust against environmental noise interference.

As shown in FIG. 4, the configuration of the apparatus for estimating the flow line from the actual walking sound is (1) sound source localization, (2) sound source separation, and (3) acoustic model likelihood calculation, as shown in FIG. Furthermore, (4) a microphone array device that detects abnormal sounds, and a sensor information integration device that estimates the number of pedestrians and flow lines using particle filters from the sound source position information and the likelihood of the acoustic model. The system is configured.
Note that the microphone array device may be configured to collect only analog acoustic signals and perform all signal processing below the sound source localization by the sensor information integration device (not shown).

  If the entire room cannot be covered with a single microphone array device due to shielding, etc. even in the same room, the other microphone array device is shielded as shown in FIG. 5 (in this example, the bed is a shield). Install on the opposite side of the object. At this time, in order to share the sound source localization coordinates between the two microphone array devices, it is necessary to measure the relative positional relationship between the microphone arrays. On the other hand, when estimating a flow line in different rooms in the same house, there is no need to share coordinates between microphone array devices placed in different rooms.

  In the device of the present invention, as shown in FIG. 6, it is also possible to use a wired / wireless network connection between the plurality of microphone array devices and the sensor information integration device. The sensor information integration device integrates the sound source position and the likelihood of the acoustic model sent asynchronously from a plurality of microphone array devices to perform flow line estimation. Therefore, it is easy to add a plurality of microphone array systems.

  The monitoring service for the elderly living alone according to the present invention is realized as follows. In addition to information on the walking flow line by acoustic sensors using a microphone array device, for example, as shown in FIG. 7, a pressure sensor is laid on the bed to determine whether an elderly person is sleeping on the bed and how many hours are spent on the bed. Such as information on the operating status of each electrical device such as a TV, radio, or light (power on / off, duration of each status, etc.), voice recognition by a voice dialogue robot, etc. Information such as direction sensors is also collected in the sensor information integration device, and when time information is insufficient for each information, the time information of the sensor information integration device is appropriately associated with each information and added to each information.

  In the microphone array apparatus, abnormal sound is also detected using the method disclosed in Patent Document 3, and when abnormal sound is detected, the information is sent to the sensor information integration apparatus together with the abnormal sound generation position. The sensor information is transferred together with the time information to an application server on a communication network or cloud as shown in FIG. Remote relatives and caregivers can use information terminals such as tablets and smartphones at hand to obtain information collected by various sensors at the elderly living alone by receiving from the communication network or accessing the application server. For example, as shown in FIG. 9, the state of daily life of an elderly person is displayed in real time in a three-dimensional CG on the screen of the information terminal at hand.

  In the figure, a sphere on a cylinder represents a person. In the figure, as an example, a person walking, a person sleeping on a bed, and a person on a wheelchair are displayed. By installing a pressure sensor on the wheelchair seat, it is possible to detect not only the position and direction of the wheelchair but also the state of whether or not a person is on board. The screen is black when the TV is off, and white when it is on. The light on / off status is also displayed. When a person or each electric device is clicked on the screen, information about that is displayed as characters.

For example, when each electric device is clicked, the current state and the duration of the state are displayed. When an abnormal sound is generated, a mark is displayed informing that the abnormal sound has occurred at the position on the CG. For example, if you are looking at the CG of the information terminal from a remote location and a mark is displayed informing you that an abnormal sound has occurred during walking, and the columnar person's symbol stops and stops moving for more than a certain time, It can be inferred that the elderly may have fallen. In the information terminal, operations such as rotation of the viewpoint and zooming are possible, and the state of the room can be observed from an arbitrary angle.
Below the 3D CG screen, there is a time-shifted scroll bar. By moving the scroll bar to the left and right, each information at the time indicated by the moved scroll bar is appropriately displayed on the 3D CG screen. You can go back to the past and check the living conditions of the elderly living alone.

  As described above, a system for monitoring the behavioral style of daily life of elderly living alone from a remote location is realized. Elderly living alone can get the peace of mind of watching the eyes of their relatives and caregivers from a distance without feeling anxiety or tension about privacy. Remote relatives and caregivers can also be assured because they can see more detailed lifestyles of elderly living alone.

に置かれた音源から出力された音響信号を、３次元空間中の任意の位置

に配置されたＱ個のマイクロフォンで受音する。音源と各マイクロフォン間の距離Ｒ_qは次式で求められる。 Hereinafter, a sound source process based on a digital sound signal recorded and collected by a microphone and subjected to AD conversion, and a flow line estimation process from the walking sound subjected to the sound source process will be described in detail.
(Step 1: Sub-step 1: Sound source localization)
The plurality of microphones can be arranged at arbitrary positions in the three-dimensional space. Arbitrary position in 3D space

An acoustic signal output from a sound source placed in

The sound is received by Q microphones arranged in the. The distance R _q between the sound source and each microphone can be obtained by the following equation.

音源から各マイクロフォンまでの伝播時間τ_qは、音速をｖとすると、次式で求められる。

各マイクロフォンで受音した中心周波数ωの狭帯域信号の、音源のそれに対する利得ｇ_qは、一般的に、音源とマイクロフォン間の距離Ｒ_qと中心周波数ωの関数として定義される。

The propagation time τ _q from the sound source to each microphone can be obtained by the following equation, where the speed of sound is v.

The gain g _q of the narrow band signal having the center frequency ω received by each microphone relative to that of the sound source is generally defined as a function of the distance R _q between the sound source and the microphone and the center frequency ω.

例えば、利得を距離Ｒ_qだけの関数として、実験的に求めた次式のような関数を用いる。

For example, a function such as the following expression obtained experimentally is used with the gain as a function of the distance R _q .

と表される。そして、位置Ｐ₀にある音源を表す位置ベクトルａ(ω，Ｐ₀)を、次式のように、狭帯域信号に関する、音源と各マイクロフォン間の伝達特性を要素とする複素ベクトルとして定義する。 The transfer characteristics between the sound source and each microphone for the narrowband signal with the center frequency ω are:

It is expressed. Then, the position vector a (ω, P ₀ ) representing the sound source at the position P ₀ is defined as a complex vector whose element is the transfer characteristic between the sound source and each microphone regarding the narrowband signal, as in the following equation.

音源位置の推定はＭＵＳＩＣ法(相関行列を固有値分解することで信号部分空間と雑音部分空間を求め、任意の音源位置ベクトルと雑音部分空間の内積の逆数を求めることにより、音源の音波到来方向や位置を調べる手法)を用いて、以下の手順で行う。ｑ番目のマイクロフォン入力の短時間フーリエ変換を (Step 1: Sub-step 2: Estimation of sound source position)

The sound source position is estimated by the MUSIC method (the signal subspace and the noise subspace are obtained by eigenvalue decomposition of the correlation matrix, and the reciprocal of the inner product of the arbitrary sound source position vector and the noise subspace is obtained. The following procedure is performed using a method for checking the position. Short-time Fourier transform of qth microphone input

で表し、これを要素として観測ベクトルを次のように定義する。

ここで、ｎはフレーム時刻のインデックスである。連続するＮ個の観測ベクトルから相関行列を次式により求める。

The observation vector is defined as follows using this as an element.

Here, n is an index of frame time. A correlation matrix is obtained from the continuous N observation vectors by the following equation.

この相関行列の大きい順に並べた固有値を

とし、それぞれに対応する固有ベクトルを

The eigenvalues arranged in descending order of this correlation matrix

And the corresponding eigenvectors

とする。そして、音源数Ｓを次式により推定する。

もしくは、固有値に対する閾値を設け、その閾値を超える固有値の数を音源数Sとすることも可能である。
雑音部分空間の基底ベクトルから行列Ｒ_n(ω)を次のように定義し、

And Then, the number S of sound sources is estimated by the following equation.

Alternatively, a threshold value for the eigenvalue may be provided, and the number of eigenvalues exceeding the threshold value may be set as the sound source number S.
Define the matrix R _n (ω) from the base vector of the noise subspace as

および音源定位の探索領域Ｕを

として、 frequency band

And search area U for sound source localization

As

を計算する。そして、関数Ｆ(Ｐ)が極大値をとる座標ベクトルを求める。ここでは仮にＳ個の極大値を与える座標ベクトルがＰ₁，Ｐ₂，・・・，Ｐ_sが推定されたとする。次にその各々の座標ベクトルにある音源のパワーを次式により求める。

Calculate Then, a coordinate vector in which the function F (P) takes a maximum value is obtained. Here, it is assumed that P ₁ , P ₂ ,..., P _s are estimated as coordinate vectors that give S maximum values. Next, the power of the sound source at each coordinate vector is obtained by the following equation.

Then, two threshold values F _thr and P _thr are prepared, and when F (P _s ) and P (P _s ) in each position vector satisfy the following conditions,

It is determined that sound has occurred in the coordinate vector P ₁ within N consecutive frame times.
In the sound source position estimation process, consecutive N frames are processed as one block. In order to estimate the sound source position more stably, it is determined that sound is generated when the number N of frames is increased and / or the condition of Expression (30) is satisfied in all of the consecutive Nb blocks. The number of blocks is set arbitrarily. When the sound source is moving at such a speed that the sound source can be seen to be approximately stationary within the time period of consecutive N frames, the moving miracle of the sound source can be captured by the above method.

(Step 1: Sub-step 3: Estimating direction of sound wave arrival of ambient noise)
A method for estimating the direction in which sound waves of a sound source at a long distance from the microphone array arrive will be described below.
The plurality of microphones can be arranged at arbitrary positions in the three-dimensional space. Sound waves coming from a long distance are considered to be observed as plane waves.

FIG. 10 is an explanatory diagram for explaining a sound receiving function using the microphone array of the present invention.
FIG. 10 shows, as an example, a case where three microphones m1, m2, and m3 arranged at arbitrary positions receive sound waves coming from a sound source. In FIG. 10, a point c indicates a reference point, and the arrival direction of the sound wave is estimated around this reference point. In FIG. 10, the plane s indicates a cross section of a plane wave including the reference point c. The normal vector n of the plane s is defined as the following equation, with the direction of the vector opposite to the propagation direction of the sound wave.

３次元空間中の音源の音波到来方向は２つのパラメータ(θ，φ)で表される。方向(θ，φ)から到来する音波を各マイクロフォンで受音し、そのフーリエ変換を求めることで受音信号を狭帯域信号に分解し、各受音信号の狭帯域信号毎に利得と位相を複素数として表し、それを要素として狭帯域信号毎に全受音信号分だけ並べたベクトルを音源の位置ベクトルと定義する。以下の処理において、方向(θ，φ)から到来する音波は、前述の位置ベクトルとして表現される。位置ベクトルは具体的に以下のように求められる。ｑ番目のマイクロフォンと平面ｓの間の距離ｒ_qを次式により求める。

The sound wave arrival direction of the sound source in the three-dimensional space is represented by two parameters (θ, φ). Sound waves arriving from the direction (θ, φ) are received by each microphone, and the received signal is decomposed into narrowband signals by obtaining the Fourier transform, and the gain and phase are determined for each narrowband signal of each received signal. A vector that is expressed as a complex number and is arranged as an element for all received sound signals for each narrowband signal is defined as a position vector of the sound source. In the following processing, the sound wave coming from the direction (θ, φ) is expressed as the position vector described above. Specifically, the position vector is obtained as follows. A distance r _q between the q-th microphone and the plane s is obtained by the following equation.

距離ｒ_qは平面ｓに関してマイクロフォンが音源側に位置すれば正となり、逆に音源と反対側にある場合は負の値をとる。音速をｖとするとマイクロフォンと平面ｓ間の伝播時間Ｔ_qは次式で表される。

The distance r _q is positive when the microphone is located on the sound source side with respect to the plane s, and is negative when the microphone is on the opposite side of the sound source. When the speed of sound is v, the propagation time T _q between the microphone and the plane s is expressed by the following equation.

平面ｓでの振幅を基準としてそこから距離ｒ_q離れた位置の振幅に関する利得を、狭帯域信号の中心周波数ωと距離ｒ_qの関数として次のように定義する。

平面ｓでの位相を基準としてそこから距離ｒ_q離れた位置の位相差は、次式で表される。

The gain related to the amplitude at a distance r _q away from the amplitude in the plane s is defined as a function of the center frequency ω of the narrowband signal and the distance r _q as follows.

Phase difference between a position away distance r _q therefrom a phase as a reference in a plane s is expressed by the following equation.

以上より、平面ｓを基準として、各マイクロフォンで観測される狭帯域信号の利得と位相差は次式で表される。

From the above, with the plane s as a reference, the gain and phase difference of the narrowband signal observed by each microphone are expressed by the following equations.

When observing sound waves coming from the (θ, φ) direction with Q microphones, the position vector of the sound source is defined as the following equation as a vector whose elements are values obtained according to Equation (26) for each microphone. The

として、 When the position vector of the sound source is defined, the direction of arrival of the sound wave is estimated using the MUSIC method. Using the matrix R _n (ω) given by Equation (15), the search region I for sound wave arrival direction estimation is

As

Calculate Then, the direction (θ, φ) in which the function J (θ, φ) gives the maximum value is obtained. Here, it is assumed that there are K sound sources, and K sound wave arrival directions ((θ ₁ , φ ₁ ),..., (Θ _k , φ _k )) that give maximum values are estimated. Next, the power of the sound source in each sound wave arrival direction is obtained by the following equation.

そして、２つの閾値Ｊ_thr，Ｑ_thrを用意し、各到来方向におけるＪ(θ_k，φ_k)とＱ(θ_k，φ_k)が次の条件を満足するときに、

Then, two threshold values J _thr and Q _thr are prepared, and when J (θ _k , φ _k ) and Q (θ _k , φ _k ) in each arrival direction satisfy the following conditions,

It is determined that there is a utterance in the direction of arrival (θ _k , φ _k ) within N consecutive frame times. In the process of estimating the direction of arrival of sound waves, N consecutive frames are processed as one block. In order to estimate the direction of arrival more stably, if the number of frames N is increased and / or if the condition of equation (31) is satisfied in all the consecutive Nb blocks, the sound wave has arrived from that direction. to decide. The number of blocks is set arbitrarily. When the sound source is moving at such a speed that the sound source can be seen to be approximately stationary within the time period of consecutive N frames, the moving miracle in the direction of arrival of the sound wave can be captured by the above method. .

(Step 2: Sound source separation processing)
The sound generation position or direction of arrival is obtained by the sound source localization process. Further, the sound source position or direction of arrival of ambient noise has already been estimated. Using these estimation results, the sound source position vector of Equations (8) and (27), and σ representing the variance of omnidirectional noise, the matrix V (ω) is defined as follows.

この相関行列の大きい順に並べた固有値を

The eigenvalues arranged in descending order of this correlation matrix

とし、それぞれに対応する固有ベクトルを

とする。

And the corresponding eigenvectors

And

そして、近距離の座標ベクトルＰで発生した音源を強調する分離フィルタＷ(ω)は、次式で与えられる。 Here, since the correlation matrix V (ω) includes (S + K) sound sources including S short-range sound sources and K long-distance sound sources, the eigenvalues and eigenvectors of (S + K) in descending order of eigenvalues. Is used to define Z (ω) as follows:

A separation filter W (ω) that emphasizes the sound source generated by the short-distance coordinate vector P is given by the following equation.

式(３６)の分離フィルタに式(１０)の観測ベクトルを乗じることで座標ベクトルＰの位置で発生した音ｖ(ω)が得られる。

The sound v (ω) generated at the position of the coordinate vector P is obtained by multiplying the separation filter of Expression (36) by the observation vector of Expression (10).

この強調された音源の波形信号は式(３７)の逆フーリエ変換を計算することで求められる。

The enhanced waveform signal of the sound source can be obtained by calculating the inverse Fourier transform of equation (37).

On the other hand, a separation filter M (ω) for emphasizing ambient noise coming from a long distance direction (θ, φ) is given by the following equation.

The ambient noise v (ω) coming from the direction (θ, φ) is obtained by multiplying the separation filter of equation (38) by the observation vector of equation (10).

とする。また各位置における分離音源は、式(３６)および式(３７)により求められる。この分離音源からＭＦＣＣ ｆ_k,jを求め、予め足音から学習した音響モデルＧＭＭまたはＨＭＭに基づいて尤度ｌ_k,jを求める。 (Step 3: Feature extraction from separated sound source, likelihood calculation of acoustic model, detection of abnormal sound)
The above-described sound source localization processing is performed on a flat equidistant grid within the walking position detection region.
The number of sound sources detected by the equation (14) at time t is Nz (t), and the coordinates are

And In addition, the separated sound source at each position is obtained by Expression (36) and Expression (37). MFCC f _{k, j} is obtained from the separated sound source, and likelihood l _{k, j} is _obtained based on the acoustic model GMM or HMM previously learned from footsteps.

ここでＬ_*()は次式のリミッタ関数である。

またδ^*，ξ^*は次式のプロセス雑音である。

The state variables of the i-th particle at time k are coordinates x ⁽ⁱ⁾ _k , y ⁽ⁱ⁾ _k and their time differences Δx ⁽ⁱ⁾ _k , Δy ⁽ⁱ⁾ _k, and these state variables are the following process equations: Assuming that The total number of particles is N _p (i = 1,..., N _p ).

Here, L _* () is a limiter function of the following equation.

Also, δ ^* and ξ ^* are process noise of the following equation.

と、状態変数の座標ｘ⁽ⁱ⁾ _k，ｙ⁽ⁱ⁾ _kの間に次の観測方程式の関係を仮定する。

ここでζ^*は次式の観測雑音である。

Observation coordinates sent from the microphone array system at time k,

And the relationship of the following observation equation between the coordinates x ⁽ⁱ⁾ _k and y ⁽ⁱ⁾ _k of the state variables.

Here, ζ ^* is the observation noise of the following equation.

The particle weight w ⁽ⁱ⁾ _k is updated by the following equation.

In order to suppress the influence on the flow line estimation caused by environmental noise other than walking sounds and footsteps, the likelihood l _{k, j} of an acoustic model representing the likelihood of walking sounds or footsteps is introduced. In addition, the sound source position obtained by the sound source localization processing may include a large error suddenly due to the influence of ambient noise, etc. In order to suppress the influence of such a sudden error, the weight at time k is updated. In addition, coordinates that have been observed in the past up to N _k before including the current time are also included in the calculation. In addition, position information sent asynchronously from a plurality of microphone array devices has a difference in estimated time between the microphone devices. In order to integrate these pieces of information, it is necessary to have a certain time width. .

However, the forgetting factor α, (0 <α ≦ 1) is introduced in order to reduce the degree of influence as it becomes the past. In general, since elderly people walk slowly, the influence of delay caused by including past walking positions is limited. The likelihood of a particle is given by

重みの降順にパーティクルをソートする。最大の重みが閾値ＴＨＲ_wより小さい場合、足音はないものと判断する。
最大の重みに一定の割合β，(０≦β＜１)を乗じた値を閾値とする。その閾値ＴＨＲ_β以上の重みを持つパーティクルを対象に、以下の手順で互いに距離が一定値Ｄ以上離れるＮ_s個のクラスＣ_k,s，ｓ＝１,…,Ｎ_sに分類する。クラスＣ_k,sは、このクラスに属するパーティクル番号ｉの集合である。この時Ｎ_sは同時に歩いている人数を表す。 Next, weight normalization is performed by the following equation.

Sort particles in descending order of weight. If the weight of the maximum threshold THR _w smaller, it is determined that there is no footsteps.
A value obtained by multiplying the maximum weight by a certain ratio β and (0 ≦ β <1) is set as a threshold value. Targeting particles having a weight of more than the threshold THR _beta, N _s number of class C _k away distance more than a predetermined value D to each other by the following _{steps, s, s = 1, ...} , classified into N _s. Class C _{k, s} is a set of particle numbers i belonging to this class. At this time, N _s represents the number of people walking at the same time.

First, _assuming that N _s = 1, particles within the distance D from the coordinates of the most weighted particle are taken out and classified into the same class. Next, a particle having the maximum weight among unclassified particles is selected and the number of classes N _s is increased by 1. Similarly, particles having a distance within D among unclassified particles are classified into the same class. .

This process is repeated until all particles having a weight equal to or greater than the threshold THR _β are classified.
Then, the estimated value of the coordinates of each class is obtained by the following equation.

Finally, resampling is performed so that all particles have a uniform weight.

(Step 4: Estimate number of people walking and traffic lines)
At time k, the number of footsteps is estimated to be N _s, and the coordinates of each footstep are estimated by equation (50). In order to obtain the walking flow line, it is necessary to link the estimation result at time k and the estimation result at time k-1. This is done as follows.

  FIG. 11 shows that particles are classified into four classes at time k−1, the probability distribution of each class is represented as a solid line, and the coordinates estimated in each class are between the probability distributions indicated by the solid line. It is assumed that it was estimated like a point. It is also assumed that at time k, the particles are classified into three classes, and the footstep coordinates are estimated as points between probability distributions indicated by dotted lines in each class. The following processing is performed for each of the three coordinates estimated at time k.

First _, taking the class C _{k, 1} as an example, consider a circle with a radius R around the estimated coordinates, and select the closest class among the classes at time k−1 in which the coordinates fall. In this example, class C _k-1,1 is selected, and the flow line is extended from the coordinates of class C _{k-1,1 to} the coordinates of class C _{k, 1} . In the case of class C _{k, 2} , there are two classes C _k-1,2 and C _{k-1,3 in} a circle with radius R. In this case, the class C _k-1,2 having the smallest distance is selected, and the flow line is extended from the coordinates of the class C _{k-1,2 to} the coordinates of the class C _{k, 2} . Class C _k-1,3 disappears without being selected.

In the case of class C _{k, 3} , the previous estimation result does not exist in the circle of radius R, so that it becomes the starting point of a new flow line. In the case of class C _{k, 3} , since the extension of the flow line is not found, it disappears.
On the other hand, there are cases where it is better that the flow line is connected without interruption, such as when walking again after stopping for a short time. In such a case, even if the above annihilation condition is satisfied, it does not immediately disappear, but a Time To Live (TTL) counter is prepared, and after the annihilation condition is satisfied, it is stored until the TTL counter becomes zero. Processing such as keeping the coordinates is performed.

  As shown in the layout shown in FIG. 2, a simple living space is constructed in the laboratory, and a sliding foot is drawn so as to draw a circle in the flow line detection area (X axis [250: 550] × Y axis [0: 300]). The footstep sound when walking several times and TV sound as environmental noise were recorded separately. The volume of the TV was set to a level considered normal by the ears of healthy individuals. The SNR calculated between the TV sound and the walking sound was over 1 dB. In the experiment, walking sound alone (Clean), mixing with normal level TV sound (SNR 1 dB), and elderly people tend to be deaf, so the volume of the TV is set higher. (SNR-5 dB) to generate a noise superimposed walking sound.

  FIG. 12 shows a plot of coordinates at each time of the flow line estimated by the proposed method from these three kinds of walking sounds with different mixing levels of television sound. In the figure, + indicates a result of Clean, x indicates a result of 1 dB, and * indicates a result of −5 dB. Further, each flow line is shown independently in FIGS. Sound source localization was performed on a grid with 10 cm intervals. In both results, a flow line that draws a circle as it walks is obtained. Further, when these three types of results are compared with each other, they are almost the same, and thus it is understood that the flow line estimation robust against the interference of the television sound can be performed.

In the method of the present invention, a portable PC having a display unit was used as a sensor information integration device, and verification was performed using a program created in C language.
However, the language of the program that executes the method of the present invention is not limited.
There may be different modules depending on the execution environment and the mounting environment.
The execution environment of the program may be a microphone array device, a sensor information integration device, a cloud or an application server, and the functions may be integrated into one device or may be distributed.
The implementation of the program in the device or system may be read at startup, read at execution time, or previously installed in the device.

The present invention has been proposed as a watch system based on elderly singles living mainly in indoor spaces, but in addition, abnormal behavior of (small) animals such as attics and under floors, (night) zoo animals ( It can also be used for remote monitoring.
Since the characteristics of walking sounds and footsteps differ between humans and animals, flow line estimation is optimized if an acoustic model of the walking sound of the target animal is created in advance (monitoring).

DESCRIPTION OF SYMBOLS 1 Microphone 2 Microphone 3 Microphone array apparatus 4 Sensor information integration apparatus 5 Hub 6 Bed pressure sensor 7 Wheelchair position direction sensor / boarding detection sensor 8 Spoken dialogue robot 9 Monitor (operation status of electrical equipment)
10 Information terminals such as tablets and smartphones 11 Cloud 12 Application server 13 Communication network

Claims (9)

A device for estimating the flow of a walking sound in an indoor space comprising a microphone array device and a sensor information integration device,
The sensor information integration device further includes a display unit for displaying the indoor space and a signal processing unit, and is connected to the microphone array device.
In the indoor space, two microphone arrays are grouped and arranged in a “C” shape,
The walking sound in the indoor space is recorded with one microphone array, the walking sound analog signal is AD converted, and the walking sound digital signal is generated,
The sound source position and direction of arrival are estimated from the walking sound digital signal using the MUSIC method (step 1),
The sound source is separated by a modified minimum dispersion beamformer (step 2),
Features are extracted from the separated sound sources, the likelihood of the acoustic model is calculated, and abnormal sounds are detected (step 3).
The number of people walking and the flow line are estimated by the particle filter (step 4),
An estimated walking flow line is displayed on a display unit of a sensor information integration device.   The apparatus of claim 1, wherein the likelihood calculation of the acoustic model is performed using an acoustic model of a walking sound / footstep.   The apparatus for estimating a flow line of walking sound in an indoor space according to any one of claims 1 to 2, wherein the weighting of the particle filter is performed by evaluating a sound source position observed in the past. . An indoor space walking sound flow line estimation method used in a indoor space walking sound flow line estimation device comprising a microphone array device and a sensor information integration device,
The sensor information integration device further includes a display unit for displaying the indoor space and a signal processing unit, and is connected to the microphone array device.
In the indoor space, two microphone arrays are grouped and arranged in a “C” shape,
The walking sound in the indoor space is recorded with one microphone array, the walking sound analog signal is AD converted, and the walking sound digital signal is generated,
The sound source position and direction of arrival are estimated from the walking sound digital signal using the MUSIC method (step 1),
The sound source is separated by a modified minimum dispersion beamformer (step 2),
Features are extracted from the separated sound sources, the likelihood of the acoustic model is calculated, and abnormal sounds are detected (step 3).
The number of people walking and the flow line are estimated by the particle filter (step 4),
A method of estimating a flow line of walking sound in an indoor space.   5. The method for estimating a flow of a walking sound in an indoor space according to claim 4, wherein the likelihood calculation of the acoustic model is performed using an acoustic model of a walking sound / footstep.   The method for estimating a flow line of walking sound in an indoor space according to any one of claims 4 to 5, wherein the weighting of the particle filter is performed by evaluating a sound source position observed in the past. . The sensor information integration device of the flow estimation device for walking sound in the indoor space according to any one of claims 1 to 3, is further connected to an operation sensor of an electric device used in the indoor space,
The output signal of each sensor is transmitted to the sensor information integration device,
The sensor information integration device converts each output signal into operation information of the electrical device,
The sensor information integration device transmits the converted operation information of the electric device together with the estimated walking flow line information to a communication network,
The information terminal which has received the transmitted walking flow line information and operation information from the communication network has a display screen, and displays the walking flow line information in the indoor space at the time indicated by the time shift scroll bar and the operation information of the electric device. A remote indoor space monitoring service providing system that displays in real time on the screen.   8. The remote indoor space monitoring system according to claim 7, wherein the electric device includes a bed pressure sensor, a position / direction sensor mounted on a wheelchair, and a voice recognition device of a voice interactive robot.   The program for performing the flow line estimation method of the walking sound of indoor space described in any one of Claims 4 thru | or 6.

Images