JP5279010B2 - Wearable sound recognition device - Google Patents

Description

  The present invention relates to a wearable sound recognition device that can be worn by a hearing-impaired person or the like and can inform the hearing-impaired person or the like of the direction and type of sound.

  The basic functions of grasping the surrounding environment by sound are sound source localization for identifying the sound source direction and sound source recognition (sound source identification) for identifying the sound source type. In order to inform a hearing-impaired person or the like of ambient sounds on a daily basis, it is desirable that the hearing-impaired person or the like be worn, that is, a wearable device. The following Patent Document 1 describes a spectacle-type head mounted display device for a hearing impaired person that can visually recognize ambient sounds.

In addition, the following patent documents 2-4 are in the related application by the applicant. The following Patent Document 5 and Non-Patent Documents 1 and 2 describe a sound source recognition device (sound source identification device) using a neural network, and the following Non-Patent Documents 3 to 5 describe sound source localization using a neural network. An apparatus is described. Furthermore, there are the following non-patent documents 6 to 9 as related documents.
JP 2007-334149 A JP 2007-164027 A JP 2008-77177 A JP 2008-85472 A Japanese Patent No. 3164100 Shinya Sakaguchi, “Research on sound source recognition using pulsed neuron model”, Nagoya Institute of Technology Graduation thesis, March 1998 Sakaguchi Shinya, Kuroyanagi Shu, Iwata Akira, "Sound Source Identification System for Understanding the Environment", IEICE Technical Committee, IEICE Technical Report, December 1999, NC99-70, p. 61-68 Kuroyanagi Shu, Akira Iwata, "Sound source direction perception by pulse transmission type auditory neural circuit model-Extraction of time difference and sound pressure difference-" Technical report of IEICE, IEICE, March 1993, NC92- 149, p. 163-170 Kuroyanagi Shu, Akira Iwata, “Supervised Learning Rules for Pulsed Neuron Model”, IEICE Technical Report, IEICE, March 1998, NC97-151, p. 95-102 Kuroyanagi Shu, Hirata Koichi, Iwata Akira, “Competitive Learning Method for Pulsed Neural Networks”, IEICE NC Study Technical Report, Institute of Electronics, Information and Communication Engineers, March 2002, NC2001-210, p. 113-120 Hiroyuki Nakao, Shu Kururoyanagi, Akira Iwata, "Sound Image Extraction Model Using Direction Information of Sound Source Using Pulse Neural Network", IEICE Technical Committee Report, IEICE, March 2001, NC2000 -108, p. 39-46 Tanaka Aihisa, Kuroyanagi Shu, Iwata Akira, “Hardware Implementation Method of Neural Network for FPGA”, IEICE NC Study Technical Report, The Institute of Electronics, Information and Communication Engineers, March 2001, NC2000-179 , P. 175-182 Noriyoshi Futaki, Susumu Kuroyanagi, Akira Iwata, “Implementation Method of Pulsed Neuron Model for FPGA”, IEICE NC Research Technical Report, The Institute of Electronics, Information and Communication Engineers, March 2002, NC2001-211, p . 121-128 Kuroyanagi Shu, Akira Iwata, “Competitive Learning Neural Network Using Pulsed Neuron Model for Auditory Information Processing System”, IEICE Transactions (D-II), July 2004, Vol. J87-D-II No. 7, p. 1496-1504

  However, since the head mounted display device for the hearing impaired person of the said patent document 1 displays the recognition result of a sound by character information on a part of visual field, there existed a possibility of disturbing a user's visual field.

  SUMMARY OF THE INVENTION The present invention solves the above-described problems, and an object thereof is to provide a sound recognition device that can inform the user of the type and direction of the sound source without interfering with the user's visual field. .

The wearable sound recognition device of the present invention is an eyeglass-integrated type worn by a user, and includes a left microphone provided on the left vine of the glasses and a right microphone provided on the right vine of the glasses. Based on the time difference between the microphone and the left input sound input from the left microphone and the right input sound input from the right microphone, which sound source direction the sound source is in a plurality of sound source directions And a sound source localization unit that outputs direction information indicating the sound source direction and a frequency pattern of at least one of the left input sound and the right input sound, and the sound source A sound source identification unit that identifies whether the type is a sound source, and outputs type information indicating the sound source type, and a plurality of vibration units provided in parts that contact the user when worn by the user, With Different vibration patterns are set for each of the sound source types, and one or more of the vibration units vibrate in a vibration pattern for the sound source type indicated by the type information based on the direction information and the type information. And at least one different vibration part for different sound source directions, the left vibration part provided on the left vine of the glasses as the vibration part, A right vibration part provided on the right vine of the glasses, and when the sound source direction is left, the left vibration part vibrates with a vibration pattern for the sound source type indicated by the type information and the right vibration parts are not vibrating, the vibrating portion of the left oscillating vibration pattern for tone generator type vibration part of the right represented by the type information when the sound source direction is right configured so as not to vibrate this The features.

  According to this, the vibration part is provided in a part that comes into contact with the user when worn by the user, and at least one different vibration part vibrates depending on the direction of the sound source, and the vibration pattern varies depending on the type of the sound source. Since the vibration part vibrates, the direction of the sound source can be determined by which vibration part vibrates, and the type of the sound source can be determined by the vibration pattern. Therefore, the user can be informed of the type and direction of the sound source without disturbing the user's visual field.

Another wearable sound recognition apparatus of the present invention is an attachment type that can be attached to ready-made glasses, and can be attached to the left vine that can be attached to the left vine of the glasses and the right vine of the glasses. A right mounting portion, a left microphone provided in the left mounting portion, a right microphone provided in the right mounting portion, a left input sound input from the left microphone, and the right A sound source localization unit that identifies a sound source direction of a plurality of sound source directions based on a time difference from the right input sound input from the microphone of the sound source and outputs direction information indicating the sound source direction; Based on a frequency pattern of at least one of the left input sound and the right input sound, the sound source is identified from among a plurality of sound source types, and type information indicating the sound source type is output. Sound An identification part; a left vibration part provided in a part of the left attachment part that comes into contact with the user when the user wears glasses wearing the wearable sound recognition device; and the right attachment And a right vibration part provided at a part that comes into contact with the user when the user wears the glasses equipped with the wearable sound recognition device, and is different for each of the sound source types. When a vibration pattern is set and the sound source direction is left based on the direction information and the type information, the left vibration unit vibrates with a vibration pattern for the sound source type indicated by the type information and The vibration unit does not vibrate, and when the sound source direction is right, the right vibration unit vibrates with a vibration pattern for the sound source type indicated by the type information, and the left vibration unit does not vibrate. Features .

  According to the wearable sound recognition apparatus of the present invention, since the direction and type of the sound source are notified by vibration, it is possible to notify the user of the type and direction of the sound source without obstructing the user's visual field.

  Hereinafter, an embodiment of the present invention will be described with reference to the drawings.

  As shown in FIGS. 1 and 2, the sound recognition apparatus 1 includes left and right microphones 2L and 2R, a recognition processing unit 4 to which the microphones 2L and 2R are connected, a power source 5 connected to the recognition processing unit 4, and a recognition unit. The left and right vibrating parts 3L and 3R are connected to the processing part 4, respectively. The sound recognition device 1 is integrated with the glasses G, the microphone 2L is attached to the outside of the left vine of the glasses G, and the microphone 2R is attached to the outside of the right vine of the glasses G. The vibration unit 3L is attached to the inside of a portion of the glasses G that contacts the user's ear of the left vine, and the vibration unit 3R contacts the user's ear of the right vine of the glasses G. It is attached inside the part. Further, the recognition processing unit 4 and the power source 5 are built in the rear end 30 of one vine of the glasses G. The recognition processing unit 4 is configured by a custom LSI, and the power source 5 is configured by a button battery.

  As shown in FIG. 2, the recognition processing unit 4 includes left and right input signal processing units 6 and 7 connected to the microphones 2L and 2R, and a sound source localization unit 8 connected to both of the input signal processing units 6 and 7, respectively. A sound source identification unit 9 connected to the input signal processing unit 7, and a sound source localization unit 8 and a vibration selection unit 18 connected to the sound source identification unit 9. The sound source identification unit 9 only needs to be connected to at least one of the input signal processing units 6 and 7. The vibration selection unit 18 is connected to the vibration units 3L and 3R.

  The sound source localization unit 8 includes a time difference detection unit 10 and a left-right direction detection unit 11, and the sound source identification unit 9 includes a frequency pattern detection unit 12 and a sound source type detection unit 13. The time difference detection unit 10, the left-right direction detection unit 11, the frequency pattern detection unit 12, and the sound source type detection unit 13 are all configured by a plurality of pulse neuron models (hereinafter referred to as “PN models”). It consists of a network. The PN model refers to a neuron model that uses a pulse train as an input / output signal, and each PN model is composed of a digital circuit (see FIG. 4 of Non-Patent Document 8).

  As shown in FIG. 3, the input signal processing units 6 and 7 include AD conversion units 14L and 14R, frequency decomposition units 15L and 15R corresponding to human cochleas, and nonlinear conversion units corresponding to hair cells. 16L and 16R, and pulse converters 17L and 17R corresponding to the cochlear nerve. The AD conversion units 14L and 14R are AD conversion circuits, and the frequency resolution units 15L and 15R, the nonlinear conversion units 16L and 16R, and the pulse conversion units 17L and 17R are each configured by a digital circuit.

  The AD converters 14L and 14R AD convert the signals input from the microphones 2L and 2R, respectively. The frequency resolving units 15L and 15R are configured by a band-pass filter (BPF) group, and a plurality of (N) frequency bands (hereinafter referred to as “frequency channels” or simply “logarithmic scales”) of AD-converted signals in a predetermined frequency range. It is broken down into “channels”). The non-linear transformation units 16L and 16R take out only the positive components by performing non-linear transformation on the signals of the respective frequency bands input from the frequency decomposition units 15L and 15R, and also by a low-pass filter (LPF). Perform envelope detection. The pulse converters 17L and 17R convert the signals in the respective frequency bands input from the nonlinear converters 16L and 16R into pulse trains each having a pulse frequency proportional to the signal intensity. Through these processes, the input signal processing units 6 and 7 convert each of the left and right input signals into a pulse train having a pulse frequency corresponding to the signal intensity, that is, the sound pressure, for each frequency band.

  The time difference detection unit 10 is composed of a time difference extraction model composed of a PN model as shown in FIG. 4 and a sequence of time delay elements 19 (see FIG. 5) for inputting to the time difference extraction model while shifting the pulse train. Yes. The time difference extraction model is the same as that described in Non-Patent Document 3 and the like and will not be described in detail. However, as shown in FIG. 4, a plurality of time difference extraction neurons (hereinafter referred to as “MSO neurons”) 20 are provided. An odd number of MSO neuron arrays are provided for each frequency channel. Each MSO neuron 20 includes a left input terminal 21 to which a left pulse signal is input, a right input terminal 22 to which a right pulse signal is input, and an output terminal 23. When the pulse signal is input almost simultaneously from the left and right by setting the weight to the common fixed value and the threshold value to be twice the weight or the value obtained by adding the reference value of the internal potential to the double weight, etc. 23 is configured to output a pulse signal. Note that “substantially simultaneous” includes, of course, simultaneous.

  Then, as shown in FIG. 5, the time difference detection unit 10 shifts the left pulse train to the right and shifts the right pulse train to the left and shifts the left and right pulses every clock (unit time) by the time delay element 19. Are input to the MSO neuron train of the corresponding frequency channel. That is, the left pulse signal is sequentially input to each MSO neuron 20 while being shifted from one end (left end in FIG. 5) to the other end (right end in FIG. 5) every unit time, and the right pulse signal is input to the MSO neuron train. Are sequentially input to each MSO neuron 20 while being shifted every other unit time from the other end (same right end) to one end (same left end).

For example, if the number of MSO neurons 20 in each MSO neuron array is 2J + 1 and each MSO neuron 20 is assigned a number from −J to J, at time t, each MSO neuron 20 has an internal potential I ^MSO according to the following [Equation 1]. _ji (t) is calculated, and y _ji (t) = 1 is output when the internal potential exceeds a predetermined threshold, and y _ji (t) = 0 is output when it does not exceed the predetermined threshold. Note that j is the number of the MSO neuron 20, and i is the frequency channel number (i = 1 to N). In the following [Equation 1], p ^left _ji (t) is a local membrane potential for the left input signal, p ^right _ji (t) is a local membrane potential for the right input signal, and w is a common connection in all neurons 20. The weight, τ, is the decay time constant.

これにより、時間差抽出モデルは、左右からパルス信号が略同時に入ってきた場合にはＭＳＯニューロン列における中央付近のニューロン２０が発火し、パルス信号が右よりも左から早く入ってきた場合にはＭＳＯニューロン列における右側のニューロン２０が発火し、パルス信号が左よりも右から早く入ってきた場合にはＭＳＯニューロン列における左側のニューロン２０が発火するというように、左右の入力信号間の時間差によって変化する発火パターンを、音の時間差情報として出力する。

Thus, in the time difference extraction model, the neuron 20 near the center in the MSO neuron array fires when a pulse signal enters from the left and right substantially simultaneously, and when the pulse signal enters from the left earlier than the right, the MSO The right neuron 20 in the neuron array fires, and when the pulse signal comes in from the right earlier than the left, the left neuron 20 in the MSO neuron array fires. The firing pattern is output as sound time difference information.

That is, as described above, if each MSO neuron 20 in each MSO neuron array is numbered from -J to J and the number of frequency channels is N, at time t, from the time difference extraction model, The vector y _MSO (t) is output as time difference information.

y _MSO (t) = (y− _J1 (t),..., y ₀₁ (t),..., y _J1 (t),
y _−J2 (t),..., y ₀₂ (t),..., y _J2 (t),
…,
y _−JN (t),..., y _0N (t),..., y _JN (t))
For example, as shown in FIG. 6, the time difference detection unit 10 can be configured by a digital circuit. Since this configuration example is also described in Chapter 5 of Non-Patent Document 8, it will not be described in detail. Note that the implementation of the refractory period can be realized by providing a counter that counts the refractory period and not firing during the period from ignition to resetting the counter along with the ignition.

  The left-right direction detection unit 11 includes a competitive learning neural network (hereinafter referred to as “CONP”) described in Non-Patent Document 9. CONP is a pulse neural network that performs quantization of an input vector. Hereinafter, CONP will be described.

  As shown in FIG. 7, the CONP is composed of a competitive learning neuron group 50 and a control neuron group 60, and the competitive learning neuron group 50 is composed of a plurality of competitive learning neurons (hereinafter referred to as “CL neurons”) 51. The control neuron group is a non-firing detection neuron (hereinafter referred to as “NFD neuron”) 61 that fires when none of the CL neurons 51 fires, and a multiple firing detecting neuron that fires when multiple CL neurons 51 fire. (Hereinafter referred to as “MFD neuron”) 62.

  The NFD neuron 61 and the MFD neuron 62 change the threshold value of each CL neuron 51 uniformly according to their firing status (actually, the internal potential of each CL neuron 51 is changed uniformly), so that the CL neuron This is a PN model for maintaining a situation where only one CL neuron 51 is fired in the group 50. Each of the NFD neuron 61 and the MFD neuron 62 includes an input terminal corresponding to the number of CL neurons 51 in the CL neuron group 50 and an output terminal. The pulse signal output from each CL neuron 51 is received by each input terminal. The NFD neuron 61 outputs “1” from the output terminal only when the signals from all the CL neurons 51 are “0”, and the MFD neuron 62 receives the signal “1” from the plurality of CL neurons 51. Only in this case, “1” is output from the output terminal.

Each CL neuron 51, as shown in FIG. 8, the input pulse _{x 1 (t), x 2} (t), ..., x i (t), ..., the input terminal _x n (t) are respectively input 551 , 552, ..., 55i, ..., 55n, and input terminals 56 and 57 to which pulse signals y _nfd (t) and y _mfd (t) output from the NFD neuron 61 and the MFD neuron 62 are input, respectively, and output terminals 58. Each input terminal 55i (i = 1 to n) branches into two, one connected to the synapse unit 53i having a variable coupling weight w _hi and the other connected to the synapse unit 54i having a fixed coupling weight “1”. Has been. Note that h is a number assigned to each CL neuron 51 in the CL neuron group 50, and h = 1 to M.

The operation of CONP will be described with reference to FIGS. Each CL neuron 51 in the CL neuron group 50 has an input vector x (t) = (x ₁ (t), x ₂ (t) _,. t),..., x _n (t)) (t: time) are input (S101). Then, the NFD neuron 61 and the MFD neuron 62 respectively output the output value y _{nfd at the} time t based on the output y _h (t−1) from each CL neuron 51 at the held time (t−1). (T), y _mfd (t) are calculated and output to each CL neuron 51 (S102, S103). Note that, in the NFD neuron 61 and the MFD neuron 62, output values y _nfd (t) and y _mfd (t) using the output y _h (t−1) from each CL neuron 51 at time (t−1), respectively. _{May be} calculated and held, and y _nfd (t) and y _mfd (t) may be output to each CL neuron 51 at time t.

Next, each CL neuron 51 calculates an internal potential I _h (t) (h = 1 to M) (S104) (see [Expression 5] below), and the internal potential I _h (t) is a threshold TH. Is exceeded and y _h (t) = 1 is output when the refractory period has elapsed since the previous ignition, and y _h (t) = 0 is output otherwise (S105) .

At the time of learning, for the CL neuron 51 that has output “1”, the connection weight w _i is updated using the local membrane potential pcw _i at the synapse 54 _i (S 106), and the CL neurons 51 around the CL neuron 51 are also updated. Similarly, the connection weight is updated (S107). As a method of determining the range of CL neurons 51 that updates the connection weight (ie, the learning range), for example, all CL neurons 51 are initially set as the learning range, the range is linearly reduced, and the winner neuron is the last. There is a method of gradually reducing the learning range. Then, the norm of the connection weight (norm of the reference vector) is normalized to 1 for the CL neuron 51 whose connection weight has been updated (S108). That is, in CONP, the self-organizing map (SOM) algorithm is realized by learning not only the winner neurons but also the neighboring neurons.

  On the other hand, when it is not at the time of learning (at the time of recognition), the connection weight is not updated. Then, the coefficient α for updating the connection weight is updated by multiplying by a constant γ (0 ≦ γ) (S109), and the processing of steps S101 to S108 is performed for the next input vector.

Here, a method of calculating the internal potential I _h (t) in CONP will be described. First, as an argument, a function F having four parameters of time t, decay time constant τ, coupling weight w, and input signal x (t) at time t is introduced and defined as [Equation 2] below. Note that Δt = 1 / Fs (Fs: sampling frequency).

すると、時刻ｔにおけるＰＮモデルの内部電位Ｉ（ｔ）は、局所膜電位ｐ_ｉ（ｔ）（ｉ＝１〜ｎ）の総和として、下記［数３］のように記述できる。τはｐ_i（ｔ）の減衰時定数である。

Then, the internal potential I (t) of the PN model at time t can be described as the following [Equation 3] as the sum of the local membrane potentials p _i (t) (i = 1 to n). τ is the decay time constant of p _i (t).

ＰＮモデルの不応期をＲＰ、時刻ｔにおける前回発火からの経過時間をＥＴ（ｔ）とし、ＥＴ（０）＞ＲＰとすると、ＰＮモデルの出力値ｙ（ｔ）は、以下のアルゴリズムにより計算される。なお、ＴＨは閾値である。

If the refractory period of the PN model is RP, the elapsed time from the previous firing at time t is ET (t), and ET (0)> RP, the output value y (t) of the PN model is calculated by the following algorithm: The TH is a threshold value.

if I (t) ≧ TH and ET (t)> RP
then y (t) = 1, ET (t) = 0
else y (t) = 0, ET (t) = ET (t−Δt) + Δt
The parameters τ, w ₁ , w ₂ ,..., W _n , TH are variable values depending on each PN model, and the operation of each PN model is determined by this combination.

Here, the outputs of the NFD neuron 61 and the MFD neuron 62 at time t are respectively y _nfd (t) and y _mfd (t), and the connection weights of the CL neurons 51 to the NFD neuron 61 and the MFD neuron 62 are respectively w _fd and − _Assuming w _fd (where w _fd > 0), the internal potential I _h (t) of the CL neuron 51 of number h at time t can be described using the function F described above as in the following [Equation 4]. In CONP, p _nfd (t) and p _mfd (t) are treated as dynamic variations of the threshold (however, instead of changing the threshold TH, the internal potential I _h (t) to be compared with the threshold TH is changed to p _nfd ( t) and p _mfd (t) to maintain the state in which only one CL neuron 51 is fired. For this reason, the attenuation time constant τ _fd is sufficiently large with respect to the time constant τ.

ところで、入力パルス列によって発生する内部電位の総量が大きく変動する場合、この変動量を吸収するために閾値の変化が生じることになり、閾値の変化が入力ベクトルの方向変化に追従できない場合がある。そこで、ＣＯＮＰでは内部電位に対して、結合重みを１に固定したシナプス部５４ｉ（ｉ＝１〜ｎ）における局所膜電位ｐｃｗ_ｉ（ｔ）の総和を一定の比率β_ｐｃｗ（但し、０≦β_ｐｃｗ≦１）であらかじめ差引くことで、入力信号のノルム変動に対する内部電位の変化を抑制している。これにより上記［数４］のＩ_ｈ（ｔ）は下記［数５］のように修正され、各ＣＬニューロン５１は［数５］に従って内部電位Ｉ_ｈ（ｔ）を演算する。

By the way, when the total amount of the internal potential generated by the input pulse train fluctuates greatly, a change in the threshold value occurs to absorb this fluctuation amount, and the change in the threshold value may not follow the change in the direction of the input vector. Therefore, in CONP, the total sum of local membrane potentials pcw _i (t) in the synapse portions i (i = 1 to n) with the coupling weight fixed to 1 is set to a constant ratio β _pcw (where 0 ≦ β _By subtracting in advance with _pcw ≦ 1), the change in the internal potential with respect to the norm fluctuation of the input signal is suppressed. As a result, I _h (t) in the above [Equation 4] is corrected as shown in [Equation 5] below, and each CL neuron 51 calculates the internal potential I _h (t) according to [Equation 5].

ＣＯＮＰは簡単なディジタル回路によりハードウェア化可能であり、その例を図１０に示す。この例では、ＣＯＮＰは、それぞれＣＬニューロン５１に相当するＭ個のＣＬニューロン部５１Ｈと、ＮＦＤニューロン６１に相当する１個のＮＦＤニューロン部６１Ｈと、ＭＦＤニューロン６２に相当する１個のＭＦＤニューロン部６２Ｈとを備え、さらに、閾値変化量生成部６３、６４と内部電位抑制量生成部６５とを１個ずつ備えている。

The CONP can be implemented by hardware with a simple digital circuit, and an example is shown in FIG. In this example, CONP includes M CL neuron units 51H corresponding to CL neurons 51, one NFD neuron unit 61H corresponding to NFD neuron 61, and one MFD neuron unit corresponding to MFD neuron 62, respectively. 62H, and further includes one threshold change amount generators 63 and 64 and one internal potential suppression amount generator 65.

Each CL neuron unit 51H includes n input terminals corresponding to the input terminals 551,..., 55n of the CL neuron 51, and n input pulses x ₁ (t),. _n AND circuits 71 each multiplying _n (t) by a weight, an adder 72 that adds the output from each AND circuit 71 to the internal potential, and an attenuation that attenuates the internal potential and outputs it to the adder 72 A generator 73 and a comparator 74 that compares the internal potential output from the adder 72 with a threshold value are provided. The comparator 74 exceeds the threshold value, and the refractory period has elapsed since the previous ignition. If y _h (t) = 1, y _h (t) = 1 is output. Otherwise, y _h (t) = 0 is output. As will be described later, the comparator 74 _receives p _nfd (t) and p _mfd (t) as dynamic threshold change amounts and S _pcw (t) as a suppression amount of the internal potential. In 74, the internal potential is adjusted with these values as in [Formula 5], and then compared with the threshold value.

The NFD neuron unit 61H includes M input terminals respectively connected to the output terminals of the M CL neuron units 51H, and M input pulses y ₁ (t) and y ₂ ( t),..., y _M (t) each multiplied by M AND circuits 76, an adder 77 for adding the output from each AND circuit 76 to the internal potential, and attenuating and adding the internal potential When the internal potential output from the adder 77 is compared with the threshold value, the internal potential exceeds the threshold value, and the refractory period has elapsed since the previous ignition. Is provided with a comparator 79 that outputs 1 and in other cases 0, and is configured to ignite when all M input pulses are 0.

  The MFD neuron unit 62H has the same configuration as that of the NFD neuron unit 61H, but is configured to fire when a plurality of M input pulses are 1 by changing weights and thresholds.

The threshold value change amount generation units 63 and 64 are portions that generate local membrane potentials p _nfd (t) and p _mfd (t) for outputs from the NFD neuron unit 61H and the MFD neuron unit 62H in each CL neuron unit 51H, respectively. Yes, it is originally a part that each CL neuron unit 51H has in common, but since the weight and decay time constant are not changed by the CL neuron unit 51H, it is taken out from each CL neuron unit 51H to be one in total. .

The threshold change amount generation unit 63 multiplies the output from the NFD neuron unit 61H by a weight w _fd , an adder 82 that adds the output from the AND circuit 81 to the local membrane potential, and the local membrane potential. An attenuation generation unit 83 that attenuates and outputs to the adder 82, and outputs the local membrane potential p _nfd (t) from the adder 82 to the comparator 74 of each CL neuron unit 51H as the dynamic change amount of the threshold. .

The threshold value change amount generation unit 64 has the same configuration as the threshold value change amount generation unit 63, generates a local membrane potential p _mfd (t) using the output from each MFD neuron unit 62H, and generates a dynamic threshold value. The change amount is output to the comparator 74 of each CL neuron unit 51H.

The internal potential suppression amount generation unit 65 is a portion that generates the suppression amount S _pcw (t) of the change in internal potential with respect to the norm fluctuation of the input signal described above. Originally, each CL neuron unit 51H has a fixed weight of 1 The total sum of local membrane potentials pcw _i (t) in the synapse part 54i is generated by multiplying by a constant ratio β _pcw , but the weight and the decay time constant are not changed by the CL neuron part 51H, so each CL neuron part 51H It is taken out from the whole to make one. The internal potential suppression amount generator 65 multiplies the n input pulses by a fixed weight β _pcw , an adder 87 that adds the output from the AND circuit 86 to the internal potential, and the internal potential. And an attenuation generation unit 88 that attenuates and outputs to the adder 87, and outputs the internal potential as the suppression amount S _pcw (t) from the adder 87 to the comparator 74 of each CL neuron unit 51H.

  Note that the CONNP hardware configuration example shown in FIG. 10 does not include a learning mechanism (a weight update mechanism in each CL neuron unit 51H). This is because learning is performed by simulation by software, weights are determined, and the weights are set on the hardware. Of course, the learning mechanism can be implemented in hardware, but it is desirable to perform learning on software in order to facilitate the circuit configuration and reduce the circuit size.

The left-right direction detection unit 11 includes the above-described hardware CONP without a learning mechanism, and includes 16 CL neurons 51 as shown in FIG. The 16 CL neurons 51 are arranged in a row from number 1 to number 16, and the closer the number, the closer the distance. The time difference information (vector y _MSO (t)) output from the time difference detector 10 is input to each CL neuron 51. As described above, the left-right direction detection unit 11 can quantize the input vector y _MSO (t) while maintaining the similarity as a result of the weight being determined by software simulation. That is, when the vectors having high similarity (close directions) are input to the left / right direction detection unit 11, the CL neurons 51 that are close to each other fire, and vectors having low similarity (distant directions) are input. Sometimes CL neurons 51 that are distant from each other fire. Therefore, the direction information indicating the sound source direction (sound source direction) in the left-right direction is indicated from the left-right direction detection unit 11 according to which CL neuron 51 fires. Here, the left / right direction detection unit 11 identifies which direction of the two left and right directions the sound source is, and the firing signals from the CL neurons numbered 1 to 8 are left, and the number 9 to number CL CL The firing signal from the neuron shall indicate the right.

  Also, the frequency pattern detection unit 12 and the sound source type detection unit 13 of the sound source identification unit 9 are each configured by hardware-based CONP without the learning mechanism described above.

  The frequency pattern detection unit 12 detects a frequency pattern present in the input signal input from the input signal processing unit 7. The frequency pattern detection unit 12 vector-quantizes the frequency pattern present in the input signal to generate a firing pattern of the CL neuron 51. It is comprised so that it may output (refer FIG. 5, FIG. 8 of a nonpatent literature 9).

  The sound source type detection unit 13 further performs vector quantization on the firing pattern input from the frequency pattern detection unit 12 by performing supervised learning using LVQ (Learning Vector Quantization) during learning (that is, during simulation by software). Thus, the CL neuron 51 indicating the sound source type (sound source type) corresponding to the firing pattern is configured to fire. Since LVQ is well known, it will not be described here. Accordingly, the sound source type detection unit 13 indicates the type information indicating the sound source type according to which CL neuron 51 fires.

  For example, in the sound of an ambulance, there are two frequency patterns that sound like “pea” and “po”, respectively, and it is the frequency pattern detection unit 12 that vector-quantizes these two frequency patterns, The sound source type detection unit 13 outputs the firing signal indicating “ambulance” by vector quantization of these two vector-quantized patterns.

  In the present embodiment, the sound source type detection unit 13 identifies whether the sound source is one of four types of “(general) automobile”, “ambulance”, “interphone”, and “phone”, and the CL neuron 51 , The firing signals from the CL neurons numbered 1 and 2 are “car”, the firing signals from the CL neurons numbered 3 and 4 are “ambulances”, and the firing signals from the CL neurons numbered 5 and 6 are “ It is assumed that the firing signal from the CL neuron of “interphone” and numbers 7 and 8 indicates “telephone”.

  The vibration selection unit 18 of the recognition processing unit 4 stores vibration patterns respectively corresponding to the four sound source types. The vibration pattern differs depending on the type of sound source. Based on the direction information output from the sound source localization unit 8 and the type information output from the sound source identification unit 9, the vibration selection unit 18 determines to the vibration unit 3L if the sound source direction indicated by the direction information is left. If it is right, the vibration control signal is output to the vibration unit 3R according to the vibration pattern for the sound source type indicated by the type information.

  Each of the vibration units 3L and 3R includes a vibration motor, and is configured to generate vibration with the vibration motor in accordance with a vibration control signal input from the vibration selection unit 18.

  When the user wears the sound recognition device 1 configured as described above, the vibration units 3L and 3R come into contact with the left ear and the right ear of the user, respectively. The microphones 2L and 2R collect the sounds on the left and right sides of the user, respectively, and input them to the input signal processing units 6 and 7, respectively. The input signal processing units 6 and 7 convert the left and right input signals into a pulse train having a pulse frequency corresponding to the sound pressure for each frequency channel.

  The time difference detection unit 10 of the sound source localization unit 8 outputs time difference information indicating the time difference between the left and right input signals based on the pulse trains input from the input signal processing units 6 and 7. The left / right direction detection unit 11 outputs direction information indicating whether the sound source direction is left or right based on the time difference information input from the time difference detection unit 10.

  On the other hand, the frequency pattern detection unit 12 of the sound source identification unit 9 vector-quantizes the frequency pattern present in the input signal based on the pulse train input from the input signal processing units 6 and 7, and determines the firing pattern of the CL neuron 51. Output. The sound source type detection unit 13 vector-quantizes the firing pattern input from the frequency pattern detection unit 12 and outputs type information indicating the sound source type corresponding to the firing pattern.

  Based on the direction information output from the sound source localization unit 8 and the type information output from the sound source identification unit 9, the vibration selection unit 18 determines that the sound source direction indicated by the direction information is to the left, the vibration unit 3L If so, a vibration control signal is output to the vibration unit 3R in accordance with the vibration pattern corresponding to the sound source type indicated by the type information, whereby the vibration units 3L and 3R vibrate according to the vibration pattern corresponding to the sound source type.

  Since the vibration parts 3L and 3R are provided in a portion that comes into contact with the user of the sound recognition device 1, the user can immediately detect vibration and know the sound source type from the vibration pattern. If the vibration unit 3L provided on the left side vibrates, the sound source is on the left, and if the vibration unit 3R provided on the right side vibrates, the sound source is on the right side, and the direction of the sound source is known sensuously. be able to.

  As described above, the sound recognizing device 1 is provided with the vibration units 3L and 3R at portions that come into contact with the user when the user wears the sound recognizing device 1, and the sound source direction indicated by the direction information. Since the vibration parts 3L and 3R of the part facing to (vibration part 3L in the left direction and vibration part 3R in the right direction) vibrate with different vibration patterns depending on the sound source type, disturb the user's visual field. In addition, it is possible to inform the user of the direction and type of the sound source, and since the sound source is in the direction in which the vibration is felt, the user can know the direction of the sound sensuously.

  In addition, you may make it vibrate a some vibration part with respect to one sound source direction. For example, in the sound recognition device 1, the sound source localization unit 8 identifies the left, right, and front three directions. When the sound source direction is left, only the vibration unit 3L is used, and when the sound source direction is right, only the vibration unit 3R is used. Although it vibrates, in the case of the former, it is good also as vibrating both the vibration parts 3L and 3R. That is, at least one different vibration part may be configured to vibrate for different sound source directions. This is because the direction of the sound source can be determined depending on which vibration part vibrates. In the example of the above three directions, the vibration part that vibrates is different between the former and the vibration part 3L, and the latter is different from the vibration part 3R. The part 3L is the same, but the vibrating part 3R is different, and the vibrating part 3R is the same in the right case and the previous case, but the vibrating part 3L is different.

  Further, in the sound recognition device 1, the vibration units 3L and 3R are respectively provided in portions facing each sound source direction (that is, left and right) when the user wears, and the sound source direction indicated by the direction information is on the left. If there is a vibration part 3L, if it is right, the vibration part 3R is vibrated, but a vibration part is not necessarily provided in a part that faces each sound source direction, and a part that faces the sound source direction indicated by the direction information. There is no need to vibrate the vibration part. However, if a vibration part is provided in each part that faces each sound source direction when the user wears it, and the vibration part in the part that faces the sound source direction indicated by the direction information is vibrated, vibration was felt. Since the direction is the sound source direction, the user can know the direction of the sound sensuously.

  In addition, the user may be able to set which vibration pattern is associated with each sound source type that can be identified by the sound recognition device 1, or the user may be able to adjust the vibration intensity.

  In addition, as shown in FIG. 11, the sound recognition device may be an attachment type sound recognition device 1B that can be attached to ready-made glasses. Hereinafter, the same components as those of the sound recognition device 1 will be described using the same reference numerals. The sound recognition device 1B includes attachment portions 31L and 31R that can be fitted to the left and right vines of the glasses, respectively, and the attachment portions 31L and 31R. Are connected to each other by a string-like member 32. A microphone 2L is attached to the outside of the attachment portion 31L, and a vibration portion 3L is attached to the inside. A microphone 2R is attached to the outside of the attachment portion 31R, and the vibration portion 3R is attached to the inside. Further, the recognition processing unit 4 and the power source 5 are built in one of the attachment units 31L and 31R. The sound recognition device 1B is used by fitting the attachment portions 31L and 31R to the left and right vines of ready-made glasses as shown by the two-dot chain lines in the figure. Then, the sound recognition device 1B operates in the same manner as the sound recognition device 1, and the vibration units 3L and 3R notify the user of the direction and type of the sound source by vibration.

  Further, the sound recognition device may be a wearable form other than glasses, for example, a hat shape integrated with a hat. In the case of a hat type, for example, the vibration part is attached to the front part, the rear part, the left part, and the right part on the inner side of the hat, and the sound source localization part 8 identifies the front, rear, left, and right directions, and the front, You may comprise so that the vibration part attached to the front part, the rear part, the left side part, and the right side part may vibrate with respect to the sound from back, left side, and right side, respectively.

  Moreover, the time difference detector described in Patent Document 2 may be used as the time difference detection unit 10, and the competition described in Patent Document 3 may be used as the left-right direction detection unit 11, the frequency pattern detection unit 12, and the sound source type detection unit 13. A learning neural network (CONP-II) may be used, or a device that is not a pulse neural network may be used. That is, the sound source localization method and the sound source identification method are not limited to the methods described above.

1 is a perspective view of a sound recognition device according to an embodiment of the present invention. It is a block block diagram of the sound recognition apparatus which concerns on the same embodiment. It is a block diagram which shows the structure of an input signal processing part. It is a schematic diagram of a time difference detection part and a left-right direction detection part. It is a figure which shows the structure of a MSO neuron row | line | column. It is the example which comprised the time difference detection part with the digital circuit, (a) is the figure for demonstrating operation | movement of the first half of 1 clock, (b) is operation | movement for the latter half. It is a schematic diagram of CONP. It is a schematic diagram of CL neuron in CONP. It is a flowchart which shows the operation | movement of CONP. It is a flowchart which shows the operation | movement of CONP. This is an example in which CONP is configured by a digital circuit. It is a sound recognition apparatus which concerns on other embodiment of this invention.

Explanation of symbols

DESCRIPTION OF SYMBOLS 1, 1B ... Sound recognition apparatus 2L, 2R ... Microphone 3L, 3R ... Vibration part 8 ... Sound source localization part 9 ... Sound source identification part

Claims (2)

An eyeglass-integrated wearable sound recognition device worn by a user,
The left microphone on the left vine of the glasses,
A right microphone provided on the right vine of the glasses;
Based on the time difference between the left input sound input from the left microphone and the right input sound input from the right microphone, the sound source is identified from among the plurality of sound source directions. A sound source localization unit that outputs direction information indicating the sound source direction;
Based on the frequency pattern of at least one of the left input sound and the right input sound, the sound source is identified from among a plurality of sound source types, and type information indicating the sound source type is output. A sound source identification unit;
A plurality of vibration parts provided in parts that come into contact with the user when the user wears each;
With
Different vibration patterns are set for each sound source type,
Based on the direction information and the type information, one or more of the vibration units are configured to vibrate with a vibration pattern for the sound source type indicated by the type information, and at least one for different sound source directions. Two different vibrating parts are configured to vibrate ;
As the vibration part, comprising a left vibration part provided on the left vine of the glasses, and a right vibration part provided on the right vine of the glasses,
When the sound source direction is left, the left vibration unit vibrates in a vibration pattern corresponding to the sound source type indicated by the type information, the right vibration unit does not vibrate, and when the sound source direction is right, the right vibration unit A wearable sound recognizing device, wherein the vibration unit vibrates in a vibration pattern corresponding to a sound source type indicated by the type information, and the left vibration unit does not vibrate . An attachment-type wearable sound recognition device that can be attached to ready-made glasses,
A left attachment that can be attached to the left vine of glasses;
A right attachment that can be attached to the right vine of the glasses;
A left microphone provided in the left mounting portion;
A right microphone provided in the right mounting portion;
Based on the time difference between the left input sound input from the left microphone and the right input sound input from the right microphone, the sound source is identified from among the plurality of sound source directions. A sound source localization unit that outputs direction information indicating the sound source direction;
Based on the frequency pattern of at least one of the left input sound and the right input sound, the sound source is identified from among a plurality of sound source types, and type information indicating the sound source type is output. A sound source identification unit;
A left vibrating portion provided on a portion of the left mounting portion that comes into contact with the user when the user wears the spectacles on which the wearable sound recognition device is mounted;
A right vibrating portion provided on a portion of the right attachment portion that comes into contact with the user when the user wears the glasses equipped with the wearable sound recognition device;
With
Different vibration patterns are set for each sound source type,
Based on the direction information and the type information, when the sound source direction is left, the left vibration unit vibrates with a vibration pattern for the sound source type indicated by the type information, and the right vibration unit does not vibrate. The wearable sound is configured such that when the sound source direction is right, the right vibration unit vibrates in a vibration pattern corresponding to the sound source type indicated by the type information, and the left vibration unit does not vibrate. Recognition device.

Images