JP2011075989A - Method and device for creating operation command signal - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To further diversify information which can be input when an information processing apparatus or an electronic device is operated by those who cannot freely use fingers, or that used under a noise environment, without making input operation complicated, and to make a device configuration simple. <P>SOLUTION: Bone conduction sound which is detected by a bone conduction microphone attached in one auricle is amplified and sampled, and it is stored in a memory. The bone conduction sound signal stored in the memory is taken out, and variable threshold processing is performed. Multi-class classification by partial space pattern classification is performed for occlusal sound candidates which are detected by the variable threshold processing. A signal according to an occlusal data of the occlusal sound candidates on which multi-class classification is performed is created as an operation command signal. <P>COPYRIGHT: (C)2011,JPO&INPIT

Description

  The present invention relates to an operation command signal generation method and an operation command signal generation device, and more specifically, an operation command signal for generating an operation command signal to a computer or a device from a signal detected by an acceleration sensor such as a bone conduction microphone. The present invention relates to a generation method and an operation command signal generation device.

  In an information processing device such as a computer or an electronic device that operates based on a command operation, as a normal form, information and contents of an operation command are input using an input device such as a keyboard, a mouse, a touch panel, or a controller. In such an input device, input is performed by pressing a key, a button, or the like, or touching on a panel. In addition, there is another voice input device, which is equipped with a voice recognition device, which recognizes and inputs a command by utterance.

  By the way, those who cannot use their fingers freely due to illness or disability are difficult to operate by operating keys and buttons, so they are not suitable for keyboard and mouse operations, but by using a voice input device, It becomes possible to operate the information processing apparatus and the electronic device. The voice input device can be used as long as it has a speech ability, but the voice input device cannot be used when there is a speech failure. Also, since the input information is known when there is another person in the surroundings, the voice input device is not suitable for inputting confidential matters, and if the surroundings are in a noisy environment, the voice input device The disadvantage is that recognition becomes inaccurate or impossible.

  There is a method using a bone conduction microphone as a means to generate an operation input signal in a form that can be input even to those who can not use fingers freely or is not perceived by surrounding people. Development has been done. The bone conduction microphone is an earphone-type acceleration sensor that detects an audio signal for detecting a vibration of the skull that is generated at the time of utterance, and is detected by attaching it to the auricle. Since the bone conduction microphone does not detect ambient noise propagating in the air, the operation input device using this can be used even in a noisy environment.

  An operation input device using a bone conduction microphone or a voice input device is disclosed in the following documents. In Patent Document 1, a bone conduction microphone is attached to the lower rear part of the auricle, and muttering sounds, tooth biting sounds, tongue hitting sounds, and the like are collected, and AD conversion, quantization, and sound using an acoustic model are collected on the collected signals. It describes that processing such as recognition is performed, and Patent Document 2 detects the sound in the oral cavity with a bone conduction microphone and compares the processing content with data registered in advance for each processing content. An input method and a processing system for determination are described.

  In Patent Document 3, an ultra-low frequency sound generated by an operator's jaw opening / closing operation is detected by an earphone, and an ultra-low frequency waveform is formed to generate a pulse signal having a number corresponding to the number of jaw opening / closing operations. It describes that the device is controlled according to the number of pulses, and Patent Document 4 picks up a voice by a voice input device using a bone conduction microphone or the like and registers it in the operation data table. An electronic device that executes a command associated with a voice when it is determined to be a device is described.

  In Patent Document 5, the bone conduction sensor detects a sound generated in the oral cavity with a bone conduction sensor, extracts a feature value as a call signal, and sends a call signal to a child device when a certain threshold is exceeded. Patent Document 6 describes an operation command input device in which microphones are attached to left and right ears to detect a tooth engagement sound and a generation position, and a command corresponding to the generation position is generated. Is described.

  Patent Document 7 describes an operation switch that can be operated even by a person who cannot use a hand due to illness, injury, etc., by detecting sound in the oral cavity with a bone conduction microphone and operating the switch by the detection signal. Further, in Patent Document 8, the acceleration transmitted through the jaw bone or the skin surface is detected by a bone motion microphone, and a trigger signal is generated when the detection signal includes a specific frequency component generated when the teeth mesh with each other. There is a description of a permanent wearable telephone device that transmits and receives a telephone call based on the time-series trigger signal.

  Among these, in Patent Documents 1 to 5, 7, and 8, the bone conduction microphone is used to detect the opening and closing operation of the jaw, and based on this, input and operation of the device are performed. One movement of the jaw is recognized as itself, and since the type of movement is not identified, it is appropriate to be used as an operation device with a switch function by on / off operation, but when used as an input device, Many operations are required according to various types of input contents, and the operation must be complicated accordingly.

  According to Patent Document 6, an ear canal insertion type headphone is provided with a left microphone and a right microphone, and the sound detected by the left and right microphones is recognized to detect the mouth sound and its detection position, and the sound is detected according to the generation position. Command is generated. Depending on the sound recognition process, it is possible to distinguish between the meshing of the right tooth and the meshing of the left tooth, and there is little possibility of further diversity as the contents of the input information. However, the sensor is assumed to be simplified as in the operation of the portable audio device, and the sensor is assumed to be provided on both the left and right sides, and the device configuration is complicated for the function.

Japanese Patent No. 3760173 JP 2002-162990 A JP 2007-11627 A JP 2004-198488 A JP 2006-180994 A JP 2006-343965 A JP 2005-130427 A Japanese Patent No. 3525665

  Even those who cannot use their fingers are allowed to perform input and command operations on information processing equipment and electronic devices, and to these devices in a noisy environment or in a form that is unknown to those around them. For the input signal generation to enable the input and operation of the input, the conventional method does not allow a great variety of information contents to be input, and therefore, the operation such as tooth meshing is repeated many times. Therefore, it is inevitable that the operation is complicated. In addition, regarding the ability to detect the positions of left and right sound generation, there is no further diversity in sound recognition, and the apparatus configuration is also complicated.

  Under such circumstances, the information that can be input to the information processing apparatus and the electronic device is made more diverse, the input operation is not complicated, and the apparatus configuration is further simplified so that the operation input can be performed. Was demanded.

  The present invention has been made to solve the above-described problems, and an operation command signal generation method according to the present invention generates a bone conduction sound signal by detecting a bone conduction sound with a bone conduction microphone attached to one auricle. Amplifying and sampling the bone conduction sound signal, storing it in a memory, taking out the stored bone conduction sound signal, performing variable threshold processing, and detecting a tooth biting sound candidate; and Performing multi-class identification by partial space pattern identification on the identified dental bite sound candidate, and generating a signal corresponding to the dental bite sound data of the multi-class identified dental bite sound candidate as an operation command signal; , Is made up of.

  Differentiating the bone conduction sound transmission path from the bone conduction sound generation position to the bone conduction microphone attached to the one auricle, and distinguishing the bone conduction sound at different positions by changing the characteristics included in the bone conduction sound It is good also as what is made.

  Regarding the bone conduction sound collected by the bone conduction sound microphone, at least the meshing operation of the front teeth, left teeth, and right teeth may be identified and recognized. Acquires data corresponding to the head movement collected by the sensor that detects the head movement, and generates an operation command signal by combining the signal corresponding to the head movement and the signal corresponding to the tooth biting sound data You may make it do.

  An operation command signal generation apparatus according to the present invention includes a bone conduction microphone, an amplification unit that amplifies a bone conduction sound signal generated by the bone conduction microphone, and samples the amplified bone conduction sound signal as bone conduction sound data. A data logger to be stored, a variable threshold processing unit for cutting out tooth biting sound candidate data from bone conduction sound data stored in the memory of the data logger, and a partial space for performing multi-class identification on the cut out tooth biting sound candidate data A pattern identification processing unit and an operation command signal generation unit that generates an operation command signal corresponding to the tooth and bite sound candidate data subjected to multi-class identification are provided.

  Differentiating the bone conduction sound transmission path from the bone conduction sound generation position to the bone conduction microphone attached to the one auricle, and distinguishing the bone conduction sound at different positions by changing the characteristics included in the bone conduction sound May be made.

  The partial space pattern identification processing unit may identify and recognize at least the engagement operation of the front teeth, left teeth, and right teeth with respect to the bone conduction sounds collected by the bone conduction sound microphone. A sensor for detecting the movement of the head; and a head movement recognition unit that acquires data corresponding to the head movement collected by the sensor and generates a signal corresponding to the head movement. The command signal generation unit may generate an operation command signal by combining a signal corresponding to the tooth biting sound data and a signal generated by the head movement recognition unit.

  In the present invention, bone conduction sound is detected by a bone conduction microphone attached to one auricle, and bones generated at different positions according to different transmission paths from the bone conduction sound generation position to the bone conduction microphone. Conducted sound can be identified in multiple classes with high accuracy by spatial pattern identification, and by identifying the meshing movements of front teeth, right teeth, left teeth, etc., those who cannot use information processing devices and electronic devices freely Information that can be input even when operated or used in a noisy environment can be diversified, the input operation is not complicated, and the apparatus configuration can be further simplified. In addition, an operation command input signal in which input information is further diversified can be generated by being configured in combination with a camera, an infrared sensor, or the like.

It is a figure which shows the signal obtained with a bone conduction microphone. It is a figure which shows the waveform and duration of a tooth biting sound. It is a figure explaining the tooth biting sound detection process using a variable threshold process. It is a figure which shows the tooth biting sound detection result by a variable threshold value process. It is a figure explaining the tooth biting sound detection process by the threshold value process of two steps. It is a figure which shows the detection result of the tooth biting sound by a threshold value process of two steps. It is a figure which shows the evaluation test result about identification of a tooth biting sound. It is a figure which shows the procedure of tooth biting sound identification. It is a figure which shows the structure of the operation command signal generation apparatus.

  In the operation command signal generation method and the operation command signal generation device according to the present invention, for example, with a bone conduction microphone inserted into either the left or right ear hole, tooth engagement and its position, tongue-like operation, swallowing operation, etc. An operation command signal is generated based on the detection and identification.

  The bone transmission microphone is an acceleration detection type sensor that detects vibration generated by an operation such as utterance through the skull when attached to the auricle. When attached to the pinna, vibration is detected through the skull. Even in a noisy environment, it is possible to detect vibrations according to body movements without detecting ambient noise or the like propagating in the air. However, as the vibration, when the operation of meshing teeth is intended, in addition to the vibration caused by this, vibration due to other unintended body movement such as tongue hitting and swinging is also detected. Therefore, in order to generate an operation command input signal, it is necessary to identify the type of bone conduction signal.

  In the present invention, the pattern of the input signal is such that the tooth bite sound signal obtained by the bone conduction microphone attached to one pinna is analyzed to identify the front tooth, right tooth, or left tooth. An operation command signal is generated using a technique of identifying. Thus, pattern identification of tooth biting sound will be described.

[Pattern identification of tooth bite]
A case will be described in which a bone conduction microphone is attached to the right ear and the teeth are meshed with the consciousness that only the front, left, and right teeth are in contact. When meshing,
・ Do not get tired no matter how many times you bite. ・ Make the contact position of the teeth the same every time. ・ Place a comfortable posture and do not move your body. When collecting a signal, bone conduction sounds other than the tooth biting sounds that are uttered regardless of the user's intention, such as speaking while swallowing the bone conduction microphone, swallowing saliva, and moving the neck are also collected.

(1) Detection sound of tooth biting sound and vibration generated by body movement are detected as bone conduction sound by a bone conduction microphone. Since the detected bone conduction sound signal is weak at about several mV, it is amplified by 35 dB with an amplifier, and then collected at a sampling frequency of 20 kHz and stored in a memory.

  The signal obtained by the bone conduction microphone is as shown in FIG. 1, and in order to automatically detect a section that is a candidate for identification of tooth bite sound from now on, as shown in FIG. And its characteristics are analyzed. As a result, it is confirmed that the tooth biting sound falls within a section of 5 msec before and 10 msec after the time when the tooth biting sound takes a peak value. Therefore, a signal included in a 15 msec section before and after the peak value is automatically detected as a tooth and sound identification candidate.

Regarding the detection method of the peak value, the variable threshold value V _th is set as follows using the average value η of the absolute values of the signals included in the detection window having a width (W) of 256 msec (FIG. 3).
V _th = η × 2 ⁴ (1)
When the maximum value of the absolute value of the signal in the detection window is larger than the variable threshold, it is assumed that the detection window includes a signal that is a candidate for a dental bite sound, and a section of 15 msec before and after the peak value is cut out.

  If this method is performed only once, there is a possibility that noise is detected as a dental bite as shown in FIG. Therefore, noise is removed by performing a process using a threshold value (minimum amplitude value of tooth-biting sound) once again on the signal detected by the variable threshold process as shown in FIG. By this two-stage threshold processing, tooth biting sound candidates as shown in FIG. 6 are detected and pattern identification is performed.

(2) Pattern identification of tooth biting sound Pattern identification is performed on a signal that becomes a detected tooth biting sound by using a CLAFIC method, which is a typical subspace pattern identification method. A time series signal for 15 msec is used as it is as a feature value for creating a partial space for identification.

の固有値を求め、大きい順に並び替える。さらに、この時の固有値λ_ｉｊに対応する固有ベクトルｕ_ｉｊ（ｊ＝１，……，ｄ）によって張られる空間をクラスｉの部分空間とする。ここで、部分空間の次元ｄ_ｉは、累積寄与率

が０．９以上になるように決定する。 In order to construct a partial space for identification, a signal whose class information is already known is used to create a partial space for each class. Class autocorrelation matrix of pattern x (number of patterns n _i ) belonging to a certain class i

The eigenvalues of are obtained and rearranged in descending order. Further, a space spanned by the eigenvector u _ij (j = 1,..., D) corresponding to the eigenvalue λ _ij at this time is defined as a subspace of class i. Here, the dimension d _i of the subspace is the cumulative contribution rate

Is determined to be 0.9 or more.

を求める。 In the actual identification stage, the similarity between the subspace of each class obtained in this way and the signal with unknown class information

Ask for.

  In addition, among the similarities between the unknown signal and the subspace of each class, if the maximum similarity is equal to or greater than a threshold, the tooth biting sound (front tooth, right tooth, left tooth) of the corresponding class is used. The signal is rejected as a swallowing sound of the saliva or a sound generated by physical movements such as a head swing.

[Effectiveness of pattern identification]
The fact that the detected tooth bite sound is identified by the pattern identification method described above is confirmed by performing pattern identification on the signal actually collected from the subject. As an actual example, for each of 15 subjects, 30 samples of tooth biting sound and 50 samples of noise were used, respectively, and signals from a mixture of three types of tooth biting sounds (anterior teeth, right teeth, left teeth) and noise. With respect to the results of multi-class discrimination by the pattern discrimination method, the discrimination performance is evaluated by cross validation, and the results are shown in FIG.

  At the time of cross-validation, the entire data is divided into five groups, and estimation evaluation using data of another group is repeated five times for one group, and an average value is calculated. In FIG. 7, the black bar shows the average value of the correct answer rate that can identify the tooth bite sound in the correct class from the mixed signal of the three kinds of tooth bite sound and noise as the discrimination rate (sensitivity%) for each subject. The white bar indicates the rate of false identification (false positive rate%) as a tooth biting sound of a different part. Table 1 shows the sensitivity, false-positive rate, and average value of 15 subjects for the front teeth, right teeth, and left teeth.

  As described above, the pattern identification with respect to the collected tooth biting sound data can identify the tooth biting sounds of the front teeth, the right teeth, and the left teeth with an accuracy of about 90%. The operation command signal can be generated based on the identification.

  In the above example, the example using the graphic method as the subspace pattern identification method has been shown. However, the multiple bite discriminant analysis method (MDA) or other nonlinear subspace identification method can also be used to identify the occlusal sound. Thus, a multi-class classifier can be configured using any one of these pattern identification methods. As for the feature amount, feature extraction means may be introduced before the identification means, and the spectrum of bone conduction sound may be used as the feature amount.

  In the foregoing, a bone conduction microphone is attached to one auricle as a sensor for collecting signals. A bone conduction microphone is an acceleration detection type sensor that detects vibrations transmitted through the skull when worn in the ear canal, and the biting sounds of the front, right, and left teeth are transmitted to the sensor as different forms of vibration. Can be identified. Therefore, a plurality of vibration generation positions can be detected by one sensor.

  In the present invention, bone conduction sound is detected by a bone conduction microphone attached to one auricle, and bones generated at different positions according to different transmission paths from the bone conduction sound generation position to the bone conduction microphone. Conducted sounds can be identified in multiple classes with high accuracy by spatial pattern identification.

  Although the case of the front tooth, the right tooth, and the left tooth has been described as the detection target, vibrations associated with operations such as humming, swallowing, and swinging can also be detected and identified. These become noise when focusing on the tooth biting sound of the front teeth, right teeth, and left teeth, but also pay attention to these other vibrations and can be treated as detection target signals if they are identified. In addition to the head, vibrations in the shoulder, abdomen, legs, etc. can also be detected by the bone conduction microphone, and these can be identified and handled as signals.

[Procedures for tooth biting sound identification]
FIG. 8 shows a general procedure for the identification of tooth biting sound and shows the processing procedure. First, (a) vibration generated by voice or body movement is detected as bone conduction sound by a bone conduction microphone.

(B) Since the detected bone conduction sound signal is weak at about several mV, it is amplified by an amplifier by about 35 dB, collected at a sampling frequency of 20 kHz, and stored in a memory.

  (C) An average value of a signal a predetermined time before the current time is held as a variable threshold value according to time, a signal stored in the memory is sequentially scanned, variable threshold processing is performed, and the amplitude value of the signal is set to a variable threshold value. Tooth bite sound candidate detection is performed by partially cutting out the tooth bite sound candidate when the signal exceeding the signal is detected and storing it in the memory. When the signal is cut out, the previous signal is erased from the memory. By this operation, audio signals other than the tooth biting sound are almost removed.

  (D) The signal cut out in (c) is subjected to normalization processing using a norm to perform multi-class identification by the subspace pattern identification method. As a result, the initial tooth biting sound data is identified as being the biting sound of the front teeth, right teeth, and left teeth.

  In the above (b) to (d), multi-class identification is performed by sequentially scanning a plurality of tooth-biting sound signals stored in the memory. It may be. In this case, the memory is a buffer that temporarily holds the collected bone conduction sound signal.

[Operation command signal generator]
By collecting bone conduction sound using a bone conduction microphone and performing multi-class discrimination by subspace pattern identification to identify the tooth bite sound, the intended tooth biting sound (anterior teeth, right teeth, left teeth) ) To generate an operation command input. Such an apparatus for generating an operation command signal is configured to execute the above-described procedures (a) to (d) and is as shown in FIG.

  The operation command signal generation device of FIG. 9 generally includes a bone conduction microphone 10, an amplifier 20, a data logger 30, a tooth biting signal processing unit 40, and an operation command input signal generation unit 50, and 60 is generated. A device (information device, electronic device, etc.) that receives an operation command signal. The amplifier 20 amplifies the bone conduction sound signal detected by the bone conduction microphone 10 to a level of about 35 dB. The data logger 30 includes an A / D conversion of the amplified signal and a sampling unit 31 for sampling at a frequency of 20 kHz and a memory 32 for storing the sampled signal as tooth biting sound data.

  The tooth biting sound signal processing unit 40 sequentially calls the tooth biting sound data stored in the memory 32, detects the tooth biting sound candidate by the variable threshold processing, and the detected tooth biting sound candidate. The pattern identification processing unit 42 performs multi-class identification of tooth biting sound by pattern identification processing using spatial pattern identification. The operation command signal generation unit 50 generates an operation command signal corresponding to the tooth biting sound signal that has been subjected to the multi-class identification process, and sends it to the device 60.

The operation command signal generation device of FIG. 9 identifies the collected tooth biting sound signals in multiple classes, and generates an input signal to the device from a signal corresponding to the meshing operation intended by the operator. As the meshing operation, the description has been made especially with respect to the meshing of the front teeth, the right teeth, and the left teeth. However, if the swallowing operation, the swinging operation, and the like are identified, the input signals are further diversified.

In addition, the operation command signal generation device of FIG. 9 has a configuration using a bone conduction microphone as a sensor, but further includes other existing sensors such as a camera, an infrared sensor, etc. It can be configured as a diversified input signal generation device.

For example, the motion of the head is recognized by a gesture recognition process from a moving image of the head taken by a camera and the function of a mouse pointer is given, and the click function of the mouse is given by the engagement operation of the front teeth, right teeth, and left teeth. Thus, various input operations can be executed by configuring as an operation command input signal generation device combining a bone conduction microphone and a camera. As described above, in the operation command input signal generation device provided with the bone conduction microphone and the other sensor, in addition to the tooth biting signal processing unit that performs other class identification on the tooth biting sound signal, the signal is collected by another sensor. A signal identification processing unit is provided, and an input signal is formed in the operation command input signal generation unit by combining the signals generated by these processing units.

  INDUSTRIAL APPLICABILITY The present invention can be used as an operation command input generation device for a trigger signal generation device by mouth sound or body movement, an information processing device such as a multi-value switch, a personal computer, a mobile phone, a portable audio device, etc. It can be used in various forms, such as a means of communication by those who cannot use their fingers freely due to general anesthesia, etc., and a means of communication and equipment operations that need to be made unnoticeable by others. is there.

DESCRIPTION OF SYMBOLS 10 Bone conduction microphone 20 Amplifier 30 Data logger 31 Sampling part 32 Memory 40 Tooth biting sound signal processing part 41 Variable threshold value processing part 42 Pattern identification processing part 50 Operation command signal generation part
60 equipment

Claims (8)

Detecting a bone conduction sound with a bone conduction microphone attached to one pinna and generating a bone conduction sound signal;
Amplifying and sampling the bone conduction sound signal and storing it in memory;
Extracting the stored bone conduction sound signal and performing a variable threshold process to detect a tooth bite sound candidate;
Performing multi-class identification by partial space pattern identification on the detected tooth biting sound candidates;
Generating a signal corresponding to the tooth biting sound data of the multi-class identified tooth biting sound candidate as an operation command signal;
An operation command signal generation method comprising:   Differentiating the bone conduction sound transmission path from the bone conduction sound generation position to the bone conduction microphone attached to the one auricle, and distinguishing the bone conduction sound at different positions by changing the characteristics included in the bone conduction sound The operation command signal generation method according to claim 1, wherein:   The bone conduction sound collected by the bone conduction sound microphone identifies and recognizes at least the engagement operation of the front tooth, the left tooth, and the right tooth according to any one of claims 1 and 2. Operation command signal generation method.   Acquires data corresponding to the head movement collected by the sensor that detects the head movement, and generates an operation command signal by combining the signal corresponding to the head movement and the signal corresponding to the tooth biting sound data The operation command signal generation method according to claim 1, wherein the operation command signal generation method is performed.   A bone conduction microphone, an amplifying unit for amplifying a bone conduction sound signal generated by the bone conduction microphone, a data logger for sampling the amplified bone conduction sound signal and storing it as bone conduction sound data, and a memory of the data logger A variable threshold value processing unit that cuts out tooth-biting sound candidate data from stored bone conduction sound data, a subspace pattern identification processing unit that performs multi-class identification from the cut-out tooth biting sound candidate data, and multi-class identification An operation command signal generation device comprising: an operation command signal generation unit configured to generate an operation command signal according to the made tooth and bite sound candidate data.   Differentiating the bone conduction sound transmission path from the bone conduction sound generation position to the bone conduction microphone attached to the one auricle, and distinguishing the bone conduction sound at different positions by changing the characteristics included in the bone conduction sound The operation command signal generation device according to claim 5, wherein:   The partial space pattern identification processing unit is configured to identify and recognize at least an engagement operation of an anterior tooth, a left tooth, and a right tooth with respect to a bone conduction sound collected by the bone conduction sound microphone. The operation command signal generation device according to any one of 5 and 6. A sensor for detecting the movement of the head; and a head movement recognition unit that acquires data corresponding to the head movement collected by the sensor and generates a signal corresponding to the head movement. The command signal generation unit generates an operation command signal by combining a signal corresponding to the tooth biting sound data and a signal generated by the head movement recognition unit. The operation command signal generation device according to any one of the above.

Images