JP2007312309A - Mobile terminal device and method of operating the same - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide a mobile terminal device with which a patting action can be extracted with high accuracy and at a low cost. <P>SOLUTION: The mobile terminal device comprises: a sound wave signal detecting means for detecting a sonic wave signal corresponding to the patting action to the relevant mobile terminal device; an extraction means for extracting the patting action based on frequency characteristics and impulse response characteristics of the detected sound wave signal; a conversion means for converting the extracted patting action into a corresponding operation command; and an execution means for executing the relevant operation command. <P>COPYRIGHT: (C)2008,JPO&INPIT

Description

  The present invention relates to a mobile terminal device and an operation method thereof, and in particular, a mobile terminal device that can be operated by a tapping operation on the mobile terminal device without using a button or a key on the mobile terminal device and the operation thereof Regarding the method.

  As mobile terminal devices such as mobile phones become more sophisticated and smaller in size, the number of buttons and keys for the operation is increasing and decreasing. On the other hand, with the widespread use of mobile phones, mobile phone users have spread from children to the elderly and from healthy people to disabled people. Many people with limited eyes and fingertips have many small buttons and keys. There is a need for an operation method that can more easily operate a high-function and small-sized mobile terminal device.

As one of the methods, for example, as described in Patent Document 1 below, a method of converting a finger hitting action and a hitting action (hereinafter referred to as “tapping action”) into an operation on the terminal has been proposed. Yes.
JP 2003-229930 A

  However, the method of Patent Document 1 has the following problems and has not been put into practical use. That is, (1) In Patent Document 1, a tapping operation on a terminal is detected by an acceleration sensor. The acceleration sensor detects vibration and movement, so even if the same tapping operation is performed, the situation where the terminal is placed (for example, on a hard desk, in a cage, held by hand) The output of the acceleration sensor differs depending on the part of the operated terminal (main body front surface, back surface, side surface, etc.), and it is difficult to determine that the operation is the same.

  (2) Since an acceleration sensor generally electrically extracts stress applied to a built-in weight, followability with respect to high-speed vibration is poor. In addition, since there are many cases where there is a resonance frequency in the audible frequency range, it is necessary to remove high frequency components as noise. From the remaining low-frequency components, when a person hears it, it is impossible to distinguish what is clearly different as “sound”. In addition, the type of object (nail, plastic, metal, etc.) that has been struck cannot be distinguished.

  (3) From the viewpoint of the design and manufacture of the mobile terminal device, the acceleration sensor is essentially a sensor that is not necessary for a call. Therefore, in order to incorporate a device called an acceleration sensor, discrete elements such as capacitors and resistors, It is necessary to add a large number of peripheral elements such as an analog amplifier, a band pass filter, and an A / D converter.

  As described above, the method of detecting the hitting operation using the acceleration sensor has low detection accuracy and high cost.

  In view of the above problems, an object of the present invention is to provide a portable terminal device that can extract a tapping operation with high accuracy and low cost and an operation method thereof.

  In order to achieve the above object, a first configuration of a mobile terminal device of the present invention is based on a sound wave signal detecting means for detecting a sound wave signal corresponding to a hitting operation on the mobile terminal device, and characteristics of the detected sound wave signal. And extracting means for extracting the hitting action, conversion means for converting the operation command corresponding to the extracted hitting action, and executing means for executing the operation command.

  A second configuration of the mobile terminal device according to the present invention is characterized in that, in the first configuration, the characteristics of the sound wave signal are a frequency characteristic and an impulse response characteristic of the sound wave signal.

  A third configuration of the mobile terminal device according to the present invention is characterized in that, in the first configuration, the conversion means encodes the extracted rhythm pattern of the hitting operation to obtain an operation command.

  According to a fourth configuration of the portable terminal device of the present invention, in the first configuration, the extraction unit obtains a frequency characteristic of the sound wave signal, and identifies a substance that has performed a tapping operation based on the frequency characteristic. It is characterized by that.

  According to a fifth configuration of the portable terminal device of the present invention, in the first configuration, the extraction unit obtains an impulse response characteristic of the sound wave signal, and a portable terminal in which a tapping operation is performed based on the impulse response characteristic. The position on the device is specified.

  A sixth configuration of the portable terminal device according to the present invention is characterized in that, in the fourth configuration, the conversion means obtains an operation command based on the extracted substance that has performed the tapping operation.

  According to a seventh configuration of the portable terminal device of the present invention, in the fifth configuration, the conversion means obtains an operation command based on a position on the portable terminal device where the extracted tapping operation is performed. Features.

  According to another aspect of the present invention, there is provided a method of operating an electronic device including a portable terminal device, comprising: detecting a sound wave signal corresponding to a hitting operation on the electronic device by a sound wave signal detecting unit built in the electronic device; A step of extracting the hitting motion based on the characteristic of the detected sound wave signal, converting it into an operation command corresponding to the extracted hitting motion, and executing the operation command. .

  According to another aspect of the present invention, there is provided a computer program for causing an electronic device to execute an operation method for an electronic device including a portable terminal device according to the present invention, wherein the computer program processes a sound wave signal detected by a sound wave signal detecting unit built in the electronic device. Based on the characteristics of the sound wave signal thus obtained, a process of extracting a hitting operation on the electronic device, a process of converting to an operation command corresponding to the extracted hitting action, and a process of executing the operation command The apparatus is characterized by being executed.

  According to the present invention, a sound wave signal corresponding to the hitting operation is detected by the sound wave signal detecting means, and analysis processing is performed using a wideband signal having a large amount of information, thereby extracting the hitting operation with high accuracy. it can.

  In addition, since the sound wave propagating through the inside of the mobile terminal device is used, the hitting operation can be extracted with high accuracy without being affected by the external environment where the terminal device is placed.

  Furthermore, since sound analysis processing is performed using a sound wave signal detection means such as an existing microphone, it is not necessary to add a dedicated device, and the tapping operation is extracted at low cost, and the operation of the portable terminal device by tapping operation is performed. Make it possible.

  Embodiments of the present invention will be described below with reference to the drawings. However, such an embodiment does not limit the technical scope of the present invention.

  In the embodiment of the present invention, in order to enable the mobile terminal device to be operated by the tapping operation, the tapping (tapping) operation with respect to the main body of the mobile terminal device incorporating a microphone (sound wave signal detecting means) such as a mobile phone is supported. The sound wave signal to be detected is detected by a microphone, and the characteristics of the sound wave signal are analyzed to extract the hitting operation, and the portable terminal device executes a function corresponding to the hitting operation.

  An impulse-type sound wave having a very short duration is generated by a tapping operation applied to the mobile terminal device body. In the present invention, the sound wave is detected by an existing built-in microphone for voice input. The frequency characteristic of the sound wave signal corresponding to the tapping operation varies depending on the material on which the tapping operation is performed.

  FIG. 1 is a diagram illustrating an example of frequency characteristics for each substance subjected to a tapping operation. As shown in FIG. 1, the frequency characteristics of the sound wave signal are different when the mobile terminal device is hit with a nail, when hit with a plastic, or when hit with a metal. The frequency characteristic of the sound wave signal is obtained by performing Fourier transform (FFT) on the sound wave signal detected by the microphone.

  When a hitting operation is applied to the mobile terminal device body, a sound wave signal (impact signal) corresponding to the hitting operation reaches the microphone through a plurality of paths.

  FIG. 2 is a diagram schematically showing a sound wave transmission path corresponding to the tapping operation. The portable terminal device has a microphone 100 and operation buttons 103 on the front surface 107 side of the main body, and incorporates a component board 104, a battery 105, and the like inside. For example, when the tapping operation position 106 on the back surface 101 of the mobile terminal device is hit, the microphone 100 is reached through a plurality of transmission paths A to E. The illustrated transmission paths are merely schematic, and the actual transmission paths differ depending on the internal configuration, material, and the like.

  FIG. 3 is a diagram schematically showing the arrival time and amplitude of a sound wave propagating through each transmission path. The microphone 100 detects sound waves that have arrived through various transmission paths A to E as shown in FIG. 2, but propagates through each transmission path according to the distance and material (including air) of each transmission path. The arrival time and amplitude of sound waves to be different are different. The amplitude of the sound wave of the shortest transmission path (the shortest distance) is the largest, and the longer the arrival time (the longer the distance), the greater the attenuation during propagation and the weaker the amplitude. Then, the arrival time and amplitude pattern (hereinafter referred to as “impulse response characteristics”) for each transmission path differ depending on the position of the terminal body (hitting operation position). In other words, the tapping operation position has a unique impulse response characteristic.

  FIG. 4 is a diagram schematically illustrating an impulse response characteristic corresponding to the tapping operation position. FIG. 4 is an example of an impulse response characteristic when a single hit operation is performed on the terminal body. The impulse response characteristic increases in amplitude and reaches as the tap operation is performed at a position closer to the microphone. Time also tends to be short. Also, when the back of the microphone near the back, which is very close to the microphone, is struck, the amplitude of the transmission path in the shortest time is prominently larger than the amplitude of other transmission paths, and the folding mobile phone In the case of such a shape, the back surface near the speaker is the farthest position from the microphone. When the back surface near the speaker is struck, the amplitude of the transmission path in the shortest time is larger than the amplitude of the other transmission paths. Since the transmission path is also greatly attenuated, the amplitude is weaker than when hitting another position, and the difference in amplitude for each transmission path is also small.

  In this way, the impulse response characteristic data of the sound wave corresponding to the tapping operation with respect to various positions of the portable terminal device and the frequency characteristic data of the sound wave for each substance that performs the tapping operation are stored in the portable terminal device. By obtaining the impulse response characteristic and frequency characteristic of the detected sound wave and collating with the stored data, the sound wave corresponding to the hitting operation can be extracted from the sound waves detected by the microphone.

  FIG. 5 is a flowchart of a process for extracting a sound wave corresponding to the hitting operation from the sound wave detected by the microphone. The processing is performed by the CPU executing the software incorporated in the mobile terminal device. When the sound wave signal detected by the microphone is monitored and a large amplitude exceeding a predetermined threshold value is detected (S100), FFT (Fast Fourier Transform) is performed on the sound wave signal with the generation time of the amplitude level exceeding the threshold value as the center. ) (S102) and frequency conversion is performed.

  The obtained frequency characteristic is compared with the frequency characteristic for each substance stored in advance. Specifically, the mobile terminal device. The frequency characteristic data table for each substance as shown in FIG. 1 is stored, the initial value is set to table number M = 0 (S104), and the frequency characteristic data of table number M = 0 and obtained in S102. The frequency characteristic data is compared (S106), and it is determined whether or not there is an identity exceeding a certain standard. If they do not match, the table number is added (S108), and the frequency characteristic data corresponding to the next table number is compared with the obtained frequency characteristic data (S106). In comparison with the frequency characteristic data corresponding to the last table number, the table number is added (S108). When the table number exceeds the maximum value (S110), the strong amplitude detected in S100 is It is determined that it is not due to the tapping operation, and the process returns to S100. When it is determined that the obtained frequency characteristic data matches the frequency characteristic data corresponding to one of the table numbers (S106), it is determined that the hitting operation has been performed with the substance having the matching table number (S112).

  Subsequently, the tapping operation position is obtained. The portable terminal device stores an impulse response characteristic data table corresponding to the hitting operation position as shown in FIG. 4, and in order to obtain the impulse response characteristic, the amplitude levels vp and the arrival in the plurality of transmission paths are reached. Time tp is obtained. First, the transmission path number is set to n = 0 (initial value) (S114), the amplitude level vp at the transmission path number n = 0 is stored (S116), and the arrival time tp is further stored (S118). Specifically, the amplitude level vp and the arrival time tp in the transmission path number n = 0 are the amplitude levels of the strong peaks (pulses) detected in S100, the amplitude levels are stored (S116), and the amplitude levels The occurrence time is stored (S118).

  Then, every time the specified time td elapses (S120), it is determined whether the specified time td has not passed the longest impulse response time (S122). The sound wave signal is subjected to FFT (Fast Fourier Transform) (S124) centering on time, and the frequency characteristic of the sound wave signal at that time is obtained each time the specified time elapses. The longest impulse response time is a time until the slowest impulse response in the impulse response pattern of each transmission path is detected while ignoring a weak impulse response below a certain level.

  Then, it is determined whether or not the frequency characteristics at the time of each specified time td coincide with the frequency characteristics of the table number determined in S112 (S126).

  Actually, there is a predetermined time difference between the arrival time tp in the transmission route of the transmission route number n = 0 and the arrival time of the next transmission route that arrives earlier, and the sound wave signal corresponding to the next transmission route is Until it arrives, no coincidence determination is made in the process of S126.

  If the match is not determined in the process of S126, the transmission path number n is added (S128), it is determined whether or not the maximum value (for example, 10) of the predetermined transmission path number is reached (S130), and the maximum value is exceeded. Steps S120 to S128 are repeated until the above. Since the sound wave signal corresponding to the hitting operation reaches the microphone through a plurality of transmission paths, the amplitude level and arrival time are different for each transmission path. By the processing of S120 to S128, the sound wave signal corresponding to the hitting operation can be extracted by comparing with the frequency characteristic of the sound wave signal corresponding to the hitting operation for each specified time td, and the amplitude level and the arrival time can be extracted. Impulse response characteristics can be obtained.

  When the maximum impulse response time has elapsed (S122), or when the transmission path number exceeds the maximum value (S130), the process proceeds to step S132, and the impulse response characteristic pattern table as shown in FIG. The impulse response characteristics obtained are compared. The portable terminal device stores in advance an impulse response characteristic pattern table when hitting various positions of the portable terminal device as shown in FIG. 4, and extracts a pattern that matches the impulse response characteristic obtained by the above processing. Then, the position of the mobile terminal device corresponding to the pattern is determined as the tapping operation position (S134).

  FIG. 6 is a diagram for explaining the flow of processing for extracting a sound wave corresponding to the hitting action from the sound wave detected by the microphone and executing a function corresponding to the hitting action. FIG. 6A shows an example of a waveform of a sound wave signal detected by the microphone.

  The tapping operation extraction process shown in FIG. 5 is performed on the detected sound wave signal. That is, if a substance that strikes the mobile terminal device is identified as the beating operation from the frequency characteristics, and a position where the beating operation is performed is identified from the impulse response characteristics, it is determined that the beating operation (striking) has been performed.

  FIG. 6B is a diagram showing the extracted tapping operation on the time axis. With respect to the waveform of FIG. 6A, a pulse signal is obtained with respect to the time when the hitting operation is extracted. When the hitting motion is extracted, the rhythm pattern of the hitting motion is subsequently determined.

  Specifically, as shown in FIG. 6 (c), a beating operation region sandwiched between blank (guard) times (start determination time and end determination time) in which no pulse corresponding to the beating operation for a certain period of time exists is cut out, The shortest interval between pulses existing in the region is obtained. In FIG. 6C, the shortest interval is between the pulse P1 and the pulse P2. For the extracted tapping motion region, the shortest interval is set as the cycle time, and the presence / absence of a pulse is determined every cycle time, and the tapping operation is performed by encoding “1” with pulse and “0” without pulse. It is converted into a digital code that is an operation command.

  FIG. 6D is a diagram illustrating a digital code that is an operation command corresponding to the tapping operation. In the tapping operation region, the first cycle time includes the pulse P1, so the digital code is “1”, and the next cycle time includes the pulse P2, so the digital code is “1”. Since the pulse P2 and the pulse P3 are more than the period time, and the next period time has no pulse, the digital code is “0”. Further, since the next cycle time includes the pulse P3, the digital code is “1”, and as a result, the digital code obtained by the tapping operation is “1101”. As described above, the presence / absence of a hitting operation for each cycle time is obtained as a rhythm pattern and encoded, whereby the hitting operation can be converted into an operation command.

  The mobile terminal device stores a table to which the function of the mobile terminal device corresponding to the digital code is assigned, and activates the function corresponding to the digital code with reference to the table. For example, as shown in FIG. 6E, if the function corresponding to the operation command (digital code) “1101” is “mail arrival check”, it is activated.

  FIG. 7 is a diagram illustrating an example of a table to which the functions of the mobile terminal device corresponding to the digital code (operation command) are assigned. FIG. 7 is a table when the mobile terminal device is a foldable mobile phone, for example, when the mobile phone is folded, opened, or in a call with respect to the same digital code. , Different functions are assigned. Of course, only one function may be assigned regardless of the usage form of the mobile phone. Different functions may be assigned depending on the hardware usage mode when folded or opened, or different functions may be assigned depending on the software usage mode (mode) such as a call or e-mail. Also good. The function to be assigned can be arbitrarily set by the user.

  FIG. 8 is a diagram showing a block configuration of the portable terminal device in the embodiment of the present invention. As an example, the configuration is almost the same as the voice input processing portion of a mobile phone. The sound wave signal input from the microphone 100 is input to the anti-aliasing filter (AAF) 112 via the level adjustment unit (TXVOL) 110 that adjusts the volume. The AAF 112 removes sound wave signals having a frequency higher than the Nyquist frequency in order to eliminate aliasing noise.

  The sound wave signal output from the AAF 112 is converted into a digital signal by the ADC 114, further noise-removed by the band pass filter (BPF) 116, and input to the PCM format conversion unit 118. PCM (Pulse Code Modulation) is a well-known method for digitizing sound, and measures the instantaneous voltage of the sound waveform every very short time (sampling rate) (approximately 44,100 times per second for CDs, etc.) This value is digitized by converting the value into, for example, a 14-digit binary number.

  The baseband CPU 120 is a CPU responsible for voice processing for calls, and converts the pulses corresponding to the tapping operation extraction process and the extracted tapping operation shown in FIG. 5 into a digital code for the sound wave signal converted into the PCM format. The processing up to this time (the processing from FIG. 6A to FIG. 6D) is executed. The baseband CPU 120 sends the obtained digital code to the multimedia CPU 24.

  The multimedia CPU 122 is a CPU responsible for functions such as e-mail, Internet, music playback, and image processing. The multimedia CPU 122 selects a function corresponding to the input digital code with reference to the table of FIG. 7 and executes it. To do.

  As described above, according to the portable terminal device in the embodiment of the present invention, the sound wave signal corresponding to the light hitting (striking operation) on the terminal body can be detected by the microphone, and the tapping operation can be extracted as the voice analysis processing. it can. By obtaining the frequency characteristic and impulse response characteristic of the sound wave signal corresponding to the tapping operation, the tapping operation can be extracted with high accuracy. Further, the present invention can be realized only by incorporating an analysis processing algorithm (the processing of FIG. 5) as software without adding a dedicated device, and an increase in cost can be prevented.

  When detecting with an acceleration sensor, there is a possibility that vibration or impact on the mobile terminal device may be detected as a tapping operation. However, in the method of the present invention, a clear tapping operation by the user is required, so that a malfunction can be prevented. .

  Also, any part of the mobile terminal device may be struck, the user can operate without visually observing the mobile terminal device, and the tapping action can be extracted regardless of the state in which the mobile terminal device is held. The operating environment is not limited. For example, the user can also operate the mobile terminal device while keeping it in a pocket.

  Since the position where the tapping operation is performed can be specified, a predetermined function may be assigned to the position where the tapping operation is performed. For example, different functions are assigned when the back of the microphone immediately behind is hit and when the vicinity of a speaker far from the microphone is hit. The mobile terminal device has a table in which functions (operation commands) are assigned to the hit positions, and executes the operation commands corresponding to the specified positions.

  Furthermore, since the substance that has performed the tapping operation can be specified, a predetermined function may be assigned to the substance that has performed the tapping operation. For example, different functions are assigned to each of a case where the user taps with a nail, a case where the user taps with a plastic, and a case where the user taps with a metal. The mobile terminal device has a table in which functions (operation commands) are assigned to the hit material, and executes an operation command corresponding to the specified material.

  The above-described method of operating the mobile terminal device by the beating operation on the mobile terminal device is not limited to the electronic device such as the mobile terminal device, but can also be applied to a stationary electronic device incorporating a microphone.

(Appendix 1)
A microphone that detects a sound wave signal corresponding to a tapping operation on the mobile terminal device;
Extraction means for extracting the hitting operation based on the characteristics of the detected sound wave signal;
Conversion means for converting into an operation command corresponding to the extracted tapping action;
A portable terminal device comprising execution means for executing the operation command.

(Appendix 2)
In Appendix 1,
The characteristic of the sound wave signal is a frequency characteristic and an impulse response characteristic of the sound wave signal.

(Appendix 3)
In Appendix 1,
The portable terminal device characterized in that the conversion means obtains an operation command by encoding the extracted rhythm pattern of the hitting action.

(Appendix 4)
In Appendix 1,
The portable terminal device characterized in that the extraction means obtains a frequency characteristic of the sound wave signal, and identifies a substance that has performed a tapping operation based on the frequency characteristic.

(Appendix 5)
In Appendix 1,
The mobile terminal device characterized in that the extraction means obtains an impulse response characteristic of the sound wave signal and specifies a position on the mobile terminal device where the tapping operation is performed based on the impulse response characteristic.

(Appendix 6)
In Appendix 4,
The portable terminal device characterized in that the conversion means obtains an operation command based on the extracted material that has performed the tapping operation.

(Appendix 7)
In Appendix 5,
The said conversion means calculates | requires an operation command based on the position on the portable terminal device in which the said tap operation | movement was performed.

(Appendix 8)
Detecting a sound wave signal corresponding to a hitting operation with respect to the electronic device by a sound wave signal detecting means built in the electronic device;
The control unit of the electronic device extracts the hitting operation based on the characteristic of the detected sound wave signal, converts it into an operation command corresponding to the extracted hitting operation, and executes the operation command. A method for operating an electronic device, comprising:

(Appendix 9)
In a computer program for processing a sound wave signal detected by a sound wave signal detecting means built in an electronic device,
Based on the characteristics of the detected sound wave signal, a process of extracting a hitting operation on the electronic device;
A computer program for causing the electronic device to execute a process of converting to an operation command corresponding to the extracted hitting operation and a process of executing the operation command.

It is a figure which shows the example of the frequency characteristic for every substance which performed the tapping operation | movement. It is a figure which shows typically the transmission path | route of the sound wave corresponding to a hit operation | movement. It is a figure which shows typically the arrival time and amplitude of the sound wave which propagates each transmission path. It is a figure which shows typically the impulse response characteristic corresponding to a tapping operation position. It is a flowchart of the process which extracts the sound wave corresponding to hitting operation from the sound wave which a microphone detects. It is a figure explaining the flow of a process which extracts the sound wave corresponding to a hit operation from the sound wave which a microphone detects, and performs the function corresponding to a hit operation. It is a figure which shows the example of the table which allocated the function of the portable terminal device corresponding to a digital code | symbol. It is a figure which shows the block configuration of the portable terminal device in the implementation of this invention.

Explanation of symbols

  100: Microphone, 118: PCM format converter, 120: Baseband CPU, 122: Multimedia CPU

Claims (3)

A sound wave signal detecting means for detecting a sound wave signal corresponding to a tapping operation on the portable terminal device;
Extraction means for extracting the hitting operation based on the characteristics of the detected sound wave signal;
Conversion means for converting into an operation command corresponding to the extracted tapping action;
A portable terminal device comprising execution means for executing the operation command. In claim 1,
The characteristic of the sound wave signal is a frequency characteristic and an impulse response characteristic of the sound wave signal. In claim 1,
The portable terminal device characterized in that the conversion means obtains an operation command by encoding the extracted rhythm pattern of the hitting action.

Images