JP4822458B2 - Interface device and interface method - Google Patents

Description

  The present invention relates to an interface device and an interface method for inputting a user operation to a computer or the like using a breath or voice for a position in a three-dimensional space.

  Conventionally, there are many pointing devices such as a mouse, a touch pad, a trackball, and a pointing stick, all of which are intended to input a position on a two-dimensional plane. However, for example, there is already a method of controlling the posture or viewpoint position of a three-dimensional model displayed on the screen with a conventional mouse or the like in a three-dimensional CAD system or the like, but a combination of two-dimensional movement Since 3D spatial information is input, there is a problem that it is difficult to specify a desired position and viewpoint.

  In order to solve such problems, it is indispensable to develop a pointing device that can intuitively input three-dimensional information. For example, in a conventional mouse or joystick, a pointing device such as a space ball or a space mouse, which can input three-dimensional information by increasing the rotation axis of a sensor, has already been put into practical use. Furthermore, development of an apparatus that indicates a three-dimensional position by combining arms has been performed.

  Conventional pointing devices require a user to operate while holding or touching any device. However, for example, it is difficult for a handicapped person or the like to handle these pointing devices, resulting in an obstacle to access to an information terminal using these pointing devices.

  In order to solve such a problem, for example, in Patent Document 1 (Japanese Patent Application Laid-Open No. 2004-280301) and the like, a user moves his / her mouth while blowing exhalation on a microphone array without wearing a special device. Alternatively, we are developing a pointing device that moves the cursor smoothly by moving the face.

In addition, the technique described in Patent Document 2 (Japanese Patent Laid-Open No. 2007-228135) estimates the utterance position of a user who uttered a voice in a specific area using a microphone array even in an environment with ambient noise.
JP 2004-280301 A JP 2007-228135 A

  However, since the method described in Patent Document 1 cannot distinguish between a user's breath sound and ambient noise, there is a problem that malfunction easily occurs due to noise. In addition, since a microphone array in which microphones are arranged on a plane is used to detect the movement of the user's mouth and face in two dimensions, a conventional pointing device for inputting a two-dimensional position and Similarly, there is a problem that it is difficult to input a three-dimensional position.

  In the method described in Patent Document 2, the utterance position of the user who uttered in a specific area is estimated by the microphone array even in an environment with ambient noise. However, since the user's utterance position is estimated as a two-dimensional position, it has not functioned as a pointing device for inputting three-dimensional information.

  In view of the above-described problems of the prior art, the present invention provides an interface device that can three-dimensionally identify a user's exhalation sound or voice utterance position even in a noisy environment.

  In order to solve the above-described problem, the invention according to claim 1 is issued from a nasal cavity of a user based on a microphone array in which a plurality of microphones are provided in a predetermined arrangement and voice data acquired by the microphone array. And an utterance position specifying means for three-dimensionally specifying the utterance position of the sound.

  According to a second aspect of the present invention, in the interface device according to the first aspect of the present invention, the interface device further includes a utterance position detection signal generating unit that generates a signal based on the utterance position specified by the utterance position specifying unit. To do.

  According to a third aspect of the present invention, there is provided a microphone array in which a plurality of microphones are provided in a predetermined arrangement, and start of utterance of a continuous sound emitted from the user's nasal cavity based on voice data acquired by the microphone array. An interface device comprising: vector specifying means for three-dimensionally specifying a vector connecting the position and the utterance end position.

  According to a fourth aspect of the present invention, there is provided a microphone array in which a plurality of microphones are provided in a predetermined arrangement, and start of utterance of a continuous sound emitted from a user's nasal cavity based on voice data acquired by the microphone array. An interface device comprising trajectory specifying means for three-dimensionally specifying a trajectory from a position to a utterance end position.

  According to a fifth aspect of the present invention, there is provided a microphone array in which a plurality of microphones are provided in a predetermined arrangement, and a direction in which sound emitted from a user's nasal cavity based on voice data acquired by the microphone array arrives. And an utterance direction specifying means for specifying an azimuth angle and an elevation angle.

  Further, the invention according to claim 6 is the interface device according to claim 5, further comprising a voice direction detection signal generation means for generating a signal based on the azimuth angle and the elevation angle specified by the voice direction specification means. It is characterized by.

  According to a seventh aspect of the present invention, there is provided a microphone array in which a plurality of microphones are provided in a predetermined arrangement, and start of utterance of a continuous sound emitted from a user's nasal cavity based on voice data acquired by the microphone array. An interface device comprising: a direction vector specifying means for specifying a vector connecting between an azimuth angle and an elevation angle of a direction, and a azimuth angle and an elevation angle of an utterance end direction.

  According to an eighth aspect of the present invention, there is provided a microphone array in which a plurality of microphones are provided in a predetermined arrangement, and start of utterance of a continuous sound emitted from a user's nasal cavity based on voice data acquired by the microphone array. It is an interface device characterized by having direction locus specifying means for specifying a locus from an azimuth angle and an elevation angle of a direction to an azimuth angle and an elevation angle in the utterance end direction.

According to a ninth aspect of the present invention, there is provided a microphone array in which a plurality of microphones are provided in a predetermined arrangement, and a start of continuous sound uttered from a user's nasal cavity based on voice data acquired by the microphone array. And a time specifying means for specifying the time from the end of utterance to the end of utterance.

  According to a tenth aspect of the present invention, there is provided a microphone array in which a plurality of microphones are provided in a predetermined arrangement, and a moving speed of a continuous sound emitted from a user's nasal cavity based on voice data acquired by the microphone array. And a moving speed specifying means for specifying the interface device.

  According to an eleventh aspect of the present invention, in the interface device according to any one of the first to tenth aspects, the feature point specifying means for specifying a feature point of a sound emitted from a user's nasal cavity, and the feature And feature point detection signal generating means for generating a signal based on the feature point specified by the point specifying means.

  According to a twelfth aspect of the present invention, in the interface device according to the eleventh aspect, the feature point of the sound emitted from the user's nasal cavity extracted by the feature point extraction means is a feature point of a fundamental frequency, an unvoiced sound, It is one of a feature point related to voiced sound, a feature point of volume, a feature point related to sound source, and a feature point related to utterance content.

  The invention according to claim 13 three-dimensionally specifies the utterance position of the sound emitted from the user's nasal cavity based on the sound data acquired by the microphone array in which a plurality of microphones are provided in a predetermined arrangement. And an utterance position specifying step.

  The invention according to claim 14 is the interface method according to claim 13, further comprising a utterance position detection signal generation step of generating a signal based on the utterance position specified by the utterance position specification step. To do.

  The invention according to claim 15 is characterized in that the utterance start position and utterance end position of a continuous sound emitted from the user's nasal cavity based on voice data acquired by a microphone array in which a plurality of microphones are provided in a predetermined arrangement. And a vector specifying step for specifying a vector connecting the two in three dimensions.

  In the invention according to claim 16, the utterance end position from the utterance start position of the continuous sound emitted from the user's nasal cavity based on the voice data acquired by the microphone array in which a plurality of microphones are provided in a predetermined arrangement. This is an interface method characterized by having a trajectory specifying step for specifying the trajectory up to three-dimensionally.

  The invention according to claim 17 is directed to an azimuth angle and an elevation angle in a direction in which a sound emitted from a user's nasal cavity arrives based on voice data acquired by a microphone array in which a plurality of microphones are provided in a predetermined arrangement. And an utterance direction specifying step for specifying.

  Further, the invention according to claim 18 is the interface method according to claim 17, further comprising an utterance direction detection signal generation step of generating a signal based on the azimuth angle and the elevation angle specified by the utterance direction specification step. It is characterized by.

  According to the nineteenth aspect of the present invention, there is provided an azimuth angle in a voice start direction of a continuous sound emitted from a user's nasal cavity based on voice data acquired by a microphone array in which a plurality of microphones are provided in a predetermined arrangement. An interface method comprising: an elevation angle and a direction vector identification step for identifying a vector connecting the azimuth angle and the elevation angle in the utterance end direction.

  The invention according to claim 20 is characterized in that an azimuth angle in the utterance start direction of a continuous sound emitted from a user's nasal cavity based on voice data acquired by a microphone array in which a plurality of microphones are provided in a predetermined arrangement. An interface method comprising a direction trajectory specifying step of specifying a trajectory from an elevation angle to an azimuth angle and an elevation angle in the utterance end direction.

  The invention according to claim 21 is from the start of utterance to the end of utterance of a continuation sound emitted from the user's nasal cavity based on voice data acquired by a microphone array in which a plurality of microphones are provided in a predetermined arrangement. It is an interface method characterized by having the time specification step which specifies time.

  According to a twenty-second aspect of the present invention, a moving speed that specifies a moving speed of a continuous sound emitted from a user's nasal cavity based on voice data acquired by a microphone array in which a plurality of microphones are provided in a predetermined arrangement. It is an interface method characterized by having a specific step.

  The invention according to claim 23 is the interface method according to any one of claims 13 to 22, wherein a feature point specifying step for specifying a feature point of a sound emitted from a user's nasal cavity; And a feature point detection signal generation step for generating a signal based on the feature point specified by the point specification step.

  According to a twenty-fourth aspect of the present invention, in the interface method according to the twenty-third aspect, the feature points of the sound emitted from the user's nasal cavity extracted by the feature point extraction step are the feature points of the fundamental frequency, the unvoiced sound, It is one of a feature point related to voiced sound, a feature point of volume, a feature point related to sound source, and a feature point related to utterance content.

  According to the interface device and the interface method of the present invention, items such as the user's breath sounds and utterance positions of utterances are specified three-dimensionally even under noisy environments, and processing corresponding to the specified items is performed on the computer side. Be able to run.

  Hereinafter, embodiments of the present invention will be described with reference to the drawings. FIG. 1 is a perspective view showing an external appearance of an interface device according to an embodiment of the present invention, and FIG. 2 is a block diagram showing the interface device according to the embodiment of the present invention. 1 and 2, 100 is an interface device, 200 is a microphone array, 201 is a silicon microphone, 202 is a wind screen, 210 is a stand, 211 is a main support, 212 is a left support, 213 is a right support, and 280 is a microphone amplifier. Reference numeral 290 denotes an AD conversion unit, 300 denotes a CPU, 400 denotes a storage unit, and 500 denotes a connection port unit.

  FIG. 1 shows the configuration of a user interface unit of the interface apparatus 100, which functions as an input device for a computer (not shown) based on the sound emitted from the nasal cavity and oral cavity of the user as shown. Here, examples of the computer include a general-purpose personal computer. However, the computer is not limited to this and can be used as an interface device for various computers. The interface device 100 of the present invention can be used not only for input to a computer but also for input to an electric product or a vehicle.

The external appearance of the interface device 100 is composed of a main column 211 standing on a stand 210, a left column 212, a right column 213, and a group of microphones provided on each column. It can be installed on a desktop. More specifically, silicon microphones 201 are provided on a substrate (not shown) at 3 cm intervals on each of the main support column 211, the left support column 212, and the right support column 213, and the microphone array 200 is composed of a total of 12 microphone groups. It is configured. The interface device 100 according to the present embodiment will be described based on the case where twelve silicon microphones 201 are used. However, the number of silicon microphones 201 may be three or more, and the present invention is twelve. However, the present invention is not limited to the use of the silicon microphone 201. Note that if the number of silicon microphones 201 is small, the noise resistance deteriorates, and if the number of silicon microphones 201 is large, the processing load of audio data becomes heavy. In this embodiment, as described above, the microphone array 200 includes twelve silicon microphones 201. The silicon microphone 201 is a small silicon microphone of about 3 mm × 5 mm.

  The four silicon microphones 201 arranged on each column are covered with a wind screen 202, and the wind noise is not input. The microphone group arranged on the left column 212 and the microphone group arranged on the right column 213 are arranged so as to have a substantially “C” -shaped layout, and the microphone group arranged on the main column 211 is vertical. Is arranged.

  FIG. 2 is a diagram showing a block configuration of the interface apparatus 100. The output of the microphone array 200 composed of twelve silicon microphones 201 is amplified by a microphone amplifier 280 and subjected to analog-digital conversion by an AD conversion unit 290 and then input to the CPU 300. The storage unit 400 includes a ROM that stores programs that run on the CPU 300 and a RAM that functions as a work area for the CPU 300. When the CPU 300 operates based on the program stored in the storage unit 400, it functions as the interface device 100 of the present invention.

  It should be noted that the category of the claims is “device”, and “speech position specifying means”, “speech position detection signal generating means”, “vector specifying means”, “trajectory specifying means”, “ "Speech direction specifying means", "Speech direction detection signal generating means", "Direction vector specifying means", "Direction locus specifying means", "Time specifying means", "Movement speed specifying means", "Feature point specifying means", etc. Each means is realized by the CPU 300 that operates based on a program stored in the storage unit 400.

  In addition, the “speech position specifying step”, “speech position detection signal generation step”, “vector specifying step”, “trajectory specifying step”, “ "Speech direction identification step", "Speech direction detection signal generation step", "Direction vector identification step", "Direction of trajectory identification step", "Time identification step", "Movement speed identification step", "Feature point identification step", etc. Each step is realized by the CPU 300 that operates based on a program stored in the storage unit 400.

  The storage unit 400 stores and holds an event database, which will be described later. The connection port unit 500 is an interface unit for connecting to other devices such as a computer, and a known device such as a USB can be used.

A usage form of the interface device 100 configured as described above will be described. Various embodiments will be individually described below, but each embodiment can be realized by changing a program stored in the storage unit 400. In addition, an interface device configured by arbitrarily combining various embodiments described below is also included in the interface device of the present invention.

  FIG. 3 is a diagram showing an example of how the interface device according to the embodiment of the present invention is used. In the interface device 100 according to the present embodiment, the interface device 100 is used to identify which region the utterance position estimated in the three-dimensional space belongs.

  Hereinafter, the term “speech” includes all kinds of sounds emitted from the user's nasal cavity. The sound emitted from the user's nose and mouth includes, for example, the sound of a tongue, but as a general use, a short utterance sound such as a user's “shush” or “pad” or “shoe” ”,“ A ”, etc., continuous continuous utterance sounds are assumed.

  In the embodiment shown in FIG. 3, a user's utterance detection area is defined, and only the user's utterance in the utterance detection area is detected, and the sound from outside the utterance detection area is processed as noise. Further, the defined user utterance detection area is divided into, for example, eight areas as shown in the figure. Then, the region in which the utterance is detected is specified in the eight divided regions. This is the “speech position specifying means” and “speech position specifying step” described in the claims. Further, when utterance is detected in the front and upper left divided areas, for example, a signal corresponding to the left click of the mouse is generated and transmitted to the computer side. The generation of such a signal is expressed as “speech position detection signal generation means” and “speech position detection signal generation step” in the claims.

  Processing of the interface device in the above embodiment will be described. FIG. 4 is a diagram showing a flowchart of processing of the interface device according to the embodiment of the present invention.

  When the process is started in step S100, the process proceeds to step S101, and audio data is captured from the microphone array 200. More specifically, in this step, an analog audio signal output from the microphone array 200 is amplified by the microphone amplifier 280, converted into a digital signal by the AD conversion unit 290, and temporarily stored in the storage unit 400.

  In the next step S102, the three-dimensional information of the user utterance position and the ambient noise arrival direction is specified. In more detail, using the method described in Japanese Patent Application Laid-Open No. 2007-228135, Japanese Patent Application Laid-Open No. 2008-67854, Japanese Patent Application No. 2006-240721, and drawings by the inventors of the present application, The ambient noise arrival direction is specified in a three-dimensional space.

  Next, in step S103, it is determined whether or not there is a user utterance. In this step, the user's utterance is detected using the method described in Japanese Patent Application No. 2006-240721, and if the user's utterance is not detected, the process is repeated from step S101. If the user utterance is detected, the process proceeds to step S104.

  In step S104, ambient noise is suppressed. In this step, sound source separation processing for suppressing ambient noise and enhancing the user's utterance is performed using the method described in Japanese Patent Application No. 2006-240721.

  In step S105, a three-dimensional utterance position is specified. More specifically, the region to which the utterance position estimated in the three-dimensional space belongs is specified. For example, as shown in FIG. 3, a user utterance detection area is defined, and the utterance detection area is further divided into eight areas. Then, the region in which the utterance is detected is specified in the eight divided regions.

  In step S106, event identification is executed. In the event database held in the storage unit 400, events corresponding to the utterance detection position and the like are stored. That is, a combination of an utterance detection position and an event is defined and held in the event database.

  In the event database, for example, an event defined as a short-time utterance in the left area before the upper row in FIG. 3 is registered in advance. In the event specifying process in step S106, it is determined whether the utterance position is the above-described position, whether the utterance duration is equal to or less than a certain threshold, and whether the utterance is an unvoiced sound. It is determined that the event has occurred when all the conditions are met.

  In step S107, it is determined whether there is a corresponding event. In step S106, it is checked whether an event that matches the event database is detected. If no event is detected, the process returns to step S101. If an event is detected, the process proceeds to step S108.

  In step S108, an event detection signal is transmitted to the application level on the computer side.

  Typical processing on the application side is shown in the dotted box. Hereinafter, typical processing assumed on the application side will be described. In step S201, it continues to wait for the reception of an event detection signal sent from the interface device of the present invention. If an event detection signal is received, the process proceeds to step S202. In step S202, an appropriate process corresponding to the received event detection signal is executed. Then, the process returns to step S201.

  As described above, according to this embodiment, the user's exhalation sound, the utterance position of the utterance, and the like can be specified three-dimensionally even in an environment with noise, and the processing corresponding to the specified matter can be executed on the computer side. become able to.

  In the embodiment described above, an example in which the utterance detection area is divided into eight areas has been described. However, the present invention is not limited to this, and for example, by dividing the utterance detection area into four areas as shown in FIG. It can also be used for cross-movement of the cursor displayed on the display unit. FIG. 5 is a diagram showing another example of area division of the interface device according to the embodiment of the present invention.

  Next, another embodiment of the present invention will be described. FIG. 6 is a diagram showing an example of how the interface device according to another embodiment of the present invention is used. In this embodiment, when a continuous sound such as “A” is uttered by the user, a vector connecting the utterance start position (A) and the utterance end position (B) of the continuous sound is specified three-dimensionally. To do. Such identification is expressed as “vector identification means” and “vector identification step” in the claims.

  Such an embodiment is realized by changing the process in step 105 of the flowchart shown in FIG. 4 to a process of recording a position at the start of utterance and a position at the end of utterance and specifying a vector connecting the two points. It becomes possible to do.

  As described above, according to the present embodiment, even in a noisy environment, the user's exhalation sound, the utterance position of the utterance, and the like are specified in a three-dimensional manner, and processing corresponding to the specified items is executed on the computer side. Will be able to.

  Next, another embodiment of the present invention will be described. FIG. 7 is a diagram showing an example of how the interface device according to another embodiment of the present invention is used. In this embodiment, when a continuous sound such as “A” is uttered by the user, the locus from the utterance start position (A) to the utterance end position (B) of the continuous sound is specified three-dimensionally. is there. Such identification is expressed as “trajectory identification means” and “trajectory identification step” in the claims.

  Such an embodiment can be realized by changing the process in step 105 of the flowchart shown in FIG. 4 to a process of recording the position at the start of utterance and the position at the end of utterance and specifying the locus between them. It becomes possible.

  As described above, according to the present embodiment, even in a noisy environment, the user's exhalation sound and the utterance position of the utterance are specified in a trajectory shape in three dimensions, and processing corresponding to the specified items is executed on the computer side. Will be able to.

  Next, another embodiment of the present invention will be described. FIG. 8 is a diagram showing an example of how the interface device according to another embodiment of the present invention is used. In the present embodiment, the azimuth angle (θ) and elevation angle (φ) in the direction in which the sound emitted from the user's nasal cavity arrives based on the audio data acquired by the microphone array 200 are specified. Such specification is expressed as “voice direction specifying means” and “voice direction specifying step” in the claims.

  Further, when the azimuth angle (θ) and elevation angle (φ) in the direction in which the user's utterance arrives are specified as described above, a signal based on the specified azimuth angle (θ) and elevation angle (φ) is generated. . Such a function is expressed as “speech direction detection signal generation means” and “speech direction detection signal generation step” in the claims.

  Such an embodiment can be realized by performing the utterance direction specifying process in step 105 of the flowchart shown in FIG. In addition, an event corresponding to the azimuth angle (θ) and the elevation angle (φ) that are the utterance directions in step S106 is defined.

  More specifically, in step 105, the azimuth angle (θ) and elevation angle (φ) of the direction of arrival of the voice uttered in the three-dimensional space are estimated, and in which area the direction of arrival is on the azimuth-elevation plane Specify whether it belongs to. For example, as shown in FIG. 10A, the azimuth-elevation plane is divided into a grid at regular intervals. Then, in which region in the azimuth-elevation plane the estimated direction of arrival is detected is specified. FIG. 10 is a diagram showing an azimuth (θ) -elevation angle (φ) plane assumed in an interface device according to another embodiment of the present invention.

  As described above, according to the present embodiment, the direction of the user's exhalation sound, the utterance position of the utterance, and the like can be specified even in a noisy environment, and processing corresponding to the specified matters can be executed on the computer side. become.

Next, another embodiment of the present invention will be described. FIG. 9 is a diagram showing an example of how to use the interface device according to another embodiment of the present invention. In this embodiment, when a user utters a continuous sound such as “A”, the azimuth angle (θ _A ) and elevation angle (φ _A ) of the continuous sound utterance start direction ( _A ) and the utterance end direction (B ) To specify a vector connecting the azimuth angle (θ _B ) and the elevation angle (φ _B ). Such identification is expressed as “direction vector identification means” and “direction vector identification step” in the claims. The “vector” in this embodiment is shown in the azimuth-elevation plane of FIG.

  Such an embodiment is realized by changing the process in step 105 of the flowchart shown in FIG. 4 to a process of recording the direction at the start of utterance and the direction at the end of utterance and specifying a vector connecting the two points. It becomes possible to do. That is, in step S105, the voice arrival direction (azimuth angle and elevation angle) at the start of utterance and the voice arrival direction (azimuth angle and elevation angle) at the end of utterance are recorded in the three-dimensional space, and on the azimuth-elevation plane To specify a vector connecting the two points.

  As described above, according to the present embodiment, the user's exhalation sound and utterance direction of the utterance can be specified in a vector even under noisy environment, and the processing corresponding to the specified matter can be executed on the computer side. become.

Next, another embodiment of the present invention will be described. FIG. 9 is a diagram showing an example of how to use the interface device according to another embodiment of the present invention. In this embodiment, when a continuous sound such as “A” is uttered by the user, the utterance end direction (θ _A ) and the elevation angle (φ _A ) of the utterance start direction (A) of the continuation sound ( B) specifies the trajectory up to the azimuth angle (θ _B ) and elevation angle (φ _B ). Such specification is expressed as “direction locus specifying means” and “direction locus specifying step” in the claims. The “trajectory” in this embodiment is shown in the azimuth-elevation plane of FIG.

  Such an embodiment is realized by changing the process in step 105 of the flowchart shown in FIG. 4 to a process of recording the direction at the start of utterance and the direction at the end of utterance and specifying the locus connecting the two points. It becomes possible to do. That is, in step S105, all the voice arrival directions (azimuth angle and elevation angle) from the start of utterance to the end of utterance are recorded in the three-dimensional space, and the movement locus is specified on the azimuth-elevation plane.

  As described above, according to the present embodiment, the user's breathing sound and the utterance direction of the utterance can be specified in a trajectory even in a noisy environment, and the processing corresponding to the specified matter can be executed on the computer side. become.

  Next, another embodiment of the present invention will be described. FIG. 11 is a diagram showing a flowchart of the process of the interface device according to another embodiment of the present invention. In the flowchart of FIG. 11, steps in which the same contents as those in the flowchart of FIG. Only the portion related to the present embodiment in the flowchart of FIG. 11 will be described.

  In the present embodiment, step S305 in FIG. 11 is performed. In step S305, processing for specifying the time from the start to the end of the continuous sound is performed. Such identification processing is indicated as “time identification means” and “time identification step” in the claims.

  As described above, according to the present embodiment, even in a noisy environment, a user's breath sound and duration of utterance are specified, and processing corresponding to the specified items can be executed on the computer side.

Next, another embodiment of the present invention will be described. FIG. 11 is a diagram showing a flowchart of the process of the interface device according to another embodiment of the present invention. In the flowchart of FIG. 11, steps in which the same contents as those in the flowchart of FIG. Only the portion related to the present embodiment in the flowchart of FIG. 11 will be described.

  In the present embodiment, step S305 in FIG. 11 is performed. In step S305, processing for specifying the moving speed of the position where the continuous sound is generated is performed. Such specifying processing is indicated as “moving speed specifying means” and “moving speed specifying step” in the claims.

  As described above, according to the present embodiment, the movement speed of the user's exhalation sound and utterance can be specified even in a noisy environment, and the processing corresponding to the specified matter can be executed on the computer side.

  Next, another embodiment of the present invention will be described. FIG. 11 is a diagram showing a flowchart of the process of the interface device according to another embodiment of the present invention. In the flowchart of FIG. 11, steps in which the same contents as those in the flowchart of FIG. Only the portion related to the present embodiment in the flowchart of FIG. 11 will be described.

  A characteristic process in the present embodiment is step S306, and in step S306, a process for specifying a feature point of a sound emitted from the user's nasal cavity is executed. Note that such a step can also be added between step S106 and step S107 in the flowchart shown in FIG. As described above, the interface device of the present invention includes a combination of any embodiments described in the specification.

  Specifying the feature points of the sound emitted from the user's nasal cavity is expressed as “feature point specifying means” and “feature point specifying step” in the claims.

  Next, the specific feature points of the sound emitted from the nasal cavity of the user specified by such feature point specifying means will be described. FIG. 12 is a diagram showing the types of feature point extraction processing in the interface device according to another embodiment of the present invention.

  As shown in FIG. 12, the feature points of the sound emitted from the user's nasal cavity include “feature points of fundamental frequency”, “feature points related to unvoiced / voiced sounds”, “feature points of volume”, “ For example, “feature points related to sound sources” and “feature points related to utterance contents”.

  The “feature point of the fundamental frequency” is specified by extracting the fundamental frequency of the sound emitted from the user's nasal cavity.

  Further, the “feature point related to unvoiced / voiced sound” is specified by discriminating unvoiced / voiced sound.

  The “volume feature point” is specified by measuring a parameter corresponding to the volume, such as power representing the volume of the sound.

  Further, the “feature point related to each sound source” is specified by identifying various types of sound sources such as whistle and tongue beat in addition to voice and breath sounds.

  Further, the “feature point related to the utterance content” is specified by performing speech recognition or the like and recognizing the utterance content.

As described above, according to the present embodiment, it is possible to identify the feature point of the sound emitted from the user's nasal cavity even in a noisy environment, and to execute the process according to the identified matter on the computer side. become.

  Next, elemental technologies in the processing of the interface device 100 will be described.

In the interface device 100, even in an environment where ambient noise exists, a three-dimensional user utterance position and user voice with separated noise are required. 5 of the interface apparatus 100 which is a three-dimensional voice pointing device necessary for extracting these pieces of information.
One process, 1. Estimation of user utterance position (estimation of short range sound source), 2. 2. Direction of arrival estimation of ambient noise (estimation of sound wave arrival direction of a sound source at a long distance), 3. User utterance detection; Sound source separation,
5). The voice recognition process (Japanese Patent Application No. 2003-320183) will be described below.
1． Estimation of user utterance position (estimation of short range sound source)
A method for estimating the position of a sound source at a short distance within about 1 m from the microphone array using the microphone array will be described below.

The plurality of microphones can be arranged at arbitrary positions in the three-dimensional space. Arbitrary position in 3D space

An acoustic signal output from a sound source placed in

The sound is received by Q microphones arranged in the. The distance Rq between the sound source and each microphone can be obtained by the following equation.

The propagation time τq from the sound source to each microphone can be obtained by the following equation, where the speed of sound is v.

The gain gq of the narrow band signal having the center frequency ω received by each microphone relative to that of the sound source is generally defined as a function of the distance Rq between the sound source and the microphone and the center frequency ω.

For example, a function such as the following expression obtained experimentally is used with the gain as a function of only the distance Rq.

The transfer characteristics between the sound source and each microphone for the narrowband signal with the center frequency ω are:

It is expressed. Then, the position vector a (ω, P0) representing the sound source at the position P0 is defined as a complex vector having a transfer characteristic between the sound source and each microphone as an element with respect to the narrowband signal, as in the following equation.

The sound source position is estimated by the MUSIC method (the signal subspace and the noise subspace are obtained by eigenvalue decomposition of the correlation matrix, and the reciprocal of the inner product of an arbitrary sound source position vector and the noise subspace is obtained. The following procedure is performed using the method for checking the position. Short-time Fourier transform of qth microphone input

The observation vector is defined as follows using this as an element.

Here, n is an index of frame time. A correlation matrix is obtained from the continuous N observation vectors by the following equation.

The eigenvalues arranged in descending order of this correlation matrix

And the corresponding eigenvectors

And Then, the number S of sound sources is estimated by the following equation.

Alternatively, a threshold value for the eigenvalue may be provided, and the number of eigenvalues exceeding the threshold value may be set as the number S of sound sources.
Define the matrix Rn (ω) from the noise subspace basis vectors as

frequency band

And a search area U for sound source position estimation

As

Calculate Then, a coordinate vector in which the function F (P) has a maximum value is obtained. Here, it is assumed that P1, P2,..., Ps are estimated as coordinate vectors giving S local maximum values. Next, the power of the sound source at each coordinate vector is obtained by the following equation.

Then, two threshold values Fthr and Pthr are prepared, and when F (Ps) and P (Ps) in each position vector satisfy the following conditions,

It is determined that there is a utterance in the coordinate vector Pl within N consecutive frame times.
In the sound source position estimation process, consecutive N frames are processed as one block. In order to more stably estimate the sound source position, the number N of frames is increased, and / or it is determined that there is a utterance when the condition of Expression (30) is satisfied in all of the consecutive Nb blocks. The number of blocks is set arbitrarily. When the sound source is moving at such a speed that the sound source can be seen to be approximately stationary within the time period of consecutive N frames, the moving miracle of the sound source can be captured by the above method.
2． Direction of arrival estimation of ambient noise (estimation of sound wave arrival direction of sound source at a long distance)
A method for estimating the direction in which sound waves of a sound source at a long distance from the microphone array arrive will be described below.
The plurality of microphones can be arranged at arbitrary positions in the three-dimensional space. Sound waves coming from a long distance are considered to be observed as plane waves.

  FIG. 13 is an explanatory diagram for explaining a sound receiving function using the microphone array of the present invention. FIG. 13 shows, as an example, a case in which three microphones m1, m2, and m3 arranged at arbitrary positions receive sound waves that have arrived from a sound source. In FIG. 13, a point c indicates a reference point, and the arrival direction of the sound wave is estimated around this reference point. In FIG. 13, the plane s indicates a cross section of a plane wave including the reference point c. The normal vector n of the plane s is defined as the following equation, with the direction of the vector opposite to the propagation direction of the sound wave.

The sound wave arrival direction of the sound source in the three-dimensional space is represented by two parameters (θ, φ). Direction (θ
, Φ) is received by each microphone, and the received signal is decomposed into narrowband signals by obtaining the Fourier transform, and the gain and phase are expressed as complex numbers for each narrowband signal of each received signal. Then, a vector in which all the received sound signals are arranged for each narrowband signal is defined as a sound source position vector. In the following processing, the sound wave coming from the direction (θ, φ) is expressed as the aforementioned position vector. Specifically, the position vector is obtained as follows. A distance rq between the q-th microphone and the plane s is obtained by the following equation.

The distance rq is positive when the microphone is located on the sound source side with respect to the plane s, and is negative when the microphone is on the opposite side of the sound source. If the speed of sound is v, the propagation time Tq between the microphone and the plane s is expressed by the following equation.

The gain related to the amplitude at a distance rq away from the amplitude in the plane s is defined as a function of the center frequency ω of the narrowband signal and the distance rq as follows.

A phase difference at a position away from the phase r with respect to the phase on the plane s is expressed by the following equation.

From the above, with the plane s as a reference, the gain and phase difference of the narrowband signal observed by each microphone are expressed by the following equations.

When observing a sound wave coming from the (θ, φ) direction with Q microphones, the position vector of the sound source is defined as the following expression as a vector whose elements are values obtained according to Expression (26) for each microphone. The

When the position vector of the sound source is defined, the direction of arrival of the sound wave is estimated using the MUSIC method. Using the matrix Rn (ω) given by equation (15), the search region I for sound wave arrival direction estimation is

As

Calculate Then, the direction (θ, φ) in which the function J (θ, φ) gives the maximum value is obtained. Here, it is assumed that there are K sound sources, and K sound wave arrival directions ((θ1, φ1),..., (ΘK, φK)) that give maximum values are estimated. Next, the power of the sound source in each sound wave arrival direction is obtained by the following equation.

Then, two threshold values Jthr and Qthr are prepared, and when J (θk, φk) and Q (θk, φk) in each arrival direction satisfy the following conditions,

It is determined that there is utterance in the direction of arrival (θk, φk) within N consecutive frame times. In the process of estimating the direction of arrival of sound waves, N consecutive frames are processed as one block. In order to estimate the direction of arrival more stably, the number of frames N is increased, and / or if the condition of equation (31) is satisfied in all the consecutive Nb blocks, the sound wave has arrived from that direction. to decide. The number of blocks is set arbitrarily. When the sound source is moving at such a speed that the sound source can be seen to be approximately stationary within the time period of consecutive N frames, the moving miracle in the direction of arrival of the sound wave can be captured by the above method. .

The short-range sound source position estimation result and the long-distance sound source direction-of-arrival direction estimation result play an important role in the subsequent speech detection process and sound source separation process. When the power of the short-distance sound source is remarkably increased with respect to the sound wave coming from the long-distance sound source, the arrival direction estimation of the sound wave of the long-distance sound source may not be performed well. Such a case is dealt with by using the arrival direction estimation result of the sound wave of the long-distance sound source estimated immediately before the short-distance sound source is generated.
3． User utterance detection When there are multiple sound sources, it is generally difficult to identify which sound source should be recognized. On the other hand, in a system that employs an interface using voice, a user utterance region that represents a position at which a user of the system utters relative to the system can be determined in advance. In this case, even if there are a plurality of sound sources around the system by the above-described method, if the position of each sound source and the arrival direction of the sound waves can be estimated, the sound source that enters the user utterance region that the system assumes in advance is selected. Thus, the user's voice can be easily identified.

  The presence of a sound source is detected when the conditions of Expression (20) and Expression (31) are satisfied, and further, the conditions of the position of the sound source and the arrival direction of the sound wave are satisfied, and the user's utterance is detected. This detection result plays an important role in the subsequent speech recognition process as the speech section information. When performing speech recognition, it is necessary to detect the start time and end time of an utterance section from an input signal. However, it is not always easy to detect an utterance section in a noise environment in which ambient noise exists. Generally, when the start time of the utterance section is shifted, the speech recognition accuracy is significantly deteriorated. On the other hand, even if there are a plurality of sound sources, the functions represented by Expression (18) and Expression (29) show a sharp peak at the position where the sound source is and the arrival direction of the sound waves. Therefore, the speech recognition apparatus of the present invention that performs speech segment detection using this information has the advantage that robust speech detection can be performed even when a plurality of ambient noises exist, and high speech recognition accuracy can be maintained. Have.

For example, a user's utterance area as shown in FIG. 14 can be defined. FIG. 14 is a functional explanatory diagram of the speech detection processing according to the present invention. In this figure, for the sake of simplicity, only the XY plane is shown, but in general, any user utterance region can be similarly defined in a three-dimensional space. In FIG. 14, a process using eight microphones m1 to m8 arranged at arbitrary positions is assumed, and a user utterance region is defined in each of a short-distance sound source search region and a long-distance sound source search region. Yes. The short-distance sound source search space is a rectangular area whose diagonal is a straight line connecting two points (PxL, PyL) and (PxH, PyH), and within that area (PTxL1, PTyL1).
) And (PTxH1, PTyH1), (PTxL2, PTyL2), and (PTxH2, PTyH2), two rectangular areas having diagonal lines connecting the two points are defined as user utterance areas. Accordingly, by selecting the position of the sound source determined to have been uttered by the expression (20) whose coordinate vector is within the user utterance area, the user can select among the sound sources existing at a short distance. Can identify voice.

On the other hand, the search space of the long-distance sound source defines the direction from the angle θL to θH with the point C as a reference, and defines the region from the angles θTL1 to θTH1 as the user's utterance region. Therefore, by selecting the arrival directions of the sound waves determined to have been uttered according to the equation (31) within the user utterance area, the sound sources existing at a long distance can be selected. User voice can be specified.
4). A sound source separation process for enhancing the user's voice and suppressing ambient noise using the sound source position estimation result or the sound wave arrival direction estimation result detected by the sound source separation utterance will be described below. The utterance position or the arrival direction of the user voice is obtained by the utterance detection process. Further, the sound source position or direction of arrival of ambient noise has already been estimated. Using these estimation results, the sound source position vectors of Equations (8) and (27), and σ representing the variance of omnidirectional noise, the matrix V (ω) is defined as follows.

The eigenvalues arranged in descending order of this correlation matrix

And the corresponding eigenvectors

And
Here, since the correlation matrix V (ω) includes (S + K) sound sources including S short-distance sound sources and K long-distance sound sources, the eigenvalues and eigenvectors of (S + K) in descending order of eigenvalues. Is used to define Z (ω) as follows:

A separation filter W (ω) that enhances the voice of the user in the short distance coordinate vector P is given by the following equation.

The voice v (ω) of the user in the coordinate vector P is obtained by multiplying the separation filter of Equation (36) by the observation vector of Equation (10).

The emphasized user speech waveform signal is obtained by calculating the inverse Fourier transform of equation (37).

  On the other hand, the separation filter M (ω) for emphasizing the voice of the user in the long distance direction (θ, φ) is given by the following equation.

By multiplying the separation filter of Expression (38) by the observation vector of Expression (10), the emphasized voice v (ω) of the user in the direction (θ, φ) is obtained.

The emphasized user speech waveform signal is obtained by calculating the inverse Fourier transform of equation (37). When the sound source is moving at such a speed that the sound source can be seen as approximately stationary within the time period of consecutive N frames, the emphasized speech of the moving user can be obtained by the above method.
5. Speech recognition processing The sound source separation processing is effective for directional noise, but noise remains to some extent for omnidirectional noise. In addition, a noise suppression effect cannot be expected even for noise that occurs in a short time such as sudden noise. Therefore, the feature correction method described in, for example, Japanese Patent Application No. 2003-320183 “Background Noise Distortion Correction Processing Method and Speech Recognition System Using the Same” is used for the recognition of the user voice emphasized by the sound source separation process. By using a built-in speech recognition engine, the effects of residual noise are reduced. Note that the present invention is not limited to Japanese Patent Application No. 2003-320183 as a speech recognition engine, and it is also possible to use a speech recognition engine in which various methods that are robust against noise are mounted.

  The feature correction method described in Japanese Patent Application No. 2003-320183 performs feature correction of noise superimposed speech based on a Hidden Markov Model (HMM) that a speech recognition engine has as a template model for speech recognition in advance. . The HMM is learned based on Mel-Frequency Cepstrum Coefficient (MFCC) obtained from clean speech with no noise. For this reason, it is not necessary to prepare a new parameter for feature correction, and there is an advantage that the feature correction method can be incorporated into an existing recognition engine relatively easily. In this method, noise is divided into a stationary component and a non-stationary component that shows a temporary change, and the stationary component of the noise is estimated from several frames immediately before the utterance.

A copy of the distribution of the HMM is generated, and the estimated noise stationary component is added to generate a feature amount distribution of the stationary noise superimposed speech. The distortion due to the stationary noise component is absorbed by evaluating the posterior probability of the observed characteristic amount of the noise superimposed speech with the feature amount distribution of the stationary noise superimposed speech. However, since distortion due to the unsteady component of noise is not taken into account only by this processing, the posterior probability obtained by the above means is not accurate when the unsteady component of noise exists. On the other hand, by using the HMM for feature correction, the temporal structure of the feature amount time series and the accumulated output probability obtained along with it can be used. By assigning the weight calculated from the accumulated output probability to the above-mentioned posterior probability, the reliability of the posterior probability deteriorated due to the non-stationary component that temporarily changes the noise can be improved.

  When performing speech recognition, it is necessary to detect the start time and end time of an utterance section from an input signal. However, it is not always easy to detect an utterance section in a noise environment in which ambient noise exists. In particular, since the speech recognition engine incorporating the feature correction estimates a steady feature of ambient noise from several frames immediately before the start of speech, the recognition accuracy is significantly deteriorated when the start time of the speech section is shifted. On the other hand, even if there are a plurality of sound sources, the functions represented by Expression (18) and Expression (29) show a sharp peak at the position where the sound source is and the arrival direction of the sound waves. Therefore, the speech recognition apparatus of the present invention that performs speech segment detection using this information can robustly perform speech segment detection even when a plurality of ambient noises exist, and can maintain high speech recognition accuracy.

  Next, an experiment for examining robustness against noise of the interface device 100 will be described.

  The experiment was conducted in a soundproof room. User voice and ambient noise are recorded separately, and the voice and noise are mixed so that the desired SNR is obtained on the computer. The sound source position estimated from the clean user voice before mixing the noise is taken as the correct answer. Similarly, the position of the sound source is estimated from the noise-mixed speech and compared with the result estimated from the clean user speech to measure how much noise affects the estimation of the sound source localization.

  In the experiment, the interface device 100 having the structure shown in FIG. 1 was used. Each microphone array unit is composed of a total of 12 microphones with 4 microphones arranged at intervals of 3 cm. The sound source position detection space (user utterance region) is performed on 10 × 10 × 10 grid-like discrete points, each side of which is 2 cm, and the X, Y, and Z axes are discretized into 10 points.

  One speaker is placed around the interface device 100. From the speaker, male and female voices are played as noise. The speakers are placed 120 cm away from the center of the interface device 100, and are sequentially installed at four locations, front, rear, left, and right, to record the noise flowing at each location. The SNR was changed from 20 dB to -10 dB in 5 dB steps to generate noise mixed speech.

  The experimental results are shown in FIGS.

  The result of FIG. 15 is that when the sound source position is not detected from the clean user voice but is detected from the noise mixed voice (false detection), the sound source position is detected from the clean user voice, but the noise mixed voice is detected. Represents the rate at which no detection (no detection) occurred. Such false detection or detection omission tends to increase as the SNR decreases. However, even under an environment where noise is dominant as low as -10 dB, misdetection or omission is about 3% or less.

  The result of FIG. 16 is that when the sound source position is estimated from both the clean user sound and the noise mixed sound, the sound source position estimated from the clean user sound and the noise mixed sound in the three-dimensional space. It represents the average error between the positions. The average error was about 2 cm to 2.5 cm. In this experiment, since the sound source position is estimated on a grid point with a side of 2 cm, this average error is an error that the position estimated on the grid point is estimated at a grid point adjacent to the correct position. .

  From the above results, it can be said that the interface device 100 of the present invention is not easily affected by ambient noise.

  Next, another experiment for examining robustness against noise of the interface apparatus 100 will be described. The actual experiment was conducted in a soundproof room. User voice and ambient noise were recorded separately and mixed on the computer to achieve the desired SNR. The sound source position estimated from the clean user voice before mixing the noise is taken as the correct answer. Similarly, by estimating the sound source position from the noise mixed user voice and comparing it with the result estimated from the clean user voice, it is measured how much noise affects the sound source localization estimation.

  In the experiment, the interface device 100 having the structure shown in FIG. 1 was used. Each microphone array unit is composed of a total of 12 microphones with 4 microphones arranged at intervals of 3 cm. The sound source position detection space (user utterance region) was performed on 10 × 10 × 10 grid-like discrete points, each side of which was 2 cm and the X, Y, and Z axes were discretized into 10 points.

One speaker is placed around the interface device 100. From the speaker, male and female voices are played as noise. Speakers are installed sequentially at four locations, front, back, left, and right, 120cm away from the center of the pointing device, and noise is recorded at each location and recorded with a microphone array. The SNR was set in steps of 5 dB from 20 dB to -10 dB, and noise mixed user speech of each SNR was generated.

The experimental results are shown in FIGS.

The result of FIG.
A. Sound source was not detected from clean user voice but detected from noise mixed user voice (false detection). Sound source was detected from clean user voice but not from noise mixed user voice. If (detection omission)
B. When the sound source positions estimated from clean user speech and noise-mixed user speech match (correct answer)
C. When the sound source positions estimated from both voices do not match (error)
The three ratios are shown.

  Although the false detection / missing detection of A tends to increase as the SNR becomes lower, the ratio was about 3% or less even in a noise-dominant environment of about -10 dB. The correct answer for B gradually decreases as the SNR decreases, and the error for C increases gradually. FIG. 18 shows the results of measuring the distance between the sound source position estimated from the noise-mixed user voice and the sound source position estimated from the corresponding clean / speech sound when the error is C, and obtaining the average and standard deviation. . The error is distributed with an average of about 2 cm to 2.5 cm and a standard deviation of about 1 cm at the maximum. In this experiment, since the sound source position is estimated on a grid point with a side of 2 cm, even if the sound source position is estimated incorrectly, the error is such that it is estimated to be shifted to a grid point substantially adjacent to the correct position. is there.

  From the above experimental results, it can be said that the sound source position estimation by the three-dimensional audio pointing device of the present invention is hardly influenced by the ambient noise.

  As described above, according to the interface device and the interface method of the present invention, items such as a user's breathing sound and the utterance position of the utterance are specified three-dimensionally even in a noisy environment, and processing corresponding to the specified items is performed by the computer. Will be able to run on the side.

1 is a perspective view showing an external appearance of an interface device according to an embodiment of the present invention. It is a figure which shows the block configuration of the interface apparatus which concerns on embodiment of this invention. It is a figure which shows the example of a utilization form of the interface apparatus which concerns on embodiment of this invention. It is a figure which shows the flowchart of a process of the interface apparatus which concerns on embodiment of this invention. It is a figure which shows the other area | region division example of the interface apparatus which concerns on embodiment of this invention. It is a figure which shows the usage example of the interface apparatus which concerns on other embodiment of this invention. It is a figure which shows the usage example of the interface apparatus which concerns on other embodiment of this invention. It is a figure which shows the usage example of the interface apparatus which concerns on other embodiment of this invention. It is a figure which shows the usage example of the interface apparatus which concerns on other embodiment of this invention. It is a figure which shows the azimuth ((theta))-elevation-angle ((phi)) plane assumed with the interface apparatus which concerns on other embodiment of this invention. It is a figure which shows the flowchart of a process of the interface apparatus which concerns on other embodiment of this invention. It is a figure which shows the kind of feature point extraction process in the interface apparatus which concerns on other embodiment of this invention. It is explanatory drawing explaining the sound reception function using the microphone array of this invention. It is function explanatory drawing of the speech detection process by this invention. It is a figure which shows the experimental result for investigating the robustness with respect to the noise of this invention. It is a figure which shows the experimental result for investigating the robustness with respect to the noise of this invention. It is a figure which shows the experimental result for investigating the robustness with respect to the noise of this invention. It is a figure which shows the experimental result for investigating the robustness with respect to the noise of this invention.

Explanation of symbols

DESCRIPTION OF SYMBOLS 100 ... Interface apparatus, 200 ... Microphone array, 201 ... Silicon microphone, 202 ... Wind screen, 210 ... Stand, 211 ... Main support, 212 ... Left support, 213 ..Right column, 280 ... Microphone amplifier, 290 ... AD converter, 300 ... CPU, 400 ... Storage, 500 ... Connection port

Claims (12)

The interface device has a microphone array and a CPU that operates based on a program that takes in the output of the microphone array and stores it in the storage unit,
The microphone array includes a main support column that stands vertically on a stand, a left support column and a right support column that branch from the main support column to the left and right, and a microphone group that is provided on each support column. The arranged microphone group and the microphone group arranged on the right column are arranged so as to have a substantially C-shaped layout,
A user's utterance detection area is defined based on the microphone array, and the area is divided into an arbitrary number of divided areas, and only the user's utterance is detected in the utterance detection area, and from outside the utterance detection area. Treat the sound as noise,
Step S101 for capturing voice data from the microphone array, Step S102 for specifying three-dimensional information of the user utterance position and the direction of arrival of ambient noise,
If the user's utterance is detected, if the user's utterance is not detected, the process repeats from step S101, and if the user's utterance is detected, the process proceeds to step S104, and sound source separation that suppresses ambient noise and emphasizes the user's utterance In step S104, the user's utterance detection area is defined in advance, the utterance detection area is divided into an arbitrary number of areas, and the divided areas are defined, and the utterance position estimated in the three-dimensional space is Step S105 for identifying in which divided area the utterance is detected in the divided areas, and a combination of the utterance detection position and the event is defined in advance and stored in the database, and the utterance position is within the divided area. Step S106a for determining whether or not the voice is in a specific position, and determining whether or not the duration of utterance is equal to or less than a predetermined threshold value And step S106b that, utterance sequentially executes step S106c that determines whether the unvoiced,
In step S106a, the utterance position is determined as a specific position in the divided region, in step S106b, the utterance duration is determined to be equal to or less than a predetermined threshold, and in step S106c, the utterance is determined to be an unvoiced sound. Step S106 in which the identification of the corresponding event is executed when
In step S106, it is checked whether an event that matches the event database is detected. If no event is detected, the process returns to step S101. If an event is detected, the process proceeds to step S108.
It functions to execute step S108 for transmitting an event detection signal to the application level on the computer side,
Set the search range of sound source position estimation as the user's utterance detection area,
In step S102,
The MUSIC method is used to estimate the sound source position.
When estimating the user utterance position,
A coordinate vector in which the following function F (P) takes a maximum value is obtained,
Obtain the power P (P _s ) of the sound source at the obtained coordinate vector,
When the function F (P) and the power P (P _s ) of the sound source are each equal to or greater than a predetermined threshold, it is determined that there is an utterance at the coordinate vector Pl.
Set the search range of sound wave arrival direction estimation as the arrival wave is a plane wave,
Use the MUSIC method to estimate the direction of arrival of sound waves,
When estimating the direction of arrival of ambient noise,
The direction (θ, φ) in which the following function J (θ, φ) takes a maximum value is obtained,
Obtain the power Q (θ _k , φ _k ) of the sound source in the obtained sound wave arrival direction,
When the function J (θ, φ) and the power Q of the sound source (θ _k , φ _k ) are each equal to or greater than a predetermined threshold, it is determined that there is a utterance in the coordinate vector,
It is determined that there is a utterance in the sound wave arrival direction (θ _k , φ _k ),
In step S104,
Using the coordinate vector Pl, the sound wave arrival direction (θ _k , φ _k ), the sound source position vector, and the variance σ of omnidirectional noise, a correlation matrix V (ω) is defined as follows:
The using eigenvalues and eigenvectors e _k of the correlation matrix V (omega), define Z a (omega) according to the following equation,
A separation filter W (ω) that enhances the voice of the user at the short distance coordinate vector P is given by

The user's voice v (ω) in the coordinate vector P is obtained by multiplying the above separation filter by the observation vector,
And an utterance position specifying means for three-dimensionally specifying an utterance position of a sound emitted from the user's nasal cavity based on the user's voice v (ω).
Where P is an arbitrary position in the three-dimensional space, ω is the frequency of the narrowband signal,
a (ω, P) is a position vector representing a sound source at position P, and is defined as a complex vector having a transfer characteristic between the sound source and each microphone regarding the narrowband signal as an element,
Rn (ω) is the number S of sound sources obtained from the eigenvalues λ of the correlation matrix R (ω) obtained by collecting N observed vectors y (ω, n) consisting of short-time Fourier transform values of the respective microphone inputs, and the eigenvalue λ. Based on the eigenvector dk corresponding to
What we asked for,
b (ω, θ, φ) is a position vector of the sound source when observing a sound wave coming from the (θ, φ) direction at the frequency ω.

The interface apparatus according to claim 1, further comprising: an utterance position detection signal generating unit that generates a signal based on the utterance position specified by the utterance position specifying unit. The utterance position specifying means three-dimensionally specifies a vector connecting the utterance start position and the utterance end position of a continuous sound emitted from the user's nasal cavity based on the voice data acquired by the microphone array. 2. The interface device according to claim 1, wherein the vector specifying means is used. Trajectory specifying means for three-dimensionally specifying the trajectory from the utterance start position to the utterance end position of a continuous sound uttered from the user's nasal cavity based on the voice data acquired by the microphone array. The interface device according to claim 1, wherein: The utterance position specifying means is a utterance direction specifying means for specifying an azimuth angle and an elevation angle in a direction in which a sound emitted from a user's nasal cavity arrives based on sound data acquired by the microphone array. The interface device according to claim 1. The interface apparatus according to claim 5, further comprising a voice direction detection signal generation unit that generates a signal based on the azimuth angle and the elevation angle specified by the voice direction specification unit. The utterance position specifying means includes an azimuth angle and an elevation angle in the utterance start direction and an azimuth angle and an elevation angle in the utterance end direction of a continuous sound uttered from the user's nasal cavity based on the audio data acquired by the microphone array. 2. The interface apparatus according to claim 1, wherein direction vector specifying means for specifying a vector connecting the two is used. From the azimuth angle and elevation angle of the utterance start direction to the azimuth angle and elevation angle of the utterance end direction of the continuous sound uttered from the user's nasal cavity based on the voice data acquired by the microphone array, 2. The interface device according to claim 1, wherein a direction locus specifying means for specifying a locus is used. The utterance position specifying means is a time specifying means for specifying the time from the start of utterance to the end of utterance of a continuous sound uttered from the user's nasal cavity based on the sound data acquired by the microphone array. The interface device according to claim 1. The utterance position specifying means is a moving speed specifying means for specifying a moving speed of a continuous sound emitted from a user's nasal cavity based on sound data acquired by the microphone array. Interface equipment. A feature point specifying unit that specifies a feature point of a sound emitted from the user's nasal cavity; and a feature point detection signal generating unit that generates a signal based on the feature point specified by the feature point specifying unit. The interface device according to claim 1, wherein The feature points of the sound emitted from the user's nasal cavity extracted by the feature point specifying means are the feature points of the fundamental frequency, the feature points related to unvoiced / voiced sounds, the feature points of the volume, the feature points related to the sound source, The interface device according to claim 11, wherein the interface device is one of feature points related to utterance contents.