JP7043081B2 - Voice recall recognition device, wearer, voice recall recognition method and program - Google Patents

Description

The present invention relates to a voice recall recognition device, a wearer, a voice recall recognition method, and a program.

As a voice language input device, a device that receives voice waves uttered so far by a microphone or a vibration pickup of bone conduction vibration and recognizes voice language information from the obtained signal is put into practical use.
In recent years, by using a huge amount of speech data and language data, and accumulating and using stochastic information about phoneme sequences (acoustic model) and word sequences (language model) on the network, high-speed and high-performance speech languages have been used. Realizes recognition. On the other hand, due to the increase in utterance-induced inconvenience / leakage to the surroundings and the increase in patients with muscle atrophy lateral sclerosis (ALS) who have difficulty speaking, the realization of language recognition by speech imagery without speech is a brain computer interface. (Brain Computer Interface; BCI) is desired from the field.

For speech language recognition from speech recall signals, speech language recognition with speech is being attempted in recent years by observing 64 to 128 points of subdural corticogram (ECoG) from the brain cortex (ECoG). See Non-Patent Document 1). However, it is not realistic to use such a method involving craniotomy except for critically ill patients. On the other hand, although the method of observing electroencephalograms (EEGs) with electrodes on the scalp has immeasurable social contribution when put into practical use, attempts to find meaningful vocal language signals in noise have been successful to date. I didn't.

In recent years, research has progressed to analyze the brain during speech using high-resolution devices such as PET and fMRI, and to observe EcoG when a patient speaks at the time of craniotomy, and speech language is processed in any part of the brain. Is becoming clearer. According to these results, after the concept preparation in the left middle temporal gyrus (MTG), the planning as a language is performed toward the superior temporal gyrus (STG) (see Non-Patent Document 2). After this, syllableation is performed in the lower left inferior frontal gyrus (IFG; Broca's area), and syllables (articulation) are performed in the left precentral gyrus (PG; motor cortex) during speech (see Non-Patent Document 3). ). From these research results, it is expected that decoding for speech languages without speech will be possible if the linguistic representation that reaches Broca's area can be captured.
Further, a technique for detecting an electroencephalogram and detecting a signal related to a motor command from the electroencephalogram has been proposed (see Patent Document 1).

Heger D. et al., Continuous Speech Recognition from ECoG, Interspeech2015, 1131-1135 (2015) Indefrey, P et al., The spatial and temporal signatures of word production components, Cognition 92, 101-144 (2004) Bouchard K.E. et al., Functional organization of human sensorimotor cortex for speech articulation, Nature 495, 327-332 (2013) Girolami M., Advances in Independent Component Analysis, Springer (2000) Durbin, J. "The fitting of time series models." Rev. Inst. Int. Stat., V. 28, pp. 233-243 (1960)

Japanese Unexamined Patent Publication No. 2008-204135

However, in speech language recognition from brain waves, it is unclear in what format the language representation is expressed, and the biggest problem is that a specific extraction method cannot be found. Furthermore, if a method for converting language representations to phoneme units is not given, many types must be targeted, for example, syllable units (syllables have many long syllables in addition to short syllables and count. Efficient speech language processing becomes very difficult (it is said to be a thousand) (24 for phonemes and 44 for English (however, weak vowels and strong vowels are separated. Normally in Japanese, it is usually). (Do not divide).

The present invention has been made in view of the above circumstances, and an object of the present invention is to provide a voice recall recognition device, a wearer, a voice recall recognition method, and a program capable of voice language recognition by electroencephalogram.

In order to achieve the above object, in order to recognize speech language from brain waves at the time of speech recall, the present invention extracts line spectrum components by a line spectrum component extractor as a language representation and convolves those components by phoneme. The most important feature is to obtain a phoneme feature vector time series by passing it through a phoneme feature vector time series converter using arithmetic.

The first invention is a voice recall recognition device that recognizes a voice language from a brain wave at the time of voice recall, analyzes and processes a discrete signal group of the brain wave for each electrode input from the electrode group, and outputs a spectral time series. Provided is a speech recall recognition device having an analysis processing unit for performing an electroencephalogram and an extraction unit for outputting an electroencephalogram feature vector time series based on the spectral time series.

The second invention is a wearer for a voice recall recognition device that recognizes a speech language from brain waves at the time of speech recall, and has an electrode group arranged around the broker field and an output for outputting a signal from the electrode group. The voice recall recognition device has an analysis process of analyzing and processing a discrete signal group of brain waves for each electrode output from the output unit and outputting a spectral time series, and the spectral time series. Based on this, a fitting is provided that executes an extraction process that outputs a phonetic feature vector time series.

The third invention is a voice recall recognition method for recognizing a speech language from a brain wave at the time of speech recall, in which a discrete signal group of the brain wave for each electrode input from the electrode group is analyzed and processed to output a spectral time series. A speech recall recognition method including an analysis processing step to be performed and an extraction step to output a phonetic feature vector time series based on the spectral time series is provided.

The fourth invention is a program for causing a computer to execute a voice recall recognition process for recognizing a speech language from a brain wave at the time of speech recall, and is a discrete brain wave for each electrode input to the computer from a group of electrodes. A program is provided for executing an analysis process of analyzing a signal group and outputting a spectral component as a linguistic representation, and an extraction process of extracting a phonetic feature group based on the spectral component of each electrode.

INDUSTRIAL APPLICABILITY According to the present invention, it is possible to provide a speech recall recognition device, a wearer, a speech recall recognition method and a program that enable speech language recognition by brain waves.

It is a model diagram which showed the structure of the recognition device of this invention. It is a figure which shows the electroencephalogram measurement electrode (10-10 system) and 9 electrodes around Broca's area. It is a figure which shows the noise reduction effect from an electroencephalogram. It is explanatory drawing of the linear prediction analysis of the EEG at the time of voice recall. It is a figure which shows the comparison between the linear predictive analysis of the EEG at the time of speech recall and the conventional Fourier analysis. It is a figure which shows the short-time sine wave group of the electroencephalogram at the time of voice recall. It is a flow chart which shows the processing procedure of a language feature extraction part. It is a figure which showed the frequency fluctuation absorption example of the electroencephalogram at the time of voice recall. It is a figure which shows the example of the line spectrum time series of the electroencephalogram at the time of voice recall. It is a figure which showed the example of the line spectrum time series which spans a plurality of electrodes. It is a flow diagram which shows the processing procedure of the design and use of the convolution operator for each phoneme. It is a figure which shows the example of the phoneme eigenvector which constitutes the phoneme-specific convolution operator. It is a figure which showed the example of the phoneme likelihood time series with respect to the electroencephalogram at the time of speech recall. It is a figure which shows the electrode position correction by a test recognition. It is a figure which shows the other configuration example of the voice recall recognition apparatus. It is a figure which shows the other configuration example of the voice recall recognition apparatus. It is a figure which shows the other configuration example of the voice recall recognition apparatus.

(Embodiment)
Hereinafter, embodiments of the voice recall recognition device according to the present invention will be described with reference to the accompanying drawings. It should be noted that the attached drawings are used to explain the technical features of the present invention, and the configuration of the apparatus described, the procedure of various processes, etc. are limited to that unless otherwise specified. Not the purpose. The same elements are designated by the same reference numerals throughout the description of the embodiments.

FIG. 1 is a model diagram showing the configuration of the voice recall recognition device 1. The configuration and operation of the voice recall recognition device 1 will be described with reference to FIG.
The voice recall recognition device 1 is for recognizing a voice language from an electroencephalogram at the time of voice recall.
The voice recall recognition device 1 includes an electroencephalogram input unit 2 that converts brain waves input from a group of electrodes installed on the scalp (not shown) into a discrete signal group, a preprocessing unit 3 that removes noise from the discrete signal for each electrode, and a preprocessing unit 3. An analysis processing unit 4 that analyzes and processes a discrete signal group for each electrode and outputs a spectral time series, a language feature extraction unit 5 that outputs a phonetic feature vector time series from the spectral time series of all electrodes, and a phonetic feature vector time series. It is composed of a word / sentence recognition unit 6 that recognizes a word / sentence that is a voice language, and a post-processing / output unit 7 that displays / outputs voice language information.

The brain wave input unit 2 converts the analog signal group x (q, t) of the multi-electrode brain wave output into a discrete signal by A / D conversion or the like, and individually uses the average value of the discrete signals of all the electrodes. A process is performed to correct the bias bias of the electrodes. At the same time, unnecessary frequency components of 70 Hz or less are blocked by a low frequency rejection filter (high frequency pass filter) from the discrete signal of each electrode, and unnecessary frequency components of 180 Hz or higher are blocked by a high frequency rejection filter (low frequency pass filter). The signal x ₁ (q, n) is output.

FIG. 2 shows the electrode arrangement of a standard international 10-10 system using 64 electrodes. Of these, voice recall signals are received from 9 electrodes {F3, F5, F7, FC3, FC5, FT7, C3, C5, T7} belonging to the Broca's area of the left brain, language features are extracted, and the recall content is recognized. It is generally said that right-handed people process language in the left brain, but a considerable number of left-handed people also process language in the left brain. It should be noted that although brain waves may be subject to large fluctuations (called artifacts) due to movements such as blinking, many unnecessary components can be removed by the above filter operation. Furthermore, for unnecessary components that cannot be removed by filter operation, a process of estimating and removing a small number of independent information sources for the discrete signals of all electrodes and then returning to the original electrode output (here, 9 electrodes) is performed. Independent Component Analysis (IPA) may be applied.

The pretreatment unit 3 removes noise passing through the filter for each electrode. An example of this process is described below. The discrete signal x ₁ (q, n), (q: electrode number, n: time) of each electrode that has completed a series of processing of the electroencephalogram input unit is first multiplied by a fixed time window, and then fast Fourier transform is performed. (FFT) maps from the time domain to the frequency domain. Then, the amplitude spectrum time series X ₁ (q, f, n') and (f is the frequency and n'is the time frame number after windowing) are obtained from the complex number components in the frequency domain as follows.

ここで、ｊは虚数単位、Ｒｅ｛ ｝、Ｉｍ{ }は各々実数部、虚数部を表す。ノイズ引き去り(Noise subtraction)では、音声想起(Speech imagery)に先立ち観測した脳波(ＥＥＧ信号)のスペクトルＮ（ｑ，ｆ,ｎ’）から次式で、平均ノイズ振幅スペクトルを求める。

Here, j represents an imaginary unit, and Re {} and Im {} represent a real part and an imaginary part, respectively. In noise subtraction, the average noise amplitude spectrum is obtained from the spectrum N (q, f, n') of the electroencephalogram (EEG signal) observed prior to the speech imagery by the following equation.

In the above equation, the average noise spectrum is calculated from 8 frames before and after the time n', but it may be set appropriately depending on the system. The time n'is usually set.
(A) The user performs voice recall following the prompt signal (signal instructing the start of recall) given from the voice recall recognition application system.
(B) Voice recall is performed following a fixed call from the user to the application system, such as "Yamada-san" (wake-up word).
In either case, N (q, f, n') is calculated from the brain waves observed in the section before or after the speech recall.
Subsequently, Nav (q, f, n') is subtracted from the speech recall signal spectrum X ₁ (q, f, n') for each electrode q as shown in the following equation.

FIG. 3 shows an example in which noise in the brain wave is removed by this processing. FIG. 3A shows before noise reduction, and FIG. 3B shows after noise reduction. Comparing FIGS. 3A and 3B, it can be seen that the effect of removing the noise spectrum is remarkable. The amplitude spectrum time series after noise reduction is returned to the waveform x ₂ (q, n) by the inverse fast Fourier transform (IFFT).

It should be noted that a process of extracting a small number of independent information sources from the 9-electrode signal after noise removal, that is, an independent component analysis (IPA) (Non-Patent Document 4) is effective. By this processing, unnecessary components that cannot be removed by the filter operation can be removed, and a small number of effective information sources can be selected from the discrete signals of 9 electrodes. However, ICA has a problem of so-called permutation in which the order of independent components of the analysis result is different each time of analysis, and a method of solving this drawback and introducing it into the present patent will be described later.

The analysis processing unit 4 may use the spectrum time series X ₂ (q, f, n') of the voice recall signal after noise removal (and after extraction of q independent components) obtained by the preprocessing unit 3. , An example in which Linear Predictive Analysis (LPA) is applied will be described below as an analysis method that brings out the effects of the present invention better. The analysis processing unit 4 can use a spectrum or a line spectrum.
Linear Predictive Coding (LPC) is currently the global standard as a voice communication method. In voice, there are two sources of information: pulse waves with a fixed period due to the vocal cords and random waves due to the narrowing of the vocal tract. Therefore, the sound source is held separately as a codebook, and all the sound sources in the codebook are passed through the linear prediction coefficient of speech (which is responsible for the transmission function of the vocal tract), and the synthetic speech is compared with the original speech. It requires a complicated process to do.

On the other hand, in the case of EEG, as shown in FIG. 4, since the information source is considered to be only a random wave, EEG synthesis is simpler than speech synthesis. Various algorithms for obtaining the linear prediction coefficient {α _m } from the autocorrelation coefficient r ₂ (τ) obtained from the electroencephalogram x ₂ (q, n) have been proposed, such as the Levinson-Durbin method (Non-Patent Document 4). As shown in FIG. 4, the voice recall electroencephalogram x (n) of each electrode is obtained by passing the white noise w (n) of the signal source through the impulse response s (n) of the nervous system. In FIG. 4, ☆ indicates a convolution integral symbol.

ここでδ(n-p)は、信号の各時刻ｎ＝ｐを表わす関数、Ｆ[ ]はフーリエ変換である。脳波に対する線形予測分析（ＬＰＡ）では、図４に示すように合成モデルＳ（ｆ）を逆フィルタとして、

と求めることができる（σは振幅バイアス値）。このように、合成過程を通して分析を精度良く行う方式は、「合成による分析(Analysis-by-Synthesis; ＡｂＳ)」と呼ばれ、脳波分析においても有効な方式である。上式のフーリエ変換Ｆ[ ]では、p個の線形予測係数(α₀=１．０)にゼロ点を付加し(0-paddingと呼ばれる)、例えば１２８点、２５６点、…と任意点数のフーリエ変換を行うことができる。このゼロ点付加によって、周波数分解精度を各々６４点、１２８点、…と任意に調整して、スペクトル成分Ａ(ｑ，ｆ，ｎ’)を求めることができる。 In the convolution integration process, the transmission (frequency) function of the impulse response s (n), which is responsible for speech language information, is S (f) in the frequency domain, and the spectrum of the brain wave is X (f) = W (f) S (f) =. It can be expressed as S (f) (however, W (f) = 1). S (f) can be obtained from the Fourier transform of the linear prediction coefficient {α _m } as shown in the following equation.

Here, δ (np) is a function representing each time n = p of the signal, and F [] is a Fourier transform. In linear predictive analytics (LPA) for EEG, as shown in FIG. 4, the synthetic model S (f) is used as an inverse filter.

(Σ is the amplitude bias value). As described above, the method of performing analysis with high accuracy through the synthesis process is called "Analysis-by-Synthesis (AbS)" and is also an effective method in EEG analysis. In the Fourier transform F [] of the above equation, zero points are added to p linear prediction coefficients (α ₀ = 1.0) (called 0-padding), for example, 128 points, 256 points, ... Fourier transform can be performed. By adding zero points, the frequency resolution accuracy can be arbitrarily adjusted to 64 points, 128 points, ..., Each, and the spectral component A (q, f, n') can be obtained.

FIG. 5 shows the spectral pattern analyzed by LPA in comparison with the spectral pattern analyzed by ordinary Fourier transform. In FIG. 5, a plurality of spectral patterns by LPA are displayed, and these show that a window function called a log window is used to attenuate the value as the delay τ increases with respect to the autocorrelation coefficient (). There is no lag window from the top, the slope of the lag window is large toward the bottom, and it becomes a sharp peak when the log window is not used). In LPA, as shown in the figure, the spectrum can be represented by a small number of essential peaks of the brain wave.

The spectrum of the EEG during speech recall through LPA analysis is represented by a small number of spectral peaks. From this, in the brain (especially Broca's area where the linguistic information of speech recall appears), the linguistic representation is composed of a group of short-time sine waves (tone-burst), in other words, the linguistic representation is peculiar. It is presumed to be represented by the line spectrum of. FIG. 6 shows an example of a tone burst wave group and their spectral shapes. A short-time sine wave is originally represented by a single parameter or single frequency, but as shown in the figure (and as shown in Figure 5), it has a normal frequency by having transients before and after the signal. The analysis has a spread in the spectrum.

The language feature extraction unit 5 extracts line spectrum components as "language representations" from a spread spectrum group, and outputs a phoneme likelihood vector time series, which is a language feature, through a phoneme unit convolution operator.
Hereinafter, the processing process will be described with reference to the processing flow diagram of the language feature extraction unit of FIG. 7. In the language feature output unit 5, the spectral time series of the electrode q is input from the analysis processing unit 4 (step S1). The spectrum of the electroencephalogram at the time of voice recall may have a fluctuation of about ± 5 Hz as shown in FIG. 8 (A). Therefore, these frequency fluctuations are absorbed by using a median filter, which is a kind of non-linear filtering (step S2).

For data within a fixed time width (several frames before and after time n') and frequency width (adjacent frequencies f-1, f, f + 1), an intermediate value in the whole is obtained and represented. Since this process can delete the value deviating from the median value, the frequency fluctuation can be absorbed. The output of the nonlinear filter is generally smoothed by a Gaussian window or the like. FIG. 8B shows the improvement results of frequency fluctuations when intermediate value filtering was performed on a total of 7 frames, 3 frames before and after the central frame n', for an electroencephalogram signal (4 msec period) of 70 Hz to 170 Hz. Indicated. It can be seen from the figure that the fluctuation is reduced. After that, the frequency analysis pattern is smoothed by multiplying it by a Gaussian window (coefficient; {1/4/1/2/1/4}) in the time direction, and the time frame is dropped from 4 msec to around 8 msec. The process of absorbing the frequency fluctuation can also be performed in the preprocessing unit 3 at a stage after removing the noise component on the amplitude spectrum and before returning to the waveform signal.

Next, the process of extracting the line spectrum will be described (step S3). In this process, the component derived from the peak appearing on the frequency axis is extracted as a line spectrum every time frame (8 msec). In particular:
(i) Frequency at which the maximum value Δ _f = 0 on the frequency axis,
(ii) When the inflection point ΔΔ _f = 0, _if Δf> 0, the frequency at which the value of _ΔΔf changes from positive to negative,
If Δ _f <0, the frequency at which the value of ΔΔ _f changes from negative to positive,
Only when these conditions are satisfied, the sinusoidal frequency component having the original amplitude, that is, the line spectrum component is used.

FIG. 9 shows an example of extracting the line spectrum component of the electroencephalogram at the time of recalling the voice. In this example, data is collected under the task of recalling / ga-gi-gu-ge-go / three times in succession as much as possible. By repeating the same sequence three times, the expert can learn the pattern of each syllable as shown in the figure, and can create a database in which the EEG data is labeled with a syllable.
In FIG. 9, the line spectrum time series of 9 electrodes is pooled in the electrode direction (processing for extracting a representative pattern from 9 electrodes. Processing such as taking p-norm (p = ∞ takes the maximum value). The result of syllable labeling for the integrated line spectrum after the processing of)) is shown. The pooling process here is performed only for reading the syllable label, and in the following phoneme feature extraction, the line spectrum component of the original 9 electrodes is targeted.

The language feature extraction unit 5 aims to finally extract phoneme features. That is, we aim to extract the phoneme component, which is the smallest unit of vocal language information, from the line spectrum component of each electrode in the form of a phoneme feature vector. Vocal language information in brain waves has a so-called tensor structure that straddles the three axes of line spectrum (frequency information) -electrodes (spatial information) -frames (time information). FIG. 10 shows an example of a line spectrum time series straddling 3 × 3 = 9 electrodes in Broca's area. This example shows an example of a single syllable / ka /. In this way, the syllable pattern that appears in Broca's area has different electrode positions each time, suggesting a flexible information processing mechanism of the cranial nerve system. On the other hand, in vocal language processing of the brain, syllables appear in Broca's area as the smallest unit of utterance, but during utterance, the speech organs are controlled by muscle movements, and this control is a tone parameter that corresponds one-to-one with phonemes. Will be done. Considering this background, it is considered that there is a process of extracting phoneme features from the syllable pattern of FIG. 10 observed in Broca's area, and the method of realizing this process on a computer is described by the phoneme-specific convolution operator of FIG. It will be described below according to the flow showing the processing procedure of design and utilization.

The flow of FIG. 11 shows the calculation of the phoneme likelihood vector by the phoneme-specific convolution operator in order to efficiently extract phonemes from the frequency-time pattern of 9 electrodes. First, syllables belonging to the same phoneme context (phoneme / s / in / sa /, / shi /, / su /, / se /, / so /, or phoneme / a / in / a /, / ka /, / sa / , / Ta /, / na /, / ha /, ..., / Ga /, / za /, ..., etc.) are stored in the memory (step S11). The method of taking in and out this accumulated information and using it for necessary information processing is called pooling.

Next, the principal component analysis is performed for each syllable (step S12), and the eigenvector for each syllable is determined for each related phoneme, phoneme / s /: {ψ ^{/ sa /} (m), ψ ^{/ shi /} (m), ψ ^{/. su /} (m), ψ ^{/ se /} (m), ψ ^{/ so /} (m)}, phoneme / a /: {ψ ^{/ a /} (m), ψ ^{/ ka /} (m), ψ ^{/ sa /} ( Group phonemes like m), ψ ^{/ ta /} (m), ψ ^{/ na /} (m),….}. Subsequently, the autocorrelation matrix is calculated from the eigenvectors of the same phoneme group and integrated into the phoneme-specific autocorrelation matrices R ^s , ^Ra , ... (Step S13). From the phoneme-specific autocorrelation matrix, the subspaces (eigenvectors) φ ^{/ s /} (m) and φ ^{/ a /} (m) for each phoneme can be obtained. FIG. 12 shows the eigenvectors of phonemes / s / and / a / (displaying the accumulation of the upper three axes).

ここでＭａｘの意味はｑ個（電極もしくはＩＣＡの成分）について最大値を取ることを意味している．また＜ ＞は内積演算を示す。なお，Ｘ（ｑ，ｆ，ｎ’）およびφ（ｆ,ｎ’）は各々予めノルムで正規化されている。
音素ｋ；ｋ＝１, ２,…, Ｋの尤度L(k)をＫ個並べたベクトルを音素特徴ベクトルとする。（７）式は、音素の固有ベクトルφ（ｆ,ｎ’）を利用して音素単位の畳み込み演算子を構成しており、音素ｋ毎に尤度としてのスカラー値Ｌ（ｋ）が得られ、これをＫ個並べたベクトルが、入力Ｘ(ｆ，ｎ’)の時刻ｎ’が推移するに従い(音素尤度ベクトル)時系列データとして言語特徴抽出部５から出力される（ステップＳ５、ステップＳ１６）。
図１３に音素の尤度（Ｌ（ｇ），Ｌ（ｏ），…)から音節の尤度（Ｌ（ｇｏ），Ｌ（ｒｏ）,…)を求めて表示した例を示した。この例は連続数字（“１，２，３，４，５，６，７，８， ９，０”）をこの順で想起した際の音節の尤度を濃淡で示している。縦軸に音節(上からｉ，ｃｈｉ，ｎｉ，ｓａ，Ｎ，ｙｏ，ｏ，ｇｏ，ｒｏ，ｋｕ，ｎａ，ｈａ，ｋｙｕ，ｕ, ｚｅ，ｅ，ｎｏｉｓｅ)を示した。連続数字を構成する音節の尤度が高い値で求められていることが分かる。 Next, by using the eigenvector group obtained for each phoneme k as the "phoneme unit convolution operator", the phoneme similarity (probability) L (probability) to the unknown 9-electrode (or minority after ICA) line spectrum time series (probability) L ( k) can be calculated (step S4, step S14, step S15).

Here, the meaning of Max means to take the maximum value for q pieces (electrode or ICA component). Further, <> indicates an inner product operation. Note that X (q, f, n') and φ (f, n') are each pre-normalized by the norm.
A phoneme feature vector is a vector in which K-likelihoods L (k) of phonemes k; k = 1, 2, ..., K are arranged. In equation (7), a phoneme-based convolution operator is constructed using the phoneme eigenvector φ (f, n'), and a scalar value L (k) as a likelihood is obtained for each phoneme k. A vector in which K of these are arranged is output from the language feature extraction unit 5 as time-series data as the time n'of the input X (f, n') changes (phoneme likelihood vector) (step S5, step S16). ).
FIG. 13 shows an example in which the likelihood of a syllable (L (go), L (ro), ...) Is obtained from the likelihood of a phoneme (L (g), L (o), ...) And displayed. In this example, the likelihood of syllables when consecutive numbers (“1, 2, 3, 4, 5, 6, 7, 8, 9, 0”) are recalled in this order is shown in shades. The vertical axis shows syllables (i, chi, ni, sa, N, yo, o, go, ro, ku, na, ha, kyu, u, ze, e, noise from the top). It can be seen that the likelihood of the syllables constituting the consecutive numbers is obtained with a high value.

Since it is difficult to collect a large amount of voice recall data at this time, an example of solving the problem in the form of a phoneme convolution operator is shown here. However, as the brain database related to speech recall is enriched in the future, it is possible to use a deep convolutional network (DCN), which is often used in the field of image processing in recent years, instead of the phoneme-specific convolution operator. It is possible.

The word / sentence recognition unit 6 recognizes a word / sentence from the time series data of the phoneme feature vector (to be exact, the phoneme likelihood vector time series data). For word / sentence recognition, a method using a hidden Markov model (HMM) that has been put into practical use in the field of speech recognition (in which triphon including front and back contexts of phonemes is used), and a method using a deep neural network ( LSTM etc.) can be applied. In addition, linguistic information (probability regarding word sequence), which is an advantage of current speech recognition, can be used as well. Furthermore, although the time axis shift becomes a problem in voice recall, the use of "spotting processing" that continuously searches for words and sentences in the time direction, which is performed in the current robust voice system, improves performance even in voice recall. It is effective for.

The post-processing / output unit 7 receives the word (column) of the recognition result and performs necessary display display and voice output. Here, based on the voice recall recognition result of a predetermined word / sentence, feedback is given to the user whether or not the multi-electrode brain wave sensor is in the correct position, and the user uses the screen of a terminal such as a smartphone or voice instruction. By moving the brain wave sensor, it is possible to have a function to support knowing the proper position.

The post-processing / output unit 7 displays a screen that assists in adjusting the optimum position of the electrode group while recalling the voice. The post-processing / output unit 7 can display a display, and FIG. 14 shows a display screen displayed by the post-processing / output unit 7. The user adjusts the position of the electrode group while looking at the screen shown in FIG.
As shown in FIG. 14, when the test voice recall (such as "Mr. Yamada") is recalled, the brain wave is input from the brain wave input unit 2, and the color and the size of 〇 are displayed on the screen displayed by the post-processing / output unit 7. , The accuracy of the recognition result can be shown by the density of the gradation (example in the figure). In FIG. 14, the first electrode position (1) is displayed in white, the next electrode position (2) is displayed in light gray, the next electrode position (3) is displayed in gray, and the next electrode position ( In 4), it is displayed in dark gray, and in the next position (5), it is displayed in light gray. Therefore, the user can know that the electrode position (4) is the optimum electrode position. While looking at the difference in accuracy in chronological order, an example was shown in which the sensor position is moved in the direction in which the correct answer is obtained to have a function to correct it.

The voice recall recognition device 1 shown in FIG. 1 can be configured by a mobile terminal. Further, the voice recall recognition device 1 can be configured by a server. At this time, the voice recall recognition device 1 may be configured by a plurality of servers. Further, the voice recall recognition device 1 can also be configured by a mobile terminal and a server. A part of the processing of the voice recall recognition device 1 can be processed by the mobile terminal, and the remaining processing can be processed by the server. At this time, the server can also be configured by a plurality of servers.

Further, as shown in FIG. 1, the voice recall recognition device 1 includes an electroencephalogram input unit 2, a preprocessing unit 3, an analysis processing unit, a language feature extraction unit 5, a word / sentence recognition unit 6, and a post-processing / output unit 7. Although it was configured by, the voice recall recognition device may include a wearer and a group of electrodes.

FIG. 15 is a diagram showing another configuration example of the voice recall recognition device.
As shown in FIG. 15, the voice recall recognition device 10 includes a wearer 11, a mobile terminal 12, and a server 13. The wearing tool 11 is a wearing tool for a voice recall recognition device that recognizes a voice language from an electroencephalogram at the time of voice recall. The mounting tool 11 has a sheet unit 21 for holding the electrode group 22, an electrode group 22 arranged around the broker field, and a processing unit 23 for outputting a signal from the electrode group 22. The electrode group 22 is composed of 9 electrodes as described above, but the number of electrodes is not limited. The processing unit 23 may have a communication function, and can process a part or all of the voice recall recognition device 1 shown in FIG.

The processing unit 23, the mobile terminal 12, and the server 13 of the mounting tool 11 are composed of, for example, a computer having a CPU (Central Processing Unit), a memory, a ROM (Read only memory), a hard disk, and the like. The terminal 12 can perform some or all processing of the voice recall recognition device 1 shown in FIG. The server 13 can perform some or all processing of the voice recall recognition device 1 shown in FIG.
The voice recall recognition method of recognizing a voice language from an electroencephalogram at the time of voice recall is executed by the wearer 11, the mobile terminal 12 and / or the server 13, and the wearer 11, the mobile terminal 12 and / or the server 13 are used alone or in cooperation with each other. Can work and run. The voice recall recognition method can be executed by the mobile terminal 12 and the server 13.

The program for causing the computer to execute the voice recall recognition process for recognizing the voice language from the brain waves at the time of voice recall is downloaded or stored in the hard disk or the like, and the brain waves for each electrode input to the computer from the electrode group. An analysis process for analyzing a discrete signal group and outputting a spectral time series, and an extraction process for extracting a phonetic feature vector time series based on the spectral components of each electrode are executed.

FIG. 16 is a diagram showing another configuration example of the voice recall recognition device.
As shown in FIG. 16, the voice recall recognition device 20 is composed of a wearer 11 and a server 13. The configuration of the mounting tool 11 is as described with reference to FIG. 15, but the processing unit 23 of the mounting tool 11 has a function of directly communicating with the server 13. The function of the voice recall recognition device can be realized by the mounting tool 11 directly exchanging information with the server 13.

FIG. 17 is a diagram showing another configuration example of the voice recall recognition device.
As shown in FIG. 17, the voice recall recognition device 30 is composed of a wearer 11. By realizing all the functions of the voice recall recognition device 30 shown in FIG. 1, the processing unit 23 of the mounting tool 11 can realize the voice recall recognition device only by the mounting tool 11.

As described above, according to the present embodiment, it is possible to directly extract a line spectrum component group representing a language from an electroencephalogram at the time of speech recall and further convert it into a phoneme feature vector time series, so that the current speech recognition framework can be used. There is an advantage that can be utilized.

The following additional notes will be further disclosed with respect to the above embodiments.

(Appendix 1)
It is a voice recall recognition method that recognizes vocal language from brain waves at the time of speech recall.
An analysis processing step that analyzes and processes a discrete signal group of brain waves for each electrode input from the electrode group and outputs a spectral time series.
An extraction step that outputs a phoneme feature vector time series based on the spectral time series,
Voice recall recognition method including.

(Appendix 2)
The voice recall recognition method according to Appendix 1, further comprising an input step of converting an electroencephalogram input from an electrode group into a discrete signal group.
(Appendix 3)
Addendum 1 or 2 further includes a preprocessing unit that performs a process of removing noise in the brain wave by subtracting an average noise amplitude spectrum from the spectrum of a voice recall signal obtained by converting a discrete signal group for each electrode into a frequency domain. The voice recall recognition method described in.

(Appendix 4)
The voice recall recognition method according to Appendix 3, further comprising an independent component analysis for extracting a small number of independent information sources from each electrode signal after noise reduction.
(Appendix 5)
The speech recall recognition method according to any one of Supplementary note 1 to Supplementary note 4, further comprising a recognition step of recognizing the speech language based on the phoneme feature vector time series.
(Appendix 6)
The voice recall recognition method according to any one of Supplementary note 1 to Supplementary note 5, further comprising an output step of outputting the recognized speech language.

(Appendix 7)
The voice recall recognition method according to Appendix 6, further comprising a step of displaying a screen that assists in adjusting the optimum position of the electrode group while recalling the voice.
(Appendix 8)
The speech recall recognition method according to any one of Supplementary note 1 to Supplementary note 7, wherein the analysis processing step extracts the spectral time series by applying linear prediction analysis.
(Appendix 9)
The voice recall recognition method according to any one of Supplementary note 1 to Supplementary note 8, wherein the analysis processing step includes a step of absorbing frequency fluctuations based on the discrete signal for each electrode.

(Appendix 10)
The speech recall recognition method according to any one of Supplementary note 1 to Supplementary note 9, wherein the analysis processing step extracts the frequency derived from the peak on the frequency axis as a line spectrum component for each time frame.
(Appendix 11)
The speech recall recognition method according to any one of Supplementary note 1 to Supplementary note 10, wherein the extraction step outputs a phoneme likelihood vector time series, which is a language feature, using a predetermined convolution operator.

(Appendix 12)
The voice recall recognition method according to any one of Supplementary note 1 to Supplementary note 11, which is executed by a mobile terminal, a server, or a mobile terminal and a server.
(Appendix 13)
The voice recall recognition method according to any one of Supplementary note 1 to Supplementary note 12, further comprising an output step of outputting a signal from a group of electrodes arranged around Broca's area provided in the mounting tool.

Thus, according to the speech recognition recognition device, wearing tool, method, and program of the present invention, it is possible to directly convert the brain wave at the time of speech recall into a line spectrum group and a phoneme feature group as a language representation. It is possible to provide a speech language that can be BCI in the framework of speech recognition.

1 Voice recall recognition device 2 EEG input unit 3 Pre-processing unit 4 Analysis processing unit 5 Language feature extraction unit 6 Word / character recognition unit 7 Post-processing / output unit

Claims (16)

It is a voice recall recognition device that recognizes vocal language from brain waves at the time of speech recall.
An analysis processing unit that analyzes and processes discrete signal groups of brain waves for each electrode input from the electrode group and outputs a spectral time series.
An extraction unit that outputs a phoneme feature vector time series based on the spectral time series,
Have,
The extraction unit is a speech recall recognition device that outputs a phoneme likelihood vector time series, which is a language feature, using a predetermined convolution operator. The voice recall recognition device according to claim 1, further comprising an electroencephalogram input unit that converts an electroencephalogram input from an electrode group into a discrete signal group. 1. Item 2. The voice recall recognition device according to Item 2. The voice recall recognition device according to claim 3, wherein the preprocessing unit performs an independent component analysis for extracting a small number of independent information sources from each electrode signal after noise removal. The voice recall recognition device according to any one of claims 1 to 4, further comprising a recognition unit that recognizes the voice language based on the phoneme feature vector time series. The voice recall recognition device according to claim 5, further comprising an output unit that outputs the voice language recognized by the recognition unit. The voice recall recognition device according to claim 6, wherein the output unit displays a screen that assists in adjusting the optimum position of the electrode group during recognition by the recognition unit. The voice recall recognition device according to any one of claims 1 to 7, wherein the analysis processing unit extracts the spectral time series by applying linear prediction analysis. The voice recall recognition device according to any one of claims 1 to 8, wherein the analysis processing unit performs processing for absorbing frequency fluctuations based on the discrete signal for each electrode. The voice recall recognition device according to any one of claims 1 to 9, wherein the analysis processing unit extracts the frequency derived from the peak on the frequency axis as a line spectrum component for each time frame. The voice recall recognition device according to any one of claims 1 to 10, further comprising a group of electrodes arranged around Broca's area. The voice recall recognition device according to claim 11, further comprising a wearer to be worn on the head. The voice recall recognition device according to any one of claims 1 to 11, wherein the voice recall recognition device is composed of a mobile terminal, a server, or a mobile terminal and a server. It is a wearer for a voice recall recognition device that recognizes a voice language from brain waves at the time of voice recall.
Electrodes placed around Broca's area and
It has a processing unit that outputs a signal from the electrode group, and has a processing unit.
The voice recall recognition device has an analysis process of analyzing and processing a discrete signal group of brain waves for each electrode output from the processing unit and outputting a spectral time series.
Based on the spectral time series, the extraction process that outputs the phoneme feature vector time series is executed.
The extraction process includes outputting a phoneme likelihood vector time series, which is a language feature, using a predetermined convolution operator. It is a voice recall recognition method that recognizes vocal language from brain waves at the time of speech recall.
An analysis processing step that analyzes and processes a discrete signal group of brain waves for each electrode input from the electrode group and outputs a spectral time series.
An extraction step that outputs a phoneme feature vector time series based on the spectral time series,
Including
The extraction step is a computer-executed speech recall recognition method comprising outputting a phoneme likelihood vector time series, which is a language feature, using a predetermined convolution operator . It is a program for causing a computer to execute a voice recall recognition process that recognizes a speech language from brain waves at the time of speech recall.
On the computer
An analysis process that analyzes and processes a discrete signal group of brain waves for each electrode input from the electrode group and outputs a spectral time series.
Extraction processing to extract phoneme feature vector time series based on the spectral components of each electrode,
To execute,
The extraction process is a program including outputting a phoneme likelihood vector time series, which is a language feature, using a predetermined convolution operator.

Images