WO2023047519A1 - Learning device, estimation device, learning method, estimation method, and program - Google Patents

Learning device, estimation device, learning method, estimation method, and program Download PDF

Info

Publication number
WO2023047519A1
WO2023047519A1 PCT/JP2021/035019 JP2021035019W WO2023047519A1 WO 2023047519 A1 WO2023047519 A1 WO 2023047519A1 JP 2021035019 W JP2021035019 W JP 2021035019W WO 2023047519 A1 WO2023047519 A1 WO 2023047519A1
Authority
WO
WIPO (PCT)
Prior art keywords
learning
estimation
microsaccade
attention
attentional
Prior art date
Application number
PCT/JP2021/035019
Other languages
French (fr)
Japanese (ja)
Inventor
慎平 山岸
茂人 古川
Original Assignee
日本電信電話株式会社
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 日本電信電話株式会社 filed Critical 日本電信電話株式会社
Priority to PCT/JP2021/035019 priority Critical patent/WO2023047519A1/en
Publication of WO2023047519A1 publication Critical patent/WO2023047519A1/en

Links

Images

Classifications

    • A—HUMAN NECESSITIES
    • A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
    • A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
    • A61B3/00—Apparatus for testing the eyes; Instruments for examining the eyes
    • A61B3/10—Objective types, i.e. instruments for examining the eyes independent of the patients' perceptions or reactions
    • A61B3/113—Objective types, i.e. instruments for examining the eyes independent of the patients' perceptions or reactions for determining or recording eye movement
    • A—HUMAN NECESSITIES
    • A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
    • A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
    • A61B5/00—Measuring for diagnostic purposes; Identification of persons
    • A61B5/16—Devices for psychotechnics; Testing reaction times ; Devices for evaluating the psychological state

Definitions

  • the present invention relates to an estimation model learning device, an estimation device using the estimation model, a learning method, an estimation method, and a program relating to attention using minute eye movements (microsaccades).
  • Non-Patent Document 1 it is possible to estimate the direction of attention not only for visual stimuli but also for auditory stimuli from the characteristics of microsaccades.
  • An object of the present invention is to estimate the state of attention to auditory stimuli based on the characteristics of microsaccades.
  • the present invention provides a learning device that learns an estimation model for estimating attentional states for auditory stimuli based on the characteristics of microsaccades, an estimation device that uses the estimation model, a learning method, an estimation method, and a program. aim.
  • a learning device extracts a microsaccade feature quantity from time-series data of eyeball positions included in learning data. and a learning unit that learns an estimation model using the feature amount of the sound and the information indicating the attention state with respect to the auditory stimulus included in the learning data, and the estimation model learns the generated microsaccade with respect to the auditory stimulus. It is a model that estimates the attentional state of the subject to auditory stimuli from the orientation.
  • an estimating device extracts a microsaccade feature amount from time-series data of eyeball positions of a subject, and includes a feature amount extracting unit and an estimating model. and an estimating unit that estimates the subject's attentional state to auditory stimuli from the microsaccade feature value using microsaccade features. It is a model learned using information indicating
  • FIG. 2 is a functional block diagram of the learning device according to the first embodiment
  • 4 is a diagram showing an example of the processing flow of the learning device according to the first embodiment
  • FIG. The functional block diagram of the estimation apparatus which concerns on 1st embodiment.
  • the time change characteristics of the microsaccade when the subject is working on the auditory attention task in the left and right directions are related to the task performance (reaction time slow vs. fast, failure vs. success). Based on the discovery that it differs depending on the subject, we estimate the direction of attention and the attention level at that time from the microsaccades obtained by measuring the eye movement of the subject.
  • ⁇ Background experiment> 1 An eye tracker is used to record microsaccades when a specific sound (standard sound) is repeatedly presented to the left and right ears of a subject. The subject pays attention to either the left or right sound, and performs a task to detect a sound (oddball sound) that occasionally appears in the sequence of sounds to which attention is directed and has characteristics different from the standard sound. For example, the subject receives the following instructions (i) to (iii). (i) Focusing attention on either left or right sound. (ii) Press the button when the oddball sound is detected from the indicated direction. (iii) ignoring oddball sounds presented to the unattended ear;
  • Figure 1 shows an example of changes in eyeball position over time and occurrence of microsaccades.
  • the characteristics of the temporal change in the direction of the generated microsaccade differ depending on the direction of attention to the auditory stimulus (see Fig. 2)
  • the characteristics of the temporal change in the direction of the microsaccade. differed depending on the performance (attention level) of the auditory attention task (see Fig. 3).
  • (1) is that microsaccades occur in the direction opposite to the direction of attention to auditory stimuli.
  • the time when the oddball sound presentation is started is set to zero.
  • the discovery of the relationship between the time-varying characteristics of microsaccades and task performance is used to estimate the state of attention (direction of attention and level of attention) to auditory stimuli from time-series data of eyeball positions.
  • FIG. 4 shows a configuration example of an estimation system according to the first embodiment.
  • the estimation system includes a learning device 100 and an estimation device 200.
  • the processing of the estimation system consists of a learning stage and an estimation stage.
  • the learning device 100 receives learning data N, learns an estimation model, and outputs a trained estimation model M.
  • the estimation device 200 receives the estimation model M prior to the estimation process.
  • the estimation device 200 receives the eyeball position time-series data S as input, estimates the state of attention (attention direction and level of attention in this embodiment) to the auditory stimulus, and outputs an estimation result T.
  • the state of attention attention direction and level of attention in this embodiment
  • the learning device 100 and the estimating device 200 are configured by reading a special program into a known or dedicated computer having, for example, a central processing unit (CPU: Central Processing Unit), a main memory (RAM: Random Access Memory), etc. It is a special device designed Learning device 100 and estimating device 200 execute each process under the control of a central processing unit, for example.
  • the data input to the learning device 100 and the estimation device 200 and the data obtained in each process are stored, for example, in a main storage device, and the data stored in the main storage device are read into the central processing unit as needed. output and used for other processing.
  • At least a part of each processing unit of learning device 100 and estimation device 200 may be configured by hardware such as an integrated circuit.
  • Each storage unit included in the learning device 100 and the estimating device 200 can be configured by, for example, a main storage device such as RAM (Random Access Memory), or middleware such as a relational database or key-value store.
  • a main storage device such as RAM (Random Access Memory), or middleware such as a relational database or key-value store.
  • middleware such as a relational database or key-value store.
  • each storage unit does not necessarily have to be provided inside the learning device 100 and the estimation device 200, and may be configured by an auxiliary storage device configured by a semiconductor memory device such as a hard disk, an optical disk, or a flash memory. , may be provided outside the learning device 100 and the estimation device 200 .
  • FIG. 5 is a functional block diagram of the learning device 100 according to the first embodiment, and FIG. 6 shows its processing flow.
  • the learning device 100 includes a feature extraction unit 110 and a learning unit 120.
  • the learning device 100 receives learning data N, learns an estimation model, and outputs a learned estimation model M.
  • the subject is repeatedly presented with standard sounds and occasionally with oddball sounds via the sound generator 80 . Furthermore, the eye movement acquiring unit 90 acquires the eye movement of the subject who is presented with the auditory stimulus. Similar processing is performed for a plurality of subjects to create learning data for the plurality of subjects. For example, as in the background experiment, subjects were instructed to turn their attention to either the left or right sound, and were instructed to press a button when an oddball sound was detected from the instructed direction. instructed to ignore oddball sounds presented to the non-attending ear.
  • the sound generator 80 is, for example, an earphone or speaker that presents auditory stimulation to the subject.
  • the reproduction signal reproduced by the sound generator 80 is a signal that repeatedly presents the standard sound and occasionally presents the oddball sound.
  • earphones we used earphones as the sound generator, but any device that can present sounds that are perceived at different spatial locations will suffice.
  • the eye movement acquisition unit 90 captures the movement of the subject's eyes while presenting the auditory stimulus with a camera or eye tracker, calculates and outputs the eye position. Because microsaccades are very fast eye movements, it is desirable to use a camera or eye tracker with a high sampling rate (at least 500 Hz).
  • the eye movement acquiring unit 90 may be configured by a device separate from the learning device 100 , or may be configured by a camera separate from the learning device 100 and a video processing unit provided in the learning device 100 .
  • the learning data includes a set of eyeball position time-series data acquired by the eye movement acquisition unit 90 and information indicating the state of attention to the auditory stimulus.
  • Information indicating the state of attention to auditory stimuli includes, for example, information indicating the direction in which the subject's attention is directed and the reaction time.
  • the information indicating the reaction time for example, the difference between the time when the oddball sound was presented and the time when the subject pressed the button when the oddball sound was detected can be used.
  • This reaction time may be calculated in the learning device 100 by using the signal presenting the auditory stimulus to the subject and the output signal of the button as inputs to the learning device 100 .
  • the information indicating the reaction time is information including, for example, the time when the oddball sound was presented and the time when the subject pressed the button when the oddball sound was detected.
  • a signal that presents the auditory stimulation to the subject is a reproduction signal that is reproduced by the sound generator 80 .
  • a configuration may be adopted in which the reaction time is obtained by an external device and the external device outputs the reaction time to the learning device 100 .
  • the information indicating the reaction time is the reaction time itself output by the external device.
  • the feature amount extraction unit 110 receives the time-series data of the eyeball position obtained by the eye movement acquisition unit 90, which is included in the learning data N, and extracts the microsaccade feature amount from the time-series data of the eyeball position ( S110), output.
  • the feature amount is extracted by the following procedure.
  • the feature quantity includes at least the above-described microsaccade amplitude information, and may also include characteristics such as microsaccade velocity, vibration characteristics, and frequency of occurrence.
  • the learning unit 120 receives the feature quantity of the microsaccade extracted by the feature quantity extraction unit 110 and the information indicating the state of attention to the auditory stimulus included in the learning data, and learns the estimation model (S120). output the estimated model of The estimation model is a model that uses microsaccade features as input and estimates attentional states for auditory stimuli.
  • the estimation model can also be said to be a model for estimating the subject's attentional state to the auditory stimulus from the direction of the generated microsaccade with respect to the auditory stimulus.
  • the learning unit 120 prepares an estimation model in which an appropriate initial value is set in advance, and the estimation result for the microsaccade feature quantity extracted by the feature quantity extraction unit 110 is the attention state for the auditory stimulus included in the learning data.
  • the estimation model is learned by machine learning so as to match the information indicating
  • the learning unit 120 repeatedly updates the parameters of the estimation model until a predetermined condition is satisfied, and outputs the estimation model at that time as a learned estimation model when the predetermined condition is satisfied.
  • the predetermined condition is a condition for determining whether or not the learning of the estimation model has converged. value or less.
  • the estimation model trained in this way will: (a) Information indicating that the subject is paying attention (information indicating the direction of attention), (b) the amplitude and direction of the microsaccade included in the input microsaccade feature amount are 0 to 0.8 after the oddball sound presentation. If it occurs in the same direction as the oddball within a second, information indicating that the attention level has decreased (information indicating the attention level) is output as the estimation result.
  • the threshold is, for example, a value similar to the value when paying attention to a sound source other than the target sound source (preferably in the opposite direction), or It may simply be an amplitude value of 0 (occurring equally in both directions).
  • FIG. 7 is a functional block diagram of the estimation device 200 according to the first embodiment, and FIG. 8 shows its processing flow.
  • the estimating device 200 includes a feature extraction unit 210 and an estimating unit 220 .
  • the estimation device 200 receives the estimation model M prior to the estimation process.
  • the estimation device 200 receives the eyeball position time series data S as input, estimates the state of attention to the auditory stimulus, and outputs the estimation result T.
  • the subject is repeatedly presented with the standard sound through the sound generator 80, and occasionally presented with the oddball sound. Furthermore, the eye movement acquiring unit 90 acquires the eye movement of the subject who is presented with the auditory stimulus. Note that in the estimation stage, it is not necessary to issue instructions to the subject unlike in the learning stage.
  • the feature amount extraction unit 210 receives the eyeball position time-series data S obtained by the eye movement acquisition unit 90, extracts the microsaccade feature amount from the eyeball position time-series data S (S210), and outputs the feature amount. .
  • the feature amount extraction unit 210 may extract the same feature amount using the same method as the feature amount extraction unit 110 .
  • the estimation unit 220 receives the microsaccade feature amount extracted by the feature amount extraction unit 210, estimates the state of attention to the auditory stimulus using an estimation model (S220), and outputs an estimation result T.
  • the estimation result T includes attention direction and attention level for auditory stimuli.
  • the "attentional state for auditory stimulation" to be estimated is the attentional direction and the attentional level, but either one of them may be used. It is only necessary to prepare learning data according to the target to be estimated, and learn the estimation model.
  • the eye movement acquisition unit 90 is configured to include a camera, but other configurations may be used as long as they are configured to acquire time-series data of eyeball positions.
  • the position of the eyeball may be obtained by arranging electrodes around the eye and measuring the electric potential generated by muscles or the like for moving the eyeball.
  • the present invention is not limited to the above embodiments and modifications.
  • the various types of processing described above may not only be executed in chronological order according to the description, but may also be executed in parallel or individually according to the processing capacity of the device that executes the processing or as necessary.
  • appropriate modifications are possible without departing from the gist of the present invention.
  • a program that describes this process can be recorded on a computer-readable recording medium.
  • Any computer-readable recording medium may be used, for example, a magnetic recording device, an optical disk, a magneto-optical recording medium, a semiconductor memory, or the like.
  • this program is carried out, for example, by selling, assigning, lending, etc. portable recording media such as DVDs and CD-ROMs on which the program is recorded.
  • the program may be distributed by storing the program in the storage device of the server computer and transferring the program from the server computer to other computers via the network.
  • a computer that executes such a program for example, first stores the program recorded on a portable recording medium or the program transferred from the server computer once in its own storage device. Then, when executing the process, this computer reads the program stored in its own recording medium and executes the process according to the read program. Also, as another execution form of this program, the computer may read the program directly from a portable recording medium and execute processing according to the program, and the program is transferred from the server computer to this computer. Each time, the processing according to the received program may be executed sequentially. In addition, the above-mentioned processing is executed by a so-called ASP (Application Service Provider) type service, which does not transfer the program from the server computer to this computer, and realizes the processing function only by its execution instruction and result acquisition. may be It should be noted that the program in this embodiment includes information that is used for processing by a computer and that conforms to the program (data that is not a direct instruction to the computer but has the property of prescribing the processing of the computer, etc.).
  • ASP
  • the device is configured by executing a predetermined program on a computer, but at least part of these processing contents may be implemented by hardware.

Landscapes

  • Health & Medical Sciences (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • Engineering & Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • Veterinary Medicine (AREA)
  • Biophysics (AREA)
  • Public Health (AREA)
  • Biomedical Technology (AREA)
  • Heart & Thoracic Surgery (AREA)
  • Medical Informatics (AREA)
  • Molecular Biology (AREA)
  • Surgery (AREA)
  • Animal Behavior & Ethology (AREA)
  • General Health & Medical Sciences (AREA)
  • Ophthalmology & Optometry (AREA)
  • Human Computer Interaction (AREA)
  • Child & Adolescent Psychology (AREA)
  • Developmental Disabilities (AREA)
  • Educational Technology (AREA)
  • Hospice & Palliative Care (AREA)
  • Psychiatry (AREA)
  • Psychology (AREA)
  • Social Psychology (AREA)
  • Pathology (AREA)
  • Measurement Of The Respiration, Hearing Ability, Form, And Blood Characteristics Of Living Organisms (AREA)

Abstract

The purpose of the present invention is to provide a learning device that learns an estimation model that estimates attention to auditory stimuli on the basis of the characteristics of microsaccades. This learning device comprises: a feature quantity extraction unit that extracts feature quantities of microsaccades from time-series data of eyeball positions included in learning data; and a learning unit that learns the estimation model by using the feature quantities of the microsaccades and information indicative of the attentional states to auditory stimuli included in the learning data. The estimation model estimates a subject's attentional state to the auditory stimuli from the direction of the generated microsaccades for the auditory stimuli.

Description

学習装置、推定装置、学習方法、推定方法、およびプログラムLearning device, estimation device, learning method, estimation method, and program
　本発明は、微小眼球運動（マイクロサッカード）を利用した注意に関する推定モデルの学習装置、推定モデルを用いた推定装置、学習方法、推定方法、およびプログラムに関する。 The present invention relates to an estimation model learning device, an estimation device using the estimation model, a learning method, an estimation method, and a program relating to attention using minute eye movements (microsaccades).
　注意に関する情報、例えば、注意方向や注意レベルが分かれば、その人の興味対象や作業への集中度を推定する技術につながる。なお、注意方向は、対象者が注意を向けている方向を表し、注意レベルは対象者がどのくらい注意を払っているかを表すものである。視覚研究の分野では、マイクロサッカードと呼ばれる不随意に生じる微小眼球運動が注意方向を反映することが報告されており、非侵襲に計測された眼球データから注意方向を推定する技術の発展が見込まれている。一方で、マイクロサッカードと聴覚的注意の関係に関する証拠は少なく、また時間的に変動する注意レベルとの関係についての証拠も確立していない。 Knowing information about attention, such as direction of attention and level of attention, will lead to technology for estimating a person's interest and degree of concentration on work. Note that the direction of attention indicates the direction in which the subject pays attention, and the attention level indicates how much attention the subject pays. In the field of vision research, it has been reported that involuntary small eye movements called microsaccades reflect the direction of attention. is On the other hand, evidence for a relationship between microsaccades and auditory attention is sparse, and evidence for a relationship with time-varying levels of attention has not been established.
　非特許文献１によれば、マイクロサッカードの特性から視覚刺激だけでなく聴覚刺激に対する注意方向を推定できる可能性がある。 According to Non-Patent Document 1, it is possible to estimate the direction of attention not only for visual stimuli but also for auditory stimuli from the characteristics of microsaccades.
　しかしながら、心理状態などに応じて時間とともに変動する注意タスクのパフォーマンスといった、人の行動や心理状態に関わる情報をマイクロサッカードの特性から推定できるかは定かではない。本発明は、マイクロサッカードの特性に基づいて聴覚的刺激に対する注意状態を推定することを目的とする。 However, it is not clear whether information related to human behavior and psychological state, such as attentional task performance that fluctuates over time according to psychological state, can be inferred from the characteristics of microsaccades. An object of the present invention is to estimate the state of attention to auditory stimuli based on the characteristics of microsaccades.
　本発明は、マイクロサッカードの特性に基づいて聴覚的刺激に対する注意状態を推定する推定モデルを学習する学習装置、推定モデルを用いた推定装置、学習方法、推定方法、及びプログラムを提供することを目的とする。 The present invention provides a learning device that learns an estimation model for estimating attentional states for auditory stimuli based on the characteristics of microsaccades, an estimation device that uses the estimation model, a learning method, an estimation method, and a program. aim.
　上記の課題を解決するために、本発明の一態様によれば、学習装置は、学習データに含まれる眼球位置の時系列データからマイクロサッカードの特徴量を抽出特徴量抽出部と、マイクロサッカードの特徴量と、学習データに含まれる聴覚刺激に対する注意状態を示す情報とを用いて、推定モデルを学習する学習部と、を含み、推定モデルは、聴覚刺激に対する、発生したマイクロサッカードの方向から、対象者の聴覚刺激に対する注意状態を推定するモデルである。 In order to solve the above problems, according to one aspect of the present invention, a learning device extracts a microsaccade feature quantity from time-series data of eyeball positions included in learning data. and a learning unit that learns an estimation model using the feature amount of the sound and the information indicating the attention state with respect to the auditory stimulus included in the learning data, and the estimation model learns the generated microsaccade with respect to the auditory stimulus. It is a model that estimates the attentional state of the subject to auditory stimuli from the orientation.
　上記の課題を解決するために、本発明の他の態様によれば、推定装置は、対象者の眼球位置の時系列データからマイクロサッカードの特徴量を抽出特徴量抽出部と、推定モデルを用いて、マイクロサッカードの特徴量から対象者の聴覚刺激に対する注意状態を推定する推定部と、を含み、推定モデルは、学習データに含まれる眼球位置の時系列データと、聴覚刺激に対する注意状態を示す情報とを用いて学習されたモデルである。 In order to solve the above problems, according to another aspect of the present invention, an estimating device extracts a microsaccade feature amount from time-series data of eyeball positions of a subject, and includes a feature amount extracting unit and an estimating model. and an estimating unit that estimates the subject's attentional state to auditory stimuli from the microsaccade feature value using microsaccade features. It is a model learned using information indicating
　本発明によれば、聴覚的刺激に対する注意状態をマイクロサッカードの計測により推定することができるという効果を奏する。 According to the present invention, it is possible to estimate the state of attention to auditory stimuli by measuring microsaccades.
眼球位置の時間変化とマイクロサッカードの発生例を示す図。The figure which shows the time change of an eyeball position, and the generation example of a micro saccade. 注意方向と、発生したマイクロサッカードの方向の時間変化との関係を示す図。The figure which shows the relationship between attention direction and the time change of the direction of the microsaccade which generate|occur|produced. 注意レベルと、発生したマイクロサッカードの方向の時間変化との関係を示す図。The figure which shows the relationship between attention level and the time change of the direction of the microsaccade which generate|occur|produced. 第一実施形態に係る推定システムの構成例を示す図。The figure which shows the structural example of the estimation system which concerns on 1st embodiment. 第一実施形態に係る学習装置の機能ブロック図。FIG. 2 is a functional block diagram of the learning device according to the first embodiment; 第一実施形態に係る学習装置の処理フローの例を示す図。4 is a diagram showing an example of the processing flow of the learning device according to the first embodiment; FIG. 第一実施形態に係る推定装置の機能ブロック図。The functional block diagram of the estimation apparatus which concerns on 1st embodiment. 第一実施形態に係る推定装置の処理フローの例を示す図。The figure which shows the example of the processing flow of the estimation apparatus which concerns on 1st embodiment. 本手法を適用するコンピュータの構成例を示す図。The figure which shows the structural example of the computer which applies this method.
　以下、本発明の実施形態について、説明する。なお、以下の説明に用いる図面では、同じ機能を持つ構成部や同じ処理を行うステップには同一の符号を記し、重複説明を省略する。 Embodiments of the present invention will be described below. It should be noted that in the drawings used for the following description, the same reference numerals are given to components having the same functions and steps that perform the same processing, and redundant description will be omitted.
＜第一実施形態のポイント＞
　本実施形態は、対象者が左右方向への聴覚注意課題に取り組んでいるときのマイクロサッカードの時間変化特性が課題成績（反応時間が遅い場合と早い場合や失敗した場合と成功した場合）に応じて異なるという発見に基づき、対象者の眼球運動計測によって得られたマイクロサッカードから注意方向やそのときの注意レベルを推定するものである。 <Points of the first embodiment>
In this embodiment, the time change characteristics of the microsaccade when the subject is working on the auditory attention task in the left and right directions are related to the task performance (reaction time slow vs. fast, failure vs. success). Based on the discovery that it differs depending on the subject, we estimate the direction of attention and the attention level at that time from the microsaccades obtained by measuring the eye movement of the subject.
　まず、本実施形態の背景となる対象者が注意課題に取り組んでいるときのマイクロサッカードの時間変化特性と課題成績の関係の発見について、根拠となる実験の内容とその結果を説明する。 First, we will explain the content and results of the experiment that is the basis for the discovery of the relationship between the time-varying characteristics of microsaccades and task performance when the subject is working on an attention task, which is the background of this embodiment.
＜背景となる実験＞
1.対象者の左右耳に特定の音（スタンダード音）を繰り返し呈示している際のマイクロサッカードをアイトラッカーで撮影する。対象者は左右どちらかの音に注意を向け、注意を向けた音列に時折現れるスタンダード音と異なる特性を持つ音（オドボール音）を検出する課題を行う。例えば、対象者は、以下の(i)～(iii)の指示を受ける。
　(i)左右どちらかの音に注意を向ける。
　(ii)指示された方向からオドボール音を検出した際には、ボタンを押下する。
　(iii)注意を向けていない耳に呈示されたオドボール音は無視する。 <Background experiment>
1. An eye tracker is used to record microsaccades when a specific sound (standard sound) is repeatedly presented to the left and right ears of a subject. The subject pays attention to either the left or right sound, and performs a task to detect a sound (oddball sound) that occasionally appears in the sequence of sounds to which attention is directed and has characteristics different from the standard sound. For example, the subject receives the following instructions (i) to (iii).
(i) Focusing attention on either left or right sound.
(ii) Press the button when the oddball sound is detected from the indicated direction.
(iii) ignoring oddball sounds presented to the unattended ear;
2.撮影した画像からマイクロサッカードの発生タイミングと振幅を計算する。なお、課題中のマイクロサッカードの振幅値の時間変化を特徴量として用いる。 2. Calculating the timing and amplitude of microsaccades from the captured images. Note that the temporal change in the amplitude value of the microsaccade during the task is used as a feature quantity.
　図１は眼球位置の時間変化とマイクロサッカードの発生例を示す。実験の結果、(1)聴覚刺激に対する注意方向に応じて、発生したマイクロサッカードの方向の時間変化の特性が異なること(図２参照)、(2)マイクロサッカードの方向の時間変化の特性が聴覚注意タスクの成績（注意レベル）に応じて異なること(図３参照)、を発見した。(1)は、具体的には、聴覚刺激に対する注意方向とは、反対方向にマイクロサッカードが発生することである。なお、図２ではオドボール音呈示開始時の時刻を0としている。(2)は、具体的には、検出が遅い場合(注意レベルが低い場合)にはオドボール音呈示後0～0.8秒の間にオドボール音と同じ方向にマイクロサッカードが発生し、その後、オドボール音とは反対方向にマイクロサッカードが発生することである。なお、図３ではオドボール音呈示開始時の時刻を0としている。 Figure 1 shows an example of changes in eyeball position over time and occurrence of microsaccades. As a result of the experiment, (1) the characteristics of the temporal change in the direction of the generated microsaccade differ depending on the direction of attention to the auditory stimulus (see Fig. 2), (2) the characteristics of the temporal change in the direction of the microsaccade. differed depending on the performance (attention level) of the auditory attention task (see Fig. 3). (1) is that microsaccades occur in the direction opposite to the direction of attention to auditory stimuli. In FIG. 2, the time when the oddball sound presentation is started is set to zero. In (2), specifically, when the detection is slow (when the attention level is low), a microsaccade occurs in the same direction as the oddball sound within 0 to 0.8 seconds after the oddball sound is presented. It is the generation of microsaccades in the opposite direction to the sound. In FIG. 3, the time when the oddball sound presentation is started is set to zero.
　本実施形態では、このマイクロサッカードの時間変化特性と課題成績の関係の発見を利用して、眼球位置の時系列データから聴覚刺激に対する注意状態(注意方向および注意レベル)を推定する。 In this embodiment, the discovery of the relationship between the time-varying characteristics of microsaccades and task performance is used to estimate the state of attention (direction of attention and level of attention) to auditory stimuli from time-series data of eyeball positions.
＜第一実施形態＞
　図４は第一実施形態に係る推定システムの構成例を示す。 <First Embodiment>
FIG. 4 shows a configuration example of an estimation system according to the first embodiment.
　推定システムは、学習装置１００と推定装置２００とを含む。 The estimation system includes a learning device 100 and an estimation device 200.
　推定システムの処理は、学習段階と推定段階とからなる。 The processing of the estimation system consists of a learning stage and an estimation stage.
　学習段階において、学習装置１００は、学習データＮを入力とし、推定モデルを学習し、学習済みの推定モデルＭを出力する。 In the learning stage, the learning device 100 receives learning data N, learns an estimation model, and outputs a trained estimation model M.
　推定装置２００は、推定処理に先立ち、推定モデルＭを受け取る。 The estimation device 200 receives the estimation model M prior to the estimation process.
　推定段階において、推定装置２００は、眼球位置の時系列データＳを入力とし、聴覚刺激に対する注意状態（本実施形態では注意方向と注意のレベル）を推定し、推定結果Ｔを出力する。 In the estimation stage, the estimation device 200 receives the eyeball position time-series data S as input, estimates the state of attention (attention direction and level of attention in this embodiment) to the auditory stimulus, and outputs an estimation result T.
　学習装置１００および推定装置２００は、例えば、中央演算処理装置（CPU: Central Processing Unit）、主記憶装置（RAM: Random Access Memory）などを有する公知又は専用のコンピュータに特別なプログラムが読み込まれて構成された特別な装置である。学習装置１００および推定装置２００は、例えば、中央演算処理装置の制御のもとで各処理を実行する。学習装置１００および推定装置２００に入力されたデータや各処理で得られたデータは、例えば、主記憶装置に格納され、主記憶装置に格納されたデータは必要に応じて中央演算処理装置へ読み出されて他の処理に利用される。学習装置１００および推定装置２００の各処理部は、少なくとも一部が集積回路等のハードウェアによって構成されていてもよい。学習装置１００および推定装置２００が備える各記憶部は、例えば、RAM（Random Access Memory）などの主記憶装置、またはリレーショナルデータベースやキーバリューストアなどのミドルウェアにより構成することができる。ただし、各記憶部は、必ずしも学習装置１００および推定装置２００がその内部に備える必要はなく、ハードディスクや光ディスクもしくはフラッシュメモリ（Flash Memory）のような半導体メモリ素子により構成される補助記憶装置により構成し、学習装置１００および推定装置２００の外部に備える構成としてもよい。 The learning device 100 and the estimating device 200 are configured by reading a special program into a known or dedicated computer having, for example, a central processing unit (CPU: Central Processing Unit), a main memory (RAM: Random Access Memory), etc. It is a special device designed Learning device 100 and estimating device 200 execute each process under the control of a central processing unit, for example. The data input to the learning device 100 and the estimation device 200 and the data obtained in each process are stored, for example, in a main storage device, and the data stored in the main storage device are read into the central processing unit as needed. output and used for other processing. At least a part of each processing unit of learning device 100 and estimation device 200 may be configured by hardware such as an integrated circuit. Each storage unit included in the learning device 100 and the estimating device 200 can be configured by, for example, a main storage device such as RAM (Random Access Memory), or middleware such as a relational database or key-value store. However, each storage unit does not necessarily have to be provided inside the learning device 100 and the estimation device 200, and may be configured by an auxiliary storage device configured by a semiconductor memory device such as a hard disk, an optical disk, or a flash memory. , may be provided outside the learning device 100 and the estimation device 200 .
　まず、学習段階について説明する。 First, I will explain the learning stage.
＜学習装置１００＞
　図５は第一実施形態に係る学習装置１００の機能ブロック図を、図６はその処理フローを示す。 <Learning Device 100>
FIG. 5 is a functional block diagram of the learning device 100 according to the first embodiment, and FIG. 6 shows its processing flow.
　学習装置１００は、特徴量抽出部１１０および学習部１２０を含む。 The learning device 100 includes a feature extraction unit 110 and a learning unit 120.
　学習装置１００は、学習データＮを入力とし、推定モデルを学習して、学習済みの推定モデルＭを出力する。 The learning device 100 receives learning data N, learns an estimation model, and outputs a learned estimation model M.
　学習データを作成するために、対象者には、音発生部８０を介して、スタンダード音が繰り返し呈示され、時折オドボール音が呈示される。さらに、聴覚刺激を呈示される対象者の眼の動きを眼球運動取得部９０で取得する。複数の対象者に対して同様の処理を行い、複数の対象者に対する学習データを作成する。例えば、背景となる実験と同様に、対象者には、左右どちらかの音に注意を向けるように指示し、さらに、指示した方向からオドボール音を検出した際には、ボタンを押下するように指示し、注意を向けていない耳に呈示されたオドボール音は無視するように指示する。 In order to create learning data, the subject is repeatedly presented with standard sounds and occasionally with oddball sounds via the sound generator 80 . Furthermore, the eye movement acquiring unit 90 acquires the eye movement of the subject who is presented with the auditory stimulus. Similar processing is performed for a plurality of subjects to create learning data for the plurality of subjects. For example, as in the background experiment, subjects were instructed to turn their attention to either the left or right sound, and were instructed to press a button when an oddball sound was detected from the instructed direction. instructed to ignore oddball sounds presented to the non-attending ear.
　音発生部８０は、例えば、対象者に聴覚刺激を呈示するイヤホンあるいはスピーカーである。音発生部８０で再生される再生信号は、スタンダード音を繰り返し呈示し、時折オドボール音を呈示する信号である。背景となる実験では音発生部としてイヤホンを用いたが、異なる空間的位置に知覚される音を呈示ですることができればどんな装置でもよい。 The sound generator 80 is, for example, an earphone or speaker that presents auditory stimulation to the subject. The reproduction signal reproduced by the sound generator 80 is a signal that repeatedly presents the standard sound and occasionally presents the oddball sound. In the background experiments, we used earphones as the sound generator, but any device that can present sounds that are perceived at different spatial locations will suffice.
　眼球運動取得部９０は、聴覚刺激を呈示しているときの対象者の眼の動きをカメラやアイトラッカーで撮影し、眼球位置を算出し、出力する。マイクロサッカードは非常に速い眼球運動のため、高いサンプリングレイト（少なくとも500 Hz）を持つカメラやアイトラッカーを用いることが望ましい。眼球運動取得部９０は、学習装置１００とは別装置により構成してもよいし、学習装置１００とは別装置のカメラと学習装置１００内に設けた映像処理部とからなる構成としてもよい。 The eye movement acquisition unit 90 captures the movement of the subject's eyes while presenting the auditory stimulus with a camera or eye tracker, calculates and outputs the eye position. Because microsaccades are very fast eye movements, it is desirable to use a camera or eye tracker with a high sampling rate (at least 500 Hz). The eye movement acquiring unit 90 may be configured by a device separate from the learning device 100 , or may be configured by a camera separate from the learning device 100 and a video processing unit provided in the learning device 100 .
　学習データは、眼球運動取得部９０で取得した眼球位置の時系列データと、聴覚刺激に対する注意状態を示す情報との組を含む。 The learning data includes a set of eyeball position time-series data acquired by the eye movement acquisition unit 90 and information indicating the state of attention to the auditory stimulus.
　聴覚刺激に対する注意状態を示す情報は、例えば、対象者が注意を向けた方向と、反応時間を示す情報を含む。反応時間を示す情報は、例えば、オドボール音を呈示した時刻と、対象者がオドボール音を検出した際にボタンを押下した時刻との差分を用いることができる。この反応時間は、対象者に聴覚刺激を呈示する信号と、ボタンの出力信号とを学習装置１００の入力とし、学習装置１００内で計算してもよい。この場合、反応時間を示す情報は、例えば、オドボール音を呈示した時刻と、対象者がオドボール音を検出した際にボタンを押下した時刻を含む情報である。対象者に聴覚刺激を呈示する信号は、音発生部８０で再生される再生信号である。また、外部装置で反応時間を求め、外部装置が学習装置１００に出力する構成としてもよい。この場合、反応時間を示す情報は、外部装置が出力する反応時間そのものである。 Information indicating the state of attention to auditory stimuli includes, for example, information indicating the direction in which the subject's attention is directed and the reaction time. As the information indicating the reaction time, for example, the difference between the time when the oddball sound was presented and the time when the subject pressed the button when the oddball sound was detected can be used. This reaction time may be calculated in the learning device 100 by using the signal presenting the auditory stimulus to the subject and the output signal of the button as inputs to the learning device 100 . In this case, the information indicating the reaction time is information including, for example, the time when the oddball sound was presented and the time when the subject pressed the button when the oddball sound was detected. A signal that presents the auditory stimulation to the subject is a reproduction signal that is reproduced by the sound generator 80 . Alternatively, a configuration may be adopted in which the reaction time is obtained by an external device and the external device outputs the reaction time to the learning device 100 . In this case, the information indicating the reaction time is the reaction time itself output by the external device.
　以下、各部について説明する。 Each part will be explained below.
＜特徴量抽出部１１０＞
　特徴量抽出部１１０は、学習データＮに含まれる、眼球運動取得部９０で得られた眼球位置の時系列データを入力とし、眼球位置の時系列データからマイクロサッカードの特徴量を抽出し（Ｓ１１０）、出力する。例えば、以下の手順で特徴量を抽出する。 <Feature quantity extraction unit 110>
The feature amount extraction unit 110 receives the time-series data of the eyeball position obtained by the eye movement acquisition unit 90, which is included in the learning data N, and extracts the microsaccade feature amount from the time-series data of the eyeball position ( S110), output. For example, the feature amount is extracted by the following procedure.
1.眼球位置の時系列データについて、瞬きにより欠損した部分を除く。 1.Exclude the part missing due to blinking from the time-series data of eyeball position.
2.オドボール音呈示中のマイクロサッカードの発生タイミングに応じた振幅値あるいは方向のバイナリデータの離散値を計算する。 2. Calculate the discrete value of the binary data of the amplitude value or the direction according to the generation timing of the microsaccade during the presentation of the oddball sound.
3.離散値から一定の窓幅(0.5～数秒)で移動平均を算出し、マイクロサッカードの発生方向の時間変化を抽出し、特徴量とする。あるいは一定の区間（例えばオドボール音呈示中の全区間）の平均値を特徴量とする。 3. Calculate the moving average from the discrete values with a certain window width (0.5 to several seconds), extract the temporal change in the direction of occurrence of the microsaccade, and use it as a feature value. Alternatively, the average value of a certain section (for example, the entire section during oddball sound presentation) is used as the feature amount.
　特徴量は、上述のマイクロサッカードの振幅情報を少なくとも含むものとし、その他に例えば、マイクロサッカードの速度、振動特性、発生頻度などといった特性を含んでもよい。 The feature quantity includes at least the above-described microsaccade amplitude information, and may also include characteristics such as microsaccade velocity, vibration characteristics, and frequency of occurrence.
＜学習部１２０＞
　学習部１２０は、特徴量抽出部１１０で抽出したマイクロサッカードの特徴量と、学習データに含まれる聴覚刺激に対する注意状態を示す情報とを入力とし、推定モデルを学習し（Ｓ１２０）、学習済みの推定モデルを出力する。推定モデルは、マイクロサッカードの特徴量を入力とし、聴覚刺激に対する注意状態を推定するモデルである。また、推定モデルは、聴覚刺激に対する、発生したマイクロサッカードの方向から、対象者の聴覚刺激に対する注意状態を推定するモデルとも言える。 <Learning unit 120>
The learning unit 120 receives the feature quantity of the microsaccade extracted by the feature quantity extraction unit 110 and the information indicating the state of attention to the auditory stimulus included in the learning data, and learns the estimation model (S120). output the estimated model of The estimation model is a model that uses microsaccade features as input and estimates attentional states for auditory stimuli. The estimation model can also be said to be a model for estimating the subject's attentional state to the auditory stimulus from the direction of the generated microsaccade with respect to the auditory stimulus.
　例えば、学習部１２０は、予め適当な初期値を設定した推定モデルを用意し、特徴量抽出部１１０で抽出したマイクロサッカードの特徴量に対する推定結果が、学習データに含まれる聴覚刺激に対する注意状態を示す情報と一致するように、推定モデルを機械学習により学習する。学習部１２０は、所定の条件を満たすまで、推定モデルのパラメータの更新を繰り返し、所定の条件を満たした場合、そのときの推定モデルを学習済みの推定モデルとして出力する。所定の条件は、推定モデルの学習が収束しているか否かを判定するための条件であり、例えば、繰り返し回数が所定の回数を超えることや、更新前後の推定モデルのパラメータの差分が所定の値以下であること、等が考えられる。 For example, the learning unit 120 prepares an estimation model in which an appropriate initial value is set in advance, and the estimation result for the microsaccade feature quantity extracted by the feature quantity extraction unit 110 is the attention state for the auditory stimulus included in the learning data. The estimation model is learned by machine learning so as to match the information indicating The learning unit 120 repeatedly updates the parameters of the estimation model until a predetermined condition is satisfied, and outputs the estimation model at that time as a learned estimation model when the predetermined condition is satisfied. The predetermined condition is a condition for determining whether or not the learning of the estimation model has converged. value or less.
　このように学習した推定モデルは、例えば図２，図３のような結果が得られた場合、(a)入力されたマイクロサッカードの特徴量に含まれるマイクロサッカードの方向とは反対方向に対象者が注意を払っていることを示す情報(注意方向を示す情報)、(b)入力されたマイクロサッカードの特徴量に含まれるマイクロサッカードの振幅や方向がオドボール音呈示後0～0.8秒の間にオドボールと同じ方向に生じた場合には、注意レベルが低下していることを示す情報(注意レベルを示す情報)を推定結果として出力する。注意方向や注意レベルを判定する際に閾値を用いる場合には、閾値は、例えば、対象とする音源以外に注意を向けているとき（反対方向が望ましい）の値と同様の値とする、あるいは単に振幅値0（どちらの方向にも均等に発生する）としてもよい。 For example, when the results shown in Figs. 2 and 3 are obtained, the estimation model trained in this way will: (a) Information indicating that the subject is paying attention (information indicating the direction of attention), (b) the amplitude and direction of the microsaccade included in the input microsaccade feature amount are 0 to 0.8 after the oddball sound presentation. If it occurs in the same direction as the oddball within a second, information indicating that the attention level has decreased (information indicating the attention level) is output as the estimation result. When a threshold is used to determine the attention direction or attention level, the threshold is, for example, a value similar to the value when paying attention to a sound source other than the target sound source (preferably in the opposite direction), or It may simply be an amplitude value of 0 (occurring equally in both directions).
　次に、推定段階について説明する。 Next, we will explain the estimation stage.
＜推定装置２００＞
　図７は第一実施形態に係る推定装置２００の機能ブロック図を、図８はその処理フローを示す。 <Estimation device 200>
FIG. 7 is a functional block diagram of the estimation device 200 according to the first embodiment, and FIG. 8 shows its processing flow.
　推定装置２００は、特徴量抽出部２１０および推定部２２０を含む。 The estimating device 200 includes a feature extraction unit 210 and an estimating unit 220 .
　推定装置２００は、推定処理に先立ち、推定モデルＭを受け取る。 The estimation device 200 receives the estimation model M prior to the estimation process.
　推定段階において、推定装置２００は、眼球位置の時系列データＳを入力とし、聴覚刺激に対する注意状態を推定し、推定結果Ｔを出力する。 In the estimation stage, the estimation device 200 receives the eyeball position time series data S as input, estimates the state of attention to the auditory stimulus, and outputs the estimation result T.
　推定段階では、対象者には、音発生部８０を介して、スタンダード音が繰り返し呈示され、時折オドボール音が呈示される。さらに、聴覚刺激を呈示される対象者の眼の動きを眼球運動取得部９０で取得する。なお、推定段階では、学習段階のように対象者に指示を出す必要はない。 In the estimation stage, the subject is repeatedly presented with the standard sound through the sound generator 80, and occasionally presented with the oddball sound. Furthermore, the eye movement acquiring unit 90 acquires the eye movement of the subject who is presented with the auditory stimulus. Note that in the estimation stage, it is not necessary to issue instructions to the subject unlike in the learning stage.
＜特徴量抽出部２１０＞
　特徴量抽出部２１０は、眼球運動取得部９０で得られた眼球位置の時系列データＳを入力とし、眼球位置の時系列データＳからマイクロサッカードの特徴量を抽出し（Ｓ２１０）、出力する。特徴量抽出部２１０は、特徴量抽出部１１０と同様の方法で同様の特徴量を抽出すればよい。 <Feature quantity extraction unit 210>
The feature amount extraction unit 210 receives the eyeball position time-series data S obtained by the eye movement acquisition unit 90, extracts the microsaccade feature amount from the eyeball position time-series data S (S210), and outputs the feature amount. . The feature amount extraction unit 210 may extract the same feature amount using the same method as the feature amount extraction unit 110 .
＜推定部２２０＞
　推定部２２０は、特徴量抽出部２１０で抽出したマイクロサッカードの特徴量を入力とし、推定モデルを用いて、聴覚刺激に対する注意状態を推定し（Ｓ２２０）、推定結果Ｔを出力する。例えば、本実施形態では、推定結果Ｔは聴覚刺激に対する注意方向および注意レベルを含む。 <estimating unit 220>
The estimation unit 220 receives the microsaccade feature amount extracted by the feature amount extraction unit 210, estimates the state of attention to the auditory stimulus using an estimation model (S220), and outputs an estimation result T. For example, in the present embodiment, the estimation result T includes attention direction and attention level for auditory stimuli.
＜効果＞
　このような構成により、聴覚刺激に対する注意状態をマイクロサッカードの計測により推定することができる。従来技術ではマイクロサッカードから視覚刺激に対する潜在的な注意方向の抽出を行う技術が報告されていたが、本実施形態は、聴覚刺激に対する注意方向を推定する手法へこれを拡張し、さらに注意レベルの推定手法を示す。 <effect>
With such a configuration, it is possible to estimate the state of attention to an auditory stimulus by measuring microsaccades. In the prior art, a technique for extracting the potential direction of attention to visual stimuli from microsaccades has been reported. shows the estimation method of
＜変形例＞
　本実施形態では、推定する対象である「聴覚刺激に対する注意状態」を、注意方向と注意レベルとにしているが、何れか一方としてもよい。推定する対象に応じた学習データを用意し、推定モデルを学習すればよい。 <Modification>
In the present embodiment, the "attentional state for auditory stimulation" to be estimated is the attentional direction and the attentional level, but either one of them may be used. It is only necessary to prepare learning data according to the target to be estimated, and learn the estimation model.
　本実施形態では、眼球運動取得部９０は、カメラを含む構成としているが、眼球位置の時系列データを取得できる構成であれば、他の構成であってもよい。例えば、カメラを含まず、眼の周りに電極を配置することによって、眼球を動かすための筋肉などが発生させる電位を測定することで、眼球位置を取得する構成としてもよい。 In the present embodiment, the eye movement acquisition unit 90 is configured to include a camera, but other configurations may be used as long as they are configured to acquire time-series data of eyeball positions. For example, without including a camera, the position of the eyeball may be obtained by arranging electrodes around the eye and measuring the electric potential generated by muscles or the like for moving the eyeball.
＜その他の変形例＞
　本発明は上記の実施形態及び変形例に限定されるものではない。例えば、上述の各種の処理は、記載に従って時系列に実行されるのみならず、処理を実行する装置の処理能力あるいは必要に応じて並列的にあるいは個別に実行されてもよい。その他、本発明の趣旨を逸脱しない範囲で適宜変更が可能である。 <Other Modifications>
The present invention is not limited to the above embodiments and modifications. For example, the various types of processing described above may not only be executed in chronological order according to the description, but may also be executed in parallel or individually according to the processing capacity of the device that executes the processing or as necessary. In addition, appropriate modifications are possible without departing from the gist of the present invention.
＜プログラム及び記録媒体＞
　上述の各種の処理は、図９に示すコンピュータの記憶部２０２０に、上記方法の各ステップを実行させるプログラムを読み込ませ、制御部２０１０、入力部２０３０、出力部２０４０などに動作させることで実施できる。 <Program and recording medium>
The various processes described above can be performed by loading a program for executing each step of the above method into the storage unit 2020 of the computer shown in FIG. .
　この処理内容を記述したプログラムは、コンピュータで読み取り可能な記録媒体に記録しておくことができる。コンピュータで読み取り可能な記録媒体としては、例えば、磁気記録装置、光ディスク、光磁気記録媒体、半導体メモリ等どのようなものでもよい。 A program that describes this process can be recorded on a computer-readable recording medium. Any computer-readable recording medium may be used, for example, a magnetic recording device, an optical disk, a magneto-optical recording medium, a semiconductor memory, or the like.
　また、このプログラムの流通は、例えば、そのプログラムを記録したＤＶＤ、ＣＤ－ＲＯＭ等の可搬型記録媒体を販売、譲渡、貸与等することによって行う。さらに、このプログラムをサーバコンピュータの記憶装置に格納しておき、ネットワークを介して、サーバコンピュータから他のコンピュータにそのプログラムを転送することにより、このプログラムを流通させる構成としてもよい。 In addition, the distribution of this program is carried out, for example, by selling, assigning, lending, etc. portable recording media such as DVDs and CD-ROMs on which the program is recorded. Further, the program may be distributed by storing the program in the storage device of the server computer and transferring the program from the server computer to other computers via the network.
　このようなプログラムを実行するコンピュータは、例えば、まず、可搬型記録媒体に記録されたプログラムもしくはサーバコンピュータから転送されたプログラムを、一旦、自己の記憶装置に格納する。そして、処理の実行時、このコンピュータは、自己の記録媒体に格納されたプログラムを読み取り、読み取ったプログラムに従った処理を実行する。また、このプログラムの別の実行形態として、コンピュータが可搬型記録媒体から直接プログラムを読み取り、そのプログラムに従った処理を実行することとしてもよく、さらに、このコンピュータにサーバコンピュータからプログラムが転送されるたびに、逐次、受け取ったプログラムに従った処理を実行することとしてもよい。また、サーバコンピュータから、このコンピュータへのプログラムの転送は行わず、その実行指示と結果取得のみによって処理機能を実現する、いわゆるＡＳＰ（Application Service Provider）型のサービスによって、上述の処理を実行する構成としてもよい。なお、本形態におけるプログラムには、電子計算機による処理の用に供する情報であってプログラムに準ずるもの（コンピュータに対する直接の指令ではないがコンピュータの処理を規定する性質を有するデータ等）を含むものとする。 A computer that executes such a program, for example, first stores the program recorded on a portable recording medium or the program transferred from the server computer once in its own storage device. Then, when executing the process, this computer reads the program stored in its own recording medium and executes the process according to the read program. Also, as another execution form of this program, the computer may read the program directly from a portable recording medium and execute processing according to the program, and the program is transferred from the server computer to this computer. Each time, the processing according to the received program may be executed sequentially. In addition, the above-mentioned processing is executed by a so-called ASP (Application Service Provider) type service, which does not transfer the program from the server computer to this computer, and realizes the processing function only by its execution instruction and result acquisition. may be It should be noted that the program in this embodiment includes information that is used for processing by a computer and that conforms to the program (data that is not a direct instruction to the computer but has the property of prescribing the processing of the computer, etc.).
　また、この形態では、コンピュータ上で所定のプログラムを実行させることにより、本装置を構成することとしたが、これらの処理内容の少なくとも一部をハードウェア的に実現することとしてもよい。 In addition, in this embodiment, the device is configured by executing a predetermined program on a computer, but at least part of these processing contents may be implemented by hardware.

Claims (7)

  1. 　学習データに含まれる眼球位置の時系列データからマイクロサッカードの特徴量を抽出特徴量抽出部と、
    　前記マイクロサッカードの特徴量と、学習データに含まれる聴覚刺激に対する注意状態を示す情報とを用いて、推定モデルを学習する学習部と、を含み、
    　前記推定モデルは、聴覚刺激に対する、発生したマイクロサッカードの方向から、対象者の聴覚刺激に対する注意状態を推定するモデルである、
    　学習装置。     a feature extraction unit that extracts microsaccade features from time series data of eyeball positions included in learning data;
    a learning unit that learns an estimation model using the feature amount of the microsaccade and information indicating the state of attention to auditory stimuli contained in the learning data,
    The estimation model is a model that estimates the attentional state of the subject for auditory stimuli from the direction of the generated microsaccade for auditory stimuli.
    learning device.
  2. 　請求項１の学習装置であって、
    　前記聴覚刺激に対する注意状態は、注意方向および注意レベルの少なくとも一方を含む、
    　学習装置。     The learning device of claim 1,
    the attentional state for the auditory stimulus includes at least one of an attentional direction and an attentional level;
    learning device.
  3. 　対象者の眼球位置の時系列データからマイクロサッカードの特徴量を抽出特徴量抽出部と、
    　推定モデルを用いて、前記マイクロサッカードの特徴量から前記対象者の聴覚刺激に対する注意状態を推定する推定部と、を含み、
    　前記推定モデルは、学習データに含まれる眼球位置の時系列データと、聴覚刺激に対する注意状態を示す情報とを用いて学習されたモデルである、
    　推定装置。     a feature extraction unit that extracts microsaccade features from time-series data of eyeball positions of a subject;
    an estimating unit that estimates the subject's attentional state to auditory stimuli from the microsaccade features using an estimation model,
    The estimation model is a model learned using time-series data of eyeball positions included in the learning data and information indicating the state of attention to auditory stimuli.
    estimation device.
  4. 　請求項３の推定装置であって、
    　前記聴覚刺激に対する注意状態は、注意方向および注意レベルの少なくとも一方を含む、
    　推定装置。     The estimating device of claim 3,
    the attentional state for the auditory stimulus includes at least one of an attentional direction and an attentional level;
    estimation device.
  5. 　学習データに含まれる眼球位置の時系列データからマイクロサッカードの特徴量を抽出特徴量抽出ステップと、
    　前記マイクロサッカードの特徴量と、学習データに含まれる聴覚刺激に対する注意状態を示す情報とを用いて、推定モデルを学習する学習ステップと、を含み、
    　前記推定モデルは、聴覚刺激に対する、発生したマイクロサッカードの方向から、対象者の聴覚刺激に対する注意状態を推定するモデルである、
    　学習方法。     a feature extraction step of extracting microsaccade features from time-series data of eyeball positions included in learning data;
    a learning step of learning an estimation model using the feature quantity of the microsaccade and information indicating the state of attention to the auditory stimulus contained in the learning data,
    The estimation model is a model that estimates the attentional state of the subject for auditory stimuli from the direction of the generated microsaccade for auditory stimuli.
    learning method.
  6. 　対象者の眼球位置の時系列データからマイクロサッカードの特徴量を抽出特徴量抽出ステップと、
    　推定モデルを用いて、前記マイクロサッカードの特徴量から前記対象者の聴覚刺激に対する注意状態を推定する推定ステップと、を含み、
    　前記推定モデルは、学習データに含まれる眼球位置の時系列データと、聴覚刺激に対する注意状態を示す情報とを用いて学習されたモデルである、
    　推定方法。     a feature extraction step of extracting a microsaccade feature from time-series data of eyeball positions of a subject;
    an estimation step of estimating the subject's attentional state to auditory stimuli from the feature quantity of the microsaccade using an estimation model,
    The estimation model is a model learned using time-series data of eyeball positions included in the learning data and information indicating the state of attention to auditory stimuli.
    estimation method.
  7. 　請求項１もしくは請求項２の学習装置、または、請求項３もしくは請求項４の推定装置としてコンピュータを機能させるためのプログラム。 A program for causing a computer to function as the learning device of claim 1 or claim 2 or the estimation device of claim 3 or claim 4.
PCT/JP2021/035019 2021-09-24 2021-09-24 Learning device, estimation device, learning method, estimation method, and program WO2023047519A1 (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
PCT/JP2021/035019 WO2023047519A1 (en) 2021-09-24 2021-09-24 Learning device, estimation device, learning method, estimation method, and program

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
PCT/JP2021/035019 WO2023047519A1 (en) 2021-09-24 2021-09-24 Learning device, estimation device, learning method, estimation method, and program

Publications (1)

Publication Number Publication Date
WO2023047519A1 true WO2023047519A1 (en) 2023-03-30

Family

ID=85719380

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/JP2021/035019 WO2023047519A1 (en) 2021-09-24 2021-09-24 Learning device, estimation device, learning method, estimation method, and program

Country Status (1)

Country Link
WO (1) WO2023047519A1 (en)

Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20180116507A1 (en) * 2011-11-22 2018-05-03 Dignity Health System and method for using microsaccade dynamics to measure attentional response to a stimulus
JP2019111291A (en) * 2017-12-26 2019-07-11 日本電信電話株式会社 Attention level estimation device, attention level estimation model learning device, attention level estimation method, attention level estimation model learning method, and program
JP2020000266A (en) * 2018-06-25 2020-01-09 日本電信電話株式会社 Reflectivity determination device, reflectivity determination method and program
US20200329316A1 (en) * 2019-04-09 2020-10-15 Gn Hearing A/S Hearing devices with eye movement detection

Patent Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20180116507A1 (en) * 2011-11-22 2018-05-03 Dignity Health System and method for using microsaccade dynamics to measure attentional response to a stimulus
JP2019111291A (en) * 2017-12-26 2019-07-11 日本電信電話株式会社 Attention level estimation device, attention level estimation model learning device, attention level estimation method, attention level estimation model learning method, and program
JP2020000266A (en) * 2018-06-25 2020-01-09 日本電信電話株式会社 Reflectivity determination device, reflectivity determination method and program
US20200329316A1 (en) * 2019-04-09 2020-10-15 Gn Hearing A/S Hearing devices with eye movement detection

Non-Patent Citations (3)

* Cited by examiner, † Cited by third party
Title
LIAO, HSIN-I ET AL.: "Evaluating auditory spatial attention through the eyes", LECTURE PROCEEDINGS OF THE 2019 AUTUMN MEETING OF THE ACOUSTICAL SOCIETY OF JAPAN; SHIGA-KEN, JAPAN; SEPTEMBER 4-6, 2019, vol. 2019, 21 August 2019 (2019-08-21) - 6 September 2019 (2019-09-06), pages 1209 - 1210, XP009545079 *
MARTIN ROLFS ; RALF ENGBERT ; REINHOLD KLIEGL: "Crossmodal coupling of oculomotor control and spatial attention in vision and audition", EXPERIMENTAL BRAIN RESEARCH, SPRINGER, BERLIN, DE, vol. 166, no. 3-4, 1 October 2005 (2005-10-01), Berlin, DE , pages 427 - 439, XP019329039, ISSN: 1432-1106, DOI: 10.1007/s00221-005-2382-y *
YAMAGASHI, SHIMPEI ET AL.: "2-P-6 Characteristics of microsaccades during warning sound presentation", PROCEEDINGS OF THE 2018 SPRING MEETING OF THE ACOUSTICAL SOCIETY OF JAPAN; SAITAMA, JAPAN; MARCH 13-15, 2018, vol. 2018, 28 February 2018 (2018-02-28) - 15 March 2018 (2018-03-15), pages 667 - 668, XP009545069 *

Similar Documents

Publication Publication Date Title
Stopczynski et al. The smartphone brain scanner: a portable real-time neuroimaging system
US8808195B2 (en) Eye-tracking method and system for screening human diseases
KR101267637B1 (en) Apparatus for measuring 3D display-induced visual discomfort, Apparatus for extracting 3D visual discomfort-inducing components automatically, and Method thereof
Zhao et al. Rapid ocular responses are modulated by bottom-up-driven auditory salience
US10817057B2 (en) Information processing device, information processing method, and program
JP6755839B2 (en) Exercise performance estimator, method, and program
JP2015132783A (en) Remarkableness estimation device for sound, and method and program thereof
JP6479708B2 (en) Feature amount extraction apparatus, estimation apparatus, method thereof, and program
US11531394B2 (en) Systems and methods for emotional-imaging composer
JP5718494B1 (en) Impression estimation device, method thereof, and program
Ki et al. Visually evoked responses are enhanced when engaging in a video game
Bœkgaard et al. In the twinkling of an eye: Synchronization of EEG and eye tracking based on blink signatures
WO2023047519A1 (en) Learning device, estimation device, learning method, estimation method, and program
JP7214986B2 (en) Reflectivity determination device, reflectivity determination method, and program
CN117271985A (en) Dream reconstruction method and device
JP5718495B1 (en) Impression estimation device, method thereof, and program
KR20210135378A (en) Method for understanding emotion dynamics in daily life and system therefore
JP6904269B2 (en) Auditory attention estimation device, auditory attention estimation method, program
CN109862822A (en) Identification influences the sense organ input of the working memory load of individual
JP5718492B1 (en) Sound saliency estimating apparatus, method and program thereof
JP2019080809A (en) State estimation apparatus
CN111568368B (en) Eyeball movement abnormality detection method, device and equipment
JP2017202048A (en) Feature amount extraction device, estimation device, method for the same and program
KR20200015169A (en) A method for analysing user&#39;s visual function who using vr contents
WO2023181931A1 (en) Storage auxiliary device, storage auxiliary method, and program

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 21958395

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 21958395

Country of ref document: EP

Kind code of ref document: A1