JP4399513B2 - EEG interface system, EEG interface apparatus, method, and computer program - Google Patents

Description

  The present invention relates to an interface (electroencephalogram interface) system capable of operating a device using an electroencephalogram. More specifically, the present invention relates to an electroencephalogram interface system having a function for efficiently selecting an option desired by a user from among a large number of options.

  In recent years, various types of information devices such as a television, a mobile phone, and a PDA (Personal Digital Assistant) have become widespread, and thus the user needs to operate the information device in many scenes in daily life. Usually, as the operation input means, for example, a method of pressing a button, determining by moving a cursor, operating a mouse while watching a screen, and the like are used.

  However, when both hands cannot be used for work other than device operation, such as housework, childcare, and driving a car, it is difficult to input using the operation input means, and device operation may not be realized. For this reason, there is an increasing need for users who want to operate information devices in all situations.

  In response to such needs, input means utilizing a user's biological signal, more specifically, an electroencephalogram interface utilizing an event-related potential of the user's electroencephalogram has been developed. As used herein, “event-related potential” refers to a transient potential fluctuation in the brain that occurs in time relation to an external or internal event. In the electroencephalogram interface, an event-related potential measured from the occurrence timing of an external event is used. For example, it is supposed that a menu option can be selected by using a P300 component of an event-related potential generated for a visual stimulus or the like. This “P300” is a positive component of the event-related potential that appears around 300 milliseconds from the starting point.

  Patent Document 1 discloses an electroencephalogram interface technique for identifying an option that a user wants to select using an event-related potential. The technique described in Patent Document 1 will be described in detail. The waveform of an event-related potential that appears at about 300 milliseconds from the timing when the option is highlighted at random and the timing when the option is highlighted is described in detail. Is used to identify the options that the user wants to select. According to this technique, the user can select the option he / she wants to select even in a situation where both hands are occupied or in a situation where the limbs cannot be moved due to illness or the like. Therefore, an interface such as device operation that meets the above-described needs is realized.

  In order to apply the event-related potential to the interface, visual stimulation such as highlighting or popping up options on the interface screen is necessary. When there are many options (for example, more than a dozen or a few dozen), it takes a very long time to highlight them individually, so it is important to highlight them efficiently.

  FIG. 18 shows an interface screen described in Non-Patent Document 1. 36 characters are displayed in a 6 × 6 matrix. In Non-Patent Document 1, each row and each column is randomly highlighted at a certain time interval, and the above-mentioned event-related potential is used to determine what row and column the user wants to select. To identify. As a result, the number of highlights required 36 times for individual highlighting is reduced to 12 (6 + 6).

  On the other hand, for example, a line-of-sight input interface technique using a line-of-sight detection device as shown in Patent Document 2 has been proposed. In Patent Literature 2, a gaze area on a user interface screen is detected, and options in the gaze area are handled as a selected state. Then, when there is an option in the selected state, if an operation (closing operation) for closing a sufficiently long time by the user is detected, it is determined that the selection of the option has been confirmed. As a result, a signal for confirming the input of information is output and the selection is confirmed.

  In this technique, if the time threshold for detecting the user's closing operation is set to a small value, for example, it is erroneously detected that selection of an option has been confirmed even in an unconscious blink, and information input that is not intended by the user is performed. There is a possibility that. On the other hand, if the time threshold for detecting the user's closing operation is set to a large value, the possibility of false detection regarding blinking is reduced, but the user consciously maintains the closing operation for a long time. Necessary, too much force can cause fatigue around the muscles around the eyes.

  As described above, the line-of-sight input interface forces the user to perform a predetermined action when confirming selection execution of an option, whereas the electroencephalogram interface does not force the user to perform a predetermined action, and the user selects This is very effective in identifying the option you want to do.

JP 2005-34620 A JP-A-10-187334

Emmanuel Donchin, two others, "The Mental Prosthesis: Assessing the Speed of P300-Based Brain-Computer Ionization Ion." 8, no. 2. June 2000

  In the above-described electroencephalogram interface technology, the options to be highlighted, the timing for starting highlighting, and the time interval between highlights are all uniquely determined by the interface system. Therefore, the user who uses the electroencephalogram interface has to wait for the desired option to be highlighted, and has to keep watching the interface screen even if the desired option is highlighted. Therefore, it was not always possible to select options efficiently.

  As a result, the following problems have occurred.

  In the first example, even if the coordinate position of the option displayed on the electroencephalogram interface is clear, the user cannot quickly select the target option. In the case of the 6 × 6 choices in Non-Patent Document 1 described above, if the highlight interval is 350 milliseconds, one selection takes 4.2 seconds (= 350 milliseconds × 12 times). Become. This is a long period of time that makes you feel uncomfortable and frustrated when you assume the operation of equipment used in daily life.

  As a second example, a user who is confused about a selection target and has not yet decided cannot perform a decision-making action smoothly. This is because highlighting a large number of choices for a user who is confused about a selection target is unnecessary and annoying highlight for the user. When an electroencephalogram interface is applied to daily use, it must be able to demonstrate its original functions without making the user feel uncomfortable or frustrated.

  An object of the present invention is to provide a system with an interface using an electroencephalogram to a user who wants to select an option whose display position on the electroencephalogram interface is clear or who is wondering which option should be selected. An object is to efficiently select an option desired by the user from among a large number of options without making the user feel uncomfortable or frustrated.

  An electroencephalogram interface system according to the present invention is used to control the operation of a device using a user's electroencephalogram signal, and the electroencephalogram interface system includes an electroencephalogram measurement unit that measures the electroencephalogram signal of the user; An eye movement measuring unit that measures the eye movement of the user, an output unit that presents options related to the operation of the device on a screen, and a point in time when the rotational angular velocity of the eye movement is equal to or less than a predetermined threshold. It is determined whether or not a predetermined time has elapsed as a starting point, and when the predetermined time has elapsed, an area in the screen that the user is gazing at is specified based on the eye movement, and an option for highlighting is selected. A highlight determination unit to be determined; and the option determined by the highlight determination unit is highlighted, and the timing is selected when the option is highlighted. The event-related potential component included in the electroencephalogram signal as a starting point, and the interface unit that determines the operation of the device based on the identified component, and the process for displaying the screen is started. And a timing adjustment unit that adjusts the highlight start timing based on the eye movements until the screen is displayed.

  The timing adjustment unit has a state in which the amount of change in the eye movement is equal to or greater than a predetermined threshold, and after the screen display, the amount of change in the eye movement is equal to or less than a predetermined threshold after a predetermined time. The highlighting of the options may be started from that point.

  The timing adjustment unit measures a gaze time in each gaze area when the user gazes an area in the screen after the screen is displayed, and an average value of the measurement time is equal to or greater than the predetermined time The highlighting of the option may be started when the amount of change in the eye movement is equal to or less than a predetermined threshold for a longer time than the predetermined time.

  The highlight determination unit may determine an option included in an area in the screen as an option for highlighting.

  The interface unit may highlight the determined option by changing at least one of luminance, hue, and size of the option determined by the highlight determination unit on the screen. .

  The interface unit may change a highlighting method according to the number of the options determined by the highlight determination unit.

  When there are a plurality of the options determined by the highlight determination unit, the plurality of options may be highlighted randomly or sequentially in a predetermined highlight interval.

  The highlight determination unit may not determine the option to be highlighted when there is no option in the area on the screen.

  The electroencephalogram interface system may further include an interval adjustment unit that adjusts a highlight time interval based on the eye movement.

  The interval adjustment unit may adjust the time interval of highlights based on the eye movement from when the process for displaying the screen is started until the screen is displayed.

  The interval adjusting unit has a state in which the amount of change in the eye movement is equal to or greater than a predetermined threshold, and after the screen is displayed, the amount of change in the eye movement is equal to or less than a predetermined threshold after a predetermined time. When this happens, the highlight time interval may be adjusted to be shorter than the set value.

  The interval adjustment unit measures the gaze time in each gaze area when the user is gazes at the area in the screen after the screen is displayed, and the average value of the measurement time is equal to or greater than the predetermined time In such a case, the highlight time interval may be adjusted to be longer than a set value.

  Another electroencephalogram interface system according to the present invention is used to control the operation of a device using a user's electroencephalogram signal, and the electroencephalogram interface system measures the electroencephalogram signal of the user. And an eye movement measurement unit that measures the user's eye movement, an output unit that presents options related to the operation of the device on the screen, and the rotational angular velocity of the eye movement is in a state equal to or less than a previously held threshold. It is determined whether or not a predetermined time has elapsed from the time point, and when the predetermined time has elapsed, an area in the screen that the user is gazing at is specified based on the eye movement and highlighted. A highlight determination unit that determines an option, and highlights the option determined by the highlight determination unit, so that the type in which the option is highlighted Identifying a component of an event-related potential contained in the electroencephalogram signal ing starting, and a interface unit for determining the operation of the device based on the components identified.

  The highlight determination unit may determine an option included in an area in the screen as an option for highlighting.

  The interface unit may highlight the determined option by changing at least one of luminance, hue, and size of the option determined by the highlight determination unit on the screen. .

  The interface unit may change a highlighting method according to the number of the options determined by the highlight determination unit.

  When there are a plurality of the options determined by the highlight determination unit, the plurality of options may be highlighted randomly or sequentially in a predetermined highlight interval.

  The highlight determination unit may not determine the option to be highlighted when there is no option in the area on the screen.

  The electroencephalogram interface apparatus according to the present invention is used in an electroencephalogram interface system that presents options related to the operation of the device on the screen and controls the operation of the device using the user's eye movements and electroencephalogram signals. The electroencephalogram interface system includes an electroencephalogram measurement unit that measures the electroencephalogram signal of the user and an eye movement measurement unit that measures the eye movement of the user. The electroencephalogram interface apparatus determines whether or not a predetermined time has elapsed from a point in time when the rotational angular velocity of the eye movement received from the eye movement measurement unit is equal to or lower than a previously held threshold. When the time has elapsed, an area in the screen on which the user is gazing is specified based on the eye movement, and a highlight determination unit that determines an option to be highlighted, and the highlight determination unit that is determined by the highlight determination unit Highlighting the option, starting from the timing when the option is highlighted, the event-related potential component included in the electroencephalogram signal received from the electroencephalogram measurement unit is identified, and based on the identified component, An interface unit that determines an operation, and a period between the start of processing for displaying the screen and the display of the screen. Based on the eye movement, and a timing adjustment unit that adjusts the start timing of the highlight.

  The method according to the present invention is a method used in an electroencephalogram interface system that controls the operation of a device using a user's electroencephalogram signal, the step of measuring the electroencephalogram signal of the user, and measuring the eye movement of the user. A step, a step of presenting options related to the operation of the device on the screen, and whether or not a predetermined time has elapsed from the time when the rotational angular velocity of the eyeball movement is equal to or lower than a previously held threshold value. Determining a region in the screen that the user is gazing at based on the eye movement when a predetermined time has elapsed, determining an option to be highlighted, and the determined option To identify the event-related potential components contained in the EEG signal, starting from the timing when the option is highlighted A step of determining the operation of the device based on the identified component, and based on the eye movement from when processing for displaying the screen is started until the screen is displayed. And the step of adjusting the start timing of the highlight.

  The method according to the present invention is a method used in an electroencephalogram interface system that controls the operation of a device using a user's electroencephalogram signal, the step of measuring the electroencephalogram signal of the user, and measuring the eye movement of the user. A step, a step of presenting options related to the operation of the device on the screen, and whether or not a predetermined time has elapsed from the time when the rotational angular velocity of the eye movement is equal to or lower than a previously held threshold. Determining a region in the screen that the user is gazing at based on the eye movement when a predetermined time has elapsed, determining an option to be highlighted, and the determined option And the component of the event-related potential contained in the electroencephalogram signal is identified from the timing when the option is highlighted. Comprising the steps of, and determining the operation of the device based on the components identified.

  A computer program according to the present invention is a computer program executed in an electroencephalogram interface apparatus, and the electroencephalogram interface apparatus is incorporated in an electroencephalogram interface system that controls the operation of a device using a user's electroencephalogram signal, The electroencephalogram interface system has an electroencephalogram measurement unit that measures the user's electroencephalogram signal, an eye movement measurement unit that measures the user's eye movement, and an output unit that presents options related to the operation of the device on the screen. The computer program is predetermined for a computer of the electroencephalogram interface apparatus, starting from the time when the rotational angular velocity of the eye movement measured by the eye movement measuring unit is equal to or lower than a pre-stored threshold value. Determining whether or not When the time elapses, a region in the screen on which the user is gazing is specified based on the eye movement, and an option to be highlighted is determined, and the determined option is highlighted, To identify an event-related potential component included in the electroencephalogram signal from the timing when the option is highlighted, to determine the operation of the device based on the identified component, and to display the screen And a step of adjusting a highlight start timing based on the eye movement from when the process is started until the screen is displayed.

  A computer program according to the present invention is a computer program executed in an electroencephalogram interface apparatus, and the electroencephalogram interface apparatus is incorporated in an electroencephalogram interface system that controls the operation of a device using a user's electroencephalogram signal, The electroencephalogram interface system has an electroencephalogram measurement unit that measures the user's electroencephalogram signal, an eye movement measurement unit that measures the user's eye movement, and an output unit that presents options related to the operation of the device on the screen. The computer program is predetermined for a computer of the electroencephalogram interface apparatus, starting from the time when the rotational angular velocity of the eye movement measured by the eye movement measuring unit is equal to or lower than a pre-stored threshold value. Determining whether or not When the time elapses, a region in the screen on which the user is gazing is specified based on the eye movement, and an option to be highlighted is determined, and the determined option is highlighted, A step of identifying a component of an event-related potential included in the electroencephalogram signal starting from a timing when an option is highlighted, and a step of determining an operation of the device based on the identified component are executed.

  According to the electroencephalogram interface apparatus, the electroencephalogram interface apparatus, the method, and the computer program of the present invention, with respect to the highlights required for the electroencephalogram interface, the target option, the timing to start, or the time interval is determined by the user's eye movement. Judgment based on. As a result, in a system equipped with an interface using an electroencephalogram, it is difficult to use the system for a user who wants to select an option whose display position on the electroencephalogram interface is clear or who is wondering which option should be selected. Without feeling frustrated, it is possible to efficiently select the option desired by the user from among a large number of options.

It is a figure which shows the structure and utilization environment of the electroencephalogram interface system. 1 is a diagram showing a functional block configuration of an electroencephalogram interface system 1 according to Embodiment 1. FIG. It is a figure which shows the structure of the eye movement measurement part 13 which measures eye movement by the EOG method. It is a figure which shows the structure of the eye movement measurement part 13 which measures eye movement by a cornea reflection method. (A) is a figure which shows the example of the data structure of the integrated 1st and 2nd calibration information, (b) is a figure which shows the example of the coordinate of the gaze position on a display screen. It is a figure which shows the example of a division | segmentation at the time of dividing | segmenting the interface screen of 6x6 choice shown in FIG. 18 into nine area | regions. It is a figure which shows the example of the data structure of the division information of each interface screen. 7 is a flowchart illustrating a processing procedure of a highlight determination unit 15; 3 is a flowchart showing a processing procedure of an electroencephalogram IF section 14; (A)-(e) is a figure which shows the example when operating the television in the electroencephalogram interface system 1 and watching the program of the channel which the user 10 wants to view. It is a figure which shows the example of the interface screen containing the content display area 111 and the menu display area 110. FIG. (A)-(b) is the gaze position and timing chart figure in the interface screen in Embodiment 2. FIG. It is a figure which shows the structure of the electroencephalogram interface system 1 by Embodiment 2. FIG. 7 is a flowchart illustrating a processing procedure of a highlight determination unit 15; 5 is a flowchart showing a processing procedure of a highlight timing adjustment unit 16; 5 is a flowchart showing a processing procedure of a highlight interval adjustment unit 17; (A) And (b) is a figure which shows the example of a division | segmentation when the division | segmentation unit of an item is changed according to the distance of the user 10 and the output part 11. FIG. It is a figure which shows the example of a screen display at the time of the interface provision by a prior art example.

  Hereinafter, embodiments of an electroencephalogram interface system and an electroencephalogram interface apparatus according to the present invention will be described with reference to the accompanying drawings.

  First, an outline of main features of an electroencephalogram interface system and an electroencephalogram interface apparatus according to the present invention will be described. Thereafter, each embodiment of the electroencephalogram interface apparatus will be described.

  In the future, the present inventors assume that an electroencephalogram interface system will be constructed in an environment in which a wearable electroencephalograph and a wearable display are combined. A user always wears an electroencephalograph and a display, and can use the wearable display to view content and operate a screen. In addition, it is assumed that an electroencephalogram interface system is constructed even in an environment such as a home in which a home television and a wearable electroencephalograph are combined. When watching a TV, the user can wear an electroencephalograph to view content and operate the screen.

  For example, FIG. 1 shows a configuration and usage environment of an electroencephalogram interface system 1 assumed by the inventors of the present application according to the latter example. This electroencephalogram interface system 1 is illustrated corresponding to the system configuration of Embodiment 1 described later.

  The electroencephalogram interface system 1 is a system for providing an interface for operating (controlling) the television 11 using the electroencephalogram signal of the user 10. The electroencephalogram signal of the user 10 is acquired by an electroencephalograph (electroencephalogram measurement unit 12) worn by the user on the head, and transmitted to the electroencephalogram interface apparatus 2 wirelessly or by wire. The eye movement of the user 10 is measured by an eye movement measuring instrument (eye movement measuring unit 13) attached to the head of the user, and transmitted to the electroencephalogram interface apparatus 2 wirelessly or by wire. The electroencephalogram interface device 2 built in the television 11 determines an option to be highlighted, a timing to start, and a time interval from the eye movement, and uses the P300 component of the event-related potential constituting a part of the electroencephalogram. Recognizes the user's intention and performs processing such as channel switching.

(Embodiment 1)
FIG. 2 shows a functional block configuration of the electroencephalogram interface system 1 according to the present embodiment. The electroencephalogram interface system 1 includes an electroencephalogram interface device 2, an output unit 11, an electroencephalogram measurement unit 12, and an eye movement measurement unit 13. The electroencephalogram interface device 2 includes an electroencephalogram interface unit (electroencephalogram IF unit) 14 and a highlight determination unit 15, and is connected to each of the output unit 11, the electroencephalogram measurement unit 12, and the eye movement measurement unit 13 by wire or wirelessly. And transmit and receive signals. The user 10 block is shown for convenience of explanation.

  The output unit 11 outputs a menu to be selected in the content and the electroencephalogram interface to the user 10. Since the television 11 shown in FIG. 1 is a specific example of the output unit, the following description will be made by assigning the reference numeral 11 to the output unit. The output unit 11 corresponds to the display screen when the output content is a moving image or a still image, and the display screen and the speaker are used as the output unit 11 when the output content includes sound. Sometimes.

  The electroencephalogram measurement unit 12 is an electroencephalograph that detects an electroencephalogram signal by measuring a potential change at an electrode attached to the head of the user 10. The electroencephalograph may be a head-mounted electroencephalograph as shown in FIG. It is assumed that the user 10 is wearing an electroencephalograph in advance.

  Electrodes are arranged in the electroencephalogram measurement unit 12 so as to come into contact with a predetermined position of the head when worn on the head of the user 10. The arrangement of the electrodes is, for example, Pz (midline parietal), A1 (earlobe), and the nasal root of the user 10. However, it is sufficient that there are at least two electrodes. For example, potential measurement is possible only with Pz and A1. This electrode position is determined from the reliability of signal measurement and the ease of mounting.

  As a result, the electroencephalogram measurement unit 12 can measure the electroencephalogram of the user 10. The measured electroencephalogram of the user 10 is sampled so as to be processed by a computer and sent to the electroencephalogram interface apparatus 2. In order to reduce the influence of noise mixed in the electroencephalogram, the electroencephalogram measured by the electroencephalogram measurement unit 12 is subjected to, for example, a band-pass filter process of 0.05 to 20 Hz in advance, for example 200 before the interface screen is presented. It is assumed that the baseline is corrected with an average potential of milliseconds.

  Next, the configuration of the two types of eye movement measurement units 13 will be described with reference to FIGS. 3 and 4. Regardless of which configuration is adopted, the eye movement measuring unit 13 may be a head-mounted measuring instrument as shown in FIG. 1, and the user 10 is previously worn before using the electroencephalogram interface system 1. To do.

  FIG. 3 shows the configuration of the eye movement measurement unit 13 that measures eye movement by the EOG method. On the other hand, FIG. 4 shows a configuration of the eye movement measuring unit 13 that measures the eye movement by the corneal reflection method. The eye movement measuring unit 13 shown in FIG. 3 measures the eye movement by measuring the corneal retinal potential. On the other hand, the eye movement measurement unit 13 shown in FIG. 4 shoots the eyeball by irradiating the eyeball with near infrared rays, and measures the position of the reflected image (corneal reflection image) of the light source on the pupil and the corneal surface on the captured image. To measure eye movements.

  As will be described later, the eye movement measurement unit 13 shown in FIGS. 3 and 4 specifies the rotational angular velocity and the gaze position of the eyeball. This means that the eye movement is defined by a physical change amount (rotation angle of the eye movement per unit time) as a rotation angular velocity when the eye moves, and a change amount of the line of sight.

  Hereinafter, how to measure the eye movement using the EOG method and the corneal reflection method will be described with reference to FIGS. 3 and 4. However, these methods are only given as examples. Other measurement methods can also be used.

  The eye movement measurement unit 13 illustrated in FIG. 3 includes a plurality of electrodes 41, a potential detection unit 42, a conversion unit 43, and a calibration information storage unit 44.

  The plurality of electrodes 41 are attached around the eyes. The potential detector 42 measures the corneal retinal potential obtained via the plurality of electrodes 41.

  The calibration information storage unit 44 stores information (first calibration information) indicating the correspondence between the corneal retinal potential and the rotation angle of the eyeball. The first calibration information is obtained by utilizing the property that the cornea of the eyeball is positively charged with respect to the retina, and is stored in advance in the calibration information storage unit 44. The calibration information storage unit 44 also stores information (second calibration information) indicating the relationship between the rotation angle of the eyeball and the gaze position of the user 10 on the display screen. The second calibration information is also stored in the calibration information storage unit 44 in advance.

  The conversion unit 43 refers to the first calibration information based on the measured corneal retinal potential, and specifies the rotation angle and rotation angular velocity of the eyeball. Then, the conversion unit 43 further refers to the second calibration information based on the identified rotation angle, and identifies the gaze position of the user 10 on the display screen.

  FIG. 5A shows an example of the data structure of the integrated first and second calibration information. The exemplified calibration information is configured such that the corneal retinal potential in the horizontal direction and the vertical direction, the rotation angle of the eyeball, and the coordinates of the gaze position on the display screen are associated with each other. The first and second calibration information may be provided independently without being integrated.

  Hereinafter, a method for specifying the rotation angle of the eyeball and the gaze position using the calibration information will be described.

  For example, when the corneal retinal potential changes by +50 μV in the horizontal direction in one second, the conversion unit 43 refers to the calibration information and specifies that the gaze position in the horizontal direction is X1. When +30 μV is changed in the vertical direction, the conversion unit 43 similarly refers to the calibration information and specifies that the gaze position in the vertical direction is Y1. As a result, the gaze position of the user 10 on the display screen is specified as coordinates (X1, Y1). FIG. 5B shows an example of the coordinates of the gaze position on the display screen.

Further, in the above example, the amount of change in the corneal retinal potential in the horizontal direction and the vertical direction is +50 μV and +30 μV, respectively, for 1 second. Therefore, according to the calibration information shown in FIG. 5A, it can be said that the eyeball has moved 5 degrees in the right direction (degrees) and has also moved 5 degrees in the upward direction. Therefore, the conversion unit 43 can specify the rotational angular velocity of the eyeball as (5 ² +5 ² ) ^1/2 ≈7.07 degrees (degrees) / second.

  Note that “eyeball rotation angle” and “gaze position” in the calibration information shown in FIG. 5A depend on the distance from the user 10 to the display screen, the size of the display screen, and the like. Therefore, these values are not necessarily fixed values, and may be changed by the user 10 in the usage environment.

  Reference is now made to FIG. The eye movement measurement unit 13 shown in FIG. 4 includes a near-infrared light source 51, a CCD camera 52, a reflected image position detection unit 53, a conversion unit 54, and a calibration information storage unit 55.

  The near-infrared light source 51 is a near-infrared point light source, and is used to irradiate the eyeball with the near-infrared light. The CCD camera 52 images the eyeball irradiated with near infrared rays. The reflected image position detecting unit 53 recognizes the pupil and the corneal surface based on the captured image of the eyeball, and further detects the position of the reflected image (corneal reflected image) of the light source on the pupil and the corneal surface.

  The calibration information storage unit 55 stores in advance information (third calibration information) indicating the relationship between the position of the reflected image and the rotation angle of the eyeball, and also the rotation angle of the eyeball and the user 10 on the display screen. Information (fourth calibration information) indicating the relationship with the gaze position is stored in advance.

  The conversion unit 54 refers to the third calibration information based on the position of the reflected image, and specifies the rotation angle and rotation angular velocity of the eyeball. Then, based on the obtained rotation angle of the eyeball, the conversion unit 54 refers to the fourth calibration information and identifies the gaze position of the user 10 on the display screen. Since the data structures of the third and fourth calibration information are similar to those in FIG. 5A, specific examples thereof are omitted.

  Refer to FIG. 2 again. The highlight determination unit 15 detects the gaze area of the user 10 on the interface screen from the gaze coordinate position of the user 10 measured by the eye movement measurement unit 13, and highlights the options from the options based on the gaze area Determine. Then, the highlight determination unit 15 determines whether or not a predetermined time (for example, 400 milliseconds) has elapsed from the time when the rotational angular velocity of the eyeball becomes equal to or less than a previously held threshold value. The electroencephalogram IF unit 14 is instructed to start highlighting options existing in the region where the line of sight of the user 10 exists.

  The threshold value determination method for the rotational angular velocity of the eyeball is as follows. According to a conventional document (Japanese Patent Laid-Open No. 2006-204855), eye movement when a human visually recognizes a still image is roughly classified into gaze movement and jumping eye movement. Here, “jumping eye movement” refers to eye movement in a state where the line of sight moves at high speed and only a little information is received from the outside world. It is reported that the jumping eye movement has a short duration of 40 to 50 milliseconds, while the rotational angular velocity exceeds 100 degrees (seconds) / second. Therefore, the eye movement threshold (X1) may be set to 5 degrees per 50 milliseconds so that the rotational angular velocity is 100 degrees / second, for example.

  Hereinafter, the function of the highlight determination unit 15 will be described in detail with reference to FIG.

  First, it is assumed that various interface screens presented to the user 10 are divided into a plurality of areas in advance for each screen. For example, FIG. 6 shows an example of division when the interface screen of 6 × 6 options shown in FIG. 18 is divided into nine areas. However, it is not necessary to display a dividing line that can be visually recognized by the user 10 on the screen.

  Assume that the highlight determination unit 15 holds in advance information regarding how an area is divided on the interface screen.

  FIG. 7 shows an example of the data structure of the division information of each interface screen. The division information of the interface screen includes the number of the target interface screen, the number of each area when the interface screen is divided, the range in the X-axis direction of each area, the range in the Y-axis direction of each area, and the area It consists of a list of choices on the interface screen included.

  Or each interface screen in advance how (in number) split is determined by the reliability and the like of the number and eye movement measurement choices. For example, instead of 9 divisions as shown in FIG. 6, coarser 4 divisions or finer 36 divisions may be used. The former is effective when the accuracy of eye movement measurement is low, and the latter is effective when the accuracy is high.

  Further, the division of the interface screen can be determined by the positional relationship between the output unit (television) 11 and the user 10. Specifically, the electroencephalograph side unit 12 measures the position information (for example, a two-dimensional position) of the user 10 and transmits the position information to the highlight determination unit 15. The highlight determination unit 15 holds the position information of the output unit 11 in advance, and calculates the distance between the user 10 and the output unit 11 based on the position information of the user 10 and the position information of the output unit 11. Then, the screen division unit is determined by comparing the distance and the threshold.

  For example, when the distance between the user 10 and the output unit 11 is large, the highlight determination unit 15 adjusts the division unit so as to be larger than a predetermined division unit. This is because it is difficult for the user 10 to see the items displayed on the output unit 11 in detail. Now, assuming that the division unit of FIG. 6 is a predetermined division unit, the highlight determination unit 15 increases the division unit as shown in FIG. Thereby, even if the user 10 is away from the output unit 11, the items displayed on the output unit 11 can be viewed in detail.

  When the distance between the user 10 and the output unit 11 is small, the highlight determination unit 15 adjusts the division unit so as to be smaller than a predetermined division unit. This is because the distance between the user 10 and the output unit 11 is short, and the user 10 can see the items displayed on the output unit 11 in detail. If it is assumed that the division unit in FIG. 6 is a predetermined division unit, the highlight determination unit 15 reduces the division unit as shown in FIG. As a result, the user 10 can view detailed items displayed on the output unit 11 and divided more.

  As described above, when the distance between the user 10 and the output unit 11 is relatively far, it is considered that the accuracy of the eye movement measurement is lowered, and the division is increased and importance is attached to the identification by the electroencephalogram signal. On the other hand, when the distance between the user 10 and the output unit 11 is considered to be relatively high, the accuracy of eye movement measurement is considered to be high.

  Accordingly, it is possible to more effectively use one that is considered to have high accuracy among discrimination by eye movement and discrimination by an electroencephalogram signal.

  The highlight determination unit 15 receives information on the rotational angular velocity of the eyeball of the user 10 and information on the gaze coordinate position on the display screen from the eye movement measurement unit 13. For example, when the rotational angular velocity of the eyeball becomes equal to or less than the threshold, the highlight determination unit 15 detects the gaze area of the user 10 on the interface screen based on the gaze position of the user 10 and the interface screen division information.

  When the highlight determination unit 15 determines that the rotational angular velocity of the eyeball is not more than a threshold value for a predetermined time or more and the gaze area of the user 10 on the interface screen is the same, at that timing The brain wave IF unit 14 is instructed to start highlighting. At the same time, the highlight determination unit 15 determines zero, one, or a plurality of options included in the region as highlight targets, and transmits information on the highlight target options to the electroencephalogram IF unit 14. Note that 0 options means that there are no options included in the area, and in this case, the highlight determination unit 15 does not determine an option to be highlighted. means.

  According to the above-mentioned Japanese Patent Application Laid-Open No. 2006-204855, there is a report that the time required for gaze movement when a human visually recognizes is approximately 200 milliseconds to 400 milliseconds. Therefore, the predetermined time (T1) until the highlighting is started is, for example, 400 milliseconds. The processing procedure of the highlight determination unit 15 will be described later with reference to the flowchart of FIG.

  The electroencephalogram IF unit 14 presents an interface screen related to device operation to the user 10 via the output unit 11. Then, the electroencephalogram IF section 14 receives the trigger from the highlight determination section 15, highlights the option determined as the highlight target, and extracts the P300 component of the event-related potential of the electroencephalogram measured by the electroencephalogram measurement section 12. Identify.

  When there are a plurality of options determined as highlight targets, the electroencephalogram IF unit 14 may select the one with the maximum maximum amplitude of the electroencephalogram signal in a certain section for each selected option. The one with the maximum average potential may be selected. Alternatively, the one having the maximum correlation coefficient value with the template may be selected.

  If there is one option determined as a highlight target, the electroencephalogram IF unit 14 determines that the maximum amplitude or average potential of the electroencephalogram signal in a certain section when the option is highlighted is greater than or equal to a predetermined threshold value. Alternatively, it may be determined if the value of the correlation coefficient with the template is equal to or greater than a predetermined threshold.

  In the event-related potential research, the same option is generally highlighted N times (for example, 5 times, 10 times, and 20 times), that is, when 4 options are selected as highlight targets, a total of 4 × The highlighting is performed N times, and the addition average for each of the same options is obtained, and then the P300 component is identified. However, the process of the electroencephalogram IF unit 14 in the present embodiment is not limited to such an addition count. The processing procedure of the electroencephalogram IF section 14 will be described later with reference to the flowchart of FIG.

  Next, processing procedures of the highlight determination unit 15 and the electroencephalogram IF unit 14 shown in FIG. 2 will be described with reference to FIG. 10 together with the flowcharts of FIGS. 8 and 9. FIG. 10 shows a display example when the user 10 selects the channel “CH21” of the television that he / she wants to watch from among the 16 channels in the electroencephalogram interface system 1.

  FIG. 8 is a flowchart illustrating a processing procedure of the highlight determination unit 15.

  First, it is assumed that the screen shown in FIG. When conditions described later are satisfied, the electroencephalogram IF section 14 displays the interface screen shown in FIG. 10B via the output section 11. As a result, the electroencephalogram interface is activated in the electroencephalogram interface system 1.

  When the interface screen shown in FIG. 10B is presented, the highlight determination unit 15 starts processing by receiving a screen number pre-assigned to the interface screen from the electroencephalogram IF unit 14. This screen number corresponds to “screen No” at the left end in the division information shown in FIG.

  In step S81, the highlight determination unit 15 determines whether or not the rotational angular velocity of the eyeball received from the eyeball movement measurement unit 13 is equal to or less than the threshold value X1. Only when the threshold value is less than or equal to X1, the subsequent processing is performed.

  In step S82, the highlight determination unit 15 reads the interface screen division information (FIG. 7) held in advance based on the screen number of the interface screen received from the electroencephalogram IF unit 14. As a result, the interface screen shown in FIG. 10B is presented by the electroencephalogram IF section 14 via the output section 11. In the interface screen shown in FIG. 10B, 16 options from the channels CH00 to CH33 of the television are displayed, and the interface screen is divided into four areas A to D.

  In step S83, the highlight determination unit 15 determines the gaze area of the user 10 on the interface screen based on the division information of the interface screen and the gaze position on the display screen of the user 10 received from the eye movement measurement unit 13. Is detected. In the example of FIG. 10B, since the gaze position of the user 10 can be estimated to be in the area C at the lower left of the screen, the gaze area of the user 10 is set as a lower left area C of the screen. In addition, the position where the user 10 is gazing is not always fixed, but varies slightly. Therefore, in FIG.10 (b), the gaze position is shown as the range shown with a dotted line instead of a point.

  In step S84, the highlight determination unit 15 determines whether or not the gaze position of the user 10 is within the same region for a predetermined time (T1) or longer. If they are in the same area, the process proceeds to step S85. If not in the same area, the process returns to step S81. Note that the “area” where the gaze position exists means an area set in the interface screen. Even if the user 10 gazes at one point outside the interface screen for a predetermined time (T1) or longer, the process does not proceed to step S85 but returns to step S81.

  In step S85, the highlight determination unit 15 transmits the list of options to the electroencephalogram IF unit 14 with the options included in the region as the options to be highlighted. The electroencephalogram IF section 14 starts highlighting with this as a trigger. This means that highlighting starts at a timing when a predetermined time (T1) has elapsed. In the example of FIG. 10B, four CH20, 21, 30, and 31 included in the area C at the lower left of the screen are determined as highlight targets.

  Next, FIG. 9 is a flowchart showing a processing procedure of the electroencephalogram IF section 14. In step S <b> 91, the electroencephalogram IF unit 14 presents an interface screen via the output unit 11. For example, when the user 10 is viewing content, a screen before selection (in this case, news) as shown in FIG. 10A is displayed on the television display. At this time, a menu icon 100 described as “menu” is displayed at the lower right of the screen and blinks at a specific frequency.

  When the user 10 looks at the menu 100, a specific frequency component corresponding to the blinking of the icon 100 is superimposed on the electroencephalogram. The electroencephalogram IF unit 14 can determine whether or not the user 10 is looking at the menu icon 100 by identifying the power spectrum of the frequency component of the blinking period in the electroencephalogram signal. If it is determined that the user 10 is viewing the menu icon 100, the electroencephalogram IF unit 14 can activate the electroencephalogram interface. “Activation of an electroencephalogram interface” means that an operation of providing an interface for performing selection or the like using an electroencephalogram is started. When the electroencephalogram interface is activated, an interface screen shown in FIG. 10B is displayed.

  In step S <b> 92, the electroencephalogram IF section 14 transmits the number of the interface screen to the highlight determination section 15 to start the process of the highlight determination section 15 described above.

  In step S <b> 93, the electroencephalogram IF unit 14 receives a list of options to be highlighted together with a highlight start trigger from the highlight determination unit 15. In the example of FIG. 10B, CH20, 21, 30, and 31, which are options in the area C, are described in the list of options to be highlighted.

  In step S94, the electroencephalogram IF unit 14 determines whether or not highlighting of all options to be highlighted has been completed. If not completed, the process proceeds to step S95. If completed, the process proceeds to step S97.

  In step S95, each target option is highlighted sequentially or randomly.

  FIG. 10C shows a state in which the electroencephalogram IF unit 14 randomly highlights each target option via the output unit 11. The interval of highlight switching time at this time is, for example, 350 milliseconds. As shown in the examples of screens (c) -1 to (c) -4 in FIG. 10, the four options belonging to the area C described in the option list are highlighted. Therefore, the process proceeds to step S97 when all the highlights of the four options have been completed.

  In the example of FIG. 10C, CH20, 21, 30, and 31 are highlighted in this order. Note that the highlight may be at least one of changes in brightness, hue, and size of options on the interface screen, and can be selected with a pointer using an auxiliary arrow instead of the highlight or together with the highlight. May be presented. In addition, the highlight order of options may not be random, and may follow a predetermined order such as a channel number order.

  In step S <b> 96, the electroencephalogram IF section 14 acquires the P300 component of the event-related potential starting from the time point when each option is highlighted, among the electroencephalogram signals measured by the electroencephalogram measurement section 12.

  In step S97, the electroencephalogram IF unit 14 identifies the P300 component of the event-related potential for each option acquired in step S96, and determines the option that the user 10 intends to select. FIG. 10D schematically shows event-related potentials starting from the time point when each option is highlighted.

  Now, assume that the user 10 wishes to view CH21. As shown in each of the screen (c) -1 to the screen (c) -4, the electroencephalogram signals 1 to 4 are acquired starting from the time when each option is highlighted. When the user 10 looks at the screen (c) -2 on which CH21 is highlighted, a characteristic positive component appears in the electroencephalogram signal in the vicinity of about 300 milliseconds from the time when CH21 is highlighted. . When the electroencephalogram IF unit 14 identifies the appearance of the P300 component, the channel that the user 10 wanted to select can be selected.

  In step S98, if the number of options determined as highlight targets is zero, or if a characteristic positive component does not appear in the P300 component of the event-related potential, the process of step S92 is executed again. Is done. Otherwise, the process proceeds to step S99.

  In step S99, the electroencephalogram IF unit 14 executes processing for the selected device operation. In the example of FIG. 10E, the brain wave IF unit 14 switches the channel to CH21 (weather forecast), so that the weather forecast program corresponding to CH21 is displayed on the output unit 11.

  According to the configuration and the processing procedure according to the present embodiment, the highlight determination unit 15 detects the gaze area on the interface screen of the user 10 from the eye movement, and highlights from the options based on the gaze area. Determine the target. Then, it is determined whether or not a predetermined time has elapsed from the time when the rotational angular velocity of the eyeball becomes equal to or less than a previously held threshold, and highlighting is started when the predetermined time has elapsed. As a result, in a system having an interface using an electroencephalogram, even if there are a dozen or dozens of options on the interface screen, the user 10 can efficiently select the option desired by the user 10 Can do.

  For example, when 6 × 6 options shown in FIG. 6 exist, the number of highlights can be reduced to four by dividing the gaze area into nine. Therefore, it becomes possible to select an option more quickly.

  The highlight determination unit 15 determines an option to be highlighted, and the electroencephalogram IF unit 14 specifies an option that the user 10 wants to select from the options determined as the highlight target. Can be reduced.

  This will be specifically described below.

  When the highlight determination unit 15 according to the present embodiment does not exist and the option is highlighted only by the electroencephalogram IF unit 14, even when the user 10 does not see the option or when the user 10 does not intend to select the option. , May highlight the subject.

  On the other hand, in this embodiment, the highlight determination unit 15 determines an option to be highlighted, and the electroencephalogram IF unit 14 highlights the option to acquire the event-related potential. The highlight target is determined based on the movement of the eyeball. Since the highlight target is not determined when the user 10 does not see the option, the user 10 always sees the option when the option is highlighted. In other words, if an option is highlighted, it does not happen that the user 10 does not see the option.

  Further, if the user 10 does not look at the option with a will of choice, it is considered that the user will not continue to see one option for a predetermined time (T1) or more, so such an option is determined as a highlight target. There is nothing. Therefore, when the user 10 is not willing to select, highlighting does not occur. Therefore, unnecessary highlights can be reduced.

  Next, a modification of the present embodiment will be described with reference to FIG. FIG. 11 shows an example of an interface screen including a content display area 111 and a menu display area 110. Options to be selected by the user 10 exist only in the menu display area 110.

  In such an interface screen, the gaze area of the user 10 is in the menu display area 110, and the gaze area of the user 10 is the same as the gaze area of the user 10 for a predetermined time (T1) or more from the time when the rotational angular velocity of the eyeball becomes the threshold value or less. Highlighting only needs to be started if it is at 110. Accordingly, unnecessary highlighting can be suppressed when the user 10 is viewing content, and highlighting can be started only when the user 10 desires a device operation such as a menu change.

(Embodiment 2)
In the electroencephalogram interface system 1 according to the first embodiment, based on the eye movements of the user 10, with respect to the highlights required for the electroencephalogram interface, the target option and the timing to start are determined. In the first embodiment, the predetermined time and the time interval between highlights from when the rotational angular velocity of the eyeball of the user 10 becomes equal to or less than the threshold value until the highlight is started are constant values.

  However, it is still preferable if flexible and more efficient selection can be performed according to the user's state. Specifically, when the user 10 knows in advance the position of the option displayed on the electroencephalogram interface screen by the user 10 using the electroencephalogram interface system 1 many times, the user 10 It can be said that it is still preferable if unnecessary highlights can be suppressed in order to select an option more quickly, and conversely to allow the user 10 who is wondering about the selection target to make a decision smoothly.

  Here, the eye movement of the user 10 with respect to the above-described state of the user 10 will be described with reference to FIG.

  (A) -1 and (b) -1 in FIG. 12 show changes in the gaze position of the user 10 and changes in the time when the user 10 knows the position of the option in advance.

  FIG. 12A-1 is an interface screen including program guide items (programs A to D) and device operation menu items. The items in the device operation menu include up / down / left / right moving direction icons for moving the program guide, “search” item icons for searching for programs such as the program guide, and programs existing in the program guide based on the viewing history of the user 10. A “recommended” item icon is displayed to display a recommended program. The arrow indicates the movement of the gaze position of the user 10 from the point A to the point B.

  The interface screen is divided into a part of items that are sequentially updated, such as a program guide, and a part where the same items are always presented at the same place, such as a device operation menu. However, the device operation menu is not limited to items that are always presented at the same place, but is updated more than items that are updated sequentially (eg, every hour or every day) such as a program guide. It is good also as an item with low frequency.

  In the present embodiment, the electroencephalogram interface screen is configured so that one item (device operation menu item) in which the same item is always presented at the same place in one divided region is one. As a result, regarding the selection of the device operation menu, the electroencephalogram IF unit 14 uses only the event-related potential to confirm whether or not there is an intention to select the option, so that the option can be selected more quickly than when there are a plurality of highlight targets. Can be determined.

  FIG. 12 (b) -1 schematically shows a time change of the point in FIG. 12 (a) -1, that is, the gaze position of the user 10. In FIG. 12B-1, time tα is time when processing for displaying the interface screen is started, and tβ is time when the interface screen is presented. Specifically, during the period from time tα to time tβ, from the screen before selection as shown in FIG. 10A to the screen as shown in FIG. 10B (screen on which items related to the operation of the device are displayed). ) Shows the period until the display is switched. That is, the interface screen of FIG. 12A is not yet displayed between time tα and time tβ. Therefore, the user 10 is in a state where he / she does not know what items are displayed on the interface screen.

  Here, the time tα may be a predetermined time before the time when the process for displaying the interface screen is started.

  When the user 10 uses the electroencephalogram interface many times and gets used to it, even before the interface screen as shown in FIG. It is thought that the position of the upper item is memorized. It is desired that the user 10 accustomed to using the electroencephalogram interface storing such items can be selected quickly.

  In the present embodiment, the user 10 can quickly select the next desired option by observing the rotational angular velocity of the eyeball at the time (time tα to time tβ) before the interface screen is presented. Details will be described below.

  It is assumed that the user 10 stores the item displayed on the interface screen and the coordinates of the item, and clearly knows the coordinate position of the target option on the electroencephalogram interface. At this time, the rotational angular velocity of the eyeball of the user 10 is determined by performing a jumping eye movement before the interface screen is presented (that is, between the time tα and the time tβ). become. This is because if the coordinate position of the target option on the interface screen is stored, the gaze position of the user 10 can be moved to a region where the target option is present before the interface screen is presented.

  For example, as understood from FIG. 12B-1, the eyeball of the user 10 performs a jumping eye movement before the time tβ when the interface screen is presented, and “search” is performed from the region where the gaze position is “program C”. To the area. Then, after the time tβ presented on the interface screen, it remains in the “search” area. It can be determined that the user 10 who has stored the coordinate position of “search” on the interface screen intends to select “search” before the interface screen is presented.

  Here, the “jumping eye movement” indicates an eye movement when the presented item is moved. The jumping eye movement and the eye movement when reading the display presented on the interface screen can be discriminated by the difference in the angular velocity of the eye movement. That is, it is possible to set an angular velocity of eye movement that can distinguish them.

  Subsequently, the eye movement of the user 10 who is wondering about the selection target is examined.

  It is considered that the eye movement of the user 10 who is wondering about the selection target will increase the gaze time at each gaze position after the interface screen is presented. For example, when the name of a program to be selected is clear, it is possible to determine whether or not the program name is immediately after observing the program name in the program table. However, if the program to be selected is not decided and the user is at a loss, the gaze time becomes longer by considering whether or not to select at each gaze position.

  FIGS. 12 (a) -2 and (b) -2 show changes in the gaze position of the user 10 and changes over time when the user 10 is confused about the selection target.

  In this example, the target option is not determined, and therefore the coordinate position of the target option on the electroencephalogram interface is not determined. Therefore, at the time before the interface screen is presented (from time tα to time tβ), no jumping eye movement is seen, and after the interface screen is presented (after time tβ), the gaze position of the user 10 is “program A”, “Program C”, “Program B”, “Recommendation”, and “Program D” move to each other, and the gaze time of each is longer.

  In the electroencephalogram interface system according to the present embodiment, a predetermined time from when the eye movement of the user 10 becomes equal to or less than a threshold value to start highlighting or a time interval of highlighting is adjusted based on the eye movement of the user 10.

  More specifically, during the period from when the interface screen is activated to when it is presented, the rotational angular velocity of the eyeball is equal to or greater than a previously held threshold (X1), and the initial time after the interface screen is presented (predetermined time) Assume that the rotational angular velocity of the eye movement is equal to or less than the threshold (X1) for a shorter time (T2) than T1).

  At this time, it is determined that the user 10 has a clear selection target and stores the coordinate position on the electroencephalogram interface. Then, highlighting starts from the time point. Therefore, by observing the rotational angular velocity of the eyeball before the interface screen is presented, the user 10 storing the coordinate position on the electroencephalogram interface can select more quickly. The highlight time interval may be adjusted to be shorter than the initial setting value (Ta) (Tb).

  In this way, by shortening the time until the highlight is started and the time interval between highlights, the user 10 can be made to select the desired option more quickly.

  On the other hand, when the average value of the gaze time at each gaze position after the interface screen is presented is equal to or longer than the predetermined time (T1), it is determined that the user 10 is confused about the selection target. Then, the time point when the rotational angular velocity of the eyeball is longer than the predetermined time (T1) (T3) and below the threshold value (X1) held in advance is determined as the highlight start timing, and the highlight time interval is also initialized. Adjust longer than the set value (Ta) (Tc). Thus, by making the time until highlight start and the time interval of highlight longer than usual, it becomes possible to suppress unnecessary highlights for the user 10 described above.

  Eric W., a conventional document. Sellers, “A P300 event-related potential brain-computer interface (BCI): The effects of matrix size and Y2O6, PS2 According to the report, an example of an electroencephalogram interface when the time intervals of highlights are 175 milliseconds, 350 milliseconds, and 1.5 seconds has been reported.

  Therefore, in the present embodiment, for example, the initial setting value (Ta) of the highlight time interval is 350 milliseconds, when adjusting short (Tb) is 175 milliseconds, and when adjusting long (Tc) is 1.5 seconds. did.

  In addition, Japanese Patent Application Laid-Open No. 2006-204855, which is a conventional document, describes that the time required for gaze movement when a human performs visual recognition is approximately 200 milliseconds to 400 milliseconds. Therefore, in the present embodiment, the initial setting value (T1) for a predetermined time until the highlight is started is 400 milliseconds, when adjusting short (T2) is 200 milliseconds, and when adjusting long (T3), there is a margin. Take it for 800 milliseconds.

  FIG. 13 shows a configuration of the electroencephalogram interface system 1 according to the present embodiment. The electroencephalogram interface system 1 according to the present embodiment includes an electroencephalogram interface apparatus 20 different from the electroencephalogram interface apparatus 2 according to the first embodiment.

  The electroencephalogram interface apparatus 20 is configured by adding a highlight timing adjustment unit 16 and a highlight interval adjustment unit 17 to the electroencephalogram interface apparatus 2 of the first embodiment. The highlight timing adjustment unit 16 and the highlight interval adjustment unit 17 may be simplified and referred to as “timing adjustment unit” and “interval adjustment unit”, respectively.

  As described above, the highlight timing adjustment unit 16 adjusts a predetermined time from when the rotational angular velocity of the eyeball is equal to or less than the threshold value (X1) held in advance until the highlight is started. The highlight interval adjustment unit 17 adjusts the highlight time interval.

  FIG. 14 shows a processing procedure of the highlight determination unit 15. The difference from the flowchart (FIG. 8) of the highlight determination unit 15 in the first embodiment is that a highlight timing adjustment process in step S134 and a highlight interval adjustment unit in step S135 are added, and an electroencephalogram in step S137. This is that a highlight interval is added to the information to be transmitted to the IF unit 14. Hereinafter, these processes will be described.

  In step S131, it is determined whether or not the rotational angular velocity of the eyeball received from the eyeball movement measurement unit 13 is equal to or less than the threshold value X1. Only when the threshold value is less than or equal to X1, the subsequent processing is performed.

  In step S132, the highlight determination unit 15 reads the interface screen division information (FIG. 7) held in advance based on the screen number of the interface screen received from the electroencephalogram IF unit 14.

  In step S133, the highlight determination unit 15 determines the gaze area of the user 10 on the interface screen based on the division information of the interface screen and the gaze position on the display screen of the user 10 received from the eye movement measurement unit 13. Is detected.

  In step S134, the highlight timing adjustment unit 16 performs a timing adjustment process for starting highlighting. Details of this processing will be described later with reference to FIG.

  In step S135, the highlight interval adjustment unit 17 performs a highlight time interval adjustment process. Details of this processing will be described later with reference to FIG.

  In step S136, the highlight determination unit 15 determines whether or not the gaze position of the user 10 is within the same region for a predetermined time (T1) or longer. If it is in the same area, the process proceeds to step S137, and if it is not in the same area, the process returns to step S131. Note that the predetermined time T until the highlighting is started is adjusted in step S134.

  In step S137, the highlight determination unit 15 assumes that the timing is the timing to start highlighting, and uses the options included in the area as the highlight target options, and the highlight adjusted in step S135. The write interval is transmitted to the electroencephalogram IF unit 14. The electroencephalogram IF section 14 starts highlighting with this as a trigger.

  FIG. 15 shows a processing procedure of the highlight timing adjustment unit 16. With the processing shown in FIG. 15, a predetermined time until highlighting is started is set to any one of T1, T2, and T3.

  It is assumed that predetermined times T1, T2, and T3 until the highlight shown below are started are 400 milliseconds, 200 milliseconds, and 800 milliseconds, respectively. That is, it is assumed that T2 <T1 <T3.

  In step S141, the highlight timing adjustment unit 16 sets the threshold value that the rotational angular velocity of the eyeball holds in advance between tα and tβ in FIG. 12B-1, that is, from when the interface screen is activated until it is presented. (X1) It is discriminated whether or not it is equal to or higher. When it becomes more than threshold value X1, it progresses to Step S144, and when that is not right, it progresses to Step S142.

  In step S142, the highlight timing adjustment unit 16 calculates the average value of the gaze time at each gaze position after the interface screen is presented.

  In step S143, the highlight timing adjustment unit 16 determines whether or not the average value calculated in step S142 is equal to or longer than a predetermined time (T1). If it is equal to or longer than the predetermined time (T1), the process proceeds to step S145; otherwise, the process proceeds to step S146.

  In step S144, the highlight timing adjustment unit 16 adjusts a predetermined time until the highlight is started to T2. That is, the adjustment is made shorter than the initial set value.

  In step S145, the highlight timing adjustment unit 16 adjusts the predetermined time until the highlighting is started to T3. That is, the adjustment is made longer than the initial set value.

  In step S146, the highlight timing adjustment unit 16 adjusts the predetermined time until the highlight is started to T1. That is, it is adjusted to the same value as the initial setting value.

  FIG. 16 shows a processing procedure of the highlight interval adjustment unit 17. With the processing shown in FIG. 16, the highlight time interval is set to one of Ta, Tb, and Tc.

  Note that the time intervals Ta, Tb, and Tc of highlights shown below are 350 milliseconds, 175 milliseconds, and 1.5 seconds, respectively. That is, it is assumed that Tb <Ta <Tc.

  From step S171 to step S173, the highlight interval adjusting unit 17 executes the same processing as that of steps S141 to S143 shown in FIG.

  In step S174, the highlight interval adjustment unit 17 adjusts the highlight time interval to Tb. That is, the adjustment is made shorter than the initial set value.

  In step S175, the highlight interval adjustment unit 17 adjusts the highlight time interval to Tc. That is, the adjustment is made longer than the initial set value.

  In step S176, the highlight interval adjusting unit 17 adjusts the highlight time interval to Ta. That is, it is adjusted to the same value as the initial setting value.

  According to the configuration and processing procedure according to the present embodiment, in a system including an interface using an electroencephalogram, a predetermined time until highlighting is started and a time interval between highlights are adjusted based on the eye movement of the user 10. To do. Therefore, when the user 10 clearly knows the coordinate position on the electroencephalogram interface of the option to be selected, the user 10 can be made to select the option more quickly. Conversely, unnecessary highlights can be suppressed in order to cause the user 10 who is confused about the selection target to perform a smooth decision-making action.

  In the present embodiment, the operation of the electroencephalogram interface apparatus 20 has been described with reference to the screen examples shown in (a) -1 and (a) -2 of FIG. However, this does not mean that it is not applicable to the screen example of FIG. 10B referred to in the description of the first embodiment. The arrow from point A to point B shown in (a) -1 of FIG. 12 can be similarly applied to the screen example of FIG. However, at this time, the point A is a gaze position at the time when the process for displaying the interface screen is started when the user 10 gazes at the menu icon 100 in FIG. It is a gaze position at the time before the interface screen of b) is presented.

  In any of the above-described embodiments, the processing described using the flowchart can be realized as a program executed by a computer. Such a computer program is recorded on a recording medium such as a CD-ROM and distributed as a product to the market or transmitted through an electric communication line such as the Internet.

  For example, the electroencephalogram interface apparatus 2 shown in FIG. 2 is realized as a general-purpose processor (semiconductor circuit) that executes a computer program. Alternatively, it is realized as a dedicated processor in which such a computer program and a processor are integrated.

  The eye movement measurement unit 13 and the electroencephalogram measurement unit 12 can also be realized as a general-purpose processor (semiconductor circuit) or a dedicated processor (semiconductor circuit) that executes a computer program. For example, the process in which the eye movement measurement unit 13 calculates the rotation angle and the rotation angular velocity of the eyeball based on the acquired corneal retinal potential can be realized as a program. In addition, the electroencephalogram measurement unit 12 can implement the band pass filter process and the baseline correction process as a program.

  According to the electroencephalogram interface apparatus and the electroencephalogram interface system in which the apparatus is incorporated according to the present invention, with respect to the highlights required for the electroencephalogram interface, the target options, the timing to start, and the time interval are determined by the user's eye movement Determine based on. This is useful for improving the operability of information equipment and audio-visual equipment equipped with equipment operation interfaces using brain waves.

DESCRIPTION OF SYMBOLS 1 EEG interface system 2, 20 EEG interface apparatus 11 Output part 12 EEG measurement part 13 Eye movement measurement part 14 EEG IF part 15 Highlight determination part 16 Highlight timing adjustment part 17 Highlight interval adjustment part

Claims (15)

An electroencephalogram interface system that controls the operation of a device using a user's electroencephalogram signal,
An electroencephalogram measurement unit for measuring the electroencephalogram signal of the user;
An eye movement measuring unit that measures the eye movement of the user;
An output unit that presents on the screen options related to the operation of the device;
It is determined whether or not a predetermined time has elapsed from the time when the rotational angular velocity of the eyeball movement is equal to or less than a predetermined threshold, and when the predetermined time has elapsed, the user is gazing A region in the screen is identified based on the eye movement, and a highlight determination unit that determines an option to highlight,
Highlighting the option determined by the highlight determination unit, identifying an event-related potential component included in the electroencephalogram signal from the timing when the option is highlighted, and based on the identified component An interface unit that determines the operation of the device;
An electroencephalogram interface system comprising: a timing adjustment unit that adjusts a highlight start timing based on the eye movement from when the processing for displaying the screen is started until the screen is displayed.   The timing adjustment unit has a state in which the amount of change in the eye movement is equal to or greater than a predetermined threshold, and after the screen display, the amount of change in the eye movement is equal to or less than a predetermined threshold for a time shorter than the predetermined time. The electroencephalogram interface system according to claim 1, wherein highlighting of the option is started from a point in time.   The timing adjustment unit measures a gaze time in each gaze area when the user gazes an area in the screen after the screen is displayed, and an average value of the measurement time is equal to or greater than the predetermined time The highlighting of the option is started when the amount of change of the eye movement is equal to or less than a predetermined threshold value for a time longer than the predetermined time. The described electroencephalogram interface system.   The electroencephalogram interface system according to claim 1, wherein the highlight determination unit determines an option included in an area in the screen as an option to be highlighted.   The interface unit highlights the determined option by changing at least one of luminance, hue, and size of the option determined by the highlight determination unit on the screen. 2. The electroencephalogram interface system according to 1.   The electroencephalogram interface system according to claim 1, wherein the interface unit changes a method of highlighting according to the number of the options determined by the highlight determination unit.   7. The electroencephalogram interface system according to claim 6, wherein when there are a plurality of the options determined by the highlight determination unit, the plurality of options are highlighted randomly or sequentially in a predetermined highlight interval.   2. The electroencephalogram interface system according to claim 1, wherein the highlight determination unit does not determine an option to be highlighted when there is no option in an area in the screen.   The electroencephalogram interface system according to claim 1, further comprising an interval adjustment unit that adjusts a time interval of highlights based on the eye movement. The gap adjusting part, based on the eye movement until the screen is displayed after the process is started to display the screen, to adjust the time interval of highlighting, claim 9 EEG interface system.   The interval adjusting unit has a state in which the amount of change in the eye movement is equal to or greater than a predetermined threshold, and after the screen display, the amount of change in the eye movement is equal to or less than a predetermined threshold for a time shorter than the predetermined time. The electroencephalogram interface system according to claim 10, wherein the highlight time interval is adjusted to be shorter than a set value. The interval adjustment unit measures the gaze time in each gaze area when the user is gazes at the area in the screen after the screen is displayed, and the average value of the measurement time is equal to or greater than the predetermined time The electroencephalogram interface system according to claim 9 , wherein the time interval of the highlight is adjusted to be longer than a set value in the case of becoming. An electroencephalogram interface apparatus used in an electroencephalogram interface system that presents options related to the operation of the device on a screen and controls the operation of the device using a user's eye movement and electroencephalogram signal,
The electroencephalogram interface system has an electroencephalogram measurement unit that measures the electroencephalogram signal of the user, and an eye movement measurement unit that measures the eye movement of the user,
It is determined whether or not a predetermined time has elapsed from a time point when the rotational angular velocity of the eye movement received from the eye movement measurement unit is equal to or lower than a previously held threshold, and when the predetermined time has elapsed, A region in the screen that the user is gazing at is specified based on the eye movement, and a highlight determination unit that determines an option to be highlighted,
Highlighting the option determined by the highlight determination unit, identifying the event-related potential component included in the electroencephalogram signal received from the electroencephalogram signal from the timing when the option is highlighted, An interface unit for determining the operation of the device based on the identified component;
An electroencephalogram interface device comprising: a timing adjustment unit that adjusts a highlight start timing based on the eye movement from when processing for displaying the screen is started to when the screen is displayed. A method used in an electroencephalogram interface system that controls the operation of a device using a user's electroencephalogram signal,
Measuring the user's electroencephalogram signal;
Measuring the eye movement of the user;
Presenting on-screen options related to the operation of the device;
Determining whether a predetermined time has elapsed from the time when the rotational angular velocity of the eye movement is in a state equal to or lower than a previously held threshold;
When a predetermined time has elapsed, identifying an area in the screen that the user is gazing based on the eye movement, and determining an option to highlight;
Highlighting the determined option determined, and identifying an event-related potential component included in the electroencephalogram signal starting from the timing when the option is highlighted;
Determining the operation of the device based on the identified component;
Adjusting the highlight start timing based on the eye movements from when the process for displaying the screen is started until the screen is displayed. A computer program executed in an electroencephalogram interface apparatus,
The electroencephalogram interface device is incorporated in an electroencephalogram interface system that controls the operation of the device using a user's electroencephalogram signal,
The electroencephalogram interface system comprises:
An electroencephalogram measurement unit for measuring the electroencephalogram signal of the user;
An eye movement measuring unit that measures the eye movement of the user;
An output unit that presents on the screen options related to the operation of the device,
The computer program is for the computer of the electroencephalogram interface device.
A step of determining whether or not a predetermined time has elapsed from the time when the rotational angular velocity of the eye movement measured by the eye movement measurement unit is equal to or less than a threshold value held in advance;
When a predetermined time has elapsed, identifying an area in the screen that the user is gazing based on the eye movement, and determining an option to highlight;
Highlighting the determined option determined, and identifying an event-related potential component included in the electroencephalogram signal starting from the timing when the option is highlighted;
Determining the operation of the device based on the identified component;
And a step of adjusting a highlight start timing based on the eye movement from when the process for displaying the screen is started until the screen is displayed.

Images