WO2018179346A1 - Système de manipulation, serveur, procédé de commande dans un système de manipulation, et programme - Google Patents

Système de manipulation, serveur, procédé de commande dans un système de manipulation, et programme Download PDF

Info

Publication number
WO2018179346A1
WO2018179346A1 PCT/JP2017/013623 JP2017013623W WO2018179346A1 WO 2018179346 A1 WO2018179346 A1 WO 2018179346A1 JP 2017013623 W JP2017013623 W JP 2017013623W WO 2018179346 A1 WO2018179346 A1 WO 2018179346A1
Authority
WO
WIPO (PCT)
Prior art keywords
data
electroencephalogram
command
database
operation system
Prior art date
Application number
PCT/JP2017/013623
Other languages
English (en)
Japanese (ja)
Inventor
哲也 田辺
Original Assignee
本田技研工業株式会社
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 本田技研工業株式会社 filed Critical 本田技研工業株式会社
Priority to PCT/JP2017/013623 priority Critical patent/WO2018179346A1/fr
Publication of WO2018179346A1 publication Critical patent/WO2018179346A1/fr

Links

Images

Classifications

    • A—HUMAN NECESSITIES
    • A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
    • A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
    • A61B5/00—Measuring for diagnostic purposes; Identification of persons
    • A61B5/24—Detecting, measuring or recording bioelectric or biomagnetic signals of the body or parts thereof
    • A61B5/316—Modalities, i.e. specific diagnostic methods
    • A61B5/369—Electroencephalography [EEG]
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer

Definitions

  • the present invention relates to an operation system using a brain wave, a server, a control method in the operation system, and a program.
  • Patent Document 1 describes that a load on an operator's work is determined based on information detected by a biological information sensor, and a work environment is controlled based on the determination result.
  • Patent Document 2 describes changing a control parameter of a two-wheeled vehicle in accordance with a human eye movement.
  • Patent Document 3 describes that a time difference between an acquired electroencephalogram signal and an electrocardiogram signal is calculated and a driver's psychological state is determined based on the calculated time difference.
  • Patent Document 4 describes that a computer or a game machine is operated using an electroencephalogram.
  • Systems and devices that use human biological information can also be applied to vehicles such as motorcycles.
  • a vehicle such as a two-wheeled vehicle, for example, it is extremely difficult for a passenger who is a driver to operate the device while traveling. It is conceivable to apply a system or apparatus using human biological information to the above-described case while traveling to enable operation of the apparatus while traveling.
  • Patent Document 1 describes a helmet equipped with an electroencephalogram sensor and a blink sensor, and describes a control configuration in accordance with the driver's fatigue level.
  • Patent Document 2 describes a helmet equipped with an eye camera, and describes a control configuration corresponding to a difference in rider's attention level.
  • Patent Document 3 describes a helmet equipped with an electroencephalogram sensor and an eye camera, and describes a control configuration according to the driver's degree of tension.
  • Patent Document 4 describes that a computer or a game machine is operated using an electroencephalogram as an electrical signal.
  • An object of the present invention is to provide an operation system, a server, a control method in the operation system, and a program that allow an occupant to operate the apparatus without causing an operation operation of the apparatus using brain waves.
  • An operation system includes a database for accumulating brain wave data, a generation unit that generates correlation data associated with a command for operating the apparatus based on the brain wave data accumulated in the database, Obtaining means for obtaining electroencephalogram data corresponding to the brain wave signal of the passenger, and sending means for sending the command based on the electroencephalogram data obtained by the obtaining means and the correlation data generated by the generating means; And a control means for operating the apparatus when the command is sent by the sending means.
  • the server includes a database for accumulating brain wave data, a generation means for generating correlation data associated with a command for operating the apparatus based on the brain wave data accumulated in the database, and a vehicle Acquiring means for acquiring the electroencephalogram data corresponding to the brain wave signals of the passengers, and transmitting means for transmitting the command based on the electroencephalogram data acquired by the acquiring means and the correlation data generated by the generating means It is characterized by providing.
  • control method corresponds to a generation step of generating correlation data associated with a command for operating the apparatus based on the electroencephalogram data stored in the database, and an electroencephalogram signal of a vehicle occupant
  • the program according to the present invention corresponds to a generation step of generating correlation data associated with a command for operating the apparatus based on the electroencephalogram data stored in the database, and an electroencephalogram signal of a vehicle occupant.
  • the command is transmitted in the transmission step, When transmitted, the computer is caused to execute a control step of operating the apparatus.
  • FIG. 1 is a diagram showing an overall configuration of an operation system using an electroencephalogram in the present embodiment.
  • the operation system according to the present embodiment can transmit an imaging command to a camera provided in the helmet of the passenger based on brain wave data transmitted from the passenger of the moving vehicle.
  • the operation system 100 includes motorcycles 101, 102, 103, radio base stations 104, 105, 106, a network 107, and a server 108.
  • the motorcycle 101 is a so-called saddle riding type vehicle, and includes a three-wheeled vehicle in addition to a two-wheeled vehicle.
  • Wireless communication is performed between each of the motorcycles 101 to 103 and the wireless base stations 104 to 106, and communication via the network 107 is performed between the wireless base stations 104 to 106 and the server 108.
  • the network 107 is, for example, the Internet or a dedicated communication network, and may partially include a telephone communication network such as a mobile phone.
  • the wireless base stations 104 to 106 are wireless communication devices that are installed in public facilities such as traffic lights and communicate with the motorcycles 101 to 103, and are configured as wireless base stations of a mobile phone communication network, for example.
  • the radio base stations 104 to 106 are configured in predetermined area units, for example. Each rider of the motorcycles 101 to 103 wears helmets 109, 110, and 111.
  • the motorcycles 101 to 103 are equipped with controllers 112, 113, and 114 for realizing the operation of the operation system 100.
  • the controllers 112 to 114 include the helmets 109 to 111, the radio base stations 104 to 106, and the like. Communicate between.
  • FIG. 1 three motorcycles 101 to 103 and three radio base stations 104 to 106 are shown. However, the number is not limited to the number shown in the figure, and a plurality of motorcycles are one radio base station. Can be supported.
  • FIG. 1 only one server 108 is shown, but it may be configured by a plurality of devices. The server 108 may be configured as a cloud.
  • the controller 112 of the motorcycle 101 can operate as a mobile terminal.
  • the controller 112 may associate, for example, the contractor ID of the rider's mobile terminal with the own device by setting via the user interface screen from the rider.
  • the server 108 may make it possible to use the rider's personal information and service information managed by the mobile phone system.
  • the motorcycle 101, the helmet 109, the controller 112, and the wireless base station 104 will be described as representative examples of the motorcycle 101 to 103, the helmet 109 to 111, the controller 112 to 114, and the wireless base station 104 to 106.
  • FIG. 2 is a diagram showing a communication network in the operation system 100.
  • FIG. 2 conceptually shows that a data communication network is included.
  • communication 201 is performed between the helmet 109 and the controller 112.
  • the communication 201 includes the electroencephalogram data transmitted from the helmet 109 and the imaging command transmitted from the controller 112.
  • communication 202 is performed between the controller 112 and the radio base station 104.
  • the communication 202 includes brain wave data and driving information of the motorcycle 101.
  • the driving information of the motorcycle 101 is speed information of the motorcycle 101, for example.
  • Communication 203 is performed between the wireless base station 104 and the server 108.
  • the radio base station 104 operates to relay between the controller 112 and the server 108, and the communication 203 includes brain wave data and driving information of the motorcycle 101.
  • FIG. 3 is a diagram illustrating an example of a block configuration of the helmet 109.
  • the helmet 109 is configured with a sensor group 304 for acquiring rider's biological information.
  • the sensor group 304 includes an electroencephalogram sensor that detects a rider's electroencephalogram signal.
  • the camera 305 is attached to the helmet 109 and photographs a front subject (for example, a landscape). In the operation system 100, the camera 305 captures a subject in accordance with a capturing command from the controller 112.
  • the eye tracking camera 306 is a camera for acquiring rider's line-of-sight information. Sensor signals from the sensor group 304 and imaging data from the camera 305 and the eye tracking camera 306 are transmitted to the control board 300.
  • the image memory 308 stores imaging data captured by the camera 305 and brain wave data at the time of imaging.
  • the image memory 308 is, for example, an SD card, and is configured so that it can be attached to and removed from the memory I / F 307.
  • the controller I / F 309 operates as an interface for performing communication with the controller 112, and transmits, for example, a sensor signal from the sensor group 304 to the controller 112. Note that communication with the controller 112 via the controller I / F 309 may be performed wirelessly or may be performed by wire.
  • the control board 300 is configured inside the helmet 109, for example.
  • a processor 301, a memory 302, a storage unit 303, a memory I / F 307, and a controller I / F 309 are configured on the control board 30, and are connected to each other via a bus 310 so as to communicate with each other.
  • the processor 301 reads out the program stored in the storage unit 303 to the memory 302 and executes it, and can control each block in the helmet 109 in an integrated manner.
  • FIG. 4 is a diagram illustrating an example of a block configuration of the controller 112.
  • the controller 112 may be configured separately from the controller including the ECU of the motorcycle 101 or may be configured as a part.
  • the display unit 404 is a display panel, for example, and displays a user interface screen such as a photo selection screen described later.
  • the GPS 405 is a GPS oscillator and is used for acquiring position information of the motorcycle 101.
  • the controller I / F 406 operates as an interface for performing communication with the helmet 109.
  • the I / F 407 operates as an interface for performing communication with the radio base station 104.
  • An image memory 308 removed from the helmet 109 can be attached to the memory I / F 408. Accordingly, the processor 401 can display a list of shooting data stored in the image memory 308 on the display unit 404.
  • the 4 further includes a processor 401, a memory 402, and a storage unit 403, and each block is connected to be communicable with each other via a bus 409.
  • the processor 401 reads out the program stored in the storage unit 403 to the memory 402 and executes it, and can comprehensively control each block on the control board.
  • the processor 401 can cooperate with the ECU of the motorcycle 101, and can control each block on the control board in accordance with, for example, the speed information of the motorcycle 101. Control can be requested. Further, the processor 401 can execute an algorithm used as AI (Artificial Intelligence).
  • AI Artificial Intelligence
  • FIG. 5 is a diagram illustrating an example of a block configuration of the radio base station 104.
  • the I / F 504 operates as an interface for performing communication with the controller 112.
  • a network (NW) I / F 505 operates as an interface for communicating with the server 108 via the network 107.
  • Each block shown in FIG. 5 includes a processor 501, a memory 502, and a storage unit 503, and is connected to be communicable with each other via a bus 506.
  • the processor 501 reads the program stored in the storage unit 503 into the memory 502 and executes it, and controls the radio base station 104 in an integrated manner.
  • the processor 501 converts a communication procedure with the controller 112 and a communication procedure with the server 108.
  • FIG. 6 is a diagram illustrating an example of a block configuration of the server 108.
  • the NWI / F 604 operates as an interface for communicating with the radio base station 104 via the network 107.
  • a database (DB) 605 stores brain wave data associated with the captured image data.
  • the DB 605 will be described later.
  • the DB 606 stores captured image data.
  • Each block shown in FIG. 6 includes a processor 601, a memory 602, and a storage unit 603, and is connected to be communicable with each other via a bus 607.
  • the processor 601 reads the program stored in the storage unit 603 into the memory 602 and executes it, and controls the server 108 in an integrated manner. Further, the processor 601 can execute an algorithm used as an AI.
  • DB 605 and DB 606 are shown as separate databases, but may be configured as an integrated database.
  • FIG. 7 is a view showing the appearance of the helmet 109.
  • a plurality of electrodes 701 corresponding to the electroencephalogram sensor 308 are configured from the top of the helmet 109 along the frontal and occipital regions and from the top of the helmet 109 along the temporal region.
  • a 20-channel EEG signal may be detected by 20 electrodes.
  • the electrode 701 detects an electrical signal of a predetermined frequency band component that appears on the scalp according to activity in the brain, for example, emotion, and transmits the detected electrical signal to the control board 704.
  • the control board 704 corresponds to the control board 300 in FIG.
  • the control board 704 converts the potential signal into digital data by an amplifier (not shown) and an AD converter, and transmits the digital data to the controller 112 as brain wave data corresponding to the brain wave signal.
  • the electroencephalogram data is acquired as a set of data corresponding to the plurality of electrodes 701 as, for example, discrete data of time and potential ( ⁇ V) acquired at a predetermined sampling period.
  • the camera 702 corresponds to the camera 305 in FIG. As described with reference to FIG. 3, the camera 702 shoots a subject according to a shooting command from the controller 112 without the rider's instruction and without causing the rider to perform a shooting operation. As shown in FIG. 3, the camera 702 is configured in front of the helmet 109 so as to generally capture the direction of the rider's line of sight.
  • the eye tracking camera 703 corresponds to the eye tracking camera 306 in FIG.
  • the eye tracking camera 306 photographs the periphery of the rider's eyes.
  • other configurations may be used as long as it is possible to acquire rider's line-of-sight information.
  • an eye tracking sensor that touches a portion corresponding to the pad portion of the glasses and acquires line-of-sight information by a change in the eye muscle resistivity may be used.
  • the DB 606 stores shooting data taken by a motorcycle rider who enjoys the service of the operation system 100 with the camera 702 of the helmet 109.
  • each imaging data is associated with brain wave data at the time of imaging and stored in the DB 605. That is, the DBs 605 and 606 are constructed as a big database of combinations of imaging data and electroencephalogram data transmitted from each rider.
  • the processor 601 of the server 108 acquires the electroencephalogram data corresponding to each electrode 701 transmitted from the controller 112 in S611, in order to acquire a continuous potential signal from the discrete data of time and potential, S612 Smoothing is performed at.
  • the processor 601 performs spectrum analysis on the smoothed potential signal.
  • the processor 601 extracts two-dimensional feature values of each frequency band and each electrode 701, and stores the extracted feature values in the DB 605. To do.
  • FIG. 8 is a diagram showing a processing sequence of the operation system 100 in the present embodiment.
  • the controller 112 may transmit the activation information of the helmet 109 and the driving information of the motorcycle 101 to the server 108 via the wireless base station 104.
  • the server 108 receives the information, the server 108 starts providing the operation system 100 service to the motorcycle 101.
  • the wireless base station 104 that relays data is not shown.
  • electroencephalogram data is transmitted from the helmet 109 to the controller 112.
  • imaging data of the eye tracking camera 306 is transmitted as line-of-sight information.
  • the controller 112 transmits driving information of the motorcycle 101 to the server 108 together with the electroencephalogram data and line-of-sight information.
  • the drive information is speed information of the motorcycle 101, for example.
  • the server 108 determines whether to transmit an imaging command based on the electroencephalogram data transmitted from the controller 112.
  • the DBs 605 and 606 are constructed as a big database of a combination of imaging data and brain wave data transmitted from each rider.
  • the imaging data captured by the camera 305 and the electroencephalogram data at the time of imaging are transmitted from each rider as needed, and the DBs 605 and 606 are updated.
  • the camera 305 is not photographed by the rider's photographing operation.
  • the rider selects what he thinks is good from the plurality of photographed data on the photograph selection screen displayed on the display unit 404 of the motorcycle 101. That is, the DBs 605 and 606 store imaging data and brain wave data that each rider thinks are really good.
  • the processor 601 of the server 108 analyzes the feature quantity of the electroencephalogram data stored in the DB 605, learns the similarity of the electroencephalogram data, and acquires the feature quantity representing the similar pattern. Then, the processor 601 holds the acquired feature amount representing the similar pattern in a predetermined storage area of the server 108 as correlation data corresponding to the shooting event.
  • the processor 601 of the server 108 determines whether or not to send the imaging command by comparing the feature quantity of the electroencephalogram data transmitted from the controller 112 with the correlation data. When it is determined that the feature amount of the electroencephalogram data transmitted from the controller 112 is within a predetermined similar range to the correlation data, it is determined that an imaging command is transmitted.
  • the server 108 transmits the shooting command to the controller 112 in S804.
  • the controller 112 transmits a shooting command from the camera 305 of the helmet 109 to the helmet 109.
  • the processor 301 of the helmet 109 performs shooting with the camera 305.
  • the captured imaging data is stored in the image memory 308 together with the electroencephalogram data at the time of imaging.
  • the feature quantities of the electroencephalogram data that each rider feels as good shooting timing are analyzed, the feature quantities that become similar patterns are learned, and the correlation data corresponding to the shooting event is obtained. Save it.
  • the electroencephalogram data newly transmitted to the server 108 and the correlation data are within a predetermined similar range, it is determined that the rider currently feels that the imaging timing is good and an imaging command is transmitted. .
  • the rider can perform shooting at a desired timing without causing a shooting operation.
  • the rider After shooting in S806, for example, it is assumed that the rider gets off the motorcycle 101 at a break point. Therefore, the rider removes the image memory 308 of the helmet 109 from the memory I / F 307 and attaches it to the memory I / F 408 of the controller 112.
  • the processor 401 displays a list of shooting data stored in the image memory 308 on the display unit 404 as a photo selection screen.
  • FIG. 11 is a diagram showing an example of the photo selection screen 1101. As shown in FIG. 11, a plurality of thumbnail images 1102 are displayed. A thumbnail image 1102 corresponds to shooting data stored in the image memory 308. The thumbnail image 1102 is displayed with date / time information 1104 added thereto. On the photo selection screen 1101, a rider's selection can be accepted by a check box 1103. In addition, when a large number of shooting data is stored in the image memory 308, it cannot be displayed on one screen. In that case, the previous page or the subsequent page can be displayed by the transition button 1105.
  • the rider can shoot at a desired timing without causing a shooting operation.
  • due to factors such as the angle of the camera 702 it is not known whether all the photographed data is truly desirable. Therefore, in this embodiment, by accepting a rider's selection on the photo selection screen 1101, only truly desirable shooting data can be specified.
  • step S807 the controller 112 accepts selection of shooting data on the photo selection screen 1101. This is not shown in FIG. 11, but the selection may be confirmed by a confirmation button or the like.
  • step S ⁇ b> 808 the controller 112 transmits the selected imaging data and electroencephalogram data at the time of imaging to the server 108.
  • the server 108 registers the transmitted imaging data and the electroencephalogram data at the time of imaging in the DB 605.
  • step S809 the server 108 analyzes the feature amounts including the brain wave data newly registered in step S808, and learns the feature amounts that are similar patterns.
  • step S810 the server 108 corrects the correlation data corresponding to the shooting event based on the feature amount obtained by learning.
  • the feature amount learning in step S809 may be performed periodically at predetermined time intervals, or may be performed every time shooting data is accumulated every predetermined number of sheets.
  • the shooting data may be selected by other configurations.
  • the shooting data may be selected on the cloud.
  • the imaging data and brain wave data are transmitted to the server 108.
  • the transmission of imaging data and electroencephalogram data may be performed directly to the server 108 or may be performed via the controller 112.
  • the rider accesses the storage location of the shooting data of the server 108 from his / her portable terminal.
  • the storage location at that time may be a photo database different from the DBs 605 and 606.
  • a photo selection screen 1101 as shown in FIG. 11 is displayed, and the rider performs a selection operation on the photo selection screen 1101.
  • the server 108 stores the imaging data and brain wave data selected in the photo database in the DB 605.
  • FIG. 9 is a flowchart showing processing of the control board 300 in the helmet 109.
  • the processor 301 determines whether or not the automatic shooting mode is set. The determination in S901 may be performed by, for example, a hard switch provided on the control board 300, or may be performed based on an instruction indicating that one of the modes is received from the controller 112. If it is determined in S901 that the mode is not the automatic shooting mode, the process of FIG. 9 is terminated. If it is determined that the mode is the automatic shooting mode, the process proceeds to S902.
  • the processor 301 converts an electrical signal (electroencephalogram signal) detected by the electrode 701 into electroencephalogram data by an amplifier and an AD converter, and transmits the electroencephalogram data to the controller 112. Then, the processor 301 transmits imaging data obtained by the eye tracking camera 306 to the controller 112 as line-of-sight information.
  • an electrical signal electroencephalogram signal
  • the processor 301 transmits imaging data obtained by the eye tracking camera 306 to the controller 112 as line-of-sight information.
  • step S ⁇ b> 903 the processor 301 determines whether a shooting command has been received from the controller 112. If it is determined that the imaging command has not been received, the process of S902 is repeated. On the other hand, if it is determined that a shooting command has been received, the processor 301 performs shooting with the camera 305 in step S904. In step S905, the processor 301 stores the imaging data in the image memory 308 together with the electroencephalogram data at the time of imaging. After S905, the processing from S901 is repeated.
  • FIG. 10 is a flowchart showing the processing of the controller 112.
  • the processing in FIG. 10 starts when the image memory 308 is attached to the memory I / F 408.
  • the processor 401 acquires imaging data and brain wave data stored in the image memory 308, and displays a photo selection screen 1101 on the display unit 404.
  • the processor 402 selects shooting data based on the selection result of the check box 1103 on the photo selection screen 1101.
  • the processor 401 transmits the selected imaging data and brain wave data to the server 108 and registers them in the DB 605 of the server 108. Then, the process of FIG. 10 is complete
  • FIG. 12 is a flowchart showing the processing of the server 108 before learning the electroencephalogram data. If the amount of electroencephalogram data stored in the DB 605 of the server 108 is not sufficient, correlation data cannot be generated by learning feature amounts of similar patterns. In that case, the determination is performed on the electroencephalogram data transmitted from the controller 112 according to a predetermined determination condition. For example, when the potential represented by the electroencephalogram data is included in a predetermined activation range, the electroencephalogram data is determined to satisfy the determination condition, and an imaging command is transmitted.
  • a predetermined determination condition For example, when the potential represented by the electroencephalogram data is included in a predetermined activation range, the electroencephalogram data is determined to satisfy the determination condition, and an imaging command is transmitted.
  • the processor 601 receives drive information.
  • the drive information here is the drive information transmitted from the controller 112 in S802 of FIG.
  • the processor 601 determines whether or not the drive information satisfies a condition. For example, the processor 601 determines whether or not the speed information of the motorcycle 101 is equal to or higher than a predetermined speed. If it is determined that the drive information satisfies the condition, the processes of S1203 and S1204 are executed. If it is determined that the drive information does not satisfy the condition, the process of FIG. In S1203, the processor 601 receives brain wave data and line-of-sight information.
  • the processor 601 determines whether or not the electroencephalogram data satisfies a condition. For example, the processor 601 determines whether or not the potential represented by the electroencephalogram data is included in a predetermined activation region.
  • FIG. 13 is a diagram illustrating an example of the case where the potential represented by the electroencephalogram data is included in the activation region.
  • the determination condition as to whether or not it is included in the activation region may be looser than the determination condition as to whether the correlation data is within a predetermined similarity range. Thereby, it is possible to easily collect brain wave data for acquiring correlation data.
  • the processor 601 transmits an imaging command to the controller 112 in step S1206. After S1206, the processing from S1201 is repeated. On the other hand, if it is determined that it is not included in the activation area, the process proceeds to S1205.
  • the processor 601 determines whether or not the line-of-sight information satisfies a condition. For example, the processor 601 determines whether the rider's line of sight is constant for a predetermined time, for example, 10 seconds. As a result, even when it is determined that the electroencephalogram data is not included in the activation area, it is possible to detect the case where the rider keeps looking at a certain direction for some reason, and the electroencephalogram data for acquiring the correlation data is collected. Can be easier. If it is determined in step S1205 that the line-of-sight information satisfies the condition, the processor 601 transmits an imaging command to the controller 112 in step S1206. If it is determined that the line-of-sight information does not satisfy the condition, the processing from S1201 is repeated.
  • FIG. 14 is a flowchart showing the processing of the server 108 after learning the electroencephalogram data, that is, after acquiring the correlation data. Steps S1401 to S1403 are the same as those described in steps S1201 to S1203 in FIG.
  • the processor 601 compares the electroencephalogram data with the correlation data.
  • the processor 601 determines whether the electroencephalogram data is included in a predetermined similarity range of the correlation data. If it is determined that it falls within the predetermined similar range, the process proceeds to S1407.
  • S1407 is the same as the description in S1206 of FIG.
  • S1406 is the same as the description in S1205 of FIG.
  • the determinations in S1205 in FIG. 12 and S1406 in FIG. 14 are performed when it is determined in S1204 and S1405 that the electroencephalogram data does not satisfy the condition.
  • the determination as to whether the line-of-sight information satisfies the condition may be performed before S1204 and S1405. In this case, if it is determined that the line-of-sight information satisfies the condition, the processing proceeds to S1206 and S1407, and the processor 601 transmits an imaging command to the controller 112. That is, imaging is performed regardless of the electroencephalogram data.
  • determinations in S1204 and S1405 are performed. When it is determined that the electroencephalogram data satisfies the condition, imaging is performed in S1206 and S1407, and when it is determined that the electroencephalogram data does not satisfy the condition, the processes from S1201 and S1401 are repeated.
  • FIG. 15 is a flowchart showing processing for storing the imaging data and brain wave data transmitted from the controller 112 in the DBs 605 and 606. The process of FIG. 15 is started when imaging data and electroencephalogram data are transmitted from the controller 112 in S808 of FIG.
  • the processor 601 monitors reception of shooting data as registration data in the DB 605, and if it is determined that shooting data has been received, the processor 601 proceeds to S1502. In S1502, the processor 601 registers imaging data and brain wave data in the DBs 605 and 606.
  • the processor 601 determines whether or not the data amount of the electroencephalogram data is sufficient for learning the similarity. If it is determined that the amount is sufficient, the processing proceeds to S1504. If it is determined that the amount is not sufficient, the processing from S1501 is repeated.
  • the processor 601 learns the similarity by analyzing the feature quantity of the electroencephalogram data corresponding to each rider registered in the DB 605, and in S1505, the feature quantity representing the similar pattern corresponds to the shooting event. Obtained as correlation data. If the process of S1504 is the first process for the electroencephalogram data registered in the DB 605, the acquired feature quantity representing the similar pattern is used as correlation data corresponding to the imaging event as a predetermined storage area of the server 108. And the process of FIG. 15 is terminated. On the other hand, if it is not the first process, the correlation data stored in a predetermined storage area of the server 108 is corrected (updated) based on the acquired feature amount representing the similar pattern, and then the process of FIG. Exit.
  • the rider can take a picture when it is considered that the rider has a good shooting timing without causing a shooting operation. Then, the rider selects shooting data that is subjectively good from the shooting result, and the selected shooting data is transmitted to the server 108. With such a configuration, only shooting data that the rider thinks is truly good shooting timing can be registered in the DB 606 of the server 108. Then, the server 108 learns the similarity of the electroencephalogram data corresponding to the imaging data that each rider thinks is good, acquires the feature amount representing the similarity pattern, and stores it as the latest correlation data. When the server 108 determines that the electroencephalogram data transmitted from the rider is similar to the correlation data, the server 108 transmits an imaging command to the controller 112 of the motorcycle 101.
  • the brain wave data is transmitted from the controller 112 of the motorcycle 101 to the server 108 and an operation command is received, so that the camera 305 performs imaging.
  • this embodiment may be applied to other operations.
  • an example applied to other operations will be described with respect to differences from the operations of the above-described embodiment.
  • FIG. 16 is a flowchart showing processing in which the server 108 transmits an operation command of the apparatus using brain wave data.
  • S1601 to S1602 are the same as those described in S1201 to S1202 of FIG.
  • the processor 601 receives the electroencephalogram data.
  • the processor 601 compares the electroencephalogram data with the correlation data.
  • the correlation data here is data corresponding to an operation event of a predetermined apparatus.
  • the predetermined device operation event is, for example, operating a music playback device (stereo or the like) or a broadcast receiving device (radio or the like).
  • the processor 601 determines whether the electroencephalogram data is included in a predetermined similarity range of the correlation data. If it is determined to be included in the predetermined similar range, the process proceeds to S1606.
  • step S ⁇ b> 1606 the processor 601 transmits an operation command for a predetermined apparatus to the controller 112. After S1606, the processing from S1601 is repeated. On the other hand, if it is determined that it is not included in the predetermined similar range, the processing from S1601 is repeated.
  • the controller 112 When the controller 112 receives the operation command in S1606, the controller 112 transmits the operation command to a predetermined device.
  • the helmet 109 is configured with a music playback device and a speaker that can be loaded with an SD card recording music, and the rider can listen to the music played back with the music playback device through the speaker. ing.
  • the controller 112 transmits a reproduction command from the music reproduction device of the helmet 109 to the helmet 109.
  • the processor 301 of the helmet 109 performs music playback by the music playback device of the helmet 109.
  • the operation of FIG. 16 described above corresponds to the processing after acquiring the correlation data of FIG.
  • the process proceeds to S1605, and it is determined whether the electroencephalogram data satisfies the condition.
  • the processor 601 may determine whether or not the potential represented by the electroencephalogram data is included in a predetermined activation area, as in S1204 of FIG.
  • the timing is not preferable for the rider even if the music reproduction is performed by transmitting the operation command in S1606.
  • the processor 601 of the server 108 does not register the electroencephalogram data when the reproduction is stopped in the DB 605 and excludes it from the similarity learning target.
  • the processor 601 of the server 108 registers the electroencephalogram data at the time of reproduction in the DB 605 and sets it as a similarity learning target.
  • the controller 112 transmits a playback command from the broadcast receiving device of the helmet 109 to the helmet 109.
  • the processor 301 of the helmet 109 performs reproduction using the radio of the helmet 109.
  • Other operations are the same as those in the music reproducing apparatus.
  • the music playback device and the broadcast receiving device have been described as being configured in the helmet 109, they may be configured in the main body of the motorcycle 101. In that case, when receiving the reproduction command, the controller 112 performs reproduction by the music reproduction device or the broadcast reception device.
  • the predetermined device may be a vehicle component of the motorcycle 101. That is, the operation command and the motorcycle 101 (vehicle) may be linked.
  • the operation command and the motorcycle 101 may be linked.
  • the command for controlling the accelerator or brake of the motorcycle 101 is operated in S1606. It may be transmitted as a command.
  • the correlation data in this case is, for example, brain wave data emitted during sleep. If it is determined in S1605 that the electroencephalogram data is included in the predetermined similar range of the correlation data, in S1606, the processor 601 transmits an accelerator or brake control command to the controller 112 as an operation command. Examples of such control commands include suppression of accelerator output and suppression of sudden changes in accelerator and brake.
  • the controller 112 When the controller 112 receives the control command, the controller 112 performs each control described above in cooperation with the ECU of the motorcycle 101 and the like. Further, the processor 601 may transmit a command for sounding a warning sound to the speaker or the like of the helmet 109 together with the accelerator or brake control command.
  • the electroencephalogram data generated during sleep is taken as an example, but electroencephalogram data representing a physical state such as fatigue may be used.
  • advice information such as prompting a break may be transmitted to the controller 112 of the motorcycle 101 in S1606.
  • the controller 112 may display guidance display and break location information on the display unit 404 based on the advice information.
  • the helmet 109 may be configured with a speaker, and the controller 112 may reproduce the voice guidance on the helmet 109 speaker.
  • the electroencephalogram data emitted during sleep and the electroencephalogram data representing a physical state such as fatigue may not be obtained by similarity learning, but may be provided as a predetermined reference.
  • the operation system (100) of the above embodiment includes: Based on the acquisition means for acquiring the electroencephalogram data corresponding to the electroencephalogram signal of the passenger of the vehicle (101, 102, 103) (S802), the database (605) for accumulating the electroencephalogram data, and the electroencephalogram data accumulated in the database Generating means for generating correlation data associated with a command for operating the apparatus (S1503, S1504), based on the electroencephalogram data acquired by the acquiring means and the correlation data generated by the generating means Transmission means for transmitting the command (S804), and control means for operating the apparatus when the command is transmitted by the transmission means (S805).
  • the determination unit further comprises a determination unit (S803) that determines whether to send the command based on the electroencephalogram data acquired by the acquisition unit and the correlation data generated by the generation unit. When it is determined that the command is transmitted by the means, the transmission means transmits the command.
  • the camera can be shot.
  • the apparatus is an imaging unit (702), and further includes a storage unit (S808) for storing brain wave data corresponding to an electroencephalogram signal at the time of imaging by the imaging unit in the database.
  • electroencephalogram data corresponding to an electroencephalogram signal at the time of imaging can be accumulated in a database.
  • the electroencephalogram data stored in the database is associated with imaging data captured by the imaging means.
  • electroencephalogram data and photographs can be associated with each other and managed in a database.
  • the electroencephalogram stored in the database further comprises selection means (S807, 1101) for the imaging means to select imaging data for generating the correlation data among the imaging data captured by the imaging means.
  • the data is associated with the imaging data selected by the selection unit, and the generation unit generates the correlation data based on the electroencephalogram data associated with the imaging data selected by the selection unit.
  • gaze detection means S902, 703 for detecting a gaze signal of the passenger, and when the gaze signal detected by the gaze detection means satisfies a condition, the transmission means transmits the command. It is characterized by doing.
  • the camera can shoot.
  • the electroencephalogram data stored in the database includes electroencephalogram data corresponding to an electroencephalogram signal of a passenger of a vehicle (102, 103) different from the passenger of the vehicle (101), and the generation means includes the database
  • the correlation data is generated by analyzing the similarity of the electroencephalogram data corresponding to the passengers of a plurality of vehicles stored in the vehicle.
  • correlation data can be generated based on brain wave signals detected from a plurality of passengers.
  • the operation system includes a server (108) and a motorcycle (101, 102, 103) as the vehicle.
  • the server includes the acquisition unit, the database, the generation unit, and the transmission unit.
  • the motorcycle includes the control unit, and the acquisition unit acquires brain wave data corresponding to a brain wave signal detected by a helmet (109, 110, 111) worn by the passenger.
  • a configuration including a motorcycle, a server, and a helmet can be realized as an operation system.
  • the apparatus is characterized in that it is a photographing means (702) configured in the helmet (109, 110, 111).
  • the photographing unit can photograph a rider in the direction of the eyes of the rider.

Landscapes

  • Engineering & Computer Science (AREA)
  • Health & Medical Sciences (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • Theoretical Computer Science (AREA)
  • General Engineering & Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • Medical Informatics (AREA)
  • Public Health (AREA)
  • Biomedical Technology (AREA)
  • Heart & Thoracic Surgery (AREA)
  • Biophysics (AREA)
  • Molecular Biology (AREA)
  • Surgery (AREA)
  • Animal Behavior & Ethology (AREA)
  • General Health & Medical Sciences (AREA)
  • Pathology (AREA)
  • Veterinary Medicine (AREA)
  • Psychology (AREA)
  • Human Computer Interaction (AREA)
  • General Physics & Mathematics (AREA)
  • Psychiatry (AREA)
  • Measurement And Recording Of Electrical Phenomena And Electrical Characteristics Of The Living Body (AREA)
  • Measurement Of The Respiration, Hearing Ability, Form, And Blood Characteristics Of Living Organisms (AREA)

Abstract

L'invention concerne un système de manipulation qui permet de faire fonctionner, à l'aide d'ondes cérébrales, un dispositif sans qu'un occupant déclenche une opération de manipulation. Des données de corrélation associées à une instruction destinée à amener le dispositif à fonctionner sont générées sur la base de données d'ondes cérébrales stockées dans une base de données. Lorsque l'instruction est transmise à la fois sur la base des données d'onde cérébrale acquises par rapport à l'occupant d'un véhicule et des données de corrélation générées, le dispositif est amené à fonctionner.
PCT/JP2017/013623 2017-03-31 2017-03-31 Système de manipulation, serveur, procédé de commande dans un système de manipulation, et programme WO2018179346A1 (fr)

Priority Applications (1)

Application Number Priority Date Filing Date Title
PCT/JP2017/013623 WO2018179346A1 (fr) 2017-03-31 2017-03-31 Système de manipulation, serveur, procédé de commande dans un système de manipulation, et programme

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
PCT/JP2017/013623 WO2018179346A1 (fr) 2017-03-31 2017-03-31 Système de manipulation, serveur, procédé de commande dans un système de manipulation, et programme

Publications (1)

Publication Number Publication Date
WO2018179346A1 true WO2018179346A1 (fr) 2018-10-04

Family

ID=63674502

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/JP2017/013623 WO2018179346A1 (fr) 2017-03-31 2017-03-31 Système de manipulation, serveur, procédé de commande dans un système de manipulation, et programme

Country Status (1)

Country Link
WO (1) WO2018179346A1 (fr)

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2021089544A (ja) * 2019-12-03 2021-06-10 富士フイルムビジネスイノベーション株式会社 情報処理システム及びプログラム

Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2004152002A (ja) * 2002-10-30 2004-05-27 Mitsubishi Electric Corp 脳波信号を利用した制御装置
WO2012060039A1 (fr) * 2010-11-02 2012-05-10 Ｎｅｃカシオモバイルコミュニケーションズ株式会社 Système de traitement d'informations et procédé de traitement d'informations
WO2013128920A1 (fr) * 2012-02-27 2013-09-06 ヤマハ発動機株式会社 Dispositif de partage d'état d'opération et système de partage d'état d'opération
JP2017042261A (ja) * 2015-08-25 2017-03-02 マツダ株式会社 脳波取得方法および脳波取得装置
JP2017061192A (ja) * 2015-09-24 2017-03-30 日産自動車株式会社 サポート装置及びサポート方法

Patent Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2004152002A (ja) * 2002-10-30 2004-05-27 Mitsubishi Electric Corp 脳波信号を利用した制御装置
WO2012060039A1 (fr) * 2010-11-02 2012-05-10 Ｎｅｃカシオモバイルコミュニケーションズ株式会社 Système de traitement d'informations et procédé de traitement d'informations
WO2013128920A1 (fr) * 2012-02-27 2013-09-06 ヤマハ発動機株式会社 Dispositif de partage d'état d'opération et système de partage d'état d'opération
JP2017042261A (ja) * 2015-08-25 2017-03-02 マツダ株式会社 脳波取得方法および脳波取得装置
JP2017061192A (ja) * 2015-09-24 2017-03-30 日産自動車株式会社 サポート装置及びサポート方法

Cited By (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2021089544A (ja) * 2019-12-03 2021-06-10 富士フイルムビジネスイノベーション株式会社 情報処理システム及びプログラム
JP7320261B2 (ja) 2019-12-03 2023-08-03 株式会社Agama-X 情報処理システム、方法、及びプログラム

Similar Documents

Publication Publication Date Title
US20220337693A1 (en) Audio/Video Wearable Computer System with Integrated Projector
EP3067827B1 (fr) Système de détection de distraction du conducteur
US11165955B2 (en) Album generation apparatus, album generation system, and album generation method
JP6862629B2 (ja) 見守り送迎システムにおける車両管理サーバ、車載端末、見守り方法、及びプログラム
JP5176311B2 (ja) 画像表示システム、表示装置、表示方法
US11057728B2 (en) Information processing apparatus, information processing method, and program
KR20170015122A (ko) 군중 기반의 햅틱
JP2016052881A (ja) 車両の走行制御システム
JP2015041969A (ja) 画像取得装置及び画像取得方法、並びに情報配信システム。
US11250875B2 (en) Behavior support system, behavior support apparatus, behavior support method, and storage medium storing program thereof
CN106060569A (zh) 一种视频直播方法、装置、设备及系统
KR101250951B1 (ko) 위험요소 감지 기능을 가지는 양귀 삽입형 보청기 및 이를 이용한 위험요소 감지 시스템
JP7287257B2 (ja) 画像処理装置、表示システム、プログラムおよび画像処理方法
CN109885368A (zh) 一种界面显示防抖方法及移动终端
CN111196230A (zh) 驾驶辅助装置、可穿戴装置、驾驶辅助方法以及程序
KR101741074B1 (ko) 차량 사고 녹화 장치 및 그것의 방법
CN111133752B (zh) 表情记录系统
KR20150137453A (ko) 뇌파를 이용한 제어 방법 및 휴대 장치
JP2019086805A (ja) 車内システム
WO2018179346A1 (fr) Système de manipulation, serveur, procédé de commande dans un système de manipulation, et programme
WO2018075523A1 (fr) Système informatique vestimentaire audio/vidéo à projecteur intégré
CN114710553B (zh) 信息获取方法、信息推送方法以及终端设备
JP2019105966A (ja) 情報処理方法及び情報処理装置
CN113965726A (zh) 处理交通视频的方法、装置以及系统
US11792484B2 (en) Information providing system, information providing method and management apparatus for information providing system for providing information obtained through a robot

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 17902623

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 17902623

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: JP