JP2006318450A - Control system - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide a control system used for controlling a device to be controlled by noninvasively acquiring information of brain activity. <P>SOLUTION: When a data processing device acquires information of brain activity such as a brain wave signal or bloodstream signal (S11: YES), the brain activity is analyzed based on the information (S12). Whether a user wearing a brain hat feels hot is determined (S13), and, when it is determined that the user feels hot (S13: YES), a control signal for operating cooling is generated (S14) and the generated control signal is transmitted to an air conditioner (S15). When it is determined that the user does not feel hot (S13: NO), whether the user feels cold is determined (S16), and, when it is determined that the user feels cold (S16: YES), a control signal for operating heating is generated (S17) and the generated control signal is transmitted to the air conditioner (S15). <P>COPYRIGHT: (C)2007,JPO&INPIT

Description

  The present invention relates to a control system that non-invasively acquires information on brain activity and controls a controlled device.

  Various electric devices such as lighting devices, television receivers, audio devices, air conditioners, home appliances such as electronic cookers, personal computers, mobile phones, and information communication devices such as PDAs (Personal Digital Assistants) have become widespread. Yes.

  Such control of the electric device is performed by the user himself / herself directly operating the electric device or by remote operation using infrared communication or the like. In recent years, there has also been proposed a system for controlling various electrical devices by giving user instructions via a communication network such as the Internet network. For example, Patent Document 1 proposes a system for remotely controlling an electric device by starting a web browser on a terminal device connected to the Internet network and accessing the electric device through the web browser.

Further, the wireless communication device is carried by the user, and it is automatically detected when the user approaches the home by detecting whether the wireless communication device exists in a predetermined area centered on the home. There has also been proposed a home control system for turning on the power of an air conditioner, a lighting device, or the like (see, for example, Patent Document 2).
JP 2001-53779 A JP-A-8-79840 International Publication No. 03/057035 Pamphlet

  However, in the remote control system described in Patent Document 1, a web browser must be started from a personal computer and an electric device installed at home must be accessed to perform remote control. It has a problem that the threshold is high for persons with physical disabilities, the elderly, and the like, and that it is troublesome for those who are not.

  In addition, the home control system described in Patent Document 2 does not require user operation, and can automatically turn on / off the power of an electrical device or control a preset value. However, there is a problem that fine control under the environment including the user, such as temperature adjustment by an air conditioner, power-on of a lighting device that is not set in advance, cannot be realized by the above-described home control system. is doing.

  By the way, recent advances in biological measurement technology are remarkable, and the measurement accuracy of the weak electric field (electroencephalogram) and the weak magnetic field (magnetoencephalogram) generated from the brain, which has been difficult to measure and has a large error, has been improved year by year. There has also been proposed an algorithm for accurately estimating the position of the current source in the brain based on the measured electroencephalogram and magnetoencephalogram (see, for example, Patent Document 3).

  Therefore, based on information obtained by estimating the position of the current source in the brain with high accuracy, biological information indicating the user's physical state, emotions, etc. is estimated, and various electrical devices are controlled according to the estimation results Therefore, it is desired to construct a control system that can create a comfortable environment for the user.

  The present invention has been made in view of such circumstances, and an object thereof is to provide a control system capable of finely controlling a controlled device without accepting an operation by a user's hand.

  Another object of the present invention is to provide a control system capable of controlling a controlled device in accordance with biological information indicating a user's physical state, emotion, and the like.

  A control system according to a first aspect of the present invention is a control system comprising a controlled device to be controlled and a control device that controls the controlled device, wherein the control device non-invasively monitors the brain activity of the observation subject. Observation means for observing and estimation means for estimating biological information related to the observation subject based on the observation result by the observation means, and controlling the controlled device based on the estimation result by the estimation means It is characterized by being.

  In the first invention, the control device observes the brain activity of the observation subject non-invasively, estimates biological information related to the observation subject based on the observation result, and controls the controlled device based on the estimation result. To control. Therefore, control according to biological information such as the physical state and emotion of the observation target person (user) is executed.

  In the control system according to a second aspect of the invention, the observation means is an fMRI apparatus that non-invasively observes the brain activity of the observation subject.

  In the second invention, since the brain activity is observed based on the fMRI having a high spatial resolution, accurate biological information can be estimated.

  In the control system according to a third aspect of the present invention, the observation means includes a first sensor that measures an electric field generated by the brain, and a second sensor that measures a blood flow state in the brain. It is characterized by providing.

  In the third invention, the signal obtained by the first sensor is excellent in time resolution, but the spatial resolution is inferior. The signal obtained by the second sensor is excellent in spatial resolution. However, the time resolution is inferior. Therefore, it is possible to accurately estimate biological information by using these signals in combination and taking advantage of each advantage.

  The control system according to a fourth aspect of the present invention is characterized in that the first sensor is EEG and the second sensor is NIRS.

  In the fourth invention, since brain activity is observed by a combination of EEG with excellent temporal resolution and NIRS with excellent spatial resolution, accurate biological information can be estimated by taking advantage of each advantage.

  The control system according to a fifth aspect of the present invention is characterized in that the first sensor is MEG and the second sensor is fMRI.

  In the fifth invention, since brain activity is observed by a combination of MEG having excellent temporal resolution and fMRI having excellent spatial resolution, accurate biological information can be estimated by taking advantage of each advantage.

  The control system according to a sixth aspect of the invention is characterized in that the controlled device is an information processing device or an electric device including a communication device.

  In the sixth aspect of the invention, the controlled device controlled by the control device is an electric device including an information processing device or a communication device. Therefore, the lighting device, the television receiver, the audio device, the air conditioner, and the electronic cooking device It is possible to control information communication devices such as home appliances such as personal computers, mobile phones and PDAs. In recent years, pet-type robots have gained popularity as information home appliances with entertainment for the purpose of interacting with people, and autonomous robots are also expected to emerge along with an aging society. However, these robots can also be controlled according to the user's biological information.

  In the case of the first invention, the brain activity of the observation subject is observed non-invasively, the biological information related to the observation subject is estimated based on the observation result, and the controlled device is controlled based on the estimation result. ing. Therefore, the controlled device can be controlled according to the biological information such as the user's physical state and emotion. For example, when the user feels "hot", not only the air conditioner is operated. By adjusting the temperature, it is possible to create a comfortable environment for the user. Moreover, since it was set as the structure which observes brain activity noninvasively, when compared with the apparatus which observes brain activity invasively, the physical burden and mental burden with respect to a user can be reduced.

  In the case of the second invention, the state of brain activity can be accurately grasped by using fMRI having a high spatial resolution, and the estimation accuracy of biological information can be increased.

  According to the third aspect of the present invention, the brain field activity is reduced by combining an electric field generated by a brain having excellent temporal resolution of brain activity and a signal indicating a blood flow state in the brain having excellent spatial resolution of brain activity. Since invasive observation is performed, the state of brain activity can be accurately grasped.

  In the case of the fourth invention, since EEG is excellent in time resolution and NIRS is excellent in spatial resolution, the state of brain activity can be accurately grasped by using both in combination, and biological information The estimation accuracy can be improved.

  In the case of the fifth invention, since MEG has excellent temporal resolution and fMRI has excellent spatial resolution, the state of brain activity can be accurately grasped by using both in combination. The estimation accuracy can be improved.

  In the case of the sixth invention, since the controlled device controlled by the control device is an electric device including an information processing device or a communication device, a lighting device, a television receiver, an audio device, an air conditioner, an electronic cooking device, etc. Home appliances, personal computers, mobile phones, PDAs and other information communication devices can be controlled. Also, pet-type robots, autonomous robots, and the like can be controlled in accordance with the user's biological information.

Hereinafter, the present invention will be specifically described with reference to the drawings illustrating embodiments thereof.
Embodiment 1 FIG.
In the present embodiment, an fMRI apparatus (fMRI: functional magnetic resonance imaging) is employed as an observation means for non-invasively observing a user's brain activity. Based on the data output from the fMRI apparatus, it is estimated whether the user feels “bright” or “dark”, and if it feels “dark”, the illumination device is activated. .

  FIG. 1 is a schematic diagram showing an example of an image obtained by an fMRI apparatus. FIG. 1A shows a brain image when a bright screen is seen, and FIG. 1B shows a brain image when a dark screen is seen. The upper left, upper right, and lower left images show brain images observed from the front, side, and top surfaces, respectively. This image has the same resolution as an actual brain image (functional image).

  When the bright screen is viewed and when the dark screen is viewed, the change in signal is only about 1%, and information is included in subtle brightness that cannot be displayed in 8 bits. Since a larger noise is applied, it is impossible to determine whether a bright screen or a dark screen is viewed from simple visual observation of both brain images.

  Therefore, in the present embodiment, hundreds of pixels are extracted from a portion corresponding to the surface of the brain in a region corresponding to the visual cortex, and a machine learning algorithm is applied. In addition, a method of predicting the inclination of the striped pattern from the brain activity analysis of the visual cortex using fMRI has been proposed (Kamitani, Y., Tong, F. (2005). Decording the visual and subjective contents Nature Neuroscience, 8, 5, 679-685), in this embodiment, the brightness of the screen being viewed is estimated by using a method similar to the above method.

  FIG. 2 is an explanatory diagram for explaining a main part of the machine learning algorithm. In this embodiment, a “dark” screen in which only one white point is arranged at the center of the black screen and a “bright” screen in which many white dots are arranged on the black screen are prepared for the user. Show any screen and train. When the “bright” screen is displayed, the output of the detector that detects “bright” is maximum, and when the “dark” screen is displayed, the output of the detector that detects “dark” is the maximum. Such a weighting is found by a machine learning algorithm. After finding the weights, the user is again shown one of the screens, and based on the outputs of the two detectors, it is estimated whether the “bright” screen is being viewed or the “dark” screen is being viewed.

  FIG. 3 is a graph showing estimation accuracy when estimation is performed using a machine learning algorithm. The horizontal axis indicates items that are predicted to be “bright” or “dark”, and the vertical axis indicates the ratio of predicted brightness. When the average of brain images when viewing the same screen for 12 seconds is taken as one trial, correct answers were obtained in 42 trials out of 44 trials, and it was found that the estimation accuracy was very high.

  Next, the configuration of the control system of the present embodiment will be described. FIG. 4 is a schematic diagram illustrating the configuration of the control system according to the first embodiment. Information on the user's brain activity is observed by the fMRI apparatus 260. The fMRI apparatus 260 is connected to the data processing apparatus 20. In the present embodiment, it is assumed that information on brain activity acquired by the fMRI apparatus 260 is analyzed by the data processing apparatus 20 and the illumination apparatus 60 as a controlled apparatus is controlled based on the analysis result.

  Specifically, the data processing device 20 is a personal computer, a workstation, or the like, and performs analysis processing on data transmitted from the fMRI device 260. The data processing device 20 includes a CPU 21, and hardware such as a ROM 23, a RAM 24, an HDD 25, a communication IF 26, an input IF 27, an output IF 28, and a control unit 29 is connected to the CPU 21 via a bus 22. The CPU 21 loads the control program stored in advance in the ROM 23 onto the RAM 24 and executes it, thereby controlling the above-described hardware units and causing them to function as a device that implements the control system according to the present invention as a whole.

  The HDD 25 is a storage device having a magnetic recording medium, and stores data received by the communication IF 26, analysis results based on the data, and the like. The input IF 27 is an interface for connecting an input device 270 such as a keyboard and a mouse to the outside of the data processing apparatus 20, and accepts operations such as an analysis start instruction and an analysis result output instruction. The output IF 28 is an interface for connecting an output device 280 such as a liquid crystal display device or a printer device, and displays and prints received data and analysis results.

  The control unit 29 includes a signal generation circuit that generates a control signal to be transmitted to the lighting device 60 that is a controlled device, a communication circuit that transmits the generated signal to the lighting device 60 through wireless communication such as infrared communication, and the like. .

  FIG. 5 is a flowchart illustrating a processing procedure of processing executed by the data processing apparatus. The CPU 21 of the data processing device 20 determines whether or not the fMRI data has been acquired by monitoring the communication IF 26 (step S1). When it is determined that the fMRI data is not acquired (S1: NO), the process waits until the fMRI data is acquired.

  When it is determined that the fMRI data has been acquired (S1: YES), the brain activity is analyzed using the fMRI data by the method described above (step S2). In this analysis, information (biological information) related to the user state such as whether the target user feels “bright” or “dark” is estimated.

  The CPU 21 determines whether or not the user feels “dark” based on the analysis result in step S2 (step S3). When it is determined that the user feels “dark” based on the analysis result of the brain activity (S3: YES), the CPU 21 gives a control signal to turn on the lighting device 60 by giving an instruction to the control unit 29. Generate (step S4). Next, the CPU 21 transmits the generated control signal to the lighting device 60 (step S5). On the other hand, if the user does not feel “dark” in step S3 (S3: NO), the process according to this flowchart is terminated without transmitting a control signal.

  In the present embodiment, when it is determined that the user feels “dark” based on the analysis of brain activity, the data processing device 20 is configured to operate the lighting device 60 to turn on the lighting. The brightness felt by the user may be estimated from the state of the brain activity of the user, and the brightness of the lighting device 60 may be adjusted based on the estimated brightness.

Embodiment 2. FIG.
In the first embodiment, the brain activity is analyzed based on the data obtained by the fMRI apparatus 260, and the illumination device 60 is operated according to the analysis result. However, the observation means for non-invasively observing the brain activity Using an electroencephalogram (EEG) with excellent temporal resolution and a near infrared sensor (NIRS) with excellent spatial resolution, combining the observation results obtained from these sensors It is good also as a structure which acquires the information of brain activity.

  FIG. 6 is a schematic diagram showing a brain hat as an observation means for non-invasively observing brain activity. The brain hat 10 includes a plurality of first sensors 101 and a plurality of second sensors 102 (for example, several tens to several hundreds) on a cloth base body 100 having a cap shape so as to cover a human head. ). The first sensor 101 and the second sensor 102 are arranged at an equal pitch (for example, at intervals of several millimeters).

  The first sensor 101 is specifically an electroencephalograph having electrodes, and detects a weak electric field (electroencephalogram) caused by a current generated by a nerve cell in the brain in response to an external stimulus. Each first sensor 101 measures and outputs a change over time of an electric field accompanying brain activity at each site where the first sensor 101 is installed. The first sensor 101 is excellent in time resolution and can measure in units of several milliseconds.

  Specifically, the second sensor 102 is a near-infrared sensor, and includes a light emitting element that emits infrared light having a relatively short wavelength and a light receiving element that receives reflected light of the infrared light. The state of cerebral blood flow is detected based on the amount of light emitted from the light emitting element in the brain. Each second sensor 102 measures and outputs cerebral blood flow at each site where it is installed. Since the second sensor 102 is not affected by other regions such as an electric field or a magnetic field, it has excellent spatial resolution and can be measured in units of several millimeters.

  Since the first sensor 101 and the second sensor 102 can observe brain activity even in a small configuration, the first sensor 101 and the second sensor 102 can be easily attached to the brain hat 10 as described above. do not need.

  FIG. 7 is a schematic diagram illustrating a configuration of a control system according to the second embodiment. The user wears the brain hat 10 shown in FIG. 6 on the head. The brain hat 10 is connected to a counting device 11 that counts signals detected by the first sensor 101 and the second sensor 102 of the brain hat 10. A communication device 12 for performing wireless communication with the data processing device 20 is connected to the counting device 11. In the present embodiment, it is assumed that information acquired by the brain hat 10 is analyzed by the data processing device 20 and the air conditioner 30 as a controlled device is controlled based on the analysis result.

  The internal configuration of the data processing device 20 is the same as that described in the first embodiment, and based on the information on the brain activity acquired through the communication IF 26, the control unit 29 transmits to the air conditioner 30 that is the controlled device. A control signal to be generated is generated, and the generated signal is transmitted to the air conditioner 30 by wireless communication such as infrared communication.

  The air conditioner 30 includes a communication circuit that receives a control signal from the data processing device 20, a CPU that interprets the received control signal, and the like (not shown). Performs heating operation, temperature control, etc. That is, it functions as a controlled device that is controlled based on the state of the brain activity of the user.

  Hereinafter, a process executed by the data processing device 20 when information related to a user's brain activity is received will be described. FIG. 8 is a flowchart for explaining a processing procedure of processing executed by the data processing device 20. The CPU 21 of the data processing device 20 determines whether or not brain activity information has been acquired by monitoring the communication IF 26 (step S11). When it is determined that the information on the brain activity is not acquired (S11: NO), it waits until the information on the brain activity is acquired.

  When it is determined that information on brain activity has been acquired (S11: YES), brain activity is analyzed based on the information (step S12). In this analysis, information (biological information) regarding the user's state such as whether the user wearing the brain hat 10 feels “hot” or “cold” is estimated.

  The CPU 21 determines whether or not the user feels “hot” based on the analysis result in step S12 (step S13). When it is determined that the user feels “hot” based on the analysis result of the brain activity (S13: YES), the CPU 21 gives an instruction to the control unit 29 to generate a control signal for operating the cooling ( Step S14). On the other hand, when it is determined in step S13 that the user does not feel “hot” (S13: NO), the CPU 21 determines whether or not the user feels “cold” (step S16). If it is determined that the user feels “cold” based on the analysis result of the brain activity (S16: YES), a control signal for operating the heating is generated (step S17).

  When a control signal for operating cooling is generated in step S14, or when a control signal for operating heating is generated in step S17, the control unit 29 transmits the generated control signal to the air conditioner 30 (step S15). ). The air conditioner 30 that has received the control signal operates either cooling or heating in accordance with the received control signal. Further, when it is determined in step S16 that it is not felt “cold” (S16: NO), the process according to this flowchart is terminated without operating the air conditioner 30.

  Thus, in this Embodiment, since the to-be-controlled device (air conditioner 30) is controlled according to the user's state with which the brain hat 10 was mounted | worn, the comfortable environment for the user is created easily. In addition, since the brain hat 10 is attached to acquire information on brain activity in a non-invasive manner rather than a method of embedding electrodes in the cerebral cortex, excessive physical and mental burdens are imposed on the user. Never give.

  In this embodiment, when it is determined that the user feels “hot” based on the analysis of the brain activity, the data processing device 20 controls the air conditioner 30 to operate the cooling, and conversely When it is determined that the user feels “cold”, the data processing device 20 is configured to control the air conditioner 30 so as to operate the heating, but further adjusts the temperature based on the state of the brain activity of the user. May be performed.

  The temperature adjustment can be realized as follows. As described above, since the data processing device 20 can generate a control signal for operating the cooling or heating according to the state of the brain activity of the user, the air conditioner 30 performs the cooling or heating at any time. It is possible to know whether it is operating or neither. Therefore, for example, when the cooling is activated, when it is determined that the user still feels “hot” based on the brain activity information newly sent from the brain hat 10, the cooling set temperature is A control signal that lowers by a predetermined temperature is generated and transmitted to the air conditioner 30. Conversely, if the user feels "cold" as a result of operating the cooling, the set temperature of the cooling is increased by the predetermined temperature. It is possible to adjust the temperature by generating a control signal and transmitting it to the air conditioner 30. The same applies to temperature adjustment when heating is activated.

  In the present embodiment, the device to be controlled (controlled device) is the air conditioner 30, but is not limited to this. For example, the present invention can be applied to home appliances such as pet robots, audio devices, lighting devices, air purifiers, and information communication devices such as personal computers and PDAs.

  In the present embodiment, EEG and NIRS are combined as a means for noninvasively acquiring information representing brain activity. However, the present invention is not limited to this. As another example, for example, a magnetoencephalograph (MEG) having a high temporal resolution and a magnetic resonance apparatus (fMRI) having a high spatial resolution may be combined.

  Furthermore, although the information acquired by the brain hat 10 is transmitted to the data processing device 20 through the communication device 12, the controlled device as described above is controlled by a control device in which the brain hat 10 and the data processing device 20 are integrated. The structure to control may be sufficient.

Embodiment 3 FIG.
In the first embodiment and the second embodiment, the control target of the data processing device 20 is one device (the air conditioner 30).

  FIG. 9 is a schematic diagram illustrating a configuration of a control system according to the third embodiment. The system configuration is the same as that of the first embodiment, but includes an air conditioner 30, a pet-type robot 40, and an audio device 50 as controlled devices controlled by the data processing device 20. Note that the internal configuration of the data processing device 20 is the same as that of the first embodiment, and thus the description thereof is omitted.

  FIG. 10 is a flowchart illustrating a processing procedure of processing executed by the data processing device 20. The CPU 21 of the data processing device 20 determines whether or not brain activity information has been acquired by monitoring the communication IF 26 (step S21). If it is determined that the brain activity information is not acquired (S21: NO), the process waits until the brain activity information is acquired.

  If it is determined that information on brain activity has been acquired (S21: YES), brain activity is analyzed based on the information (step S22). As a method for analyzing brain activity, the same method as in the first embodiment can be used. Next, the CPU 21 selects a controlled device to be controlled according to the state of brain activity obtained as an analysis result (step S23). For example, when it is determined that the user wearing the brain hat 10 feels “hot” or “cold”, the air conditioner 30 is selected as a controlled device to be controlled. In addition, when the user is in a state of being beaten or in a state of reaching anger, the pet robot 40 is selected as a controlled device to be controlled. Further, when the user is in the “softened” state or in the “concentrated work” state, the audio device 50 is selected as the controlled device to be controlled.

  Next, the CPU 21 gives an instruction to the control unit 29 to generate a control signal to be transmitted to the selected controlled device (step S24), and transmits the generated control signal to the selected controlled device (step S25). ). For example, if it is determined that the user wearing the brain hat 10 feels “hot” and the air conditioner 30 is selected, a control signal for operating the cooling is generated and transmitted to the air conditioner 30. When it is determined that the air conditioner 30 is felt as being “cold”, a control signal for operating the heating is generated and transmitted to the air conditioner 30. In addition, when the user selects the pet robot 40 as being in a “striking” state, a control signal for controlling the pet robot 40 to play is generated and transmitted to the pet robot 40. When the pet type robot 40 is selected as being in the state of “getting angry,” a control signal for controlling the pet type robot 40 so as not to approach the user is generated and transmitted to the pet type robot 40. Further, if the user is in a “softened” state or selects the audio device 50 as being “concentrated on work”, it generates a control signal that controls to play classical music, The generated control signal is transmitted to the audio device 50.

  As described above, the controlled device to be controlled is selected according to the brain activity state of the user wearing the brain hat 10, and the control signal for controlling the selected control device is generated and transmitted. A comfortable environment is automatically created without the user directly operating each device.

It is a schematic diagram which shows an example of the image obtained by an fMRI apparatus. It is explanatory drawing explaining the principal part of a machine learning algorithm. It is a graph which shows the estimation precision when estimating using a machine learning algorithm. 1 is a schematic diagram illustrating a configuration of a control system according to Embodiment 1. FIG. It is a flowchart explaining the process sequence of the process which a data processor performs. It is a schematic diagram which shows the brain hat as an observation means which observes brain activity noninvasively. 1 is a schematic diagram illustrating a configuration of a control system according to Embodiment 1. FIG. It is a flowchart explaining the process sequence of the process which a data processor performs. 6 is a schematic diagram illustrating a configuration of a control system according to Embodiment 2. FIG. It is a flowchart explaining the process sequence of the process which a data processor performs.

Explanation of symbols

10 Brain Hat
11 Aggregation device
12 Communication device
20 Data processing device 21 CPU
DESCRIPTION OF SYMBOLS 29 Control part 30 Air conditioner 40 Pet type robot 50 Audio apparatus 101 1st sensor 102 2nd sensor

Claims (6)

In a control system comprising a controlled device to be controlled and a control device that controls the controlled device,
The control device includes observation means for non-invasively observing the brain activity of the observation subject, and estimation means for estimating biological information related to the observation subject based on an observation result by the observation means, the estimation means A control system characterized in that the controlled device is controlled on the basis of an estimation result obtained by.   The control system according to claim 1, wherein the observation unit is an fMRI apparatus that non-invasively observes the brain activity of the observation subject.   The said observation means is equipped with the 1st sensor with which the said observation subject is mounted | worn and which measures the electric field which a brain generate | occur | produces, and the 2nd sensor which measures the blood-flow state in a brain, The control system described in.   The control system according to claim 3, wherein the first sensor is an EEG, and the second sensor is a NIRS.   The control system according to claim 3, wherein the first sensor is an MEG and the second sensor is an fMRI.   The control system according to claim 1, wherein the controlled device is an information processing device or an electric device including a communication device.

Images