JP2023048016A - Sensation transmission system and method for transmitting sensation - Google Patents

Abstract

To allow subjects to properly transmit a sensation to one another when there are individual differences in the subjects and there are differences in the environments around the subjects.SOLUTION: A sensation transmission system includes: a first device for detecting brain activation information of a first subject when the first subject perceive a thing; an estimation device for estimating reference sensation information as sensation information thought of in relation to the perception on the basis of the detected brain activation information of the first subject and estimating corresponding sensation information as sensation information corresponding to the reference sensation information of a second subject different from the first subject on the basis of the estimated reference sensation information; and a second device for stimulating the second subject so that the second subject will think of the estimated corresponding sensation information.SELECTED DRAWING: Figure 2

Description

The present invention relates to a sensory transmission system and a sensory transmission method.

In recent years, technologies such as functional magnetic resonance, near-infrared spectroscopy, etc., have been developed to non-invasively measure activation information in the brain, and brain-machine interface technology, which is the interface between the brain and the outside world, is becoming a reality. be. As an example of using such a technique, the brain activation information of a sleeping subject is detected to determine the sleep state of the subject. (see, for example, Patent Document 1).

JP-A-2003-332251

The correspondence relationship between the activity state of the subject's brain and the measured brain activation information varies depending on the individuality of the subject, the environment surrounding the subject at the time of measurement, and the like. For example, in the technique described in Patent Document 1, the type of dream that occurs when a dream-inducing stimulus is applied differs for each subject or depending on the timing of applying the stimulus, etc., and it is difficult to control the content of the dream itself. be. On the other hand, when judging or controlling the state of brain activity based on brain activation information of a subject, there is a need for a technique that considers factors such as individual differences and differences in the environment surrounding the subject.

The present invention has been made in view of the above, and is a sensory transmission system capable of appropriately transmitting sensations between subjects when there are individual differences between subjects or differences in the environment surrounding the subjects. and to provide a sensory transmission method.

A sensory transmission system according to the present invention includes a first device for detecting brain activation information of the first subject when perceived by the first subject, and based on the detected brain activation information of the first subject. and estimating reference sensory information, which is sensory information recollected for the perception, and sensory information corresponding to the reference sensory information for a second subject different from the first subject based on the estimated reference sensory information. and a second device for stimulating the second subject so as to recall the estimated corresponding sensory information.

The sensory transmission method according to the present invention includes detecting brain activation information of the first subject when sensed by the first subject, and based on the detected brain activation information of the first subject. Reference sensory information that is sensory information recollected for perception is estimated, and sensory information corresponding to the reference sensory information for a second subject different from the first subject based on the estimated reference sensory information. estimating corresponding sensory information; and stimulating the second subject to recall the estimated corresponding sensory information.

ADVANTAGE OF THE INVENTION According to the present invention, it is possible to appropriately transmit sensations between subjects when there are individual differences between subjects or differences in the environment surrounding the subjects.

FIG. 1 is a schematic diagram showing an example of a sensory transmission system according to the first embodiment. FIG. 2 is a functional block diagram showing an example of the sensory transmission system according to the first embodiment. FIG. 3 is a diagram showing an example of a neural network. FIG. 4 is a diagram schematically showing an example of the operation of the sensory transmission system according to the first embodiment; FIG. 5 is a flow chart showing an example of the operation of the sensory transmission system according to the first embodiment. FIG. 6 is a diagram schematically showing another example of the operation of the sensory transmission system according to the second embodiment. FIG. 7 is a diagram schematically showing another example of the operation of the sensory transmission system according to the third embodiment; FIG. 8 is a functional block diagram showing an example of a sensory transmission system according to the fourth embodiment. FIG. 9 is a diagram schematically showing an example of the operation of the sensory transmission system according to the fourth embodiment; FIG. 10 is a flow chart showing an example of the operation of the sensory transmission system according to the fourth embodiment. FIG. 11 is a functional block diagram showing an example of a sensory transmission system according to the fifth embodiment. FIG. 12 is a diagram schematically showing an example of the operation of the sensory transmission system according to the fifth embodiment; FIG. 13 is a flow chart showing an example of the operation of the sensory transmission system according to the fifth embodiment.

BEST MODE FOR CARRYING OUT THE INVENTION An embodiment of the present invention will be described below with reference to the drawings. In addition, this invention is not limited by this embodiment. In addition, components in the following embodiments include components that can be easily replaced by those skilled in the art, or components that are substantially the same.

[First embodiment]
FIG. 1 is a schematic diagram showing an example of a sensory transmission system 100 according to the first embodiment. FIG. 2 is a functional block diagram showing an example of the sensory transmission system 100. As shown in FIG. As shown in FIGS. 1 and 2, the sensory transmission system 100 includes a first device 10, an estimating device 20, and a second device 30. FIG.

The first device 10 detects brain activation information of the first subject R1 when perceived by the first subject R1. The first device 10 has a detection unit 11 , a communication unit 12 , a processing unit 13 , a stimulus applying unit 14 and a storage unit 15 .

The detection unit 11 detects brain activation information. Examples of brain activation information include oxygenated hemoglobin concentration, deoxidized hemoglobin concentration, total hemoglobin concentration, and the like contained in the cerebral blood flow of a subject. As the detection unit 11, for example, a measuring device that performs measurement based on the principle of fMRI (functional Magnetic Resonance Imaging), fNIRS (functional Near-Infrared Spectroscopy), etc. , a measuring device using an invasive electrode, a measuring device in which micromachines are placed in the blood vessels of the brain and the micromachines are used for measurement, and the like can be used. Note that the detection unit 11 is not limited to the device described above, and other types of devices may be used. For example, when the brain of the first subject R1 is partitioned by a three-dimensional matrix composed of voxels of several millimeters or less, the intracerebral activation information can be indicated as the level of activity for each voxel.

The communication unit 12 is an interface that performs wired communication or wireless communication. The communication unit 12 transmits the brain activation information detected by the detection unit 11 to the estimation device 20 . The communication unit 12 includes an interface for communicating with an external device via a so-called wireless LAN conforming to the IEEE802.11 standard. The communication unit 12 may realize communication with an external device under the control of the processing unit 22 or the like. Note that the communication method is not limited to a wireless LAN, and may include, for example, an infrared communication method, a Bluetooth (registered trademark) communication method, a wireless communication method such as Wireless USB, and the like. Furthermore, a wired connection such as a USB cable, HDMI (registered trademark), IEEE1394, or Ethernet may be employed.

The processing unit 13, the stimulation applying unit 14, and the storage unit 15 will be described later. In the first embodiment, the processing unit 13, the stimulus applying unit 14, and the storage unit 15 may not be provided.

The estimation device 20 has a communication unit 21 , a processing unit 22 and a storage unit 23 . The communication unit 21 is an interface that performs wired communication or wireless communication. The communication unit 21 receives brain activation information transmitted from the first device 10, for example. The communication unit 21 transmits, for example, correspondence sensation information, which will be described later and is estimated by the processing unit 22 , to the second device 30 . Note that the communication unit 21 may have the same configuration as the communication unit 12 described above.

The processing unit 22 has a processing device such as a CPU (Central Processing Unit) and a storage device such as a RAM (Random Access Memory) or a ROM (Read Only Memory). The processing unit 22 performs various types of processing including the estimation processing described below.

The storage unit 23 stores various information. The storage unit 23 has storage such as a hard disk drive and a solid state drive. Note that an external storage medium such as a removable disk may be used as the storage unit 23 .

The processing unit 22 estimates sensory information (reference sensory information) about the sensation recalled by the first subject R1 based on the detected brain activation information of the first subject R1. The sensory information may be, for example, information related to at least one of the so-called five senses of sight, hearing, touch, taste, and smell, or may be a sense of balance or other somatic sensations. Specifically, when the sensory information is information related to vision, the sensory information is image data perceived by the first subject R1, but is not limited to this. It may be sampled image data or image data obtained by filtering the image data. Furthermore, when the sensory information is information about vision, it may be information about light entering the eyeball of the first subject R1. In this case, for example, information about light may be obtained by a contact lens equipped with an optical sensor. Alternatively, an artificial retina may be used to acquire information about light. Specifically, information about light from a CCD sensor provided on the artificial retina may be used. Moreover, when the sensory information is information related to hearing, the sensory information may be voice signal data perceived by the first subject R1. Moreover, when the sensory information is information about taste, the sensory information may be data indicating indices of a plurality of chemical substances that reproduce the taste perceived by the first subject R1. Further, when the sensory information is information related to tactile sensation, the sensory information is data indicating which part of the development view the whole surface of the body of the first subject R1 is developed and to what extent the stimulus occurred. I wish I had. Note that these sensory information are only examples, and the present invention is not limited to these. Regarding the first subject R1, it is possible to obtain a correspondence relationship by conducting an experiment in advance as to what kind of brain activation information is generated when what kind of sensory information is recollected. For example, brain activation information detected from the first subject R1 and sensory information corresponding to the brain activation information are associated to form a set of learning data sets, and machine learning is performed on the learning data sets. to generate the first learning model. The first learning model can be stored in the storage unit 23, for example.

In addition, the processing unit 22 estimates corresponding sensation information, which is sensation information corresponding to the reference sensation information for the second subject R2 different from the first subject R1, based on the estimated reference sensation information. A specific example of the corresponding sense information may correspond to a specific example of the sense information. The second subject R2 is a subject to whom the sensation of the first subject R1 is transmitted. Regarding the relationship between the reference sensory information and the corresponding sensory information, it is possible to associate them by conducting an experiment in advance. For example, the sensory information corresponding to the first subject R1 and the second subject R2 is used as a set of learning data sets, and machine learning is performed on the learning data sets to generate the second learning model. The second learning model can be stored in the storage unit 23, for example.

The second device 30 stimulates the second subject R2 to recall the estimated corresponding sensory information. The second device 30 has a detection unit 31 , a communication unit 32 , a processing unit 33 , a stimulus applying unit 34 and a storage unit 35 .

The detector 31 will be described later. In 1st Embodiment, the detection part 31 does not need to be provided.

The communication unit 32 is an interface that performs wired communication or wireless communication. The communication unit 32 receives correspondence sensation information transmitted from the estimation device 20 . The communication unit 32 also transmits the brain activation information detected by the detection unit 31 to the estimation device 20 . Note that the communication unit 32 may have the same configuration as the communication unit 12 described above.

The stimulation applying unit 34 applies an electromagnetic wave signal to the target site of the brain of the second subject R2 to activate the target site, thereby stimulating the second subject R2. In this case, the brain of the second subject R2 is partitioned, for example, by a three-dimensional matrix composed of voxels of several millimeters or less, and each voxel is irradiated with an electromagnetic wave. The stimulus imparting unit 34 can irradiate electromagnetic waves based on stimulation image information indicating to what extent the intensity of the electromagnetic waves is to be radiated to which voxel of the three-dimensional matrix. Voxels in the three-dimensional matrix of stimulus image information may correspond, for example, to voxels in the three-dimensional matrix of brain activation information in size, position, and the like. As to which voxel in the brain of the second subject R2 is irradiated with an electromagnetic wave of what intensity and what kind of sensory information is recollected, it is possible to obtain a correspondence relationship by conducting an experiment in advance. . For example, the stimulation image information for the second subject R2 and the sensory information recollected by the second subject R2 when electromagnetic waves are irradiated based on the stimulation image information are associated to form a set of learning data sets, and the learning data A third learning model can be generated by subjecting the set to machine learning. The third learning model can be stored in the storage unit 35 of the second device 30, for example. The processing unit 33 can calculate stimulus image information corresponding to the received corresponding sense information based on the corresponding sense information received by the communication unit 32 and the third learning model.

The above-described first learning model, second learning model and third learning model can be generated using a neural network (convolutional neural network) represented by VGG16, for example. FIG. 3 is a diagram showing an example of a neural network. As shown in the upper part of FIG. 3, the neural network NW has 13 convolutional layers S1, 5 pooling layers S2, and 3 fully connected layers S3. The neural network sequentially processes input information in a convolutional layer S1 and a pooling layer S2, and the processed results are combined in a fully connected layer S3 and output.

When generating the first learning model, the second learning model, and the third learning model, as shown in the middle part of FIG. Learn the correlation of the learning data set by machine learning such as deep learning. That is, the learning model is generated by optimizing the neural network NW through learning. For example, when one of the information constituting each learning data set is input, learning is performed so as to solve a problem for obtaining one of the information.

When inference is performed using the first learning model, the second learning model, and the third learning model, as shown in the lower part of FIG. Input to NW. From the first learning model, the second learning model, and the third learning model, the other information I2 corresponding to the input information I1 is output based on the learning result of the correlation of the information constituting the learning data set. be done. In this embodiment, an example in which a learning model is generated using a convolutional neural network represented by VGG16 is described, but the present invention is not limited to this, and a learning model is generated using another type of neural network. You may

Next, a sensation transmission method using the sensation transmission system 100 configured as described above will be described. FIG. 4 is a diagram schematically showing an example of the operation of the sensory transmission system 100. As shown in FIG. As shown in FIG. 4, the first subject R1 is made to perceive to recall the reference sensory information. A case where the first subject R1 visually perceives a cat's face and recalls it as visual information will be described below as an example.

As shown in the upper part of FIG. 4, the detection unit 11 detects the brain activation information 42 of the first subject R1 who perceives the cat's face and recalls the sensory information 41 . The communication unit 12 transmits the brain activation information 42 detected by the detection unit 11 to the estimation device 20 .

In the estimation device 20 , the communication unit 21 receives the brain activation information 42 transmitted from the first device 10 . The processing unit 22 estimates reference sensory information 43 based on the received brain activation information 42 . In this case, the processing unit 22 inputs the brain activation information 42 of the first subject R1 to the first learning model. The first learning model outputs reference sensory information 43 corresponding to the input brain activation information 42 based on the learning result of the correlation between the brain activation information 42 and the reference sensory information 43 . The processing unit 22 acquires the output reference sensation information 43 as an estimation result.

Based on the estimated reference sensation information 43, the processing unit 22 estimates corresponding sensation information 44 corresponding to the reference sensation information 43 for the second subject R2. In this case, the processing unit 22 inputs the acquired reference sensation information 43 to the second learning model. From the second learning model, corresponding sensation information 44 corresponding to the input reference sensation information 43 is output based on the learning result of the correlation between the reference sensation information 43 and the corresponding sensation information 44 . The processing unit 22 acquires the output corresponding sense information 44 as an estimation result. The communication unit 21 transmits the acquired corresponding sensation information 44 to the second device 30 .

In the second device 30 , the communication unit 32 receives the corresponding sensation information 44 transmitted from the estimation device 20 . The processing unit 33 inputs the received corresponding sense information 44 to the third learning model stored in the storage unit 35 . The third learning model outputs stimulation image information 45 corresponding to the inputted corresponding sensation information 44 based on the learning result of the correlation between the corresponding sensation information 44 and the stimulation image information. The stimulus imparting unit 34 stimulates the second subject R2 by irradiating the brain of the second subject R2 with electromagnetic waves based on the output stimulus image information 45 . As a result, the second subject R2 to whom the stimulation is applied by the stimulation applying unit 34 recalls the corresponding sensation information 46 corresponding to the stimulation image information 45. FIG. That is, the second subject R2 recalls the visual information of the cat's face as the corresponding sensory information 46. FIG.

On the other hand, since there is a difference in personality between the first subject R1 and the second subject R2, the correspondence between the brain activity state and the brain activation information is different. For this reason, as shown in the lower part of FIG. 4 (indicated by a dashed line), the brain activation information 42 of the first subject R1 is directly transmitted to the second device 30, and the brain activation information 42 is transmitted to the second device 30. If the brain of the second subject R2 is stimulated so as to correspond to , there is a high possibility that the second subject R2 will recall, as the sensory information 47, visual information different from the cat's face. In this case, the sensory information of the first subject R1 is not properly transmitted to the second subject R2.

On the other hand, in the sensory transfer system 100 of the present embodiment, the estimation device 20 estimates the corresponding sensory information 44 of the second subject R2. Visual information is properly conveyed.

FIG. 5 is a flow chart showing an example of the operation of the sensory transmission system 100. As shown in FIG. As shown in FIG. 5, in the sensory transmission system 100, the first device 10 detects brain activation information of the first subject R1 when perceived by the first subject R1 (step S101). Next, the estimation device 20 estimates reference sensory information recalled by the first subject R1 with respect to perception based on the brain activation information of the first subject R1 (step S102). Next, the estimation device 20 estimates corresponding sensory information corresponding to the reference sensory information for the second subject R2, which is different from the first subject R1, based on the estimated reference sensory information (step S103). Then, the second device 30 gives a stimulus to the second subject R2 so as to recall the estimated corresponding sensory information (step S104).

As described above, the sensory transmission system 100 according to the present embodiment includes the first device 10 that detects the brain activation information of the first subject R1 when the first subject R1 perceives it, and the detected first subject R1 Based on the brain activation information of R1, reference sensory information, which is sensory information recollected for perception, is estimated, and based on the estimated reference sensory information, a reference for the second subject R2 different from the first subject R1 is calculated. An estimating device 20 for estimating corresponding sensory information, which is sensory information corresponding to sensory information, and a second device 30 for stimulating the second subject R2 so as to recall the estimated corresponding sensory information.

The sensory transmission method according to the present embodiment detects the brain activation information of the first subject R1 when the first subject R1 perceives it, and based on the detected brain activation information of the first subject R1 Then, based on the estimated reference sensory information, corresponding sensory information corresponding to the reference sensory information of the second subject R2, which is different from the first subject R1, is calculated. estimating; and stimulating the second subject R2 to recall the estimated corresponding sensory information.

According to this configuration, instead of causing the second subject R2 to recall the reference sensory information of the first subject R1 as it is, the corresponding sensory information of the second subject R2 is estimated based on the reference sensory information, and the estimated correspondence is obtained. In order to stimulate the second subject R2 to recall the sensory information, even if there is a difference in brain activity when recalling the sensory information between the first subject R1 and the second subject R2, the first Sensory information can be appropriately transmitted from the subject R1 to the second subject R2.

In the sensory transmission system 100 according to this embodiment, the reference sensory information is sensory information recalled by the first subject R1 who has detected the brain activation information. In this configuration, by directly associating the sensory information of the first subject R1 with the sensory information of the second subject R2, the sensory information can be appropriately transmitted.

In the sensory transmission system 100 according to the present embodiment, stimulation includes irradiating a target region of the brain of the second subject R2 with an electromagnetic wave signal to activate the target region. In this configuration, by directly activating the brain of the second subject R2, it is possible to make the second subject R2 more directly recall sensory information.

[Second embodiment]
Next, a second embodiment will be described. In the first embodiment, the case where the sensory transmission system 100 transmits sensations in one direction from the first subject R1 to the second subject R2 has been described as an example. On the other hand, in the second embodiment, the sensory transmission system 100 also transmits sensations from the second subject R2 to the first subject R1. In other words, the sensory transmission system 100 is configured to bi-directionally transmit sensory sensations between the first subject R1 and the second subject R2.

The overall configuration of the sensory transmission system 100 is similar to that of the first embodiment. The configuration of the sensory transmission system 100 will be described below from the side of the second device 30 with reference to FIGS. 1 and 2. FIG.

The second device 30 has a detection unit 31 , a communication unit 32 , a processing unit 33 , a stimulus applying unit 34 and a storage unit 35 . The processing unit 33, the stimulation applying unit 34, and the storage unit 35 are the same as in the first embodiment. The detection unit 31 detects brain activation information of the second subject R2, like the detection unit 11 in the first embodiment. The communication unit 32 transmits the brain activation information detected by the detection unit 31 to the estimation device 20 .

The estimation device 20 has a communication unit 21, a processing unit 22, and a storage unit 23, like the estimation device 20 of the first embodiment. The communication unit 21 is capable of wired communication or wireless communication. In this embodiment, the communication unit 21 receives brain activation information transmitted from the second device 30, for example. The communication unit 21 transmits, for example, correspondence sensation information, which will be described later and is estimated by the processing unit 22 , to the first device 10 .

The processing unit 22 estimates sensory information (reference sensory information) about the sensation recalled by the second subject R2 based on the detected brain activation information of the second subject R2. Regarding the second subject R2, it is possible to obtain a correspondence relationship by conducting an experiment in advance as to what kind of brain activation information is generated when what kind of sensory information is recollected. For example, brain activation information detected from the second subject R2 and sensory information corresponding to the brain activation information are associated to form a set of learning data sets, and machine learning is performed on the learning data sets. can generate a fourth learning model. The fourth learning model can be stored in the storage unit 23, for example.

The processing unit 22 also estimates corresponding sensation information, which is sensation information corresponding to the reference sensation information for the first subject R1, based on the estimated reference sensation information. In this case, the processing unit 22 can perform estimation based on the second learning model stored in the storage unit 23 .

The first device 10 has a detection unit 11 , a communication unit 12 , a processing unit 13 , a stimulus applying unit 14 and a storage unit 15 . The detection unit 11 and the communication unit 12 have the same configuration as in the first embodiment. Note that the communication unit 12 receives correspondence sensation information transmitted from the estimation device 20 .

The processing unit 13 estimates stimulus image information corresponding to the corresponding sensation information received by the communication unit 12 . The stimulus image information is information indicating the content of the stimulus to be applied to the first subject R1 by the stimulus applying unit 14 .

The stimulus imparting unit 14 stimulates the first subject R1 by irradiating the target site of the brain of the first subject R1 with an electromagnetic wave signal to activate the target site. In this case, the brain of the first subject R1 is partitioned, for example, by a three-dimensional matrix composed of voxels of several millimeters or less, and an electromagnetic wave is applied to each voxel, as in the stimulation applying unit 34 in the first embodiment. The stimulus imparting unit 14 can irradiate electromagnetic waves based on stimulation image information indicating the intensity of electromagnetic waves to which voxels in the three-dimensional matrix are to be irradiated. As to which voxel in the brain of the first subject R1 is irradiated with an electromagnetic wave of what intensity and what kind of sensory information is recollected, it is possible to obtain a correspondence relationship by conducting an experiment in advance. . For example, the stimulation image information for the first subject R1 and the sensory information recalled by the first subject R1 when electromagnetic waves are irradiated based on the stimulation image information are associated to form a set of learning data sets, and the learning data A fifth learning model can be generated by subjecting the set to machine learning. The fifth learning model can be stored in the storage unit 15 of the first device 10, for example.

The above-described fourth learning model and fifth learning model can be generated using a neural network represented by VGG16, for example, like the first to third learning models. When generating the fourth learning model and the fifth learning model, each learning data set is input to the neural network, and the correlation of the learning data set is learned by machine learning such as deep learning. In other words, the learning model is generated by optimizing the neural network through learning. Note that the learning model may be generated using other types of neural networks, without being limited to the convolutional neural network represented by VGG16.

Next, a sensation transmission method using the sensation transmission system 100 configured as described above will be described. The sensation transmission method for transmitting the sensation from the first subject R1 to the second subject R2 is the same as in the first embodiment. In this embodiment, a case will be described where sensation is transmitted from the second subject R2 to the first subject R1.

FIG. 6 is a diagram schematically showing an example of the operation of the sensory transmission system 100. As shown in FIG. As shown in FIG. 6, the second subject R2 is made to perceive to recall the reference sensory information. A case where the second subject R2 visually perceives a cat's face and recalls it as visual information will be described below as an example. In the following examples, the cat's face is the cat's face as in the first embodiment.

As shown in the upper part of FIG. 6, the detection unit 31 detects the brain activation information 52 of the second subject R2 who perceives the cat's face and recalls the sensory information 51 . The communication unit 32 transmits the brain activation information 52 detected by the detection unit 31 to the estimation device 20 .

In the estimation device 20 , the communication unit 21 receives the brain activation information 52 transmitted from the third device 130 . The processing unit 22 estimates reference sensory information 53 based on the received brain activation information 52 . In this case, the processing unit 22 inputs the brain activation information 52 of the second subject R2 to the fourth learning model. The fourth learning model outputs reference sensory information 53 corresponding to the input brain activation information 52 based on the learning result of the correlation between the brain activation information 52 and the reference sensory information 53 . The processing unit 22 acquires the output reference sensation information 53 as an estimation result.

Based on the estimated reference sensation information 53, the processing unit 22 estimates corresponding sensation information 54 corresponding to the reference sensation information 53 for the first subject R1. In this case, the processing unit 22 inputs the acquired reference sensation information 53 to the second learning model. From the second learning model, corresponding sensation information 54 corresponding to the input reference sensation information 53 is output based on the learning result of the correlation between the reference sensation information 53 and the corresponding sensation information 54 . The processing unit 22 acquires the output corresponding sense information 54 as an estimation result. The communication unit 21 transmits the acquired corresponding sensation information 54 to the first device 10 .

As shown in the lower part of FIG. 6 , in the first device 10 , the communication unit 12 receives the corresponding sensation information 54 transmitted from the estimation device 20 . The processing unit 13 inputs the received corresponding sense information 54 to the fifth learning model stored in the storage unit 15 . The fifth learning model outputs stimulation image information 55 corresponding to the inputted corresponding sensation information 54 based on the learning result of the correlation between the corresponding sensation information 54 and the stimulation image information 55 . The stimulation applying unit 14 gives stimulation to the first subject R1 by irradiating the brain of the first subject R1 with electromagnetic waves based on the output stimulation image information 55 . As a result, the first subject R1 to whom the stimulus is given by the stimulus applying section 14 recalls the corresponding sensation information 56 corresponding to the stimulus image information 55 . That is, the first subject R1 recalls the visual information of the cat's face as the corresponding sensory information 56 . Thus, the visual information of the cat's face is transmitted from the second subject R2 to the first subject R1.

As described above, in the sensory transmission system 100 according to the present embodiment, the reference sensory information is the sensory information recalled by the second subject R2 who has detected the brain activation information. In this configuration, by directly associating the sensory information of the second subject R2 with the sensory information of the first subject R1, it is possible to appropriately transmit sensory information between subjects having different brain activities. can.

[Third embodiment]
Next, a third embodiment will be described. In the first and second embodiments described above, the sensory information recalled by the subject who detected the brain activation information was described as the reference sensory information. In contrast, in the third embodiment, an example will be described in which standard sensory information extracted based on corresponding sensory information among a plurality of subjects is used as standard sensory information. The overall configuration of the sensory transmission system 100 is similar to that of the first embodiment.

In the estimation device 20, when estimating the corresponding sensation information from the reference sensation information, the standard sensation information is used as the reference sensation information. Standard sensory information can be the average value of brain activation information detected in a plurality of subjects when, for example, a plurality of detectors perform the same perception such as viewing the same image. Therefore, the standard sensory information has little individual difference due to acquired memory, etc., and is the sensory information of an average person.

Standard sensory information can be extracted from learning results when learning corresponding sensory information among multiple subjects. For example, each of brain activation information of a specific subject (for example, first subject R1 or second subject R2) and standard sensory information (average value of brain activation information of a plurality of subjects) is set as a learning data set. , a sixth learning model can be generated by performing machine learning on the learning data set. When generating the sixth learning model, standard sensory information is extracted from a plurality of sensory information included in the learning data set, and the extracted standard sensory information is associated with individual brain activation information included in the learning data set. You may do so. As a result, different types of sensations are generated, such as sensory information about sight when looking at an object, sensory information about hearing when hearing a sound, and sensory information about touch when touching an object. For the information, a sixth learning model is generated by machine-learning correspondence relationships between individual brain activation information and standard sensory information of a plurality of subjects. The sixth learning model can be stored in the storage unit 25, for example.

Next, a sensation transmission method using the sensation transmission system 100 configured as described above will be described. FIG. 7 is a diagram schematically showing an example of the operation of the sensory transmission system 100. As shown in FIG. For example, when transmitting the sensory information 61 when the first subject R1 perceives to the second subject R2, the first device 10 acquires the brain activation information 62 of the first subject R1, and the estimation device 20 Send.

In the estimation device 20, the processing unit 33 causes the storage unit 35 to store the brain activation information 62 transmitted from the first device 10 and the identification information of the second subject R2 to whom the sensory information is transmitted. Enter the sixth learning model. In the sixth learning model, standard sensory information 63 corresponding to brain activation information 62 is calculated, and sensory information of second subject R2 associated with the standard sensory information 63 is output as corresponding sensory information 64 . The communication unit 21 transmits the output corresponding sensation information 64 to the second device 30 .

Similarly to the first embodiment, the second device 30 receives the corresponding sensation information 64 transmitted from the estimation device 20 and acquires the stimulation image information 65 based on the received corresponding sensation information 64 . The stimulus imparting unit 34 stimulates the second subject R2 by irradiating the brain of the second subject R2 with electromagnetic waves based on the output stimulus image information 65 . The second subject R<b>2 to whom the stimulation is applied by the stimulation applying unit 34 recalls the corresponding sensation information 66 corresponding to the stimulation image information 65 .

As described above, in the sensory transmission system 100 according to the present embodiment, the reference sensory information is the standard sensory information 63 extracted based on sensory information corresponding among a plurality of subjects. In this configuration, the standard sensory information 63 extracted based on the corresponding sensory information among a plurality of subjects is used as the standard sensory information, so sensory information can be appropriately transmitted among many subjects.

[Fourth Embodiment]
FIG. 8 is a diagram showing an example of a sensory transmission system 200 according to the fourth embodiment. In the sensory transmission system 100 of the above embodiment, the case of transmitting sensory information between different subjects has been described. On the other hand, in the sensory transmission system 200 described in the fourth embodiment, a case where sensory information is transmitted between the same subjects will be described as an example.

As shown in FIG. 8 , the sensory transmission system 200 includes a detection stimulator (detector, stimulator) 110 and an estimator 120 . The detection stimulus device 110 has, for example, the same configuration as the first device 10 described in the above embodiment, and includes a detection unit 11, a communication unit 12, a processing unit 13, a stimulus application unit 14, and a storage unit 15. have. The detection unit 11 detects brain activation information. The communication unit 12 performs wired communication or wireless communication, and transmits the brain activation information detected by the detection unit 11 to the estimation device 20 . The processing unit 13 calculates stimulus image information based on the corresponding sensation information received by the communication unit 12 . Based on the calculated stimulation image information, the stimulus applying unit 14 irradiates the target region of the brain of the target subject R4 with an electromagnetic wave signal to activate the target region, thereby stimulating the target subject R4. The storage unit 15 stores various information.

Estimation device 120 includes communication unit 21 , processing unit 22 , and storage unit 23 . The communication unit 21 is capable of wired communication or wireless communication. In this embodiment, the communication unit 21 receives brain activation information transmitted from the detection stimulation device 110, for example. The communication unit 21 transmits corresponding sensation information, which will be described later and is estimated by the processing unit 22, to the detection stimulus device 110, for example.

The processing unit 22 estimates reference sensory information recollected for perception based on the brain activation information of the target subject R4 detected at the first time point. The first time point may be a time point when the target subject R4 is young, for example, the target subject R4 may be less than 3 years old.

As the reference sensory information, for example, sensory information recalled by the target subject R4 at the first time point can be used. In this case, the processing unit 22 can estimate the reference sensation information based on the seventh learning model similar to the first learning model described above. The seventh learning model can be stored in the storage unit 25, for example.

Further, the reference sensory information may be standard sensory information extracted based on sensory information corresponding among a plurality of subjects, for example. The standard sensory information can be, for example, information extracted based on sensory information recalled by a plurality of subjects whose brain growth states correspond to those of the target subject R4. Examples of such multiple subjects include, for example, multiple subjects with corresponding ages, multiple subjects with corresponding growth environments (latitude, cultural environment, language used, etc.), multiple subjects with the same or corresponding occupations, etc. mentioned. The processing unit 22 can use, for example, an eighth learning model that estimates standard sensory information based on the brain activation information of the target subject R4 at the first time point. The eighth learning model can be stored in the storage unit 25, for example.

Further, based on the estimated reference sensation information, the processing unit 22 estimates corresponding sensation information corresponding to the reference sensation information for the target subject R4 at a second time point after the first time point. For example, the degree of brain growth of target subject R4 may be significantly different between the first time point and the second time point. For example, the corresponding sensory information between the target subject R4 at the first time point and the target subject R4 at the second time point is set as a set of learning data sets, and the learning data set is subjected to machine learning to generate a ninth learning model. can do. The ninth learning model can be stored in the storage unit 23, for example.

Next, a sensation transmission method using the sensation transmission system 200 configured as described above will be described. FIG. 9 is a diagram schematically showing an example of the operation of the sensory transmission system 100. As shown in FIG. As shown in the upper part of FIG. 9, the detection unit 11 of the detection stimulation device 110 detects brain activation information 72 of the target subject R4 who perceives the cat's face and recalls the sensory information 71 . The communication unit 12 transmits the brain activation information 72 detected by the detection unit 11 to the estimation device 120 . In the estimation device 120 , the communication unit 21 receives the brain activation information 72 transmitted from the detection stimulation device 110 . The processing unit 22 inputs the received brain activation information to the seventh learning model or the eighth learning model. From the seventh learning model or the eighth learning model, reference sensory information 73 corresponding to the input brain activation information 72 is output. The processing unit 22 acquires the output reference sensation information 73 as an estimation result. The storage unit 25 stores the acquired reference sensation information 73 .

As shown in the lower part of FIG. 9 , at a second point in time after the first point in time, for example, when the target subject R4 tries to relive the perception of the cat's face, the subject R4 uses the detection stimulation device 110 Prepare to be in a state in which stimulation can be applied from the stimulation applying unit 14 of . Based on the reference sensation information 73 stored in the storage unit 25, the processing unit 22 estimates corresponding sensation information 74 corresponding to the reference sensation information 73 for the target subject R4. In this case, the processing unit 22 inputs the acquired reference sensation information 73 to the ninth learning model. Corresponding sense information 74 corresponding to the input reference sense information 73 is output from the ninth learning model. The processing unit 22 acquires the output corresponding sense information 74 as an estimation result. The communication unit 21 transmits the acquired corresponding sensory information 74 to the detection stimulation device 110 .

In the detection stimulus device 110 , the communication unit 12 receives the corresponding sensory information 74 transmitted from the estimation device 120 . The processing unit 13 inputs the received corresponding sensation information 74 to the third learning model stored in the storage unit 15 . The third learning model outputs stimulus image information 75 corresponding to the input corresponding sense information 74 . The stimulus applying unit 14 gives stimulation to the target subject R4 by irradiating the brain of the target subject R4 with electromagnetic waves based on the output stimulation image information 75 . As a result, the target subject R4 to whom the stimulus is applied by the stimulus applying unit 14 recalls the corresponding sensation information 76 corresponding to the stimulus image information 75. FIG. That is, the target subject R4 recalls the visual information when he saw the cat's face at the first time point as the corresponding sensory information 76 at the second time point. Thus, the visual information of the cat's face is transmitted from the target subject R4 at the first time point to the target subject R4 at the second time point. This allows the target subject R4 to relive seeing the cat's face.

FIG. 10 is a flow chart showing an example of the operation of the sensory transmission system 200. As shown in FIG. As shown in FIG. 10, in the sensory transmission system 200, the detection stimulation device 110 detects brain activation information perceived by the target subject R4 at a first time point (step S201). Next, the estimation device 20 estimates reference sensory information at the first time point based on the brain activation information of the target subject R4 (step S202). Next, the estimation device 20 estimates corresponding sensory information corresponding to the reference sensory information for the second subject R2, which is different from the first subject R1, based on the estimated reference sensory information (step S103). Then, the second device 30 gives a stimulus to the second subject R2 so as to recall the estimated corresponding sensory information (step S104).

As described above, the sensory transmission system 200 according to the present embodiment includes a detection device (detection stimulation device 110) that detects activation information in the brain of the target subject R4 when the target subject R4 perceives it, and at the first time point Based on the detected brain activation information of target subject R4, reference sensory information, which is sensory information recollected for perception, is estimated, and based on the estimated reference sensory information, time has passed since the first time point. an estimating device 120 for estimating corresponding sensory information corresponding to the reference sensory information for the target subject R4 at the second time point, and a stimulus that stimulates the target subject R4 at the second time point so as to recall the estimated corresponding sensory information. device (detection stimulator 110).

In addition, the sensory transmission method according to the present embodiment detects the brain activation information of the target subject R4 when the target subject R4 perceives it, and detects the brain activation information of the target subject R4 detected at the first time point. Based on the information, reference sensory information, which is sensory information recollected for perception, is estimated, and based on the estimated reference sensory information, a reference for the target subject R4 at a second point in time after the first point in time. estimating corresponding sensory information corresponding to the sensory information; and stimulating target subject R4 at a second time point to recall the estimated corresponding sensory information.

According to this configuration, the reference sensory information of the target subject R4 at the first time point is not directly recollected in the target subject R4 at the second time point, but the corresponding response of the target subject R4 at the second time point is based on the reference sensory information. The sensory information is estimated, and the target subject R4 is stimulated to recall the estimated corresponding sensory information. Therefore, even if there is a difference in brain activity when recalling sensory information between the target subject R4 at the first time point and the target subject R4 at the second time point, the sensory information can be appropriately transmitted. .

In the sensory transmission system 200 according to the present embodiment, the reference sensory information is sensory information recalled by the target subject R4 at the first time point. In this configuration, by directly associating the sensory information of the target subject R4 at the first time point with the sensory information of the target subject R4 at the second time point, the sensory information can be appropriately transmitted.

In the sensory transmission system 200 according to this embodiment, the reference sensory information is standard sensory information extracted based on sensory information corresponding among a plurality of subjects. In this configuration, the standard sensory information extracted based on the corresponding sensory information among a plurality of subjects is used as the standard sensory information, so that the sensory information can be efficiently transmitted between the target subjects R4 at different points in time. can be done.

In the sensory transmission system 200 according to the present embodiment, standard sensory information is extracted based on sensory information recalled by a plurality of subjects whose brain growth states correspond to those of the target subject R4. In this configuration, sensory information can be efficiently transmitted between target subjects R4 at different points in time based on sensory information recalled by a plurality of subjects whose brain growth states correspond to that of target subject R4.

[Fifth embodiment]
FIG. 11 is a diagram showing an example of a sensory transmission system 300 according to the fifth embodiment. As shown in FIG. 10 , a sensory transmission system 300 according to the fifth embodiment includes a sensory estimation device 220 and a stimulation device 210 .

Sense estimation device 220 includes communication unit 221 , processing unit 222 , storage unit 223 , and input unit 224 . The communication unit 221 performs wired communication or wireless communication with the stimulation device 210 . The storage unit 223 stores reference sensory information, which is sensory information recalled by perception. The storage unit 223 has storage such as a hard disk drive and a solid state drive. Note that an external storage medium such as a removable disk may be used as the storage unit 223 . The reference sensory information may be sensory information recalled by the subject R5 or a person different from the subject R5, or may be standard sensory information extracted based on corresponding sensory information among a plurality of subjects. There may be.

Based on the reference sensation information stored in the storage unit 223, the processing unit 222 estimates corresponding sensation information, which is sensation information corresponding to the reference sensation information for the subject R5. For example, the sensory information corresponding to the sensory information of the subject R5 and the reference sensory information may be used as a set of learning data sets, and the learning data set may be machine-learned to generate the tenth learning model. The tenth learning model can be stored in the storage unit 223, for example. For example, when reference sensation information is specified by an input unit 224, which will be described later, the processing unit 222 estimates corresponding sensation information corresponding to the specified reference sensation information.

The input unit 224 is capable of a predetermined input operation for inputting information. As the input unit 224, for example, an input device such as a keyboard or touch panel is used. As the input unit 224, buttons, levers, dials, switches, or other input devices may be used in addition to or instead of these. The input unit 224 can input sensory information that the subject R5 wants to experience from, for example, a plurality of reference sensory information stored in the storage unit 223 .

The stimulation device 210 has a communication section 212 , a processing section 213 , a stimulation applying section 214 and a storage section 215 . The communication unit 212 is capable of wired communication or wireless communication. The communication unit 212 receives corresponding sensation information transmitted from the sensation estimation device 220 .

The processing unit 213 generates a stimulus image corresponding to the received corresponding sense information based on the corresponding sense information received by the communication unit 212 and the eleventh learning model that has learned the correspondence relationship between the corresponding sense information and the stimulus image information. Calculate information. This eleventh learning model can be, for example, a learning model similar to the third learning model in the above-described first embodiment. The stimulation applying unit 214 applies an electromagnetic wave signal to the target region of the brain of the subject R5 to activate the target region, thereby stimulating the subject R5.

Next, a sensation transmission method using the sensation transmission system 300 configured as described above will be described. FIG. 12 is a diagram schematically showing an example of the operation of the sensory transmission system 300 according to this embodiment. The subject R5 prepares to be able to receive stimulation from the stimulation applying section 214 of the stimulation device 210 . When the subject R5 selects the reference sensation information 81 via the input unit 224, the processing unit 222 in the sensation estimation device 220 generates the corresponding sensation information 82 for the subject R5 based on the selected reference sensation information 81. presume. The processing unit 222 inputs, for example, the selected reference sense information 81 to the tenth learning model. Corresponding sense information 82 corresponding to the input reference sense information 81 is output from the tenth learning model. The processing unit 222 acquires the output corresponding sense information 82 as an estimation result. The communication unit 221 transmits the acquired corresponding sensation information 82 to the stimulation device 210 .

In the stimulation device 210 , the communication section 212 receives the corresponding sensation information 82 transmitted from the sensation estimation device 220 . The processing unit 213 inputs the received corresponding sense information 82 to the eleventh learning model stored in the storage unit 215 . The eleventh learning model outputs stimulus image information 83 corresponding to the input corresponding sensory information 82 . The stimulation applying unit 214 gives stimulation to the subject R5 by irradiating the brain of the subject R5 with electromagnetic waves based on the output stimulation image information 83 . As a result, the subject R5 to whom the stimulus is given by the stimulus applying section 214 recalls the corresponding sensation information 84 corresponding to the stimulus image information 83. FIG. In other words, subject R5 recalls the visual information of the cat's face as the corresponding sensory information 84 . Thus, the visual information of the cat's face is transmitted from the sensation estimation device 220 to the subject R5.

FIG. 13 is a flow chart showing an example of the operation of the sensory transmission system 300. As shown in FIG. As shown in FIG. 13, when the subject R5 selects the reference sensation information 81 via the input unit 224, the processing unit 222 in the sensation estimation device 220 acquires the selected reference sensation information 81 (step S301). ), the corresponding sensory information 82 for the subject R5 is estimated based on the acquired reference sensory information 81 (step S302). Then, the stimulation device 210 stimulates the fifth subject R5 so as to recall the estimated corresponding sensory information 82 (step S303).

As described above, the sensation estimation device 220 according to the present embodiment includes the storage unit 223 that stores the reference sensation information, which is the sensation information that is recollected by perception, and based on the reference sensation information that is stored in the storage unit 223 . and a processing unit 222 for estimating corresponding sensory information, which is sensory information corresponding to the reference sensory information for the subject R5.

In addition, the sensation estimation method according to the present embodiment acquires reference sensation information from the storage unit 223 that stores reference sensation information, which is sensation information recalled by perception, and based on the acquired reference sensation information. , and estimating corresponding sensory information that is sensory information corresponding to the reference sensory information for subject R5.

According to this configuration, since the corresponding sensory information corresponding to the subject R5 is estimated based on the reference sensory information of the sensory estimation device 220, appropriate sensory information can be estimated for each subject R5.

In the sensation estimation device 220 according to the present embodiment, the reference sensation information is sensation information recalled by a person other than the subject R5. According to this configuration, appropriate sensory information can be estimated between subjects having different brain activities.

In the sensation estimation device 220 according to this embodiment, the reference sensation information is standard sensation information extracted based on corresponding sensation information among the plurality of subjects R5. According to this configuration, sensory information can be appropriately transmitted among many subjects.

The sensory transfer system 300 according to this embodiment includes the sensory estimator 220 and the stimulator 210 that stimulates the subject R5 so as to recall the corresponding sensory information estimated by the sensory estimator 220. FIG. According to this configuration, appropriate sensory information can be estimated for each subject R5 based on the reference sensory information stored in the storage unit 223, and can be transmitted to the subject R5.

The technical scope of the present invention is not limited to the above embodiments, and modifications can be made as appropriate without departing from the scope of the present invention. For example, in each of the above-described embodiments, visual sensory information was described as an example, but the present invention is not limited to this. good. Further, for example, detailed sensory information obtained by interlocking sensory information of the five senses may be used.

When generating the learning model, the content presented to the subject may be an instruction such as "raise your right hand", and the response of the subject's motor cortex may be learned. Also, when generating the learning model, the content presented to the subject may be viewing content such as a movie, and the subject's entire brain reaction may be learned.

Further, an ID may be assigned to an element of the standard sense information obtained when generating the learning model, and the closer the correlation with the content, the closer the ID may be. For example, if there are contents of "dog", "cat", and "paper", assign close IDs to "dog" and "cat", and assign distant IDs to "dog", "cat" and "paper". may

Also, if there is no corresponding sense information corresponding to the standard sense information, content having a similar ID set as described above may be used as the corresponding sense information, or the corresponding sense information may not be transmitted.

Corresponding sensory information derived from certain standard sensory information may be replaced with other sensory information and presented. For example, based on the standard sensory information corresponding to the visual information of the orange, the corresponding sensory information replaced with the olfactory information, the taste information, and the tactile information of the orange may be transmitted to the transmission destination.

Also, when the sensory information is transmitted from the first subject R1 to the second subject R2, the command information "send this sensory information to the second subject R2" is deleted, and "the first subject R1 feels like this" is deleted. Even if communication control information corresponding to the packet header such as "I am recalling information" and "The first subject R1 is requesting the second subject R2 to recall sensory information in this way" good.

Further, when extracting standard sense information, learning may be done online so that the contents are constantly updated. Moreover, each estimated sensory information may be stored in a storage unit and may be transmitted to a transmission destination after a predetermined period of time has passed.

NW... neural network, R1... first subject, R2... second subject, R4... subject subject, R5... subject, S1... convolution layer, S2... pooling layer, S3... coupling layer, 10... first device, 11, 31 Detecting unit 12, 21, 32, 212, 221 Communication unit 13, 22, 33, 213, 222 Processing unit 14, 34, 214 Stimulation applying unit 15, 23, 25, 35, 215, 223... Storage unit 20, 120... Estimation device 30... Second device 41, 47, 51, 61, 71... Sensory information 42, 52, 62, 72... Brain activation information 43, 53, 73 , 81... reference sensory information, 44, 46, 54, 56, 64, 66, 74, 76, 82, 84... corresponding sensory information, 45, 55, 65, 75, 83... stimulus image information, 63... standard sensory information , 100, 200, 300... Sensory transmission system, 110... Detection stimulator, 130... Third device, 210... Stimulator, 220... Sensory estimation device, 224... Input unit

Claims (5)

a first device for detecting brain activation information of the first subject when perceived by the first subject;
Based on the detected brain activation information of the first subject, reference sensory information that is sensory information recollected for the perception is estimated, and the first subject is based on the estimated reference sensory information. an estimation device for estimating corresponding sensory information, which is sensory information corresponding to the reference sensory information for a different second subject;
a second device for stimulating the second subject to recall the estimated corresponding sensory information. The second device detects brain activation information of the second subject when the second subject is stimulated,
The estimation device estimates the reference sensory information based on the detected brain activation information of the second subject, and estimates the corresponding sensory information for the first subject based on the estimated reference sensory information. death,
2. The sensory transmission system of claim 1, wherein the first device stimulates the first subject to recall the estimated corresponding sensory information. 3. The sensory information according to claim 1, wherein the reference sensory information is the sensory information recalled by the subject who detected the brain activation information among the first subject and the second subject. sensory transmission system. 3. The sensory transmission system according to claim 1, wherein the reference sensory information is standard sensory information extracted based on the sensory information corresponding among a plurality of subjects. Detecting brain activation information of the first subject when perceived by the first subject;
Based on the detected brain activation information of the first subject, reference sensory information that is sensory information recollected for the perception is estimated, and the first subject is based on the estimated reference sensory information. estimating corresponding sensory information that is sensory information corresponding to the reference sensory information for a different second subject;
and stimulating the second subject to recall the estimated corresponding sensory information.

Images