JP7075045B2 - Estimating system and estimation method - Google Patents

Description

The present invention relates to an estimation system and an estimation method.

In recent years, a technique for measuring the brain activity of a subject and estimating the internal state of the subject such as emotions has been known (for example, Patent Document 1). For example, in the prior art described in Patent Document 1, the brain activity induced by a training stimulus such as a moving image is measured, and the brain activity and the perceptual meaning content are expressed based on the set data of the training stimulus and the brain activity. Learn the correspondence with the perceptual meaning space (brain information expression space). Then, in the prior art, the perceptual meaning content (internal state of the subject) is estimated from the brain activity measured when the new stimulus is given to the subject by using the learned correspondence.

Japanese Unexamined Patent Publication No. 2016-195716

However, in the above-mentioned conventional technique, it is necessary to measure the brain activity for the new sensory information every time the internal state of the subject (estimation reference person) is estimated for the stimulus information which is the new stimulus, which is rapid. Moreover, it was difficult to easily estimate the internal state of the subject (estimation reference person) using brain information. Further, in the above-mentioned conventional technique, prior knowledge such as a large-capacity corpus is required in order to express the meaning and content of the internal state.

The present invention has been made to solve the above problems, and an object thereof is an estimation system capable of quickly and easily estimating the internal state of an estimation reference person for stimulation information using brain information. , And to provide an estimation method. In addition, the estimation of the internal state directly linked to the information expression in the brain is achieved without using prior knowledge such as a large-scale corpus for the internal state.

In order to solve the above problem, one aspect of the present invention includes a feature amount extraction unit that extracts a plurality of feature amounts related to a plurality of sensations from stimulus information to be estimated, each feature amount related to the plurality of sensations, and the above. Based on the first correspondence information indicating the correspondence relationship with the first brain information corresponding to each of the plurality of sensations, the plurality of feature quantities extracted from the stimulus information of the estimation target by the feature quantity extraction unit can be obtained. The correspondence between the first conversion unit that converts a plurality of the first brain information corresponding to each, the first brain information, and at least one second brain information corresponding to the internal state of the estimation reference person. Based on the second correspondence information shown, the plurality of first brain information converted by the first conversion unit is converted into the second brain information by the second conversion unit and the second conversion unit. It is an estimation system including an estimation unit that estimates the internal state corresponding to the stimulus information of the estimation target by using the second brain information.

Further, in one aspect of the present invention, in the estimation system, the first brain information is information that defines the dissimilarity of the information pattern of the brain activity caused by the sensation as a distance, and is the second. The brain information is information that defines the dissimilarity of the information pattern of the brain activity caused by the internal state as a distance, and the second conversion unit is based on the second correspondence information. The position in the distance space indicated by the plurality of first brain information is converted into the position in the distance space indicated by the second brain information, and the estimation unit uses the position information in the distance space indicated by the second brain information. , The internal state may be estimated.

Further, in one aspect of the present invention, in the above estimation system, a plurality of feature quantities relating to the plurality of sensations extracted from the stimulus information for learning and a stimulus by the stimulus information for learning are given to the estimation reference person. Based on the measurement result of the brain activity at the time, the first generation unit that generates the first correspondence information corresponding to each of the plurality of sensations, and the internal unit corresponding to the stimulus information for learning. Based on the information indicating the state and the measurement result of the brain activity when the estimation reference person is stimulated by the stimulus information for learning, the information indicating the internal state and the second brain information Based on the second generation unit that generates the third correspondence information indicating the correspondence relationship, the position in the distance space indicated by the plurality of first brain information corresponding to the stimulus information for learning, and the third correspondence information. A third generation unit that generates the second correspondence information may be provided based on the position in the distance space indicated by the second brain information generated from the information indicating the internal state. ..

Further, in one aspect of the present invention, in the estimation system, the feature amount extraction unit extracts the feature amount every predetermined unit time, and the estimation unit extracts the feature amount every predetermined unit time. 2. Based on the position in the distance space indicated by the brain information, the internal state may be estimated every predetermined unit time corresponding to the stimulus information of the estimation target.

Further, in one aspect of the present invention, in the estimation system, the plurality of sensations are visual and auditory, and the feature amount extracting unit includes the visual feature amount related to the visual sense and the auditory feature amount related to the auditory sense. Is extracted, and the first conversion unit converts the visual feature amount into a position in the visual space indicated by the first brain information corresponding to the vision, based on the first correspondence information corresponding to the vision. Based on the first corresponding information corresponding to the visual sense, the auditory feature amount is converted into a position in the auditory space indicated by the first brain information corresponding to the visual sense, and the second conversion unit performs the second conversion unit. Based on the corresponding information of 2, the position in the visual space and the position in the auditory space converted by the first conversion unit may be converted into the position in the distance space indicated by the second brain information.

Further, in one aspect of the present invention, in the estimation system, the internal state of the estimation reference person includes the emotion of the estimation reference person, and the second conversion unit is the plurality of first brains. The plurality of first brain information indicates based on the second correspondence information indicating the correspondence relationship between the position in the distance space indicated by the information and the position in the emotion space indicated by the second brain information corresponding to the emotion. The position in the distance space is converted into the position in the emotion space, and the estimation unit corresponds to the stimulus information of the estimation target based on the position information in the emotion space converted from the positions in the plurality of distance spaces. The emotion may be estimated.

Further, in one aspect of the present invention, in the estimation system, the feature quantities relating to the plurality of sensations and the information indicating the internal state are multidimensional information, and the information indicating the internal state is the information. , Information having a dimension number smaller than the dimension number of the feature quantity relating to the plurality of sensations may be used.

Further, one aspect of the present invention is a measurement result of a plurality of feature quantities relating to a plurality of sensations extracted from the stimulus information for learning and a brain activity when the estimation reference person is stimulated by the stimulus information for learning. Based on the above, the first brain information, which is information that defines the dissimilarity between each feature amount of the plurality of sensations and the information pattern of the brain activity caused by the sensations as a distance, is the plurality of pieces. The internal state of the estimation reference person corresponding to the stimulus information for learning and the first generation unit that generates the first correspondence information indicating the correspondence relationship with the first brain information corresponding to each of the sensations of. Based on the information shown and the measurement result of the brain activity when the estimation reference person is stimulated by the stimulus information for learning, the information indicating the internal state and the internal state are used as factors. The second brain information, which is information that defines the dissimilarity of the information pattern of the induced brain activity as a distance, and is the second correspondence information showing the correspondence with the second brain information corresponding to the internal state. The second generation unit, the plurality of first brain information corresponding to the stimulus information for learning generated from the plurality of feature quantities based on the first correspondence information, and the second It is the third correspondence information which shows the correspondence relation with the 2nd brain information generated from the information which shows the internal state based on the correspondence information, and when explaining each element included in the 2nd brain information. Based on the third generation unit that generates the contribution degree of each element included in the plurality of first brain information, the plurality of feature quantities, the information indicating the internal state, and the contribution degree of each element. A model building unit that builds a mathematical model that estimates information indicating the internal state from the plurality of feature quantities, and a feature quantity extraction unit that extracts a plurality of feature quantities related to a plurality of sensations from the stimulus information to be estimated. And, based on the mathematical model constructed by the model building unit, the above-mentioned items corresponding to the stimulus information of the estimation target from the plurality of feature quantities extracted from the stimulus information of the estimation target by the feature amount extraction unit. It is an estimation system including an estimation unit that estimates a target state.

Further, in one aspect of the present invention, the feature amount extraction unit extracts a plurality of feature amounts related to a plurality of sensations from the stimulus information to be estimated, and the first conversion unit extracts the plurality of sensations. The feature amount extracted from the stimulus information of the estimation target by the feature amount extraction step based on the first correspondence information showing the correspondence relationship between the feature amount of the above and the first brain information corresponding to each of the plurality of sensations. The first conversion step of converting a plurality of feature quantities into a plurality of the first brain information corresponding to each, and the second conversion unit correspond to the first brain information and the internal state of the estimation reference person. A second that converts the plurality of first brain information converted by the first conversion step into the second brain information based on the second correspondence information indicating the correspondence relationship with at least one second brain information. An estimation including a conversion step and an estimation step in which the estimation unit estimates the internal state corresponding to the stimulus information of the estimation target by using the second brain information converted by the second conversion step. The method.

According to the present invention, the internal state of the subject with respect to the stimulus information can be quickly and easily estimated.

It is a functional block diagram which shows an example of the estimation system by 1st Embodiment. It is a flowchart which shows an example of the learning process of the estimation system by 1st Embodiment. It is a figure explaining an example of the metric space construction process in 1st Embodiment. It is a figure explaining an example of the construction process of the spatial correspondence relation in 1st Embodiment. It is a flowchart which shows an example of the estimation process of the estimation system by 1st Embodiment. It is a figure explaining an example of the estimation process of the estimation system by 1st Embodiment. It is a figure explaining the effect of the estimation system by 1st Embodiment. It is a functional block diagram which shows an example of the estimation system by 2nd Embodiment. It is a flowchart which shows an example of the learning process of the estimation system by 2nd Embodiment. It is a figure explaining an example of the construction process of the regression model which estimates the emotion rating in 2nd Embodiment. It is a flowchart which shows an example of the estimation process of the estimation system by 2nd Embodiment. It is a figure explaining an example of the estimation process of the estimation system by 2nd Embodiment.

Hereinafter, the estimation system according to the embodiment of the present invention will be described with reference to the drawings.

[First Embodiment]
FIG. 1 is a functional block diagram showing an example of the estimation system 1 according to the first embodiment.
As shown in FIG. 1, the estimation system 1 includes an analysis device 10, a display device 21, a speaker 22, an fMRI (functional Magnetic Resonance Imaging) 30, and an estimation device 40.

The display device 21 is, for example, a liquid crystal display or the like, and displays a learning image based on learning moving image data (an example of learning stimulus information, hereinafter referred to as learning moving image data). It is assumed that the moving image data includes image data and sound data.
The speaker 22 (an example of a sound emitting device) outputs a learning sound based on the learning moving image data.

The display device 21 and the speaker 22 output an image and a sound based on the learning moving image data, and allow the estimation reference person S1 to view the image and the sound. The estimation reference person S1 is, for example, a subject who serves as a reference in the estimation of emotion rating.

The fMRI30 (an example of the brain activity measurement unit) measures the brain activity of the estimation reference person S1 when the estimation reference person S1 is stimulated. The stimulus is, for example, an image and a sound based on learning moving image data output by the display device 21 and the speaker 22. The fMRI 30 outputs an fMRI signal (brain activity signal) that visualizes the blood flow dynamic response related to the brain activity of the estimation reference person S1. The fMRI 30 measures the brain activity of the estimation reference person S1 at predetermined unit times (for example, every 2 seconds), and outputs the measured measurement result to the analysis device 10 as an fMRI signal.

The analysis device 10 includes a storage unit 11 and a control unit 12. The storage unit 11 stores various information used by the control unit 12 of the analysis device 10. The storage unit 11 stores the measurement result measured by the fMRI 30, the feature amount of the learning moving image data, and each information of the emotion evaluation of the learning moving image data. Based on each information stored in the storage unit 11, the control unit 12 executes, for example, a learning process by machine learning, and generates various information used for estimating an emotion rating from the feature amount of the video data for learning. .. The emotion rating is an example showing the internal state of the estimation reference person S1, and is information indicating the emotion of the estimation reference person S1.

The storage unit 11 includes, for example, a moving image storage unit 111, an emotion rating storage unit 112, a feature amount storage unit 113, a measurement result storage unit 114, and a construction result storage unit 115.
The moving image storage unit 111 stores learning moving image data.

The emotion rating storage unit 112 stores the emotion rating corresponding to the learning moving image data. In the emotion evaluation, for example, the internal state when the estimation reference person S1 watches the moving image based on the learning moving image data is given and stored at predetermined time intervals (every predetermined unit time). The emotion rating is, for example, a label that numerically expresses emotions. The emotion rating storage unit 112 stores the emotion rating corresponding to the learning moving image data at predetermined time intervals (every predetermined unit time) in time series at predetermined time intervals.

The feature amount storage unit 113 stores the feature amount related to the sensation extracted from the learning moving image data. The feature amount is extracted from the learning moving image data by the control unit 12 at predetermined time intervals (at predetermined unit times). Senses include, for example, vision and hearing. Of the moving image data for learning, the visual feature amount, which is a feature amount related to vision, is extracted from the image data, and the auditory feature amount, which is a feature amount related to hearing, is extracted from the sound data. For example, the feature amount storage unit 113 stores the visual feature amount and the auditory feature amount in a time series in association with each other at predetermined time intervals.

The measurement result storage unit 114 stores the measurement result measured by the fMRI 30. When the measurement result storage unit 114 allows the estimation reference person S1 to watch a moving image based on the learning moving image data, the measurement result storage unit 114 obtains the measurement result of the brain activity of the estimation reference person S1 measured by the fMRI 30 at predetermined time intervals. Store in series.

The construction result storage unit 115 stores the result of learning processing by the control unit 12 of the analysis device 10. In the learning process by the control unit 12, the brain according to the sensation is based on the feature amount related to the sensation and the measurement result of the brain activity when the estimation reference person S1 is stimulated by the learning moving image data for a plurality of sensations. Build a feature space that expresses information expression in a distance space. Alternatively, for emotions, an emotional space in which the brain information expression according to the emotions is expressed in a distance space based on the emotional evaluation and the measurement result of the brain activity when the estimation reference person S1 is allowed to watch the learning video data. To construct. The construction of the feature space or the emotional space may, for example, represent a brain information expression corresponding to a certain sensation or a certain emotion by a position in the space.

The construction result storage unit 115 stores various construction results constructed by the learning process and the correspondence information between the constructed feature space and the emotional space. Specifically, the construction result storage unit 115 may include, for example, a construction result (weighted information Wv) of a visual feature space (an example of a visual space), a construction result (weighted information Ws) of an auditory feature space (an example of an auditory space), and the like. The construction result of the emotional space (weighting information We) and the correspondence information between the visual / auditory feature space (hereinafter referred to as visual / auditory space) and the emotional space (weighting information A) are stored.

The construction result of the visual feature space is the correspondence information showing the correspondence between the visual feature amount and the visual feature space (or the position in the visual feature space) representing the brain activity according to the vision, and the construction result of the auditory feature space is. , Correspondence information showing the correspondence between the auditory feature amount and the auditory feature space (or the position in the auditory feature space) representing the brain activity corresponding to the auditory sense. The result of constructing the emotional space is correspondence information showing the correspondence between the emotional rating and the emotional space (or the position in the emotional space).
In the present embodiment, the construction result of the visual feature space and the construction result of the auditory feature space are examples of the first correspondence information, and the correspondence information between the visual / auditory space and the emotional space is the second correspondence information. This is just one example. Moreover, the construction result of the emotional space is an example of the third correspondence information.

The control unit 12 is, for example, a processor including a CPU (Central Processing Unit) and the like, and controls the analysis device 10 in an integrated manner. The control unit 12 includes a measurement control unit 121, a feature amount extraction unit 122, a feature space construction unit 123, an emotion space construction unit 124, and a space correspondence construction unit 125.

The measurement control unit 121 controls the measurement of the brain activity when the estimation reference person S1 is stimulated by the learning moving image data. The measurement control unit 121 outputs the learning moving image data stored in the moving image storage unit 111 to the display device 21 and the speaker 22, and gives the brain activity when the image and sound by the learning moving image data are given to the estimation reference person S1. Let fMRI30 measure. The measurement control unit 121 acquires the fMRI signal of the brain activity from the fMRI 30 at predetermined time intervals, and stores the measurement result in the measurement result storage unit 114 in time series as the measurement result. Various methods have been proposed for extracting visual features, and for example, AlexNet, VGG 19 layer, etc. can be used. The extracted feature amount is represented by, for example, an n-dimensional vector (n is a natural number), and n = 4096, n = 1000, etc. are assumed, but are not limited to these numbers. In addition, various methods have been proposed for extracting auditory features, and for example, soundNet can be used. The extracted feature amount is represented by, for example, an m-dimensional (m is a natural number) vector, and m = 362, m = 3133, etc. are assumed, but the feature amount is not limited to these numbers.
For example, the measurement control unit 121 may give an image and a sound to the estimation reference person S1 at the same time, a case where only an image is given to the estimation reference person S1, and a case where only a sound is given to the estimation reference person S1. The measurement result of the brain activity may be acquired.

The feature amount extraction unit 122 extracts the feature amount related to the sensation from the learning moving image data. The feature amount extraction unit 122 extracts the visual feature amount from the image data included in the learning moving image data and the auditory feature amount from the sound data included in the learning moving image data at predetermined time intervals. The feature amount extraction unit 122 stores the extracted visual feature amount and auditory feature amount for each predetermined time interval in the feature amount storage unit 113 in chronological order.

The feature space construction unit 123 (an example of the first generation unit) is based on a plurality of feature quantities related to a plurality of sensations and measurement results of brain activity when the estimation reference person S1 is stimulated by learning video data. Therefore, the first correspondence information corresponding to each of the plurality of sensations is generated. For example, the feature space construction unit 123 is based on a plurality of sets of data of the visual feature amount stored in the feature amount storage unit 113 in time series and the measurement result stored in the measurement result storage unit 114 in time series. We construct a visual feature space, which is a distance space (semantic space) for expressing brain information about. The feature space construction unit 123 constructs a model of the visual feature space by using, for example, the following equation (1) based on a linear regression model.

Rv = Sv × Wv ・ ・ ・ (1)

Sv indicates a visual feature amount, and Rv is an estimated value of brain activity information when Sv is used as an input. The estimated value of this brain activity information corresponds to the position information (first brain information) in the visual feature space. Wv is coefficient information (weighting information) when associating a pattern of brain activity information with a visual feature amount, and is acquired as a result of learning. For this Wv estimation, for example, ridge regression considering the L2 norm may be used. Further, the position information Rv, the visual feature amount Sv, and the coefficient information Wv in the visual feature space are multidimensional information, such as a vector, a matrix, and array data. As an example, when the visual feature amount Sv is an n-dimensional vector and the estimated value Rv of the brain activity information is an n-dimensional vector, it can be regarded as a position vector on the n-dimensional space determined by the n-dimensional visual feature amount. can. In this example, the number of dimensions of the visual feature amount Sv and the number of dimensions of the estimated value Rv of the brain activity information are the same, but they do not have to be the same and may be completely different numbers.

The feature space construction unit 123 executes learning processing based on a plurality of set data of visual feature quantities and measurement results of brain activity, and equation (1), and constructs coefficient information Wv (first correspondence information). ). The measurement result used when constructing the visual feature space is the measurement result when the image and the sound are given to the estimation reference person S1 at the same time, or the measurement result when only the image is given to the estimation reference person S1. Further, the feature space construction unit 123 stores the generated coefficient information Wv in the construction result storage unit 115.

Further, for example, the feature space construction unit 123 is based on a plurality of sets of data of the auditory feature amount stored in the feature amount storage unit 113 in time series and the measurement result stored in the measurement result storage unit 114 in time series. We construct an auditory feature space, which is a distance space (semantic space) for expressing brain information related to auditory features. The feature space construction unit 123 constructs a model of the auditory feature space by using, for example, the following equation (2) based on a linear regression model.

Rs = Ss × Ws ・ ・ ・ (2)

Ss indicates a visual feature amount, and Rs is an estimated value of brain activity information when Ss is used as an input. The estimated value of this brain activity information corresponds to the position information (first brain information) in the auditory feature space. Ws is coefficient information (weighting information) when associating a pattern of brain activity information with an auditory feature amount, and is acquired as a result of learning. For this Ws estimation, for example, ridge regression considering the L2 norm may be used. Further, the position information Rs, the auditory feature amount Ss, and the coefficient information Ws in the auditory feature space are multidimensional information, for example, a vector, a matrix, an array data, or the like. As an example, when the auditory feature Ss is an m-dimensional vector and the estimated value Rs of the brain activity information is an m-dimensional vector, it can be regarded as a position vector in the m-dimensional space determined by the m-dimensional auditory feature. can. In this example, the number of dimensions of the auditory feature Ss and the number of dimensions of the estimated value Rs of the brain activity information are the same, but they do not have to be the same and may be completely different numbers.

The feature space construction unit 123 executes a learning process based on a plurality of set data of the auditory feature amount and the measurement result of the brain activity and the equation (2), and constructs the coefficient information Ws (first correspondence information). ). The measurement result used when constructing the auditory feature space is the measurement result when the image and the sound are given to the estimation reference person S1 at the same time, or the measurement result when only the sound is given to the estimation reference person S1. Further, the feature space construction unit 123 stores the generated coefficient information Ws in the construction result storage unit 115.

The emotion space construction unit 124 (an example of the second generation unit) is based on the emotion rating corresponding to the learning video data and the measurement result of the brain activity when the estimation reference person S1 is stimulated by the learning video data. Therefore, a third correspondence information showing the correspondence between the emotion rating and the position information (second brain information) in the emotion space is generated. As an emotional rating, a plurality of emotions and emotions such as "pleasant", "unpleasant", "fun", and "fear", for example, i (i is a natural number, and for example, several tens to 100 can be assumed). A combination of words (not limited to this number) and numbers (eg 1 to 5) representing the degree of each word (eg (fear, 3), (fun, 5), etc.) can be used. .. Then, this emotional evaluation can be obtained by the subject who has viewed the learning moving image data in advance, but it can also be acquired by the estimation reference person S1 himself. When using this example for emotion rating, it can be thought of as an i-dimensional vector.
For example, the emotion space construction unit 124 is based on a plurality of sets of data of an emotion rating stored in advance in the emotion rating storage unit 112 in time series and a measurement result stored in the measurement result storage unit 114 in time series. Build an emotional space, which is a distance space (semantic space) for expressing brain information about emotions. The emotional space construction unit 124 constructs a model of the emotional space by using, for example, the following equation (3) based on a linear regression model.

Re = Se × We ・ ・ ・ (3)

Se indicates an emotional rating, and Re is an estimated value of brain activity information when Se is used as an input. The estimated value of this brain activity information corresponds to the position information (second brain information) in the estimated emotional space. We is coefficient information (weighting information) when associating a pattern of brain activity information with an emotional rating, and is acquired as a result of learning. For this We estimation, for example, ridge regression considering the L2 norm may be used. Further, the position information Re, the emotion rating Se, and the coefficient information We in the emotion space are multidimensional information, for example, a vector, a matrix, an array data, or the like. When the emotion rating is an i-dimensional vector and the second brain information Re is an i-dimensional vector, it can be regarded as a position vector in the i-dimensional space determined by the i-dimensional emotion rating. In this example, the number of dimensions of the emotion rating and the number of dimensions of the second brain information are the same, but they do not have to be the same and may be completely different numbers. The emotion rating is, for example, information having a dimension number smaller than the dimension number of the feature amount relating to a plurality of sensations (for example, the total feature amount of the visual feature amount Sv and the auditory feature amount Ss).

The emotion space construction unit 124 executes a learning process based on a plurality of set data of emotion evaluation and measurement results of brain activity, and equation (3), and constructs coefficient information We (third correspondence information). Generate as. The measurement result used when constructing the emotional space is, for example, the measurement result when the image and the sound are given to the estimation reference person S1 at the same time, or the measurement result when only the image is given to the estimation reference person S1. This is a measurement result when only sound is given to the estimation reference person S1. Further, the emotional space construction unit 124 stores the generated coefficient information We in the construction result storage unit 115.

The spatial correspondence construction unit 125 (an example of the third generation unit) has a position in a distance space indicated by a plurality of first brain information (position information in the visual / auditory space) corresponding to the video data for learning, and a second brain information (an example). Based on the position in the distance space indicated by the position information in the emotional space), the second correspondence information which is the correspondence information between the visual / auditory space and the emotional space is generated. The plurality of first brain information is, for example, position information in the visual / auditory space generated from the visual feature amount and the auditory feature amount corresponding to the learning moving image data based on the above-mentioned coefficient information Wv and coefficient information Ws. .. Further, the second brain information is position information in the emotion space generated from the emotion rating corresponding to the learning moving image data based on the above-mentioned coefficient information We.

The space correspondence construction unit 125 uses the following equation (4) by a linear regression model from the set data of the position information in the visual / auditory space and the position information in the emotional space, for example, to use the following equation (4) in the visual / auditory space and the emotional space. Learn the correspondence with.

Re = [Rv, Rs] × A ... (4)

Re indicates the estimated position information in the emotional space (second brain information), and [Rv, Rs] means the result of combining the position information in the visual and auditory spaces. Further, A is coefficient information (weighting information), and corresponds to the second correspondence information (learning result) showing the correspondence between the visual / auditory space and the emotional space. Then, Re can be represented by, for example, a position vector in i-dimensional space, and [Rv, Rs] can be represented by, for example, a position vector in a (n + m) dimensional space that combines visual and auditory senses. .. As a result, the relationship between the two position vectors can be related via the measurement result of the brain activity, and this relationship becomes the second correspondence information. Further, when estimating A, ridge regression considering the L2 norm may be used.
In this way, the spatial correspondence construction unit 125 associates the positional information in the visual / auditory space with the positional information in the emotional space, and generates the second correspondence information. The spatial correspondence construction unit 125 stores the generated second correspondence information in the construction result storage unit 115.

The estimation device 40 executes an estimation process of estimating an emotion rating from the moving image data to be estimated by using various construction results constructed by the analysis device 10 by the learning process and the corresponding information thereof. The estimation device 40 includes a storage unit 41 and a control unit 42.

The storage unit 41 includes, for example, a feature amount storage unit 411, a projection result storage unit 412, an emotion information storage unit 413, and an estimation result storage unit 414.
The feature amount storage unit 411 stores the visual feature amount extracted from the moving image data to be estimated and the auditory feature amount. Specifically, the feature amount storage unit 411 stores the visual feature amount and the auditory feature amount extracted from the moving image data to be estimated at predetermined time intervals (at predetermined unit time) by the feature amount extraction unit 421. Store in the series.

The projection result storage unit 412 stores various projection results which are intermediate information in the estimation process of emotion evaluation. The projection result storage unit 412 stores, for example, a projection result projected from the visual feature amount to the visual feature space, a projection result projected from the auditory feature amount to the auditory feature space, and a projection result projected from the visual / auditory space to the emotional space. do. Further, the projection result storage unit 412 stores various projection results in time series, for example, at predetermined time intervals.

The emotion information storage unit 413 has an emotion rating associated with a stimulus such as a moving image, an image, or a sound at predetermined time intervals, and one of a plurality of types such as "sad" and "fear". It is stored in association with the emotional name (label information).
The estimation result storage unit 414 stores the estimation result of the emotion rating corresponding to the moving image data to be estimated, which is estimated by the estimation device 40. The estimation result storage unit 414 stores, for example, the estimation result of the emotion evaluation in a time series at predetermined time intervals.

The control unit 42 is a processor including, for example, a CPU, and controls the estimation device 40 in an integrated manner. The control unit 42 includes a feature amount extraction unit 421, a feature space projection unit 422, an emotion space projection unit 423, and an estimation processing unit 424.

The feature amount extraction unit 421 acquires the moving image data of the estimation target, and extracts a plurality of feature amounts related to a plurality of sensations from the moving image data of the estimation target. The moving image data to be estimated includes image data and sound data. For example, the feature amount extraction unit 421 extracts the visual feature amount for each predetermined time interval from the image data of the moving image data to be estimated, and extracts the auditory feature amount for each predetermined time interval from the sound data. The feature amount extraction unit 421 stores the visual feature amount and the auditory feature amount for each predetermined time interval extracted from the moving image data to be estimated in the feature amount storage unit 411 in chronological order.

The feature space projection unit 422 (an example of the first conversion unit) corresponds to each of a plurality of feature quantities extracted from the moving image data to be estimated by the feature quantity extraction unit 421 based on the above-mentioned first correspondence information. Convert to multiple first brain information. The feature space projection unit 422 first acquires the construction results (coefficient information Wv and coefficient information Ws) stored in the construction result storage unit 115 of the analysis device 10. The feature space projection unit 422 projects the visual feature amount Sv stored in the feature amount storage unit 411 onto the visual feature space using the acquired coefficient information Wv, and converts it into position information Rv in the visual feature space. The feature space projection unit 422 projects the auditory feature amount Ss stored in the feature amount storage unit 411 onto the auditory feature space using the acquired coefficient information Ws, and converts it into position information Rs in the visual feature space. The feature space projection unit 422 executes these projection processes for the feature quantities (Sv, Ss) at predetermined time intervals, and stores the projection results (Rv, Rs) in the projection result storage unit 412 in chronological order. Let me.

The emotion space projection unit 423 (an example of the second conversion unit) converts a plurality of first brain information converted by the feature space projection unit 422 into second brain information based on the above-mentioned second correspondence information. .. Specifically, the emotion space projection unit 423 first acquires the construction result (coefficient information A) stored in the construction result storage unit 115 of the analysis device 10. The emotional space projection unit 423 acquires the projection results (Rv, Rs) stored in the projection result storage unit 412, and for example, the position information Rv in the visual feature space and the position information Rs in the auditory feature space are combined to provide visual perception. -Generate position information Rv and Rs in the auditory space. The emotion space projection unit 423 projects the position information Rv and Rs in the visual / auditory space to the emotion space using the acquired coefficient information A, and converts it into the position information Re in the emotion space. The emotion space projection unit 423 executes these projection and conversion processes on the position information Rv and Rs in the visual / auditory space at predetermined time intervals, and stores the converted result (Re) in the projection result storage unit. Store in 412 in chronological order.

The estimation processing unit 424 (an example of the estimation unit) estimates the emotion rating corresponding to the moving image data to be estimated based on the second brain information (position information Re in the emotion space) converted by the emotion space projection unit 423. .. The estimation processing unit 424 uses, for example, the coefficient information We (third correspondence information) stored in the construction result storage unit 115 of the analysis device 10 to reversely convert the position information Re in the emotion space to perform emotion evaluation. It may be estimated, or the emotion rating may be estimated based on the distance from the position in the emotion space corresponding to the emotion rating known in advance. The estimation processing unit 424 stores the estimation result of the emotion evaluation in the estimation result storage unit 414.

The estimation processing unit 424 outputs the estimation result of the emotion evaluation to the estimation device 40 to the outside. For example, when the estimation processing unit 424 outputs the estimation result of the emotion rating to the outside, the estimation processing unit 424 may convert the information stored in the emotion information storage unit 413 into an emotion name and output it.

Next, an example of the operation of the estimation system 1 according to the present embodiment will be described with reference to the drawings.
FIG. 2 is a flowchart showing an example of the learning process of the estimation system 1 according to the present embodiment. Further, FIG. 3 is a diagram illustrating an example of the metric space construction process in the present embodiment. Further, FIG. 4 is a diagram illustrating an example of the spatial correspondence relationship construction process in the present embodiment.

As shown in FIG. 2, the analysis device 10 of the estimation system 1 measures the brain activity in response to the visual stimulus and the auditory stimulus by the moving image for learning (step S101). The measurement control unit 121 of the analysis device 10 outputs the moving image data stored in the moving image storage unit 111 to the display device 21 and the speaker 22, and gives the image and sound of the moving image data to the estimation reference person S1 as a stimulus. The activity is measured by fMRI30. The measurement control unit 121 acquires the measurement result (fMRI signal) of the brain activity measured by the fMRI 30 at a predetermined time interval, and stores the measurement result in the measurement result storage unit 114 in time series.

Next, the feature amount extraction unit 122 of the analysis device 10 extracts the visual feature amount and the auditory feature amount from the learning moving image data (step S102, see FIG. 3). The feature amount extraction unit 122 extracts the visual feature amount and the auditory feature amount at predetermined time intervals and stores them in the feature amount storage unit 113 in chronological order.

Next, the feature space construction unit 123 of the analysis device 10 constructs a visual feature space from the visual feature amount and the measurement result of the brain activity (see step S103, FIG. 3). The feature space construction unit 123 describes a plurality of sets of data of the visual feature amount stored in the feature amount storage unit 113 in time series and the measurement result stored in the measurement result storage unit 114 in time series by the above-mentioned equation (1). To build a model of the visual feature space by learning processing using. The feature space construction unit 123 generates coefficient information Wv as a learning result, and stores the generated coefficient information Wv in the construction result storage unit 115.

Next, the feature space construction unit 123 constructs an auditory feature space from the auditory feature amount and the measurement result of the brain activity (step S104, see FIG. 3). The feature space construction unit 123 describes a plurality of sets of data of the auditory feature amount stored in the feature amount storage unit 113 in time series and the measurement result stored in the measurement result storage unit 114 in time series by the above-mentioned equation (2). A model of the auditory feature space is constructed by learning processing using. The feature space construction unit 123 generates coefficient information Ws as a learning result, and stores the generated coefficient information Ws in the construction result storage unit 115.

Next, the emotional space construction unit 124 of the analysis device 10 constructs an emotional space from the emotional evaluation and the measurement result of the brain activity (step S105, see FIG. 3). The emotion space construction unit 124 describes a plurality of set data of the emotion rating stored in advance in the emotion rating storage unit 112 in time series and the measurement result stored in the measurement result storage unit 114 in time series by the above-mentioned formula ( A model of the emotional space is constructed by learning processing using 3). The emotional space construction unit 124 generates the coefficient information We as a learning result, and stores the generated coefficient information We in the construction result storage unit 115.

Next, the spatial correspondence construction unit 125 of the analysis device 10 learns the correspondence relationship between the visual feature space and the auditory feature space and the emotional space (see step S106, FIG. 4). The spatial correspondence construction unit 125 learns the correspondence relationship between the visual / auditory space and the emotional space by using the above-mentioned equation (4) from the set data of the positional information in the visual / auditory space and the positional information in the emotional space. do. The spatial correspondence construction unit 125 generates coefficient information A as a learning result, and stores the generated coefficient information A in the construction result storage unit 115. After the process of step S106, the spatial correspondence construction unit 125 ends the learning process.

The process from step S102 to step S105 described above corresponds to the metric space construction process shown in FIG. As shown in FIG. 3, the analysis device 10 extracts the visual feature amount and the auditory feature amount from the learning moving image data, and acquires the emotion rating corresponding to the learning moving image data. Then, the analysis device 10 constructs a visual feature space from the visual feature amount based on the measurement result of the brain activity, generates coefficient information Wv (first correspondence information), and constructs the auditory feature space from the auditory feature amount. And generate coefficient information Ws (first correspondence information). Further, the analysis device 10 constructs an emotional space from the emotional evaluation based on the measurement result of the brain activity, and generates the coefficient information We.

Further, the process of step S106 described above corresponds to the process of constructing the spatial correspondence relationship shown in FIG. As shown in FIG. 4, the analysis device 10 describes, for example, a linear regression of the correspondence between the visual feature space and the auditory feature space (visual / auditory space) constructed by the above-mentioned distance space construction process and the emotional space. Learn using a model.

Next, the estimation process of the estimation system 1 according to the present embodiment will be described with reference to the drawings.
FIG. 5 is a flowchart showing an example of the estimation process of the estimation system 1 according to the present embodiment. Further, FIG. 6 is a diagram illustrating an outline of an example of the estimation process of the estimation system 1 according to the present embodiment.

As shown in FIG. 5, the feature amount extraction unit 421 of the estimation device 40 first acquires the moving image data to be estimated (step S201).
Next, the feature amount extraction unit 421 extracts the visual feature amount and the auditory feature amount from the moving image data to be estimated (step S202, see FIG. 6). The feature amount extraction unit 421 extracts the visual feature amount Sv from the image data included in the moving image data to be estimated, for example. Further, the feature amount extraction unit 421 extracts the auditory feature amount Ss from the sound data included in the moving image data to be estimated, for example. The feature amount extraction unit 421 stores the extracted visual feature amount Sv and auditory feature amount Ss in the feature amount storage unit 411.

Next, the feature space projection unit 422 of the estimation device 40 projects the visual feature amount onto the visual feature space (step S203, see FIG. 6). The feature space projection unit 422 projects the visual feature amount Sv stored in the feature amount storage unit 411 onto the visual feature space using the coefficient information Wv generated by the analyzer 10 and converts it into the position information Rv in the visual feature space. do. The feature space projection unit 422 stores the position information Rv in the visual feature space in the projection result storage unit 412.

Next, the feature space projection unit 422 projects the auditory feature amount onto the auditory feature space (step S204, see FIG. 6). The feature space projection unit 422 projects the auditory feature amount Ss stored in the feature amount storage unit 411 onto the auditory feature space using the coefficient information Ws generated by the analyzer 10 and converts it into position information Rs in the auditory feature space. do. The feature space projection unit 422 stores the position information Rs in the auditory feature space in the projection result storage unit 412.

Next, the emotion space projection unit 423 of the estimation device 40 estimates the position of the emotion space from the positions of the visual feature space and the auditory feature space (see step S205, FIG. 6). The emotion space projection unit 423 combines the position information Rv in the visual feature space and the position information Rs in the auditory feature space to generate the position information Rv and Rs in the visual / auditory space. The emotion space projection unit 423 projects the position information Rv and Rs in the visual / auditory space to the emotion space using the coefficient information A generated by the analysis device 10, and converts the position information Rv and Rs into the emotion space Re.

Next, the estimation processing unit 424 of the estimation device 40 back-projects the emotion rating from the position of the emotion space (step S206, see FIG. 6). The estimation processing unit 424 back-projects the position information Re in the emotion space to the emotion rating using the coefficient information We generated by the analysis device 10, and estimates the emotion rating.
Next, the estimation processing unit 424 outputs the estimated emotion rating (step S207).
The estimation device 40 repeatedly executes the processes of steps S201 to S207 described above for the moving image data to be estimated at predetermined time intervals.

As described above, the estimation system 1 according to the present embodiment includes a feature amount extraction unit 421, a feature space projection unit 422 (first conversion unit), an emotion space projection unit 423 (second conversion unit), and estimation processing. A unit 424 (estimation unit) is provided. The feature amount extraction unit 421 extracts a plurality of feature amounts (for example, visual feature amount and auditory feature amount) related to a plurality of sensations from the stimulus information (for example, moving image data) to be estimated. The feature space projection unit 422 uses a plurality of first brain information (corresponding to each of the plurality of features related to the sensation extracted from the stimulus information to be estimated by the feature quantity extraction unit 421 based on the first correspondence information. For example, it is converted into position information in the visual feature space and position information in the auditory feature space). The first correspondence information is information showing the correspondence relationship between each feature amount related to a plurality of sensations and the first brain information corresponding to each of the plurality of sensations. The emotional space projection unit 423 converts a plurality of first brain information converted by the feature space projection unit 422 into second brain information (for example, position information in the emotional space) based on the second correspondence information. The second correspondence information is information showing a correspondence relationship between the first brain information and at least one second brain information corresponding to the internal state (for example, emotion) of the estimation reference person S1. The estimation processing unit 424 estimates the internal state corresponding to the stimulus information to be estimated based on the second brain information converted by the emotion space projection unit 423. Here, the first brain information and the second brain information are treated independently, and even if they are expressed independently in the brain, the estimation from the sensory features to the emotional space is effective. It can be carried out.

As a result, the estimation system 1 according to the present embodiment uses the first correspondence information and the second correspondence information to estimate the internal state of the estimation reference person S1 such as emotions, so that it is internal. It is not necessary to measure the brain activity by fMRI30 every time the state is estimated. Therefore, the estimation system 1 according to the present embodiment can quickly and easily estimate the internal state of the estimation reference person S1 with respect to the stimulus information. For example, the estimation system 1 according to the present embodiment has an image or audio without actually viewing the emotional effect of the video / audio content such as a movie, drama, or home video on the viewer (for example, the estimation reference person S1). It can be evaluated from.

Further, FIG. 7 is a diagram illustrating an example of the effect of the estimation system 1 according to the present embodiment.
In this figure, the horizontal axis of the graph shows an example of the correlation coefficient between the estimated emotion rating (estimated value) and the actual emotion rating (true value) in the estimation system 1 of the present embodiment. Here, the correlation coefficient in the present embodiment shows the case where the emotion rating is estimated by using the knowledge about the brain information. In addition, the vertical axis of the graph shows the correlation coefficient between the estimated emotion rating and the actual emotion rating when the emotion rating is simply estimated from the visual features and the auditory features without using brain information. There is.

The term “without going through the brain (brain information)” refers to the visual / auditory features as opposed to the method of the present embodiment in which the emotional rating is estimated from the visual / auditory features as shown in FIG. 4 via the visual / auditory space and the emotional space. It is a method of estimating the emotional rating directly from the quantity. More specifically, without using brain information, we show a linear regression model that explains emotional evaluation from visual and auditory features. In addition, each circle corresponds to an emotion rating item (name or label information). In addition, this result is subjected to learning processing using learning video data under certain conditions (as an example, an example of learning processing of visual / auditory features and emotional ratings acquired from a time zone related to "excitement"). , Shown using the conventional estimation system constructed.

In FIG. 7, the black circle indicates an emotion rating item having a high correlation coefficient and good estimation accuracy in the present embodiment as compared with the case where the brain activity is not measured by fMRI30 and is estimated without using brain information. ing. The circles with vertical stripes have a correlation coefficient of 0.2 or more in the present embodiment, but indicate an emotion rating item with poor estimation accuracy as compared with the case of estimation without using brain information. The white circles indicate emotion rating items with a correlation coefficient of 0.2 or less, both with and without brain information, and with low estimation accuracy.

As shown in FIG. 7, the number of black circles is larger than the number of vertical stripes, and the estimation system 1 according to the present embodiment tends to have higher estimation accuracy than the case of estimation without intervention of brain activity. It shows that it is in. As described above, under this condition, it is shown that the estimation system 1 according to the present embodiment has higher estimation accuracy than the case where the internal state of the estimation reference person S1 is simply estimated from the feature amount related to the senses. ing. The correlation coefficient of the estimation system 1 according to the present embodiment was significantly higher than that of the case of estimation without using brain information by the sine rank test (p <0.01).

Further, in the present embodiment, the first brain information is information that defines the dissimilarity of the information pattern of the brain activity caused by the sensation as the distance (for example, the position information in the metric space corresponding to the sensation). The second brain information is information that defines the dissimilarity of the information pattern of the brain activity caused by the internal state as a distance (for example, the position information in the metric space corresponding to the internal state). The emotional space projection unit 423 converts the position in the metric space indicated by the plurality of first brain information into the position in the metric space indicated by the second brain information based on the second correspondence information. The estimation processing unit 424 estimates the internal state based on the position in the metric space indicated by the second brain information.
The information regarding the distance of the information pattern of the brain activity is obtained as an estimated value by the estimation system 1 according to the present embodiment. Therefore, the estimation system 1 according to the present embodiment uses the estimated distance space for the video data to be estimated even when the brain activity when the video data is viewed by the estimation reference person S1 is not newly measured by the fMRI30. Therefore, the internal state of the estimation reference person S1 with respect to the stimulus information can be estimated quickly and easily.

Further, the estimation system 1 according to the present embodiment includes a feature space construction unit 123 (first generation unit), an emotion space construction unit 124 (second generation unit), and a space correspondence construction unit 125 (third generation unit). Be prepared. The feature space construction unit 123 is based on the visual feature amount and the auditory feature amount extracted from the learning video data and the measurement result of the brain activity when the estimation reference person S1 is stimulated by the learning video data. Generate a first correspondence information corresponding to each of visual and auditory. The emotion space construction unit 124 is based on the information indicating the emotion corresponding to the learning video data (emotion rating) and the measurement result of the brain activity when the estimation reference person S1 is stimulated by the learning video data. Generate a third correspondence information. The third correspondence information shows the correspondence between the emotion rating and the second brain information. The spatial correspondence construction unit 125 is generated from the emotion rating based on the position in the distance space (visual / auditory space) indicated by the first brain information about a plurality of sensations corresponding to the learning video data and the third correspondence information. The second correspondence information is generated based on the position in the distance space (sensory space) indicated by the second brain information.
Thereby, the estimation system 1 according to the present embodiment can generate the first correspondence information and the second correspondence information for estimating the emotion (internal state), and corresponds to the moving image data to be estimated. The emotion (internal state) of the estimation reference person S1 can be estimated quickly and easily.

In the present embodiment, the feature amount extraction unit 421 extracts the feature amount at predetermined unit times. The estimation processing unit 424 estimates the emotion for each predetermined unit time corresponding to the video data to be estimated based on the position in the metric space (position in the emotion space) indicated by the second brain information for each predetermined unit time. ..
Thereby, the estimation system 1 according to the present embodiment can estimate, for example, the time change of emotion (internal state).

In this embodiment, the plurality of sensations are visual and auditory. The feature amount extraction unit 421 extracts a visual feature amount related to vision and an auditory feature amount related to hearing. The feature space projection unit 422 converts the visual feature amount into a position in the visual feature space indicated by the first brain information corresponding to the visual sense based on the first corresponding information corresponding to the visual sense, and the first corresponding information corresponding to the auditory sense. Based on the corresponding information, the auditory feature amount is converted into the position in the auditory feature space indicated by the first brain information corresponding to the auditory sense. The emotional space projection unit 423 indicates the position in the visual feature space and the position in the auditory feature space converted by the feature space projection unit 422 based on the second correspondence information, in the distance space (emotion space) indicated by the second brain information. Convert to the position in.
Thereby, the estimation system 1 according to the present embodiment can appropriately estimate the internal state of the estimation reference person S1 from the moving image data of the estimation target by the visual sense and the auditory sense.

In the present embodiment, the internal state of the estimation reference person S1 includes the emotion of the estimation reference person S1. The emotion space projection unit 423 is based on the second correspondence information showing the correspondence relationship between the position in the distance space indicated by the plurality of first brain information and the position in the emotion space indicated by the second brain information corresponding to the emotion. The position in the distance space indicated by the plurality of first brain information is converted into the position in the emotional space. The estimation processing unit 424 estimates the emotion corresponding to the moving image data to be estimated based on the position in the emotion space.
As a result, the estimation system 1 according to the present embodiment can appropriately estimate emotions from, for example, moving image data to be estimated.

Further, in the present embodiment, the feature amounts related to a plurality of sensations (for example, visual feature amount and auditory feature amount) and the information indicating the internal state (for example, emotion rating) are multidimensional information and are internal. The information indicating the state is information having a dimension number smaller than the dimension number of the feature quantity relating to a plurality of sensations.
As a result, the estimation system 1 according to the present embodiment estimates information (for example, emotion rating) indicating a state having a smaller number of dimensions than the multidimensional feature amount (for example, visual feature amount and auditory feature amount). It can be expected that the internal state of the estimation reference person S1 can be estimated more accurately than the case of estimating the information of the dimension number equal to or more than the feature of.

Further, in the present embodiment, the estimation processing unit 424 estimates the emotion corresponding to the moving image data to be estimated based on the third correspondence information from the position in the emotion space. That is, the estimation processing unit 424 estimates the emotion from the position in the emotion space by the back projection (reverse transformation) using the third correspondence information.
As a result, the estimation system 1 according to the present embodiment can appropriately estimate emotions from a position in the emotional space by a simple method called back projection (reverse transformation).

Further, the estimation method according to the present embodiment includes a feature amount extraction step, a first conversion step, a second conversion step, and an estimation step. In the feature amount extraction step, the feature amount extraction unit 421 extracts a plurality of feature amounts related to a plurality of sensations from the stimulus information of the estimation target. In the first conversion step, the feature space projection unit 422 is based on the first correspondence information showing the correspondence relationship between the feature quantities relating to the plurality of sensations and the first brain information corresponding to each of the plurality of sensations. A plurality of feature quantities extracted from the stimulus information to be estimated by the feature quantity extraction step are converted into a plurality of first brain information corresponding to each. In the second conversion step, the emotional space projection unit 423 becomes the second correspondence information showing the correspondence relationship between the first brain information and at least one second brain information corresponding to the internal state of the estimation reference person S1. Based on this, the plurality of first brain information converted by the first conversion step is converted into the second brain information. In the estimation step, the estimation processing unit 424 estimates the internal state corresponding to the stimulus information to be estimated based on the second brain information converted by the second conversion step.
Thereby, the estimation method according to the present embodiment has the same effect as the estimation system 1 described above, and can quickly and easily estimate the internal state of the estimation reference person S1 with respect to the stimulus information.

[Second Embodiment]
Next, the estimation system 1a according to the second embodiment will be described with reference to the drawings.
In the first embodiment described above, an example of estimating emotions from visual features and auditory features extracted from moving image data by two-step conversion between a first conversion step and a second conversion step has been described. In this embodiment, an example of estimating emotions from visual features and auditory features by one-step conversion using a mathematical model will be described.

FIG. 8 is a functional block diagram showing an example of the estimation system 1a according to the second embodiment.
As shown in FIG. 8, the estimation system 1a includes an analysis device 10a, a display device 21, a speaker 22, an fMRI 30, and an estimation device 40a.
In this figure, the same reference numerals are given to the same configurations as those in FIG. 1 described above, and the description thereof will be omitted.

The analysis device 10a includes a storage unit 11a and a control unit 12a. Further, the storage unit 11a stores various information used by the analysis device 10a. The storage unit 11a includes, for example, a moving image storage unit 111, an emotion rating storage unit 112, a feature amount storage unit 113, a measurement result storage unit 114, and a construction result storage unit 115a.

The construction result storage unit 115a stores various construction results constructed by the learning process of the analysis device 10a. The construction result storage unit 115a is, for example, a construction result (weighted information Wv) of a visual feature space (an example of a visual space), a construction result (weighted information Ws) of an auditory feature space (an example of an auditory space), and a construction result of an emotional space. (Weighting information We) and the model parameters (weighting information A) of the regression model for estimating the emotion rating from the visual feature amount and the auditory feature amount are stored. Details of the model parameters (weighting information A) will be described later.

The control unit 12a is a processor including, for example, a CPU, and controls the analysis device 10a in an integrated manner. The control unit 12a includes a measurement control unit 121, a feature amount extraction unit 122, a feature space construction unit 123, an emotion space construction unit 124, a space correspondence construction unit 125a, and a model construction unit 126.

The spatial correspondence construction unit 125a (an example of the third generation unit) generates the third correspondence information (the second correspondence information of the first embodiment) as in the first embodiment. In this embodiment, for convenience of explanation, the second correspondence information of the first embodiment is described as the third correspondence information, and the third correspondence information of the first embodiment is described as the second correspondence information. do.
In addition, the spatial correspondence construction unit 125a calculates the degree of contribution in explaining the coordinate position of the emotional space for each coordinate dimension of the visual / auditory space when generating the third correspondence information. That is, the spatial correspondence construction unit 125a is in the second brain information for each element (each dimension) included in the plurality of first brain information (position information in the visual / auditory space) when generating the third correspondence information. Generate a contribution in explaining the position information of each emotion.

This processing is necessary for the visual feature space, the auditory feature space, and the emotional space, and the correlation between the audiovisual feature amount and each element of the emotion rating (between each dimension) is applied to the visual feature space, the auditory feature space, and the emotional space. This is because there is a possibility that the correlation between the visual and auditory features before positioning and each element of the emotion rating (between each dimension) is different. In other words, above the visual and auditory feature spaces and the emotional space, there is an element (dimension) with audiovisual features that is strongly correlated with the element (dimension) with emotion rating, or another element (dimension) that is not correlated. ) Is to be quantified. As an example of considering the correlation between these elements, that is, the degree of contribution, preferentially using the audiovisual features with strong correlation can be expected, and accurate estimation can be expected from this.

The model building unit 126 constructs a mathematical model that estimates information indicating an internal state from a plurality of features based on a plurality of features, information indicating an internal state, and the degree of contribution of each element. do. The model construction unit 126 uses the following equation based on a linear regression model, for example, based on the visual and auditory features, the emotion rating, and the contribution of each element (each dimension) generated by the spatial correspondence construction unit 125a. Using (5), a learning process for estimating an emotional rating from visual features and auditory features is executed.

Se ^ = Svs × A ... (5)
L = (Se-Svs × A) ^T (Se-Svs × A) -2λ (RA-r) ・ ・ ・ (6)

Se ^ indicates the estimated emotional rating, Se indicates the emotional rating, and Svs indicates the result of the combination of the visual feature amount and the auditory feature amount. Further, A is coefficient information (weighting information, model parameter) estimated so as to minimize L in the equation (6), and corresponds to a learning result (construction result) in the present embodiment. Further, 2λ (RA-r) is a term relating to a constraint on the relationship between the contribution degree of each element of the visual / auditory feature amount and the coefficient information A, thereby limiting the range of the coefficient information A. The constraint condition is, for example, to estimate the element pattern of the coefficient information A corresponding to the emotion so as to be orthogonal to the element pattern of the visual / auditory feature having a weak co-occurrence relationship with a certain emotion rating. At this time, R and r are determined so as to satisfy the conditions.

Equation (6) will be described in more detail. In the equation (6), the first term represents the cosine distance (inner product) for the difference between the actual emotion rating Se and the estimated emotion rating, and the parameter A is usually set so that this value is minimized. Will be. The second term is a regularization term when solving a regression model. For example, in the covariance matrix of the audiovisual feature amount Svs and the actual emotion rating Se, a vector orthogonal to the vector corresponding to each emotion rating dimension can be calculated, and a combination of these orthogonal vectors for the emotion rating dimension is used. Let it be a matrix R. As a result, R becomes a matrix expressing the contribution of each element of Sv when explaining Se, and in the process of minimizing L in the equation (6), the parameter A is orthogonal to R (RA = 0). Is estimated, and the contribution of audiovisual features will be reflected.
As described above, the model construction unit 126 generates the weighting information A, which is a model parameter for estimating the emotion rating from the visual features and the auditory features, by the regression model. The model construction unit 126 stores the generated weighting information A in the construction result storage unit 115a.

The estimation device 40a executes an estimation process of estimating an emotion rating from the moving image data to be estimated by using the learning result learned by the analysis device 10a by the learning process. The estimation device 40a includes a storage unit 41a and a control unit 42a.
The storage unit 41a stores various information used by the estimation device 40a. The storage unit 41a includes, for example, a feature quantity storage unit 411, an emotion information storage unit 413, and an estimation result storage unit 414.

The control unit 42a is a processor including, for example, a CPU, and controls the estimation device 40a in an integrated manner. The control unit 42a includes a feature amount extraction unit 421 and an estimation processing unit 424a.

The estimation processing unit 424a considers information indicating an internal state from a plurality of feature quantities extracted from the stimulus information to be estimated by the feature quantity extraction unit 421 based on the mathematical model constructed by the model construction unit 126. Using the weighted information A estimated in the above, the internal state corresponding to the stimulus information to be estimated is estimated. That is, the estimation processing unit 424a first acquires the weighting information A stored in the construction result storage unit 115a of the analysis device 10. The estimation processing unit 424a estimates the emotion rating of the visual feature amount and the auditory feature amount (Svs) stored in the feature amount storage unit 411 by the above-mentioned equation (5) using the weighting information A. The estimation processing unit 424a stores the estimated emotion rating as an estimation result in the estimation result storage unit 414.

Next, the operation of the estimation system 1a according to the present embodiment will be described with reference to the drawings.
FIG. 9 is a flowchart showing an example of the learning process of the estimation system 1a according to the present embodiment. Further, FIG. 10 is a diagram illustrating an example of a regression model construction process for estimating an emotion rating in the present embodiment.

Since the process from step S301 to step S305 shown in FIG. 9 is the same as the process from step S101 to step S105 shown in FIG. 2 described above, the description thereof will be omitted here.
In the process of step S305, the emotional space construction unit 124 generates the coefficient information We as the second correspondence information in the present embodiment.

In step S306, the spatial correspondence construction unit 125a of the analysis device 10a learns the correspondence relationship between the visual feature space and the auditory feature space and the emotional space, and explains each emotion for each dimension related to the visual / auditory feature. The degree of contribution is calculated (step S306, see FIG. 10). The space correspondence construction unit 125a obtains the coordinate dimension of the emotional space for each coordinate dimension of the visual / auditory space, which is the third correspondence information, from the set data of the position information in the visual / auditory space and the position information in the emotional space. Calculate the degree of contribution when explaining.

Next, the model building unit 126 builds a regression model that estimates the emotion rating from the visual features and the auditory features with the contribution of each dimension as a constraint (see step S307, FIG. 10). The model building unit 126 stores a plurality of sets of data of the visual feature amount and the auditory feature amount stored in the feature amount storage unit 113 and the emotion rating stored in the emotion rating storage unit 112 for each coordinate dimension of the visual / auditory space. With knowledge about the degree of contribution as a constraint condition for weighting adjustment, learning processing is performed so as to minimize L in the above equation (6), and a regression model for estimating emotion rating from visual features and auditory features is constructed. Here, the knowledge about the degree of contribution is, for example, information (value) representing the co-occurrence relationship between each dimension of the audiovisual feature in the information expression in the brain and the emotion rating. As a learning result, the model construction unit 126 generates weighting information A, which is a model parameter for estimating emotion rating from visual features and auditory features, and stores the generated weighting information A in the construction result storage unit 115a. After the process of step S307, the model building unit 126 ends the learning process.

Next, the estimation process of the estimation system 1a according to the present embodiment will be described with reference to the drawings.
FIG. 11 is a flowchart showing an example of the estimation process of the estimation system 1a according to the present embodiment. Further, FIG. 12 is a diagram illustrating an example of estimation processing of the estimation system 1a according to the present embodiment.

Since the processing of step S401, step S402, and step S404 shown in FIG. 11 is the same as the processing of step S201, step S202, and step S204 shown in FIG. 5 described above, the description thereof will be omitted here.

In step S403, the estimation processing unit 424a of the estimation device 40a estimates the emotion rating by the regression model (see FIG. 12). The estimation processing unit 424a evaluates the visual feature amount and the auditory feature amount (Svs) stored in the feature amount storage unit 411 by using the weighted information A generated by the analysis device 10a and the above-mentioned equation (5). presume. The estimation processing unit 424a stores the estimated emotion rating as an estimation result in the estimation result storage unit 414.

As described above, the estimation system 1a according to the present embodiment includes the feature space construction unit 123 (first generation unit), the emotion space construction unit 124 (second generation unit), and the space correspondence construction unit 125a (third generation unit). Section), a model building section 126, a feature quantity extraction section 421, and an estimation processing section 424a. When the feature space construction unit 123 gives a stimulus to the estimation reference person S1 with a plurality of feature quantities (visual feature quantity and auditory feature quantity) related to a plurality of sensations extracted from the stimulus information for learning and the video data for learning. Based on the measurement results of the brain activity of the above, the first correspondence information showing the correspondence relationship between each feature amount related to a plurality of sensations (visual and auditory senses) and the first brain information corresponding to each of the plurality of sensations is provided. Generate. The first brain information is information that defines the dissimilarity of information patterns of brain activity caused by sensation as a distance.

Further, the emotion space construction unit 124 is based on the emotion rating of the estimation reference person S1 corresponding to the learning video data and the measurement result of the brain activity when the estimation reference person S1 is stimulated by the learning video data. , Generates a second correspondence information (coefficient information We) showing a correspondence relationship between the emotion rating and the second brain information corresponding to the emotion. The second brain information is information that defines the dissimilarity of the information pattern of brain activity caused by emotion as a distance.

Further, the spatial correspondence construction unit 125a includes a plurality of first brain information corresponding to the learning moving image data generated from a plurality of feature quantities (visual feature quantity and auditory feature quantity) based on the first correspondence information. Based on the second correspondence information, a third correspondence information showing a correspondence relationship with the second brain information generated from the emotion rating is generated. Here, the third correspondence information is the degree of contribution of each element (each dimension) in explaining the coordinate dimension of the emotional space for each coordinate dimension of the visual / auditory space. The model building unit 126 has emotions from a plurality of features (visual features and auditory features) based on a plurality of features (visual features and auditory features), an emotion rating, and the contribution of each element. Build a mathematical model (eg, regression model) to estimate the rating.

In addition, the feature amount extraction unit 421 extracts a plurality of feature amounts (visual feature amount and auditory feature amount) related to a plurality of sensations (visual and auditory) from the moving image data to be estimated. The estimation processing unit 424a uses a plurality of feature amounts (visual feature amount and auditory feature amount) extracted from the moving image data to be estimated by the feature amount extraction unit 421 based on the mathematical model constructed by the model construction unit 126. The emotion rating is estimated, and the emotion corresponding to the video data to be estimated is estimated based on the emotion rating.
As a result, the estimation system 1a according to the present embodiment has the same effect as that of the first embodiment described above, and the emotion (internal state) of the estimation reference person S1 with respect to the stimulus information such as moving image data can be quickly and quickly obtained. It can be easily estimated.

The present invention is not limited to each of the above embodiments, and can be modified without departing from the spirit of the present invention.
For example, in each of the above embodiments, examples of applying visual sense and auditory sense as sensations have been described, but the present invention is not limited to this, and other sensations such as taste, smell, and touch may be used. ..

Further, in each of the above embodiments, an example of estimating emotions has been described as an example of the internal state of the estimation reference person S1, but the present invention is not limited to this, and instead of emotions, for example, self such as good and evil. You may try to estimate other internal conditions such as judgment of whether or not it is in line with your values and whether you are in good physical condition.
Further, in each of the above embodiments, an example of using moving image data as an example of stimulus information has been described, but the present invention is not limited to this, and features related to sensations such as sight, hearing, taste, smell, and touch are included. Other information may be used as long as it is stimulus information that can be extracted.

Further, in each of the above embodiments, an example in which the first correspondence information, the second correspondence information, and the third correspondence information are coefficient information (weighting information) has been described, but the present invention is not limited thereto. Instead, it may be correspondence information or coefficient information such as a conversion table, a correspondence table, or other learning results of machine learning.
Further, in each of the above embodiments, an example of executing the learning process by using the linear regression model has been described, but the present invention is not limited to this, and a non-linear function may be used. A mathematical model other than the regression model may be used.

Further, in each of the above embodiments, an example in which the estimation system 1 (1a) includes an analysis device 10 (10a), a display device 21, a speaker 22, an fMRI 30, and an estimation device 40 (40a) has been described. However, the estimation device 40 (40a) may include a part or all of the analysis device 10 (10a). Further, a part of the analysis device 10 (10a) and the estimation device 40 (40a) may be provided outside each device. For example, a part or all of the storage unit 11 (11a) and the storage unit 41 (41a) may be provided externally. Further, when a part or all of the storage unit 11 (11a) and the storage unit 41 (41a) are provided externally, the storage unit 11 (11a) and the storage unit 41 (41a) may be configured to be connectable to each device via a network.

Further, in each of the above embodiments, the estimation system 1 (1a) has described an example in which the speaker 22 is provided as an example of the sound emitting device, but the present invention is not limited to this, and the estimation system 1 (1a) is a sound emitting device such as headphones. You may. Further, although the example in which the estimation system 1 (1a) includes fMRI30 as a measuring device for measuring brain activity has been described, another measuring device may be provided.

Each configuration included in the estimation system 1 (1a) described above has a computer system inside. Then, a program for realizing the functions of each configuration included in the estimation system 1 (1a) described above is recorded on a computer-readable recording medium, and the program recorded on the recording medium is read into the computer system and executed. By doing so, the processing in each configuration provided in the estimation system 1 (1a) described above may be performed. "Loading a program recorded on a recording medium into a computer system and executing it" includes installing the program in the computer system. The "computer system" shall include hardware such as an OS and peripheral devices.
Further, the "computer system" may include a plurality of computer devices connected via a network including a communication line such as the Internet, WAN, LAN, and a dedicated line. Further, the "computer-readable recording medium" refers to a portable medium such as a flexible disk, a magneto-optical disk, a ROM, or a CD-ROM, and a storage device such as a hard disk built in a computer system. As described above, the recording medium in which the program is stored may be a non-transient recording medium such as a CD-ROM.

1,1a ... estimation system, 10,10a ... analyzer, 11,11a, 41,41a ... storage unit, 12,12a, 42,42a ... control unit, 21 ... display device, 22 ... speaker, 30 ... fMRI, 40 , 40a ... estimation device, 111 ... moving image storage unit, 112 ... emotion rating storage unit, 113,411 ... feature quantity storage unit, 114 ... measurement result storage unit, 115, 115a ... construction result storage unit, 121 ... measurement control unit, 122, 421 ... Feature quantity extraction unit, 123 ... Feature space construction unit, 124 ... Emotion space construction unit, 125, 125a ... Spatial correspondence construction unit, 126 ... Model construction unit, 412 ... Projection result storage unit, 413 ... Emotion information storage Unit 414 ... Estimated result storage unit 422 ... Feature space projection unit 423 ... Emotional space projection unit, 424, 424a ... Estimating processing unit, S1 ... Estimating reference person

Claims (9)

A feature amount extraction unit that extracts multiple feature amounts related to multiple sensations from the stimulus information of the estimation target,
The stimulus of the estimation target is stimulated by the feature amount extraction unit based on the first correspondence information indicating the correspondence relationship between each feature amount relating to the plurality of sensations and the first brain information corresponding to each of the plurality of sensations. A first conversion unit that converts the plurality of feature quantities extracted from the information into a plurality of the first brain information corresponding to each.
The above-mentioned converted by the first conversion unit based on the second correspondence information indicating the correspondence relationship between the first brain information and at least one second brain information corresponding to the internal state of the estimation reference person. A second conversion unit that converts a plurality of first brain information into the second brain information,
An estimation system including an estimation unit that estimates the internal state corresponding to the stimulus information of the estimation target by using the second brain information converted by the second conversion unit. The first brain information is information that defines the dissimilarity of the information pattern of the brain activity caused by the sensation as a distance.
The second brain information is information that defines the dissimilarity of the information pattern of the brain activity caused by the internal state as a distance.
Based on the second correspondence information, the second conversion unit converts the position in the metric space indicated by the plurality of first brain information into the position in the metric space indicated by the second brain information.
The estimation system according to claim 1, wherein the estimation unit estimates the internal state by using the position information in the metric space indicated by the second brain information. Based on the plurality of feature quantities related to the plurality of sensations extracted from the stimulus information for learning and the measurement result of the brain activity when the estimation reference person is stimulated by the stimulus information for learning. The first generation unit that generates the first correspondence information corresponding to each of the senses of
The internal state is based on the information indicating the internal state corresponding to the learning stimulus information and the measurement result of the brain activity when the estimation reference person is stimulated by the learning stimulus information. A second generation unit that generates a third correspondence information showing a correspondence relationship between the information indicating the state and the second brain information.
The second brain information generated from the position in the distance space indicated by the plurality of first brain information corresponding to the stimulus information for learning and the information indicating the internal state based on the third corresponding information. The estimation system according to claim 2, further comprising a third generation unit that generates the second correspondence information based on the position in the distance space indicated by. The feature amount extraction unit extracts the feature amount at predetermined unit times.
The estimation unit estimates the internal state for each predetermined unit time corresponding to the stimulus information to be estimated based on the position in the metric space indicated by the second brain information for each predetermined unit time. The estimation system according to claim 2 or claim 3. The plurality of sensations are visual and auditory.
The feature amount extraction unit extracts the visual feature amount related to the visual sense and the auditory feature amount related to the auditory sense.
Based on the first correspondence information corresponding to the vision, the first conversion unit converts the visual feature amount into a position in the visual space indicated by the first brain information corresponding to the vision, and converts it into the auditory sense. Based on the corresponding first correspondence information, the auditory feature amount is converted into a position in the auditory space indicated by the first brain information corresponding to the hearing.
Based on the second correspondence information, the second conversion unit sets the position in the visual space and the position in the auditory space converted by the first conversion unit in the metric space indicated by the second brain information. The estimation system according to any one of claims 2 to 3. The internal state of the estimator includes the emotions of the estimator,
The second conversion unit shows the correspondence relationship between the position in the distance space indicated by the plurality of first brain information and the position in the emotion space indicated by the second brain information corresponding to the emotion. Based on, the position in the distance space indicated by the plurality of first brain information is converted into the position in the emotional space.
One of claims 2 to 4, wherein the estimation unit estimates the emotion corresponding to the stimulus information to be estimated based on the position information in the emotion space converted from the positions in the plurality of distance spaces. The estimation system described in paragraph 1. The feature quantities related to the plurality of sensations and the information indicating the internal state are multidimensional information.
The estimation system according to any one of claims 1 to 6, wherein the information indicating the internal state is information having a dimension number smaller than the dimension number of the feature amount relating to the plurality of sensations. The plurality of sensations are based on the plurality of feature quantities related to the plurality of sensations extracted from the stimulus information for learning and the measurement results of the brain activity when the estimation reference person is stimulated by the stimulus information for learning. It is the first brain information which is the information which defines the dissimilarity of the information pattern of the brain activity caused by the sensation as a distance, and is the first which corresponds to each of the plurality of sensations. The first generation unit that generates the first correspondence information showing the correspondence relationship with the brain information,
Based on the information indicating the internal state of the estimation reference person corresponding to the learning stimulus information and the measurement result of the brain activity when the estimation reference person is stimulated by the learning stimulus information. The second brain information, which is information that defines the dissimilarity between the information indicating the internal state and the information pattern of the brain activity caused by the internal state as a distance, and is the internal state. A second generation unit that generates a second correspondence information showing a correspondence relationship with the second brain information corresponding to the state, and
Based on the plurality of first brain information corresponding to the learning stimulus information generated from the plurality of feature quantities based on the first correspondence information, and the internal information based on the second correspondence information. It is the third correspondence information which shows the correspondence relation with the 2nd brain information generated from the information which shows a state, and is included in the plurality of 1st brain information when explaining each element included in the 2nd brain information. A third generator that generates the contribution of each element
Based on the plurality of features, the information indicating the internal state, and the contribution of each element, a mathematical model for estimating the information indicating the internal state from the plurality of features is constructed. Model building department and
A feature amount extraction unit that extracts multiple feature amounts related to multiple sensations from the stimulus information of the estimation target,
Based on the mathematical model constructed by the model building unit, the internal features corresponding to the stimulus information of the estimation target from the plurality of feature quantities extracted from the stimulus information of the estimation target by the feature amount extraction unit. An estimation system with an estimation unit that estimates the state. A feature amount extraction step in which the feature amount extraction unit extracts a plurality of feature amounts related to a plurality of sensations from the stimulus information to be estimated, and a feature amount extraction step.
The first conversion unit sets the feature amount extraction step based on the first correspondence information showing the correspondence relationship between each feature amount relating to the plurality of sensations and the first brain information corresponding to each of the plurality of sensations. The first conversion step of converting the plurality of feature quantities extracted from the stimulus information of the estimation target into the plurality of the first brain information corresponding to each.
The first conversion unit is based on the second correspondence information indicating the correspondence relationship between the first brain information and at least one second brain information corresponding to the internal state of the estimation reference person. A second conversion step of converting the plurality of first brain information converted by the step into the second brain information,
An estimation method including an estimation step in which the estimation unit estimates the internal state corresponding to the stimulus information of the estimation target by using the second brain information converted by the second conversion step.

Images