JP6154728B2 - Viewing state estimation apparatus and program thereof - Google Patents

Description

  The present invention relates to a viewing state estimation device and a program for estimating a viewing state such as a degree of understanding of a content of a viewer who views the content and subjective evaluation.

Conventionally, comprehension and subjective evaluation of content such as broadcast programs are based on questionnaires (emails, postcards, etc.) for viewers after viewing the content, and questionnaires (subjects such as answer sheets) for subjects after having multiple subjects view the content. Answer).
Such an understanding level and subjective evaluation for content are useful for grasping viewer's preference and the like, and are expected to be useful for information recommendation and the like for viewers. However, in order to perform understanding and subjective evaluation of content, it is necessary to count a large number of questionnaires, and it is not realistic to conduct a questionnaire every time after viewing the content.
Therefore, in recent years, a technique has been proposed in which an arbitrary subject is subjected to objective and automatic grasping of a viewer's viewing state such as an understanding level from measurement data such as a biological signal (see Patent Documents 1 to 6). .
In these techniques, the degree of understanding or the like is estimated based on whether or not a measured biological signal includes a predetermined biological signal pattern.

Japanese Patent No. 4370209 Japanese Patent No. 4441345 Japanese Patent No. 4686299 Japanese Patent No. 3991066 Japanese Patent No. 4189440 Japanese Patent No. 5119375

However, there are various types (genre, content, etc.) of contents such as broadcast programs, and the response to them is complicated.
The technique using a predetermined biological signal pattern as described above cannot be applied to a wide variety of contents because the biological signal pattern is fixed.
That is, even if the conventional technology can estimate the viewing state of the viewer based on a predetermined biological signal pattern for specific content, the biological signal pattern is different for different content. There is a problem that it cannot be estimated.

  The present invention has been made in view of such a problem, and it is possible to estimate the viewing state of the viewer with respect to various contents without using a biological signal pattern predetermined for each content. It is an object of the present invention to provide a simple viewing state estimation device and a program thereof.

  In order to solve the above-mentioned problem, the viewing state estimation apparatus according to the present invention is a result of the learning subject answering the biometric signals of a plurality of learning subjects who are viewing content and the question asking about the viewing state of the content. A viewing state estimation device for estimating the viewing state of the target subject from the biological signal of the target subject who is viewing the content based on the learning answer data, A configuration comprising learning viewing state similarity calculating means, similarity mapping information calculating means, channel selecting means, biological signal extracting means, selected biological signal similarity calculating means, and viewing state estimated value calculating means. It was.

In such a configuration, the viewing state estimation apparatus uses the learning biosignal similarity calculation unit to calculate the similarity between the learning subjects with respect to the biosignals of a plurality of learning subjects input for each measurement channel that is a measurement portion of the biosignal. The degree (biological signal similarity) is calculated.
In addition, the viewing state estimation apparatus calculates the similarity (viewing state similarity) of the answer data for learning between the learning subjects by the learning viewing state similarity calculating unit. Are considered to have a strong correlation.
Therefore, the viewing state estimation device associates the biological signal similarity with the viewing state similarity, so that the similarity signal mapping information calculation means calculates the biological signal similarity between a predetermined number of learning subjects for each measurement channel. And the weighting coefficient that maximizes the correlation between the viewing state similarity and the viewing state similarity are calculated as mapping information.

Then, the viewing state estimation device uses the channel selection means to obtain mapping information for biosignal similarity and viewing state similarity between a plurality of other learning subjects different from the learning subject for which mapping information is calculated. A measurement channel whose correlation error when used is lower than a predetermined threshold is selected as a selected channel.
That is, the viewing state estimation apparatus estimates mapping information from a predetermined number of learning subjects by means of similarity mapping information calculation means. Then, the viewing state estimation device applies the mapping information to other learning subjects by the channel selection means, and uses the measurement channel that has calculated the mapping information whose estimation error is smaller than a predetermined threshold as the selection channel. select.

Then, the viewing state estimation device extracts the biological signal of the selected channel from the biological signal for each measurement channel for the subject subject by the biological signal extraction means. As a result, a biological signal having a strong correlation with the viewing state is extracted.
Then, the viewing state estimation apparatus calculates the similarity between the biological signal of the target subject extracted by the biological signal extraction means and the biological signals of the plurality of learning subjects by the selected biological signal similarity calculation means.

  Further, the viewing state estimation device uses the viewing state estimation value calculation means to calculate the similarity between the unknown learning answer data of the target subject and the learning answer data of the plurality of learning subjects, and the selected biological signal similarity calculation means. With respect to the calculated similarity, an unknown number of learning answer data having a maximum correlation when using mapping information corresponding to the selected channel is calculated as a viewing state estimation value.

In this way, the viewing state estimation device is configured to map the mapping information (weighting coefficient) that associates the similarity of the biological signal between the plurality of learning subjects with the similarity of the viewing state (learning answer data) between the plurality of learning subjects. ) And the corresponding unknown answer data (viewing state) is estimated from the biological signal of the subject subject based on the mapping information.
Thereby, the viewing state estimation device estimates the viewing state of the target subject without determining the viewing state based on whether or not the biological signal pattern of the target subject includes a predetermined biological signal pattern as in the past. Can do.

  The viewing state estimation apparatus according to the present invention includes a computer of the viewing state estimation apparatus, a learning biological signal similarity calculation unit, a learning viewing state similarity calculation unit, a similarity mapping information calculation unit, a channel selection unit, It is possible to operate with a viewing state estimation program for functioning as a biological signal extraction unit, a selected biological signal similarity calculation unit, and a viewing state estimation value calculation unit.

The present invention has the following excellent effects.
According to the present invention, rather than associating a specific biological signal pattern generated during content viewing with a viewing state, the similarity of the biological signal between the learning subjects and the similarity of the viewing state (learning answer data) The viewing state of the subject subject can be estimated from the relevance of. Therefore, the present invention can estimate the viewing state for any content.
Further, according to the present invention, it is possible to objectively grasp the influence of the content on the viewer. Therefore, the present invention can quantitatively evaluate the effect of content distribution.

It is explanatory drawing for demonstrating the outline | summary of the viewing-and-listening state estimation apparatus which concerns on embodiment of this invention, Comprising: (a) shows the learning stage of viewing-and-listening state, (b) shows the outline | summary of the viewing-and-listening state estimation stage. It is a block block diagram which shows the structure of the viewing-and-listening state estimation apparatus which concerns on embodiment of this invention. It is a flowchart which shows operation | movement of the learning stage of the viewing condition of the viewing condition estimation apparatus which concerns on embodiment of this invention. It is a flowchart which shows operation | movement of the estimation stage of the viewing condition of the viewing condition estimation apparatus which concerns on embodiment of this invention.

Embodiments of the present invention will be described below with reference to the drawings.
[Outline of viewing state estimation device]
Initially, with reference to FIG. 1, the outline | summary of the viewing-and-listening state estimation apparatus 1 which concerns on embodiment of this invention is demonstrated.
The viewing state estimation device 1 estimates a viewing state such as a degree of comprehension and subjective evaluation of a subject's content from a biological signal of a subject (target subject) who is viewing the content such as a broadcast program.

As shown in FIG. 1 (a), the viewing state estimation device 1 is viewing content presented on a monitor M such as a television receiver from a plurality of subjects (learning subjects Hs, Hs,...) As a learning stage. In addition, a biological signal (learning biological signal) measured at a plurality of measurement points (channels) of the biological signal measuring device (here, electroencephalograph B) is acquired.
Moreover, the viewing state estimation apparatus 1 acquires the answer (questioning answer data) of the question (questionnaire) which asks for a viewing state from a plurality of subjects (learning subjects Hs, Hs,...) As a learning stage. Then, the viewing state estimation apparatus 1 estimates the channel of the biological signal that is strongly related to the question by machine learning.

Then, as shown in FIG. 1 (b), the viewing state estimation device 1 performs the biological signal measurement device (in this case, an electroencephalograph B) from the target subject Ht while viewing the content presented on the monitor M as an estimation stage. The biological signal (estimation target biological signal) of the channel estimated in the learning stage is acquired from the biological signals measured in step (1), and answer data (viewing state estimation value) indicating the viewing state of the target subject Ht is estimated.
Note that the content viewed by the subject (learning subject Hs, target subject Ht) is the same content. Moreover, the content of the questionnaire performed with respect to several test subjects Hs, Hs,... Is the same.
Hereinafter, the configuration and operation of the viewing state estimation apparatus 1 that identifies the channel to be used by learning and estimation and estimates the viewing state of the target subject Ht will be described below.

[Configuration of viewing state estimation device]
First, the configuration of the viewing state estimation apparatus 1 according to the embodiment of the present invention will be described with reference to FIG.

  As shown in FIG. 2, the viewing state estimation device 1 has a biological signal input means 10, a biological signal storage means 11, and a learning biological signal for selecting a channel to be used in the estimation stage in the prior learning stage. Similarity calculation means 12, viewing state input means 13, viewing state storage means 14, learning viewing state similarity calculation means 15, similarity mapping information calculation means 16, mapping information storage means 17, channel The selection unit 18 includes a biological signal extraction unit 20, a selection biological signal similarity calculation unit 21, and a viewing state estimation value calculation unit 22 for estimating the viewing state of the subject subject in the estimation stage.

  The biological signal input means 10 inputs the biological signal of the subject for learning for each channel for measuring the biological signal via a biological signal measuring device (not shown) connected to the outside. The biological signal input means 10 digitizes biological signals input from a plurality of learning subjects as time series signals, and writes them into the biological signal storage means 11 in association with the learning subjects and channels.

Here, the biological signal is a signal reflecting the state of the biological tissue of the learning subject (the same applies to the subject subject) and information when the tissue is functioning. This biological signal is, for example, cerebral blood oxygen content, electroencephalogram (EEG), body surface temperature of the human body, and the like.
For example, by using an fMRI apparatus as a biological signal measuring apparatus, the biological signal input means 10 can calculate the oxygen content in the brain blood as luminance data of an image measured by functional nuclear magnetic resonance imaging (fMRI: Magnetic Resonance Imaging). Can be acquired.

In addition, for example, by using a NIRS brain measuring device as a biological signal measuring device, the amount of oxygen in the brain blood as data indicating the degree of absorption of near infrared light measured by near infrared spectroscopy (NIRS) Can be obtained.
In addition, for example, by using an electroencephalograph as the biological signal measuring device, an electroencephalogram can be acquired as potential data measured by an electrode placed on the subject's scalp or the like.
In addition, for example, by using thermography as the biological signal measuring device, the body surface temperature can be acquired as image data indicating the heat distribution.

  Here, the channel indicates a measurement point of the biological signal of the subject measured by the biological signal measurement device. For example, when data measured by fMRI is used as a biological signal, since the fMRI apparatus generates a plurality of cross-sectional images (three-dimensional images) of the brain, the channel is a pixel unit (three-dimensional image). More precisely, it corresponds to the brain part of the subject expressed in voxel units). In addition, for example, when using data or brain waves measured by near infrared spectroscopy as a biological signal, the channel corresponds to a portion of the brain expressed in probe units or electrode units placed on the subject's head surface. .

The fMRI apparatus outputs the amount of oxygen in the brain as a cross-sectional image of the brain, but the image differs in size, shape, etc. depending on the subject. Therefore, the biological signal input unit 10 compresses, expands, and rotates the cross-sectional image by affine transformation or the like in order to align the parts pointed as channels among a plurality of subjects, and the template brain image called a standard brain and the size, shape, and the like It is preferable to perform a normalization process for matching.
Moreover, it is preferable that the biological signal input unit 10 removes noise from the biological signal by performing a smoothing process or the like on the measured biological signal.
In this way, the biological signal input unit 10 writes and stores the biological signal subjected to normalization processing, noise removal, and the like in the biological signal storage unit 11.

The biological signal storage unit 11 stores the biological signal input by the biological signal input unit 10. The biological signal storage means 11 can be configured by a general storage device such as a semiconductor memory or a hard disk.
The biological signal storage unit 11 stores biological signals of a plurality of learning subjects as time-series signals associated with the learning subjects and channels. For example, the biological signal storage means 11 stores biological signals measured by 10,000 channels respectively for ten learning subjects. Here, it is assumed that the number of learning subjects who have measured the biological signals stored in the biological signal storage unit 11 is M.
The biological signal stored in the biological signal storage unit 11 is referred to by the learning biological signal similarity calculation unit 12, the channel selection unit 18, and the selected biological signal similarity calculation unit 21.

The learning biosignal similarity calculation means 12 calculates the similarity between learning subjects for the biosignals of a plurality of learning subjects who have viewed the same content stored in the biosignal storage means 11.
The learning biosignal similarity calculation unit 12 outputs the calculated similarity to the similarity mapping information calculation unit 16.

Specifically, the learning biosignal similarity calculating unit 12 calculates the similarity as follows. Here, the learning biosignal similarity calculation unit 12 uses a correlation coefficient as the similarity.
Here, when the channel identifier is c and the learning subject identifier is i, the time-series biological signal x ^c _i stored in the biological signal storage means 11 is expressed by a vector of the following equation (1). be able to.

Note that t represents transposition (the same applies to the following equations).
In this equation (1), x ^c _{i, 1} is a biological signal first measured during content viewing, and indicates a biological signal sequentially measured in time series up to x ^c _{i, T.}
Then, for each channel c, the learning biological signal similarity calculating means 12 correlates the correlation coefficient M ^c _{x, i} between the biological signal x ^c _i of the learning subject _i and the biological signal x ^c _j of the learning subject j. _{, J} are calculated by the following equation (2).

Here, X - ^c _i is a vector with the mean value of all elements of the vector x ^c _i in the formula (1) to each element, specifically, the following equation (3) and (4) (X￣ ^c _j is also the same). In Equation (3), T represents the number of elements of the vector x ^c _i shown in Equation (1). Expression (4) is a T-dimensional vector in which all elements are “1”.

Then, the learning biosignal similarity calculation means 12 uses the following expression having the similarity (correlation coefficient ^Mc _{x, i, j} ) between the N learning subjects calculated by the expression (2) as an element. The N × N-dimensional matrix (biological signal similarity matrix M ^c _x ) shown in (5) is output to the similarity mapping information calculation means 16. This equation (5) is a symmetric matrix, and the values of the diagonal components are all “1”.

Here, the biometric signal similarity calculation means 12 for learning is different by sequentially changing the combination of selecting N persons from M persons for N persons (N <M) smaller than the total number of learning subjects (M persons). The similarity of the biological signal is calculated between the learning subjects in combination.
When the learning biosignal similarity calculation means 12 calculates the similarity for a certain N, the channel selection means 18 applies the remaining (MN) human biosignals corresponding to the N. This is used when verifying the accuracy of mapping, and the explanation thereof will be described later.

  Here, the biometric signal similarity calculation means 12 for learning uses a correlation coefficient as the biosignal similarity, but other similarity indices such as an inner product between vectors and a cosine distance may be used. Absent.

  The viewing state input means 13 is a result obtained by answering a question (questionnaire) in which a plurality of learning subjects whose biological signals are measured during content viewing asks the viewing state such as the degree of understanding and subjective evaluation of the content. Learning answer data is input and the viewing state is digitized. The viewing state input means 13 writes and stores the digitized viewing state (answer pattern) in the viewing state storage means 14 in association with the learning subject. Here, the number of learning subjects who have acquired the viewing state stored in the viewing state input unit 13 is the same as that of the biological signal storage unit 11.

  Here, the questions that are asked to the learning subject are questions that ask the understanding of the content, questions that ask the subjective evaluation of the content, and the like. For example, when the content is a news program, the following examples (questions Q1 and Q2) are given as examples of questions that ask the understanding of the content.

Question Q1: What is the topic of the content?
Option A1: Currency euro crisis Option A2: EU political instability Option A3: China's investment in EU region Option A4: Economic conflict between Germany and France

Question Q2: What is the current state of the euro?
Option A1: Reliability of the euro system itself is declining Option A2: Recovering trust through financial support from the IMF Option A3: The EU is showing a face of devaluation due to political conflict between countries Option A4: Many countries are considering returning to their own currency

  In addition, for example, when the content is a drama program, the following examples (questions Q3 and Q4) are given as examples of questions about the subjective evaluation of the content.

Question Q3: Was this content interesting? Please evaluate in 3 stages (1-3).
(Interesting: 0, normal: 1, not interesting: 2)

Question Q4: Can you agree with the way of life of the protagonist A? Please evaluate in 3 stages (1-3).
(I can agree: 0, Neither: 1, I can't agree: 2)

Here, the viewing state input means 13 inputs the results answered by a plurality of subjects for learning in response to questions about the content, for example, via input means (not shown) such as a keyboard and a touch panel. Then, the viewing state input means 13 compares the answer with a predetermined correct answer for each question, and quantifies it to a predetermined value corresponding to the correct answer or the incorrect answer. . For example, it is digitized as “1” for correct answer and “0” for incorrect answer.
The viewing state input means 13 uses the input numerical value as it is for a question that is answered numerically, such as the subjective evaluation level.

Then, when the number of questions is N _q , the viewing state input means 13 corresponds to the answer to the question as shown in the following formula (6) as the viewing state y _i for each learning subject i. , correctness, or to produce a N _{q-dimensional} vector obtained by digitizing the evaluation levels (viewing state vector).

Each element of the viewing state y _i is “0” or “1” or a numerical value of the evaluation level depending on the answer of the learning subject i.
The question (questionnaire) may be only a question asking for understanding of the content, a question asking for subjective evaluation of the content, or a mixture of them.
This viewing state input means 13 writes and stores the viewing state y _i shown in Equation (6) in the viewing state storage means 14.

  Here, the viewing state input means 13 inputs the answer to the question of the learning subject as the answer data for learning, but the data (6) that has already been subjected to the correctness determination etc. (Answer pattern) may be directly input.

The viewing state storage unit 14 stores learning answer data input by the viewing state input unit 13. The viewing state storage unit 14 can be configured by a general storage device such as a semiconductor memory or a hard disk.
The viewing state storage means 14 stores the viewing state y _i shown in the equation (6) associated with the learning subject i as learning answer data.
The learning answer data (viewing state) stored in the viewing state storage unit 14 is referred to by the learning viewing state similarity calculation unit 15, the channel selection unit 18, and the viewing state estimated value calculation unit 22.

The learning viewing state similarity calculating unit 15 calculates the similarity between learning subjects with respect to learning answer data of a plurality of learning subjects who viewed the same content stored in the viewing state storage unit 14. is there.
The learning viewing state similarity calculating unit 15 outputs the calculated similarity to the similarity mapping information calculating unit 16.

Specifically, the learning viewing state similarity calculating unit 15 calculates the similarity as follows. Here, the learning viewing state similarity calculating unit 15 uses an inner product between vectors as the similarity.
That is, the learning viewing state similarity calculating means 15 calculates the similarity My _{, i, j} between the viewing state y _i of the learning subject _i and the viewing state y _j of the learning subject _j using the following formula (7 ).

Then, the learning viewing state similarity calculating means 15 is represented by the following equation (8) having the similarity My _{, i, j} between the N learning subjects calculated by the equation (7) as an element. The × N-dimensional matrix (viewing state similarity matrix M _y ) is output to the similarity mapping information calculation means 16.
This equation (8) is a symmetric matrix, and the values of the diagonal components are all “1”.

Here, the learning viewing state similarity calculating means 15 is similar to the learning biosignal similarity calculating means 12 for learning N persons (N <M) less than the total number of learning subjects (M persons). The similarity between subjects will be calculated. The N persons selected by the learning viewing state similarity calculating unit 15 correspond to the N persons selected when the learning biosignal similarity calculating unit 12 calculates the biosignal similarity. That is, the learning viewing state similarity calculating unit 15 sequentially calculates the viewing state similarity between the learning subjects for the same combination of selecting N from M persons in the learning biosignal similarity calculating unit 12. .
When the learning viewing state similarity calculating unit 15 calculates the similarity for a certain N, the learning answer data (viewing state) for the remaining (MN) people corresponding to the N is as follows. The channel selection means 18 is used when verifying the accuracy of mapping, and the description thereof will be described later.

  Here, the learning viewing state similarity calculating unit 15 uses the inner product between vectors as the viewing state similarity, but other similarity indices such as a cosine distance may be used.

  The similarity-to-similarity mapping information calculating unit 16 calculates the similarity between the biological signals calculated by the learning biological signal similarity calculating unit 12 and the viewing state similarity calculated by the learning viewing state similarity calculating unit 15. It is related. Here, the similarity-to-similarity mapping information calculation means 16 calculates mapping information (weighting coefficient) that maximizes the correlation between the similarity between the biological signals and the similarity in the viewing state between the same learning subjects.

That is, the similarity-to-similarity mapping information calculation unit 16 calculates the biological signal similarity matrix M ^c _x (see the above equation (5)) calculated by the learning biological signal similarity calculation unit 12 and the learning viewing state similarity. From the viewing state similarity matrix M _y calculated by the means 15 (see the above equation (8)), the weight vector w ^c (first weight vector), v ^c (the first weight vector) satisfying the relationship of the following equation (9): 2). Both w ^c and v ^c are N-dimensional vertical vectors corresponding to the channel c.

  When this equation (9) is satisfied, the correlation coefficient is “1”, and the following equation (10) is also satisfied. Note that ‖ / ‖ represents the norm of the vector.

  Here, if the norm of the denominator is a constant, for example, “1”, Expression (10) becomes the following Expression (11).

^Here, because it contains a measurement error in ^M _{c x} and _{M y,} it is difficult to obtain a ^w c, ^{v c} as the left-hand side of this equation (11) becomes "1".
Therefore, the similarity between the mapping information calculating unit ^16, a maximum correlation with ^M _{c x} and _{M y,} ^{w c} (first weight vector) that maximizes the left side of the equation (11), ^{v c} (No. 2).

Here, the similarity-to-similarity mapping information calculation means 16 uses the first vector that is the product of the biological signal similarity matrix M ^c _x and the unknown first weight vector w ^c of the number of learning subjects, and the viewing state similarity. inner product of the second vector is the product of the matrix M _y an unknown second weight vector v ^c of the learning sample size dimensions, constraints to predetermined constant norm of the first vector and second vector The first weight vector w ^c and the second weight vector v ^c that are maximum at the beginning are calculated as weight coefficients for each measurement channel.

That is, the similarity-to-similarity mapping information calculation unit 16 calculates w ^c and v ^c that satisfy the following expression (12).

The maximization problem of equation (12) can be solved using Lagrange's undetermined constant method. Here, the error function (Lagrangian) L is expressed by the following equation (13). Note that λ _w and λ _v are undetermined multipliers.

Incidentally, when the data fluctuation of the biological signal and the viewing state (noise) is large, there is a tendency that w ^c, v norm of ^c increases. Therefore, here, the error function L ′ shown in the following formula (14), in which a penalty term depending on the norms of w ^c and v ^c is added to the error function L, is set so that the norm does not become excessive. . Note that ρ _w and ρ _v are parameters for controlling the vulnerability to data noise, and are preset depending on the data (ρ _w > 0, ρ _v > 0).

Here, in this equation (14), when L ′ is differentiated by w ^c and v ^c and each is set to “0”, this results in a fixed value problem shown in the following equation (15). I represents an N × N-dimensional unit matrix. The derivation of equation (15) will be described later.

_{Here, r w = 2ρ w / λ} w, r v = 2ρ v / λ v, is a λ ² = 4λ _w λ _v.
Here,, [rho was set _w and [rho _v advance, [rho instead of setting the _w and [rho _v, may be possible to preset the _{r w} and _{r v (e.g.,} r w = 1, _{r v} = 1).
By solving the eigenvalue problem of equation (15), w ^c , v ^c and λ can be determined.
However, w ^c and v ^c are also eigenvectors, and usually a plurality of them are determined by the number of eigenvalues λ. Therefore, the similarity-to-similarity mapping information calculation unit 16 selects a predetermined number (here, P) of eigenvalues in descending order, determines the corresponding eigenvectors w ^c and v ^c as P pairs, and from the P pair vectors, As mapping information, mapping matrices W ^c and V ^c as shown in the following equation (16) are generated.

As a result, the biological signal similarity matrix M ^c _x (see Equation (5)) calculated by the learning biological signal similarity calculation unit 12 and the viewing state similarity calculated by the learning viewing state similarity calculation unit 15. The relationship with the degree matrix M _y (see the above equation (8)) can be related as shown in the following equation (17) through the P-dimensional space by the mapping matrix of the equation (16).

This inter-similarity mapping information calculation means 16 calculates mapping information (mapping matrices W ^c , V ^c ) for all channels c, and writes and stores them in the mapping information storage means 17.
Note that the similarity mapping information calculation unit 16 sequentially obtains similarities corresponding to N learning subjects in different combinations from the learning biosignal similarity calculation unit 12 and the learning viewing state similarity calculation unit 15. Entered.
Therefore, the similarity-to-similarity mapping information calculation means 16 stores mapping information (mapping matrices W ^c , V ^c ) in the mapping information storage means 17 for each combination of learning subjects.

The mapping information storage unit 17 stores the mapping information (mapping matrix) calculated by the similarity mapping information calculation unit 16. The mapping information storage means 17 can be configured by a general storage device such as a semiconductor memory or a hard disk.
The mapping information storage means 17 stores mapping information (mapping matrices W ^c , V ^c ) calculated for different combinations of learning subjects (N persons).
The mapping information (mapping matrix) stored in the mapping information storage unit 17 is referred to by the channel selection unit 18 and the viewing state estimation value calculation unit 22.

The channel selection means 18 selects a channel closely related to the answer to the question (questionnaire result) from the channels where the biological signal is measured.
This channel selection means 18 uses mapping information from learning subject data different from the data (biological signal, learning answer data [viewing state]) used when the mapping information calculation means 16 calculates the mapping information. Calculate the accuracy of and select the channel with the highest accuracy.
Here, the channel selecting unit 18 includes a similarity calculating unit 181, an error calculating unit 182, and a channel specifying unit 183.

The similarity calculation means 181 uses the biological signal of the learning subject i (2 ≦ i ≦ N) used when calculating the mapping information and the biological signal of the learning subject j (N + 1 ≦ j ≦ M) different from the viewing state. The degree of similarity between the learning subjects is calculated with respect to the viewing state.
Here, the similarity calculation unit 181 refers to the biological signal stored in the biological signal storage unit 11 for each channel c, and the biological signal x ^c _{i of} the learning subject i (N + 1 ≦ i ≦ M). Then, the correlation coefficient M ^c _{x, i, j} with the biological signal x ^c _j of the learning subject j (N + 1 ≦ j ≦ M) is calculated by the above equation (2). For the similarity, another similarity index such as an inner product of vectors or a cosine distance may be used.

Then, the similarity calculation means 181 is similar to the biological signal shown in the following formula (18) having the similarity (correlation coefficient M ^c _{x, i, j} ) between (MN) learning subjects. A degree matrix M ′ ^c _x (second biological signal similarity matrix) is calculated.

Further, the similarity calculation unit 181 refers to the viewing state stored in the viewing state storage unit 14, and the viewing state y _i of the learning subject i (N + 1 ≦ i ≦ M) and the learning subject j (N + 1). The similarity M _{y, i, j} with the viewing state y _j of ≦ j ≦ M) is calculated by the equation (7). For the similarity, another similarity index such as a cosine distance may be used.

Then, the similarity calculation means 181 includes a viewing state similarity matrix M ′ _y (Equation 19) having the similarity M _{y, i, j} between (MN) learning subjects as an element. (Second viewing state similarity matrix) is calculated.

The similarity calculation means 181 uses the biometric signal similarity matrix M ′ ^c _x calculated by the expression (18) and the viewing state similarity matrix M ′ _y calculated by the expression (19) as an error calculation means 182. Output to.
The similarity calculation means 181 calculates the similarity between the biological signals and the viewing state similarity between the N learning subjects and the remaining (MN) learning subjects when calculating the similarity. The degree will be calculated. In general, there are many combinations for selecting N people from M people. Therefore, the similarity calculation unit 181 and the error calculation unit 182 described later repeatedly perform calculation under a sufficient number of combinations (for example, 100 types).

The error calculation unit 182 converts the biometric signal similarity (biological signal similarity matrix M ′ ^c _x ) calculated by the similarity calculation unit 181 and the viewing state similarity (viewing state similarity matrix M ′ _y ). The prediction error is calculated by applying the mapping information (weighting coefficient) stored in the mapping information storage means 17.
Here, the error calculation unit 182 stores the mapping information storage unit 17 corresponding to the combination of the learning subjects whose similarity is calculated by the learning biological signal similarity calculation unit 12 and the learning viewing state similarity calculation unit 15. The prediction error E is calculated by the following equation (20) for each channel c using the mapping matrices W ^c and V ^c (see the equation (16)).

  The error calculating unit 182 outputs the prediction error E and the channel c when the prediction error E is calculated to the channel specifying unit 183. When there are a plurality of combinations of learning subjects, the error calculation means 182 calculates the prediction error E for each combination, and sets the average as the prediction error E again.

The channel specifying unit 183 specifies a channel based on the prediction error calculated by the error calculating unit 182.
Here, the channel specifying means 183 is a channel suitable for estimating the viewing state of a channel whose prediction error is lower than a predetermined threshold value (for example, “0.1”), that is, a small error. Identify. Note that the channel specifying means 183 may select one channel with the smallest error, or may select a plurality of channels below a predetermined threshold.

The channel specifying unit 183 outputs (notifies) the specified channel (selected channel C) to the biological signal extracting unit 20, the selected biological signal similarity calculating unit 21, and the viewing state estimated value calculating unit 22.
Thus, the channel selection means 18 verifies various mapping information and selects a channel corresponding to mapping information with high accuracy.
Thus, the viewing state estimation apparatus 1 can estimate a channel (biological signal) suitable for estimating the viewing state by machine learning.

The biological signal extraction means 20 is selected by the channel selection means 18 from biological signals for each measurement channel input from an externally connected biological signal measuring device (not shown) for the subject subject whose viewing state is to be estimated. The biological signal of the selected channel is extracted.
The biological signal extraction unit 20 has the same function as the biological signal input unit 10 except that the biological signal of the selected channel is extracted. That is, the biological signal input means 10 digitizes the biological signal of the selected channel as a time series signal.
The biological signal extraction unit 20 outputs a time-series biological signal associated with the selected channel (selected channel C) to the selected biological signal similarity calculation unit 21.

The selected biological signal similarity calculating unit 21 is configured to obtain the biological signal in the selected channel of the subject subject extracted by the biological signal extracting unit 20 and the biological signal in the selected channel of the learning subject stored in the biological signal storage unit 11. The similarity is calculated.
Here, when the identifier of the selected channel is C, the time-series biological signal x ^C _new in the selected channel C of the subject subject extracted by the biological signal extracting unit 20 is expressed by a vector of the following equation (21). Can do.

In this equation (21), x ^C _{new, 1} is a biological signal first measured during content viewing, and indicates a biological signal measured sequentially in time series up to x ^C _{new, T.}
Then, the selected biological signal similarity calculating unit 21 outputs the biological signal x ^C _new of the subject subject in the selected channel C selected by the channel selecting unit 18 and the biological signal x of all the learning subjects represented by the above formula (1). The similarity (selected biological signal similarity) with ^C _i (i = 1 to M) is calculated.
Here, the selected biological signal similarity calculating unit 21 calculates the correlation coefficient s _xi as the similarity for each learning subject i (i = 1 to M) as shown in the following formula (22).

^Here, _{X - C new new} is a vector with the mean value of all elements of the vector ^x _{C new new} of the formula (21) to each element, specifically, the following equation (23) and (24) (X￣ ^C _j is also the same). In Expression (23), T represents the number of elements of the vector x ^C _new shown in Expression (21). Expression (24) is a T-dimensional vector in which all elements are “1”.

When the selected channel C is selected by the channel selecting unit 18, the selected biological signal similarity calculating unit 21 is an M-dimensional vector (similarity vector) having the correlation coefficient s _xi of the equation (22) as the i-th element. ) S _x is output to the viewing state estimated value calculation means 22. When a plurality of channels are selected in the channel selection unit 18, the selected biological signal similarity calculation unit 21 calculates a similarity vector s _x for each selected channel, and sends it to the viewing state estimation value calculation unit 22. Output.

The viewing state estimated value calculation means 22 is a degree of similarity between the unknown learning answer data (viewing state) of the target subject and the learning answer data (viewing state) of the learning subject stored in the viewing state storage means 14. And the unknown number of learning answer data having the maximum correlation with the similarity of the biological signal calculated by the selected biological signal similarity calculating unit 21 is calculated as the viewing state estimation value.
The viewing state estimation value is data obtained by quantifying the questionnaire result estimated that the target subject will answer the questionnaire conducted on the learning subject.

Specifically, the viewing state estimated value calculation means 22 calculates the viewing state estimated value as follows.
Here, an unknown questionnaire result (correctness corresponding to the answer to the question or data obtained by quantifying the evaluation level) of the subject subject whose number of questions is N _q is expressed as N _q dimension as shown in the following formula (25). (Viewing state vector y _new ).

At this time, the similarity between the viewing state vector y _new and the viewing state vectors y _i (i = 1 to M) of all the learning subjects stored in the viewing state storage unit 14 (see the above equation (6)). Can be expressed as a similarity vector s _y shown in the following equation (26).

Note that the biological information and the viewing state are associated with the mapping information storage unit 17 by mapping matrices W ^c and V ^c (see Expression (16)). That is, the similarity vector s _x of the calculated selected biological signal similarity calculation unit 21 biological signal, and the similarity vector s _y of view condition of the equation (26), the relationship shown in the following equation (27) Have C indicates a channel selected by the channel selection means 18.

Therefore, the viewing state estimated value calculation means 22 calculates the viewing state vector y _new that satisfies the equation (27) as the viewing state estimated value y ^ _new .
That is, the viewing state estimated value calculation means 22 calculates a viewing state vector satisfying the following equation (28) among the viewing state vectors y _new as the viewing state estimated value y ^ _new .

There are various methods for solving the equation (28). For example, the viewing state estimation value calculating unit 22 calculates the similarity vector s _{y according} to the equation (26) for all combinations of y _new values. generated may be the right side of the norm squared ^{_{(‖s x t W C -s y}} t V C ‖ ²⁾ _{y new new} which is minimum viewing state estimate _{y ^ new new} equation (28).
Depending on the questionnaire items, it may be difficult to list all combinations of values. Therefore, the viewing state estimated value calculation means 22 may calculate the viewing state estimated value ＾ _new using the following equation (29) obtained by solving the equation (28) by the least square method.

In this equation (29), the matrix Y indicates a matrix in which the i-th column is the viewing state vector y _i of the learning subject i (see the above equation (6)). The matrix YV ^C V ^Ct Y ^t indicates a generalized inverse matrix of YV ^C. The derivation of this equation (29) will be described later.

In this way, the viewing state estimated value calculation means 22 outputs the viewing state estimated value y ^ _new satisfying the equation (28) to the outside as the viewing state of the target subject.
When a plurality of channels are selected by the channel selection unit 18 and a plurality of viewing state estimation values y ^ _new are calculated, the viewing state estimation value calculation unit 22 includes elements of the respective viewing state estimation values y ^ _new . Each time, an average value is calculated.

At this time, when correct / incorrect values (for example, correct answer “1”, incorrect answer “0”) are associated with the answer to the question, the viewing state estimated value calculation means 22 determines the average value of the target subject's understanding. Estimate as a degree. Thereby, if it is closer to “1”, it can be estimated that the subject subject understands the question more.
When correct / incorrect values are associated with the answer to the question, the viewing state estimated value calculation means 22 estimates the most frequent result for each element of the plurality of viewing state estimated values y ^ _new as the answer to the question. It is good to do.

As described above, by configuring the viewing state estimation device 1, the viewing state estimation device 1 allows the similarity of the biological signal between the plurality of learning subjects and the viewing state (questionnaire of the questionnaire) between the plurality of learning subjects. By associating the similarity of the answer pattern) with the mapping information, the unknown viewing state can be estimated from the biological signal of the new target subject.
Note that the viewing state estimation apparatus 1 can operate the computer with a program (viewing state estimation program) for causing the computer to function as each unit configured as described above.

(Derivation of equation (15))
A procedure for deriving the equation (15), which is an eigenvalue problem, from the equation (14) in the similarity mapping information calculation unit 16 will be described.
First, L ′ in the equation (14) is differentiated by w ^c as shown in the following equation (30) and set to “0”.

Here, since M _x ^c is a symmetric matrix, there is a relationship of M _x ^c = M _x ^ct . Therefore, this equation (30) can be transformed into the following equation (31).

Here, I represents an N × N-dimensional unit matrix.
Similarly, the L 'in the formula (14), differentiated by ^{v c} as shown in the following equation (32), put "0".

Since _{M y} is a symmetric _matrix, a relationship of M y _{= M} ^{y t.} Therefore, this equation (32) can be transformed into the following equation (33).

  Further, the equation (33) can be transformed into the following equation (34).

The ^{v c,} are substituted into the equation (31) by the following expression (35).

Here, if r _w = ρ _w / λ _w , r _v = ρ _v / λ _v , 4λ _w λ _v = λ ² (that is, r _w = 2ρ _w / λ, r _v = 2ρ _v / λ). The equation (15) can be derived from the equations (34) and (35).

(Derivation of Equation (29))
Derivation of the expression (29) for calculating the viewing state estimated value ＾ _new in the viewing state estimated value calculating means 22 will be described below.
The equation (28) can be modified as shown in the following equation (36).

Here, the square of the norm on the right side of the equation (36) is differentiated by y _new as shown in the following equation (37), and set to “0”.

  Then, the equation (29) can be derived by modifying the equation (37) as shown in the following equations (38) to (41).

[Operation of viewing state estimation device]
Next, with reference to FIG. 3 and FIG. 4, operation | movement of the viewing-and-listening state estimation apparatus 1 which concerns on embodiment of this invention is demonstrated. Here, the learning stage operation of the viewing state estimation apparatus 1 will be described with reference to FIG. 3, and the estimation stage operation will be described with reference to FIG.

(Learning stage operation)
First, the learning stage operation of the viewing state estimation apparatus 1 will be described with reference to FIG.
The viewing state estimation device 1 receives a biological signal from a learning subject who is viewing content for each channel for measuring a biological signal by means of a biological signal input means 10 via a biological signal measuring device (not shown) connected to the outside. (Biological signal for learning) is input (step S1).

  Then, the viewing state estimation device 1 writes and stores the learning biological signal input in step S1 by the biological signal input means 10 in the biological signal storage means 11 as a time series signal in association with the learning subject and the channel. (Step S2).

  In addition, the viewing state estimation device 1 is a result of a plurality of learning subjects whose biological signals are measured during content viewing by the viewing state input unit 13 answering a question (questionnaire) asking about the viewing state of the content. The answer data for learning (viewing state) is input (step S3).

  Then, the viewing state estimation device 1 writes and stores the learning answer data input in step S3 by the viewing state input unit 13 in the viewing state storage unit 14 as a numerical answer pattern in association with the learning subject. (Step S4).

  Then, the viewing state estimation device 1 returns to step S1 if the learning biosignal and the learning answer data are not input by the predetermined number of learning subjects (M) (No in step S5). Repeat the operation.

On the other hand, when the learning biosignal and the learning answer data are input by the predetermined number of learning subjects (M) (Yes in step S5), the viewing state estimation device 1 calculates the biometric similarity for learning. For the biological signals of the plurality of learning subjects stored in step S2 by means 12, the similarity of the biological signals between the learning subjects for each channel (biological signal similarity matrix M ^c _x [see Equation (5) above)] ) Is calculated (step S6).
Here, the biometric signal similarity calculation means 12 for learning uses a combination of N persons sequentially for N persons (N <M), which is smaller than the total number of learning persons (M), and learners with different combinations. A biosignal similarity (N × N-dimensional biosignal similarity matrix) is calculated.

In addition, the viewing state estimation device 1 uses the learning viewing state similarity calculation unit 15 to determine the viewing state between the learning subjects for the learning answer data (viewing state) of the plurality of learning subjects stored in step S4. Similarity (viewing state similarity matrix M _y [see the above equation (5)])) is calculated (step S7).
Here, the learning viewing state similarity calculating means 15 changes the combination of N persons sequentially for N persons (N <M), which is smaller than the total number of learning persons (M persons), and learners with different combinations. The similarity (N × N-dimensional viewing state similarity matrix) of the answer data for learning (viewing state) is calculated.

  Then, the viewing state estimation device 1 uses the similarity mapping information calculation unit 16 to calculate the similarity (biological signal similarity matrix) calculated in step S6 and the similarity (viewing state similarity matrix) calculated in step S7. Mapping information (weighting coefficient) is calculated (step S8).

Here, the similarity-to-similarity mapping information calculation means 16 calculates the product vector of the N × N-dimensional biological signal similarity matrix M ^c _x and the N-dimensional weight vector w ^c for each combination of channel and learning subject, and viewing. as the inner product of the vector product of the state similarity matrix M _y and the number of dimensions of the same weight vector v ^c is the norm of each vector is maximized under the constraint that "1", a plurality (P number ) Weight vector pair (W ^c , V ^c ) is calculated.

  Then, the viewing state estimation device 1 writes the mapping matrix composed of the plurality of weight vectors calculated in step S8 in the mapping information storage unit 17 by the similarity mapping information calculation unit 16 (step S9).

  Then, the viewing state estimation apparatus 1 calculates the accuracy of the mapping information from the data of the learning subject different from the data (biological signal, viewing state) used when the mapping information is calculated in step S8 by the channel selection unit 18. Then, a highly accurate channel is selected (step S10).

Here, the channel selection means 18 uses the similarity calculation means 181 to learn learning subject combinations (N people) used for the mapping information calculation in step S6 and step S7, and other learning subjects ([MN] people). ) Between the biological signal (biological signal similarity matrix M ′ ^c _x [refer to the above equation (18)]) and the viewing state similarity (viewing state similarity matrix M ′ _y [the above equation ( 19)]]) is calculated for each channel.

Then, the channel selection means 18 is the product of the biological signal similarity matrix M ′ ^c _x for each channel calculated for the learning subject (N + 1 to M) by the error calculation means 182 and the mapping matrix W ^c calculated in step S8. And the product of the viewing state similarity matrix M ′ _y and the mapping matrix V ^c calculated in step S8 is calculated (see the above equation (20)).

Then, the channel selecting unit 18 selects a channel for which the error calculated by the error calculating unit 182 is lower than a predetermined threshold by the channel specifying unit 183.
Thereby, the viewing state estimation apparatus 1 can select a channel suitable for estimating the answer data (viewing state estimation value) indicating the viewing state of the subject subject.

(Estimation stage operation)
Next, referring to FIG. 4 (refer to FIG. 2 as appropriate for the configuration), the operation in the estimation stage of the viewing state estimation apparatus 1 will be described.
The viewing state estimation device 1 is selected by the biological signal extraction means 20 from the biological signal of the target subject input via a biological signal measuring device (not shown) connected to the outside at the learning stage described in FIG. The biological signal of the selected channel is extracted (step S20).

Then, the viewing state estimation apparatus 1 uses the selected biological signal similarity calculating unit 21 to detect the biological signal in the selected channel of the target subject extracted in step S20 and the learning subjects (1 to 1) stored in the biological signal storage unit 11. M) The similarity with the biological signal in the selected channel (similarity vector s _x with the correlation coefficient s _xi [see Equation (22) above) as the i-th element) is calculated (step S21).

  Then, the viewing state estimation apparatus 1 is calculated in step S21 by the viewing state estimated value calculation means 22 based on mapping information (weighting factor) between the similarity of the biological signal between the learning subjects and the similarity of the viewing state. The viewing state (viewing state estimation value) of the target subject is calculated from the similarity of the biological signal between the target subject and the learning subject (step S22).

Here, the viewing state estimated value calculation means 22 uses the unknown questionnaire result of the subject subject as the viewing state vector y _new, and the learning subject viewing state vector y _i (i = 1) stored in the viewing state storage means 14. formula similarity vector _{s y} with ~M) and (26).
Then, the viewing state estimated value calculation unit 22 obtains the viewing state vector y _new that satisfies the expression (27) from the mapping matrix stored in the mapping information storage unit 17 corresponding to the selected channel C. ^ Calculated as _new .
Thereby, the viewing state estimation apparatus 1 can estimate the viewing state of the subject subject from the biological signal of the channel suitable for estimating the viewing state.

  As described above, the viewing state estimation device 1 uses the similarity of the biological signal (time-series data) between the plurality of learning subjects and the similarity of the viewing state (learning answer data) between the plurality of learning subjects. Thus, the corresponding unknown answer data (viewing state) can be newly estimated from the biological signal (time-series data) of the subject subject.

In this way, the viewing state estimation device 1 does not associate a specific biological reaction that occurs during content viewing with the viewing state as in the past, but the similarity of the biological signal (time-series data) between the learning subjects. Since the viewing state of the target subject can be estimated from the relevance between the similarity of the viewing state and the viewing state, the viewing state can be estimated for any content.
Also, by using the viewing state estimation device 1, the content creator can objectively grasp the influence of the content on the viewer and quantitatively evaluate the effect of the content.

DESCRIPTION OF SYMBOLS 1 Viewing state estimation apparatus 10 Biological signal input means 11 Biological signal storage means 12 Learning biological signal similarity calculation means 13 Viewing state input means 14 Viewing state storage means 15 Learning viewing state similarity calculation means 16 Similarity mapping information calculation means 17 mapping information storage means 18 channel selection means 181 similarity calculation means 182 error calculation means 183 channel identification means 20 biological signal extraction means 21 selected biological signal similarity calculation means 22 viewing state estimation value calculation means

Claims (5)

Based on the biological signals of a plurality of learning subjects who are viewing the content and the learning answer data which is a result of the learning subject answering the question asking the viewing state of the content A viewing state estimation device that estimates a viewing state of the target subject from a biological signal of the subject,
Learning biosignal similarity calculating means for calculating biosignal similarity, which is a similarity between learning subjects, with respect to biosignals of a plurality of learning subjects input for each measurement channel that is a measurement site of the biosignal. ,
With respect to the learning answer data, learning viewing state similarity calculating means for calculating a viewing state similarity that is a similarity between the learning subjects;
A similarity-to-similarity mapping information calculation unit that calculates, as mapping information, a weighting coefficient that maximizes the correlation between the biological signal similarity between the predetermined number of learning subjects and the viewing state similarity for each measurement channel; ,
The correlation error when using the mapping information is determined in advance for biosignal similarity and viewing state similarity between a plurality of other learning subjects different from the learning subject for which the mapping information is calculated. Channel selection means for selecting a measurement channel below a threshold as a selection channel;
For the subject subject, a biological signal extraction means for extracting a biological signal of the selected channel from a biological signal for each measurement channel;
A selected biological signal similarity calculating means for calculating a similarity between the biological signal of the target subject extracted by the biological signal extracting means and the biological signals of the plurality of learning subjects;
The selection of the similarity between the unknown learning answer data of the target subject and the learning answer data of the plurality of learning subjects, and the similarity of the biological signal calculated by the selected biological signal similarity calculating means Viewing state estimation value calculating means for calculating, as a viewing state estimation value, the unknown number of learning answer data having a maximum correlation when using the mapping information corresponding to a channel;
A viewing state estimation apparatus comprising: The similarity mapping information calculation means includes:
A first vector that is a product of a biosignal similarity matrix having biosignal similarity between learning subjects as a matrix element and an unknown first weight vector of the number of learning subjects, and learning between the learning subjects The inner product of the second state vector, which is the product of the viewing state similarity matrix having the similarity of the answer data for the matrix as an element of the matrix and the unknown second weight vector of the number of learning subjects, is the first vector and the first The first weight vector and the second weight vector, which are maximized under a constraint condition in which a norm of two vectors is a predetermined constant, are calculated as the weight coefficients for each measurement channel. The viewing state estimation apparatus according to Item 1. The channel selection means includes
With respect to the biological signals of the plurality of other learning subjects and the answer data for learning of the learning subjects, the similarity between the plurality of other learning subjects is used as a matrix element for each measurement channel. Similarity calculating means for calculating a second biological signal similarity matrix and a second viewing state similarity matrix having the similarity of the answer data for learning among the plurality of other learning subjects as elements of the matrix;
Error calculation for calculating an error between the product of the second biological signal similarity matrix and the first weight vector and the product of the second viewing state similarity matrix and the second weight vector for each measurement channel. Means,
Channel specifying means for specifying, as the selected channel, a measurement channel in which the error calculated by the error calculating means falls below a predetermined threshold;
The viewing state estimation apparatus according to claim 2, further comprising: The learning biological signal similarity calculating unit and the learning viewing state similarity calculating unit calculate a biological signal similarity and a viewing state similarity with different numbers of the learning subjects, respectively.
The channel selection means is for learning except for all the learning subjects excluding the learning subjects for which the biometric signal similarity calculating means for learning and the learning viewing state similarity calculating means calculate the similarity. The viewing state estimation apparatus according to claim 3, wherein the second biological signal similarity matrix and the second viewing state similarity matrix are calculated for a subject.   A viewing state estimation program for causing a computer to function as the viewing state estimation apparatus according to claim 1.

Images