JP2015188649A - A plurality of physiological index and visual line analysis support apparatus, programs - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a device outputting the physiological index obtained through a viewpoint tracking device and physiological measuring so that analysts can analyze them easily.SOLUTION: A control part of an analysis support apparatus: creates time-series graph data through the index D80 of subject's emotional state which is calculated for every predetermined sampling intervals (S41); coordinates viewpoint coordinate data D20 with every one frame of scene images D50 (S42); synchronizes the time-series according to the time difference D40 and display a graph of the index D80 of the emotional state created in S41 and the scene image coordinated with the viewpoint index created in S42 in the display part in the same screen (S43).

Description

  The present invention analyzes sensibility / emotional state from a plurality of physiological indices obtained by acquiring biological information, and simultaneously displays the index of the emotional state as the index and the analysis result and the measurement result of the line of sight by the viewpoint tracking device. The present invention relates to an analysis support apparatus for displaying.

  Conventionally, items such as whether a user is easy to use or a good visual impression are evaluated for products such as commercial products or products to be put on the market in the future. In addition, for advertisements such as television commercials and advertising flyers on paper media, items such as whether they are noticed by the user or whether purchase motivation is improved are numerically evaluated. These are sometimes referred to as so-called market surveys. As a typical evaluation method, there is known a method of analyzing data obtained through self-reporting such as a questionnaire or interview survey.

  However, in the self-reporting method such as questionnaires, answers are induced by the bias of the question contents, or the answers are unconsciously created due to the potential fixed idea or position of the user. It is difficult to obtain accurate information. This creation of answers is particularly noticeable in face-to-face surveys such as interview surveys, and there is a problem that it is difficult to reflect the honest feeling of the user in the survey results.

  On the other hand, a method for acquiring the user's biometric information, for example, a method for acquiring and analyzing data such as the user's viewpoint, psychological state, and stress when viewing a certain advertisement is known. This user's biometric information can be obtained in real time, compared to methods based on questionnaires and interviews, etc., and accurate information is obtained because the user's answers are not created and the answers are not arbitrarily selected. It is easy to acquire.

  Various methods are publicly known for the type, acquisition method, and acquisition purpose of the biometric information of the acquired user. The biometric information handled here includes the gaze, brain waves, and autonomic / respiratory physiological indices (heart rate, heart rate variability, blood pressure, cardiac output, blood volume pulse, respiratory rate, tidal volume, skin conductance response) Skin potential response, core temperature, skin temperature, etc.).

  In Patent Document 1, a subject who is a user wears a viewpoint tracking device equipped with an electroencephalograph and a markerless eye tracking function, browses a scene image or a snapshot image related to the content, and is measured by the viewpoint tracking device. An analysis support apparatus is disclosed in which the measurement results of both the subject's viewpoint coordinate data and the electroencephalogram data generated along with brain activity are associated with each other and displayed on the same screen for easy analysis.

JP 2013-180076 A

  As described above, it is known to simultaneously display and analyze the measurement results of both one physiological index obtained from physiological measurements such as brain waves and the line of sight. However, it is easy to analyze the brain wave, autonomic nerves, and respiratory system physiological indices together with the line of sight, and further display the emotional state index data and the line of sight obtained by analyzing the sensitivity and emotional state obtained from those indices. There was no mechanism to output like this.

  The present invention has been made in view of the above-described problems, and the object of the present invention is to provide a measurement result by the viewpoint tracking device and at least one arbitrary physiological index and / or emotions obtained from these indices. It is an object of the present invention to provide an analysis support apparatus capable of outputting both measurement results of state indicators so as to be easily analyzed.

  The first invention for achieving the object described above supports the analysis of measurement results by at least one physiological index obtained from at least one physiological measurement and a viewpoint tracking device for a subject who is browsing the content. An analysis support apparatus that performs at least one physiological index of the subject obtained from the at least one physiological measurement, viewpoint coordinate data of the subject measured by the viewpoint tracking apparatus, and the viewpoint tracking A memory for storing a scene image or a snapshot image related to the content imaged by a device, and a time difference between a measurement start time of each physiological index of the at least one physiological measurement and a measurement start time by the viewpoint tracking device And an emotional state of the subject based on the at least one physiological index An emotional state calculating means for calculating each of the marks, an associating means for associating the viewpoint coordinates of the subject with the scene image or the snapshot image based on the viewpoint coordinate data, and synchronizing a time series based on the time difference. Analysis support comprising: the scene image or the snapshot image to which the viewpoint coordinates of the subject are associated, and display means for displaying the calculated emotional state indicators on the same screen. Device. According to the first invention, it is possible to output at least one physiological measurement and both measurement results by the viewpoint tracking device so as to be easily analyzed.

  The data of each physiological index in the first invention includes a measurement start time by the at least one physiological measurement, and the viewpoint coordinate data includes a measurement start time by the viewpoint tracking device. It is desirable to further comprise time difference calculating means for calculating the time difference based on physiological index data and the viewpoint coordinate data, and storing the time difference in the storage means. As a result, the time difference between the data of each physiological index and the viewpoint coordinate data can be automatically calculated, thereby reducing the load on the analyst. Here, the emotional state index is the amount of the consciousness (unconsciousness) of the subject who is browsing the content on a predetermined scale based on the data of each physiological index acquired by at least one physiological measurement. Data.

  For example, the association means in the first invention associates the viewpoint coordinates with the scene image frame by frame, and the display means reproduces the scene image while superimposing a viewpoint mark indicating the position of the viewpoint coordinates. At the same time, the timeline indicating the playback position of the scene video is superimposed, and at least one or more emotional state index changes are drawn as a time-series graph. Thus, in the case of dynamic content such as a moving image such as a television commercial or a subject moving, the analyst can simultaneously view at least one emotional state indicator and a change in viewpoint of the subject. For example, the analyst can easily visually recognize which scene of the dynamic content is being viewed by the subject when at least one emotional state indicator changes characteristically. .

  Further, for example, the association means in the first invention associates the viewpoint coordinates with the scene image frame by frame, and further sets at least one degree of decoration of a viewpoint mark indicating the position of the viewpoint coordinates. The display means reproduces the scene video while superimposing the viewpoint mark decorated according to the degree of decoration, and displays a timeline indicating the reproduction position of the scene video. At least one or more emotional state index changes are drawn as a time-series graph while being superimposed. Thus, in the case of dynamic content such as a moving image such as a TV commercial or a subject moving, the analyst can simultaneously view the indicator of the emotional state of the subject and the change in the viewpoint. In particular, since the viewpoint mark is decorated and displayed according to the emotional state index, the analyst can visually recognize the fluctuation of the emotional state index on the dynamic content.

   Further, for example, the associating means according to the first aspect of the present invention performs coordinate transformation of the viewpoint coordinates into a coordinate system of the snapshot image and associates the viewpoint coordinates with a single snapshot image, and further determines the position of the viewpoint coordinates. The degree of decoration of the viewpoint mark to be displayed is determined according to at least one of the emotional state indicators, the viewing time is calculated for each viewpoint coordinate, and the area degree of the viewpoint mark is determined according to the viewing time. The display means displays the snapshot image by superimposing the viewpoint mark having an area of the area degree decorated according to the degree of decoration for each viewpoint coordinate. Thus, in the case of static content such as a paper-based advertising flyer, the analyst can simultaneously view the emotional state indicator and the change in viewpoint of the subject. For example, the analyst can easily visually recognize which position in the static content the subject is watching in what emotional state for how long.

   A second invention is a program for causing a computer to function as an analysis support device that supports analysis of measurement results by at least one physiological measurement and viewpoint tracking device for a subject who is browsing content, At least one physiological index of the subject measured by at least one physiological measurement, viewpoint coordinate data of the subject measured by the viewpoint tracking device, and the content imaged by the viewpoint tracking device Based on the scene image or snapshot image, storage means for storing the measurement start time by the at least one physiological measurement and the time difference between the measurement start times by the viewpoint tracking device, and the at least one physiological index Emotional state for calculating an index of the emotional state of the subject An output unit, an association unit that associates the viewpoint coordinates of the subject with the scene video or the snapshot image based on the viewpoint coordinate data, and a time series based on the time difference to synchronize the viewpoint coordinates of the subject. It is a program for causing the scene video or the snapshot image to be associated and the emotional state index to function as display means for displaying on the same screen. By installing the program of the second invention in a general-purpose computer, an analysis support apparatus capable of easily outputting both measurement results by at least one physiological measurement and viewpoint tracking apparatus is obtained. Can do.

  According to the present invention, it is possible to provide an analysis support apparatus capable of outputting the measurement results of at least one physiological measurement and the viewpoint tracking apparatus so as to be easily analyzed.

Schematic diagram of component device of this embodiment Example of wearing eyeglass-type viewpoint tracking device on a subject An example of data stored in the analysis support apparatus 10 The flowchart which shows the flow of a process in the structure apparatus of this embodiment. The flowchart which shows the process of the analysis assistance apparatus 10 in 1st Embodiment. Example of curve data for one or more emotional state indicators 80 An example of a display method when displaying an indicator 80 of a certain emotional state Graph representing emotional state called "Russel's ring" A graph to express the emotional state called the “castle three-dimensional emotional state” An example of one or more emotional state indicators 80 represented as a radar chart Example of measurement result display screen 200 in the first embodiment The flowchart which shows the process of the analysis assistance apparatus 10 in 2nd Embodiment. Example of measurement result display screen 200 in the second embodiment The flowchart which shows the process of the analysis assistance apparatus 10 when a still image is displayed.

  Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings. In addition, the emotional state index refers to the consciousness (unconsciousness) of the subject who is browsing the content according to a predetermined scale based on at least one physiological index acquired by at least one physiological measurement. This is quantified data (details will be described later).

  The contents common to all the embodiments will be described with reference to FIGS.

  FIG. 1 is a schematic diagram of a configuration apparatus according to the present embodiment. As shown in FIG. 1, the configuration device of the present embodiment includes an analysis support device 10, a viewpoint tracking device 20, and various physiological index measurement devices. Here, typical examples of the physiological index measuring device include an electroencephalograph 30, a heart rate meter 40, a skin electrometer 50, a pulse wave / Korotkoff sound measuring device 60, and the like. These physiological index measuring devices measure the viewpoint coordinate data and at least one physiological index of the subject who is browsing the content, and support the analyst to analyze using the measurement results.

  In the example illustrated in FIG. 1, the analysis support apparatus 10 and the physiological index measurement apparatus transmit and receive data via a wired cable. However, when the analysis support apparatus 10 and the physiological index measuring apparatus have a wireless communication function, the physiological index measured via a radio wave may be transmitted and received.

  In the example illustrated in FIG. 1, the analysis support apparatus 10 and the viewpoint tracking apparatus 20 transmit and receive data via the storage medium 27. Examples of the storage medium 27 include a storage card typified by an SD memory card, a storage disk typified by a CD-R (Compact Disk-Recordable), a USB (Universal Serial Bus) memory, and the like. There are auxiliary storage devices with terminals. When the analysis support apparatus 10 and each physiological index measurement device include a wired cable I / O (Input / Output Interface), the analysis support apparatus 10 and each physiological index measurement apparatus transmit data via the wired cable. Transmission and reception may be performed. When the analysis support apparatus 10 and each physiological index measurement device are provided with a wireless communication device, the analysis support apparatus 10 and each physiological index measurement device may transmit and receive data via a radio wave.

  The analysis support apparatus 10 supports the analysis of the measurement result by each physiological index measurement apparatus and the viewpoint tracking apparatus 20 for the subject who is browsing the content. The analysis support apparatus 10 is a computer. Examples of the computer include a desktop PC (Personal Computer), a notebook PC, a smartphone, a tablet terminal, a mobile phone terminal, and a game terminal. Hereinafter, the analysis support apparatus 10 will be described as a general-purpose computer such as a desktop PC or a notebook PC.

  The analysis support apparatus 10 includes a control unit 11, a storage unit 12, a display unit 13, an input unit 14, an I / O 15 and the like.

  The control unit 11 includes a CPU (Central Processing Unit), a ROM (Read Only Memory), a RAM (Random Access Memory), and the like. The CPU calls a program stored in the storage unit 12, ROM, storage medium, or the like to a work memory area on the RAM, executes it, controls the driving of each device, and realizes processing to be described later performed by the analysis support device 10. The ROM is a non-volatile memory, and permanently stores programs such as a boot program and BIOS of the analysis support apparatus 10, data, and the like. The RAM is a volatile memory, and temporarily holds programs and data loaded from the storage unit 12, ROM, storage medium, and the like, and includes a work area used by the control unit 11 to perform various processes.

  The storage unit 12 is, for example, an HDD (Hard Disk Drive), and stores a program executed by the control unit 11, data necessary for program execution, an OS (Operating System), and the like. As for the program, a control program corresponding to the OS and an application program for causing the analysis support apparatus 10 to execute processing to be described later are stored. Each of these program codes is read by the control unit 11 as necessary, transferred to the RAM, read by the CPU, and executed as various means.

  The display unit 13 includes a display device such as a liquid crystal panel, and a logic circuit (such as a video adapter) for realizing the video function of the analysis support device 10 in cooperation with the display device. The input unit 14 inputs data and includes, for example, a keyboard, a pointing device such as a mouse, and an input device such as a numeric keypad. An operation instruction, an operation instruction, data input, and the like can be performed on the analysis support apparatus 10 via the input unit 14. The display unit 13 and the input unit 14 may be integrated like a touch panel display.

  The I / O 15 is a port for connecting peripheral devices (for example, each physiological index measuring device, an external storage device such as the storage medium 27, a printer (not shown), a network device (not shown)) to the analysis support device 10. Or a transceiver. The analysis support apparatus 10 transmits / receives data to / from peripheral devices via the I / O 15. Examples of the I / O 15 include ports such as USB, IEEE 1394, and RS-232C, drive devices for reading and writing the storage medium 27, wireless LAN devices, Bluetooth (registered trademark) transceivers, and the like. The connection form with the peripheral device may be wired or wireless.

  The viewpoint tracking device 20 generally includes a main body 21 and a recorder 22. The main body 21 and the recorder 22 may be integrated or separate.

  The main body 21 includes a scene camera 23, a viewpoint tracking sensor 24, a marker sensor 25, and the like. The marker sensor 25 is not essential. When the marker sensor 25 is used, a plurality of markers (not shown) are also required. The marker is an infrared marker, for example, and is installed to define an area to be analyzed.

  The scene camera 23 captures content and scenery that the subject is viewing. In the present embodiment, the scene camera 23 is a camera that captures scene videos and snapshot images, which will be described later.

  The viewpoint tracking sensor 24 is a sensor for tracking the viewpoint of the subject. The viewpoint tracking sensor 24 includes, for example, an infrared light source and a receiving unit, and tracks the subject's viewpoint based on the principle of infrared corneal reflection detection. Since the principle of infrared corneal reflection detection is known, the description thereof is omitted.

  The marker sensor 25 communicates with the marker and detects the analysis area. Content that needs to detect an analysis area using a marker is, for example, a paper medium such as a magazine advertisement or a flyer.

  The recorder 22 includes a control unit 26, a storage medium 27, and the like. The storage medium 27 is as described above. The control unit 26 stores, in the storage medium 27, scene images and snapshot images captured by the scene camera 23, subject coordinate data tracked by the viewpoint tracking sensor 24, analysis area data detected by the marker sensor 25, and the like. To do.

  As an example of the viewpoint tracking device 20, for example, there are a spectacle type represented by “Tobii Glass Eye Tracker” of Toby Technology, a display type represented by “Tobii TX300 Eye Tracker” of Toby Technology, etc. .

  FIG. 2 shows an example in which a spectacle type viewpoint tracking device is attached to a subject. A scene camera 23 is provided in the main body 21 of the eyeglass-type viewpoint tracking device 20. The viewpoint tracking sensor 24 is not visible from the outside because it is inside the eyeglass type viewpoint tracking apparatus.

  When content can be output to a display, such as a television commercial or a website, the subject may use a spectacle type or a display type. When the subject is assumed to move, or when viewing an object such as a paper medium, the subject uses a glasses type. ?

  Returning to FIG. 1, an electroencephalograph 30 that is one of the physiological index measuring devices includes a headband 31, an electrode 32, a transducer 33, an amplifier 34, and the like. The electroencephalograph 30 has a structure in which an electrode 32 fixed by a headband 31 is in contact with a head of a subject, specifically, a position corresponding to Fp1 in the international 10-20 method (left frontal lobe). Fp1 is generally known to be able to measure reactions related to emotions and emotions. The converter 33 converts the analog signal output from the electrode 32 into a digital signal. The amplifier 34 amplifies the digital signal output from the converter 33 and outputs it to the I / O 15 of the analysis support apparatus 10. The analysis support apparatus 10 stores the data in the storage unit 12 using a storage unit. The sampling frequency in the A / D conversion may be 1024 Hz, for example, and may be the same value as or different from the sampling frequency of other physiological index measuring devices.

  Details of the electroencephalograph 30 are described in, for example, Japanese Unexamined Patent Application Publication No. 2010-131328 and Japanese Unexamined Patent Application Publication No. 2011-45541. As an example of each physiological index measuring apparatus, for example, a simple electroencephalograph such as “alphatec-IV” and “Brain Builder Unit” of the Brain Power Development Laboratory Co., Ltd. may be mentioned. In the electroencephalogram by a simple electroencephalograph, for example, one electrode 32 is attached to an electrically inactive earlobe, and the other electrode 32 is attached to an electrically active scalp to derive a potential difference between two electrodes. Measured by monopolar induction method.

  The heart rate monitor 40, which is one of the physiological index measuring devices, includes a wristwatch type device and a stationary type device that can be wound around a wrist, and in this example, a stationary device is assumed. The heart rate monitor 40 includes an electrode 41, a converter 42, an amplifier 43, and the like. The heart rate monitor 40 is configured such that the electrode 41 is in contact with a site where a heartbeat sound is easily captured, such as a wrist or neck of a subject. The converter 42 converts the analog signal output from the electrode 41 into a digital signal. The amplifier 43 amplifies the digital signal output from the converter 42 and outputs it to the I / O 15 of the analysis support apparatus 10. The analysis support apparatus 10 stores the data in the storage unit 12 using a storage unit. The sampling frequency in A / D conversion is arbitrary.

  The skin electrometer 50, which is one of the physiological index measuring devices, normally includes three electrodes, with the reference electrode in the center of the forearm and the other two electrodes (positive electrode and negative electrode) sandwiched between the reference electrodes. In this way, the potential difference is taken. Skin electrometer 50 includes electrode 51, converter 52, amplifier 53, and the like. The converter 52 converts the analog signal output from the electrode 51 into a digital signal. The amplifier 53 amplifies the digital signal output from the converter 52 and outputs it to the I / O 15 of the analysis support apparatus 10. The analysis support apparatus 10 stores the data in the storage unit 12 using a storage unit. The sampling frequency in A / D conversion is arbitrary.

  A pulse wave / Korotkoff sound measuring device 60, which is one of physiological index measuring devices, is generally the same device as a device known as a blood pressure measuring device. There is usually one cuff (arm band) (two cuffs for the left arm and right arm may be provided), and the pulse wave and Korotkoff sound are wrapped around the upper arm's elbow joint. Is to measure. The pulse wave / Korotkoff sound measuring device 60 includes a cuff 61, a converter 62, an amplifier 63, and the like. The converter 62 converts an analog signal output from a microphone built in the cuff 61 into a digital signal. The amplifier 63 amplifies the digital signal output from the converter 62 and outputs it to the I / O 15 of the analysis support apparatus 10. The analysis support apparatus 10 stores the data in the storage unit 12 using a storage unit. The sampling frequency in A / D conversion is arbitrary.

  Next, emotional state indices obtained from several physiological indices will be described. Here, examples of emotion state indicators include the following. "Interest / concentration" (degree of interest / concentration and concentration when viewing the object), "stress / relaxed state" (stress / relaxed state when viewing the object), "pleasant / uncomfortable" (when viewing the object Comfort / discomfort), “arousal level” (awakening level when viewing objects), “basic emotions” (basic emotional states such as emotions when browsing objects), “autonomic nerve activity level” ( Autonomic nerve indices (LF (Low Frequency), HF (Hi Frequency), etc.) are calculated from physiological indices, and autonomic nerve activity is calculated). Further, the value of the “physiological index value” itself may be handled in the same manner as the analysis index.

  Any of the above-described methods for calculating the emotional state index is known. For example, in Patent Document 1 cited above, it is assumed that the user's subject's brain wave is measured, the brain wave data is subjected to frequency analysis, the average for each frequency is calculated from the power spectrum data, and the user is not interested. A method of calculating the “interest” of a subject by calculating a difference between the average of the obtained data and the data of each subject used for filter calculation is disclosed. According to this, it is possible to obtain an analysis index related to “degree of interest” using the physiological index of the electroencephalograph 30.

  In addition, JP 2009-66017 A and JP 2012-50711 A improve accuracy by reducing a burden (stress) associated with measurement of a pulse wave, which is a wave accompanying a heartbeat of a subject who is a user. A stress evaluation apparatus and a stress evaluation system are disclosed in which a chaos analysis is performed on the data and the psychological state and stress of the subject are analyzed. According to this, it is possible to accurately diagnose the mental and physical state “stress level” of the subject based on the calculated stress evaluation value.

  Similarly, Japanese Laid-Open Patent Publication No. 9-70399 discloses a relaxation degree determination device that can easily and accurately determine the degree of relaxation using a heartbeat (pulse) signal of a subject who is a user. . This is the sequential calculation of the rate of increase of the required time for the predetermined heart rate pulsation required to hit a predetermined heart rate, and the emotional state such as “relaxation” by comparing the rate of increase and the criterion value Can be determined. About these, the analysis parameter | index which concerns on "stress degree" and "relaxation degree" can be obtained by utilizing the physiological index obtained by the heart rate monitor 40, the pulse wave and the Korotkoff sound measuring device 60, and the like.

  Japanese Patent Application Laid-Open No. 2006-149470 collects biological signals such as pulse waves and heartbeats of a subject who is a user by using a device for observing peripheral circulation such as a fingertip volume pulse wave measuring device. An apparatus for extracting a frequency component including peak values of a muscle fatigue signal and a sleep onset signal and evaluating whether or not the subject feels pleasant / discomfort is disclosed. Similarly, Japanese Patent Laid-Open No. 2000-116614 discloses a comfort evaluation that determines whether the subject feels comfortable by determining the degree of activation and a state that is likely to become sleepy from skin temperature, skin impedance, and pulse. An apparatus is disclosed. About these, the analysis parameter | index which concerns on "pleasantness / discomfort" can be obtained by utilizing the physiological parameter | index obtained with the heart rate meter 40, the skin electrometer 50, the pulse wave and the Korotkoff sound measuring device 60, etc.

  Also, in the re-published WO2010 / 140273, a voltage is applied to an electrode in contact with a subject who is a user, a heartbeat signal of the subject is obtained from a potential difference of each electrode, and a heartbeat interval calculated from the heartbeat signal Has disclosed a wakefulness determination device that performs frequency analysis and determines the wakefulness. According to this, even when the subject feels drowsy and resists not to sleep, the arousal level can be appropriately determined. About this, the analysis parameter | index which concerns on a "wakefulness degree" can be obtained by utilizing the physiological parameter | index obtained with the skin electrometer 50, the pulse wave and the Korotkoff sound measuring device 60, etc.

  Japanese Patent Laid-Open No. 2002-112969 discloses a sensor for detecting biological signals such as blood pressure, skin temperature, and skin resistance for a subject who is a user (a pressure sensing sensor such as a piezo sensor, a comparator connected to an electrode, a thermocouple, and an electrode). Etc.) and a device and method for recognizing basic emotions such as emotions of subjects are disclosed. About this, the analysis parameter | index which concerns on a "basic emotion" can be obtained by utilizing the physiological parameter | index obtained by the skin electrometer 50, the pulse wave and the Korotkoff sound measuring device 60, etc.

  Furthermore, algorithms for measuring fluctuations in heart rate, performing power spectrum analysis of frequency components using the measurement data, and determining the activity level of sympathetic and parasympathetic nerve activities are disclosed. Internet <URL: http: // www. take-clinic. com / psm / hrv / hrv_autonomic2. htm>, [ONLINE] [retrieval on March 12, 2014]. According to this, an analysis index relating to “autonomous nerve activity” can be obtained by using a physiological index obtained by the heart rate monitor 40, the pulse wave / Korotkoff sound measuring device 60, or the like.

  As described above, the method for obtaining the emotional state index obtained from several physiological indices is exemplified, but these operate as the emotional state calculating means of the analysis support apparatus. It should be noted that, when obtaining these emotional state indicators, the physiological index measuring device presented here is not limited to use, and even if a better combination of measuring devices exists, it is included in this embodiment. It is.

  FIG. 3 is an example of data stored in the analysis support apparatus 10. Data stored in the storage unit 12 of the analysis support apparatus 10 includes viewpoint coordinate data D20, physiological index measurement data D30, and a time difference D40 between the measurement start time and the viewpoint measurement start time in each physiological index measurement device (hereinafter referred to as “physiological index measurement”). Device time difference D42), scene video D50, snapshot image D60, and the like. ? ?

  The viewpoint coordinate data D20 has a header part and a main body part. The main body is data with repetition. The header portion includes at least the viewpoint measurement start time D21. The viewpoint measurement start time D21 is a time when measurement by the viewpoint tracking device 20 is started. The main body includes at least a measurement elapsed time D22, a viewpoint x coordinate D23, and a viewpoint y coordinate D24. The measurement elapsed time D22 is an elapsed time from the viewpoint measurement start time D21. The viewpoint x-coordinate D23 and the viewpoint y-coordinate D24 are an x-coordinate and a y-coordinate indicating the viewpoint position of the subject in the coordinate system of the scene image D50 or the snapshot image D60, respectively, measured by the viewpoint tracking sensor 24. The viewpoint x coordinate D23 and the viewpoint y coordinate D24 are stored at the sampling interval of the viewpoint tracking sensor 24 (for example, 30 frames / second). ?

  The physiological index measurement data D30 has a header part and a main body part. The main body is data with repetition. The header part includes at least a physiological index measuring device identifier (ID) D31 and a measurement start time D32. The measurement start time D32 is a time when measurement by each physiological index measurement device is started. The main body includes at least a measurement elapsed time D33 and a measurement value (data) D34. There may be a plurality of measurement values (data) (D34 to D39 in FIG. 2), and the measurement elapsed time D33 is the elapsed time from the measurement start time D32. The measured values (data) D34 to D39 are digital signals sent from the respective physiological index measuring devices to the analysis support apparatus 10. As will be described later, the measurement values (data) D34 to D39 are converted into emotional state indices at a predetermined time interval (for example, 1 second). A data structure organized at intervals may be used. ? ?

  The time difference D42 is a difference between the time when each physiological index measurement device starts measurement and the time when the viewpoint tracking device 20 starts measurement. There are two methods for measuring the time difference D42. ?

  In the first method, a subject or an analyst visually confirms a time point when each physiological index measurement device starts measurement and a time point when the viewpoint tracking device 20 starts measurement, and the time difference between the relative times is measured. D42 is input to the analysis support apparatus 10. For example, the subject or the analyst starts either the physiological index measuring device or the viewpoint tracking device 20 first, and the 0th second of the device started later is the number of the devices started first. Whether the second time is reached is input to the analysis support apparatus 10. In the first method, the measurement start time D32 of the physiological index measurement data D30 and the viewpoint measurement start time D21 of the viewpoint coordinate data D20 are not necessary. Therefore, the subject or the analyst does not need to match the clock of each physiological index measuring device and the clock of the viewpoint tracking device 20. ?

  In the second method, the control unit 11 of the analysis support apparatus 10 determines the time difference between the physiological index measurement devices based on the measurement start time D32 of the physiological index measurement data D30 and the viewpoint measurement start time D21 of the viewpoint coordinate data D20. Time difference calculating means for calculating D42 is provided. For example, a value obtained by subtracting the viewpoint measurement start time D21 of the viewpoint coordinate data D20 from the measurement start time D32 of the physiological index measurement data D30 in a certain physiological index measurement apparatus is set as the time difference D42 of the physiological index measurement apparatus. When the time difference D42 is positive, it means that the viewpoint measurement start time D21 is earlier by the time difference D42 than the measurement start time D32 of the physiological index measurement device. When the time difference D42 is negative, it means that the measurement start time D32 of the physiological index measuring device is earlier than the viewpoint measurement start time D21 by the time difference D42. In the second method, the analysis support apparatus 10 can automatically calculate the time difference D42 between the physiological index measurement data D30 and the viewpoint coordinate data D20, thereby reducing the load on the analyst. However, the subject or the analyst needs to match the clock of each physiological index measuring device with the clock of the viewpoint tracking device 20. It should be noted that the physiological index measuring device identifier (ID) D31 is obtained every time the physiological index measuring data D30 is generated. The physiological index measuring device identifier (ID) D41 may be set every time the time difference D42 of the physiological index measuring device is calculated.

  The scene video D50 is a moving image file (for example, an avi file) captured by the scene camera 23. The scene video D50 is stored at a sampling interval comparable to the sampling interval of the viewpoint tracking sensor 24 (for example, 30 frames / second). In the case of the first embodiment and the second embodiment, the scene video D50 is essential. ?

  The snapshot image D60 is a still image file (for example, a jpeg file) captured by the scene camera 23. The snapshot image D60 is a single image file in which the entire content is included in the imaging range. ?

  FIG. 4 is a flowchart showing the flow of processing in the configuration apparatus of this embodiment. Although the measurement process of the time difference D42 is not shown in FIG. 4, the time difference D42 is measured by the first measurement method or the second measurement method described above, and the time difference D42 is stored in the storage unit 12 of the analysis support apparatus 10. Shall be stored in ?

  Each physiological index measuring apparatus measures the physiological index of the subject based on the measurement start and measurement end instructions from the analysis support apparatus 10 (S31). Then, each physiological index measuring device performs AD conversion of the measured value at predetermined sampling intervals (S32), amplifies (S33), and transmits the amplified result to the analysis support device 10 as physiological index measured data D30. (S34). ?

  The main body 21 of the viewpoint tracking device 20 measures the viewpoint of the subject based on an instruction to start or end measurement from the analyst or the subject (S21). Then, the recorder 22 of the viewpoint tracking device 20 stores the viewpoint coordinate data D20 in the storage medium 27 at predetermined sampling intervals (S22). ?

  The control unit 11 of the analysis support apparatus 10 receives the physiological index measurement data D30 from each physiological index measurement device, and stores the physiological index measurement data D30 in the storage unit 12 (S11). And the control part 11 accumulate | stores the related physiological parameter measurement data D30 in order to calculate the parameter | index of the emotion state designated or predetermined by the analyst. Specifically, physiological index measurement data D30 having a physiological index measurement device identifier necessary for calculating an index of a certain emotional state and having the same measurement time in time series is accumulated (S12), which is necessary for the calculation. When the measured physiological index measurement data D30 is prepared, the specified or predetermined emotional state index is calculated. As described above, any known calculation method is used. And When the emotional state index is calculated, information on the subject and content is added and stored in the RAM or the storage unit 12 (S13).

  The control unit 11 of the analysis support apparatus 10 inputs data from the storage medium 27, and stores the viewpoint coordinate data D20 and the scene video D50 or the snapshot image D60 in the storage unit 12 using the storage unit (S14). Next, the control unit 11 associates the subject's viewpoint coordinates with the scene video D50 or the snapshot image D60 based on the viewpoint coordinate data D20 (S15). Further, the control unit 11 synchronizes the time series based on the time difference D42, and displays the scene video D50 or snapshot image D60 associated with the subject's viewpoint coordinates and one or more emotional state indicators on the display unit 13. Are displayed on the same screen (S16). The control unit 11 may not only display on the display unit 13 but also output it to a printer or send it to another computer. The display process will be described in detail in each embodiment. ?

<First Embodiment>
The first embodiment will be described with reference to FIGS. ?

  FIG. 5 is a flowchart showing the processing of the analysis support apparatus 10 in the first embodiment. As illustrated in FIG. 5, the control unit 11 of the analysis support apparatus 10 generates time-series graph data from one or more emotion state indicators 80 calculated at predetermined sampling intervals (for example, 1 second). (S41).

  The control unit 11 associates the viewpoint coordinate data D20 with the scene video D50 frame by frame (S42). If the sampling intervals of the viewpoint coordinate data D20 and the scene video D50 are the same, the control unit 11 makes a one-to-one correspondence in time series. When the sampling intervals of the viewpoint coordinate data D20 and the scene video D50 are different, the control unit 11 performs an appropriate interpolation process or thinning process, and then associates them in a one-to-one order in time series. ?

  The control unit 11 synchronizes the time series based on the time difference D42, and the curve data of the one or more emotion state indicators 80 generated in S41 and the scene video D50 associated with the viewpoint coordinates in S52, It displays on the display part 13 of the same screen (S43). The “same screen” does not need to have the same display area (window), and may be a display mode in which an analyst can simultaneously view the curve data of one or more emotion state indicators 80 and the scene video D50. For example, the display area (window) may be different. ?

  FIG. 6 is an example of curve data of one or more emotional state indices 80 created in this way. The y axis indicates the magnitude (strongness) of the emotional state index, and the x axis is the time axis.

  FIG. 7 is an example of a display method for displaying an indicator 80 of a certain emotional state. In the example shown in FIG. 7, the emotional state index is indicated by a pointer on an analog scaled disk. You may make it display the parameter | index 80 of several emotion states by using a plurality of scales and pointers.

  FIG. 8 is an example of a display method for displaying an indicator 80 of a certain emotional state. The example shown in FIG. 8 is a graph suitable for representing an emotional state called “Russel's torus”. The coordinates of the emotional state index 181 shown in this graph are two-dimensionally shown from the emotional state index of “pleasant / unpleasant” and the emotional state index of “arousal level”.

  FIG. 9 is a graph for representing another emotional state called “castle three-dimensional emotional state”. In the example shown in FIG. 9, the coordinates of the emotional state index 191 include an emotional state index of “pleasant / unpleasant”, an emotional state index of “arousal level”, and an emotional state index of “degree of interest / concentration”. It is shown three-dimensionally. The position of the emotional state index 191 is projected on the xy plane so that the coordinates of the emotional state index 191 shown here are easy to understand (192), and the position of the emotional state index 191 is also projected on the z axis ( 193). ?

  FIG. 10 is an example in which one or more emotion state indicators 80 are represented as a radar chart. In the example shown in FIG. 10, five emotion state indicators 80 are shown on the radar chart. Here, the index shown in this chart is not limited to the analyzed emotional state index, but may be the physiological index value itself. The same applies to the indices displayed in FIGS. ?

  The measurement result display screen 200 shown in FIG. 11 displays the curve data of one or more emotion state indicators 80, the above-described graph, and the radar chart and the scene video D50 so that they can be simultaneously viewed in time series synchronization. This is an example of the display means. Each display area (window) may be different as long as it can be displayed at the same time. It includes an emotional state indicator curve display region 210, a scene video display region 220, an emotional state indicator graph display region 230, and a measurement information display region 240. The emotional state indicator curve display region 210 is a region for displaying one or more emotional state indicator curve data. The scene video display area 220 is an area for displaying the scene video D50, and the viewpoint mark 221 at that time is displayed. In this example, an emotion state indicator graph display area 230 displays a radar chart of emotion state indicators, and the measurement information display area 240 displays various information of measurement results obtained by the respective physiological index measurement devices and the viewpoint tracking device 20. Is an area for displaying. Here, as described above, it is shown in the emotional state indicator curve display region 210 and the emotional state indicator graph display region 230? The index is not limited to the analyzed emotional state index, but may be the physiological index value itself.

  In this way, by displaying a plurality of physiological indices at the same time, it is possible to visually confirm the correlation between the indices and the quantitative and temporal relations of the undulations for analysis. Further, by estimating an emotional state from a plurality of physiological indices and displaying the estimated emotional state, it is possible to visually confirm complex information that cannot be obtained from only the physiological indices and use it for analysis. Further, by utilizing these, it is possible to acquire and analyze the user's psychological state, its degree, or the tendency of the change as information that is not affected by the bias of the question contents by the questionnaire.

  The analysis support apparatus 10 displays a graph in which the horizontal axis indicates time and the vertical axis indicates a change in the emotional state index value in the emotional state indicator curve display area 210. Here, the emotional state index indicated by the solid line 212 is curve data of one emotional state index generated in S41. The emotional state index indicated by the white line 213 is curve data of another emotional state index generated in S41. The timeline 211 shows the playback position of the scene video D50 played back in the scene video display area 220. If the scene video D50 is being continuously reproduced, the timeline 211 follows the playback position of the scene video D50 and slides from left to right. ?

  For example, the analysis support apparatus 10 embeds a general-purpose moving image playback module in the scene video display area 220. The scene video display area 220 includes buttons for playback, pause, fast forward, rewind, mute (silence), volume adjustment, and the like. The viewpoint mark 221 indicates the position of the viewpoint coordinate data D20 associated in S42. ?

  As shown in FIG. 11, in the first embodiment, the control unit 11 of the analysis support apparatus 10 associates the viewpoint coordinates of the subject with the scene video D50 frame by frame. Then, the control unit 11 reproduces the scene video D50 on the display unit 13 while superimposing the viewpoint mark 221 indicating the position of the viewpoint coordinates, and also superimposes the timeline 211 indicating the reproduction position of the scene video D50 on the emotional state. The change of the index is drawn on the display unit 13 as a time-series graph. ?

  According to the first embodiment, in the case of a moving image such as a television commercial or dynamic content such as a subject moving, the analyst can simultaneously view the indicator of the subject's emotional state and the change in viewpoint. For example, the analyst can easily visually recognize which scene of the dynamic content is being viewed by the subject when the emotional state index is at a peak. ?

  Second Embodiment A second embodiment will be described with reference to FIGS. 12 and 13. In 2nd Embodiment, the same number is attached | subjected to the element same as the element of 1st Embodiment, and duplication description is abbreviate | omitted. ?

  FIG. 12 is a flowchart showing processing of the analysis support apparatus 10 in the second embodiment. As illustrated in FIG. 12, the control unit 11 of the analysis support apparatus 10 generates time-series graph data from one or more emotion state indicators 80 calculated at predetermined sampling intervals (for example, one second). (S51). ?

  The control unit 11 associates the viewpoint coordinate data D20 with the scene video D50 frame by frame (S52). Furthermore, the control unit 11 decorates the mapped viewpoint coordinates according to one or more emotion state indicators 80 (S53). As a decoration mode, it is conceivable to change the shape, pattern, or color of the viewpoint mark or add another mark in accordance with one or more emotion state indicators 80. ?

  The control unit 11 synchronizes the time series based on the time difference D40, and the scene video D50 in which the curve data of the one or more emotion state indicators 80 generated in S51 and the viewpoint mark decorated in S53 are superimposed. Are displayed on the display unit 13 of the same screen (S54). ?

  FIG. 13 is a display example in the second embodiment. In the example illustrated in FIG. 13, the content is dynamic content on the assumption that the subject moves as in the example illustrated in FIG. 11. ?

  The measurement result display screen 200b shown in FIG. 13 includes an emotional state indicator curve display region 210, a scene video display region 220, an emotional state indicator graph display region 230, a measurement information display region 240, and a degree of decoration 250. The decoration degree 250 indicates the degree of decoration in S53. ?

  The analysis support apparatus 10 superimposes the decorated viewpoint mark 221b on the scene video D50 in the scene video display area 220. In the example shown in FIG. 13, the decoration degree 250 is expressed by the difference in the viewpoint mark pattern. That is, the pattern of the viewpoint mark 221b is changed according to the emotional state index 80. Although an example of changing the color cannot be illustrated due to limitations of the patent drawing, for example, as in thermography, “red”, “orange”, “yellow”, “ When the color is changed such as “yellowish green” or “green”, it is easy to understand intuitively. ?

  As shown in FIG. 13, in the second embodiment, the control unit 11 of the analysis support apparatus 10 associates the viewpoint coordinates of the subject with the scene video D50 frame by frame, and further, the viewpoint mark 221b indicating the position of the viewpoint coordinates is displayed. The degree of decoration is determined according to the emotional state index. Then, the control unit 11 reproduces the scene video D50 while superimposing the viewpoint mark 221b decorated according to the degree of decoration, and superimposes the timeline 211 indicating the reproduction position of the scene video D50 while superimposing the emotional state index. The fluctuation is drawn on the display unit 13 as a time-series graph. ?

  According to the second embodiment, in the case of dynamic content such as a moving image such as a television commercial or a subject moving, the analyst can simultaneously view the emotional state indicator and the change in viewpoint of the subject. In particular, since the viewpoint mark 221b is decorated and displayed according to the emotional state index, the analyst can visually recognize the fluctuation of the emotional state index on the dynamic content. ?

  In addition, the subject can also display a still image such as a paper medium advertising flyer using the snapshot image D60 instead of the moving image such as the scene image D50 in the first and second embodiments. The change of the viewpoint and the index of the emotional state can be visually recognized at the same time.

  FIG. 14 is a flowchart showing the processing of the analysis support apparatus 10 when the subject's viewpoint change and the emotional state index are simultaneously viewed when a still image is displayed. As shown in FIG. 14, the control unit 11 of the analysis support apparatus 10 associates the coordinates of the viewpoint coordinate data D20 with the coordinate system of the snapshot image D60 (S61). The control unit 11 synchronizes the time series based on the time difference D40, calculates a viewing time for each viewpoint coordinate, and displays a snapshot image D60 on which a viewpoint mark whose area degree is changed according to the time is superimposed. It is displayed on the part 13 (S62).

  Furthermore, the viewpoint mark indicating the subject viewpoint may be decorated according to the emotional state index 80 as described in the second embodiment. In this way, the analyst can use the snapshot image D60 to display a still image such as a paper-based advertising flyer and change the subject's viewpoint, gaze time, and emotional state, as in the first embodiment. Can be simultaneously viewed.

  The preferred embodiments of the analysis support apparatus and the like according to the present invention have been described above with reference to the accompanying drawings, but the present invention is not limited to such examples. It will be apparent to those skilled in the art that various changes or modifications can be conceived within the scope of the technical idea disclosed in the present application, and these are naturally within the technical scope of the present invention. Understood. ?

10 …… Analysis support device 20 …… Viewpoint tracking device 30 …… Physiological index measurement device (electroencephalograph)
40 …… Physiological index measurement device (heart rate monitor)
50 …… Physiological index measurement device (skin electrometer)
60 …… Physiological index measurement device (pulse wave / Korotokoff sound meter)
D20 ... viewpoint coordinate data D30 ... physiological index measurement data D40 ... time difference between physiological index measurement start time and viewpoint measurement start time D50 ... scene image D60 ... snapshot image

Claims (6)

An analysis support device that supports analysis of measurement results by at least one physiological index obtained from at least one physiological measurement for a subject viewing content and a viewpoint tracking device,
The at least one physiological index of the subject obtained from the at least one physiological measurement, the viewpoint coordinate data of the subject measured by the viewpoint tracking device, and the content imaged by the viewpoint tracking device Storage means for storing the scene image or snapshot image, and the time difference between the measurement start time of each physiological index of the at least one physiological measurement and the measurement start time by the viewpoint tracking device;
An emotional state calculating means for calculating an index of the emotional state of the subject based on the at least one physiological index,
Association means for associating the subject's viewpoint coordinates with the scene video or the snapshot image based on the viewpoint coordinate data;
Display means for synchronizing the time series based on the time difference and displaying the scene image or the snapshot image associated with the viewpoint coordinates of the subject and the calculated emotional state indicators on the same screen; An analysis support apparatus comprising: The data of each physiological index includes a measurement start time by the at least one physiological measurement, and the viewpoint coordinate data includes a measurement start time by the viewpoint tracking device,
2. The analysis according to claim 1, further comprising a time difference calculating unit that calculates the time difference based on the data of each physiological index and the viewpoint coordinate data, and stores the time difference in the storage unit. Support device. The association means associates the viewpoint coordinates with the scene image frame by frame,
The display means reproduces the scene video while superimposing a viewpoint mark indicating the position of the viewpoint coordinates, and superimposes a timeline indicating the reproduction position of the scene video while at least one indicator of the emotional state The analysis support apparatus according to claim 1, wherein the fluctuation is drawn as a time-series graph. The association means associates the viewpoint coordinates with the scene image frame by frame, and further determines the degree of decoration of a viewpoint mark indicating the position of the viewpoint coordinates according to at least one emotion state index. ,
The display means reproduces the scene video while superimposing the viewpoint marks decorated according to the degree of decoration, and at least one or more emotions while superimposing a timeline indicating a reproduction position of the scene video The analysis support apparatus according to claim 1, wherein the fluctuation of the state index is drawn as a time-series graph. The association means performs coordinate conversion of the viewpoint coordinates into a coordinate system of the snapshot image and associates the viewpoint coordinates with a single snapshot image, and at least one degree of decoration of a viewpoint mark indicating the position of the viewpoint coordinates Determining according to two or more indicators of the emotional state, calculating a viewing time for each viewpoint coordinate and determining an area degree of the viewpoint mark according to the viewing time,
The display means displays the snapshot image by superimposing the viewpoint mark having an area of the area degree decorated according to the degree of decoration for each viewpoint coordinate. The analysis support apparatus according to claim 2. A program for causing a computer to function as an analysis support device that supports analysis of measurement results by at least one physiological measurement and viewpoint tracking device for a subject viewing content,
At least one physiological index of the subject measured by the at least one physiological measurement, viewpoint coordinate data of the subject measured by the viewpoint tracking device, and the content imaged by the viewpoint tracking device Storage means for storing the scene image or snapshot image according to the above, and the time difference between the measurement start time by the at least one physiological measurement and the measurement start time by the viewpoint tracking device;
An emotional state calculating means for calculating an index of the emotional state of the subject based on the at least one physiological index,
Association means for associating the subject's viewpoint coordinates with the scene video or the snapshot image based on the viewpoint coordinate data;
Synchronizing the time series based on the time difference, and functioning as a display means for displaying the scene video or the snapshot image associated with the subject's viewpoint coordinates and the emotional state indicator on the same screen program.

Images