JP5445981B2 - Viewer feeling judgment device for visually recognized scene - Google Patents

Description

  The present invention relates to a viewer feeling determination apparatus for a visually recognized scene, and particularly relates to a technique for analyzing a change in physiological reaction of a viewer when visually recognizing the visually recognized scene and evaluating an object and the ability of the viewer himself.

  Conventionally, in order to evaluate video content such as a television program, the audience rating is measured. However, the audience rating alone does not know whether or not the viewer actually watched even if the TV channels were combined, and in addition, a huge number of survey viewers is required to improve reliability. There were problems such as. In view of this, there has been disclosed a technique for capturing an eye movement of a viewer and efficiently evaluating video content from the obtained data.

JP 2004-282471 A

  In the technique described in the above-mentioned patent document, the video content is visually recognized using the movement of the line of sight, the pupil area, the blink frequency, etc. as data obtained by imaging the eye movement. Judging the degree of interest and interest and evaluating video content. As a result, a higher-level evaluation that is more reliable than an evaluation method that uses the audience rating as an index has been realized with a small number of subjects.

However, as in the technology described in the above-mentioned patent document, it is possible to accurately grasp the physiological reaction of the viewer only with data such as eye movement, pupil area, and blink occurrence frequency obtained from an eye movement image. It was difficult. That is, in the technique described in the above-mentioned patent document, when the blink occurrence frequency decreases, it is determined that the object is focused, and the change in the emotion of the viewer is analyzed from the position of the eyeball center and the pupil area. However, these merely determine the degree of attention of the viewer with respect to the object, and do not determine the viewer's emotion (emotion and emotion) in more detail. The definition of emotion and emotion in the text of this patent is shown below. Emotion is a physiological reaction that reacts unconsciously to a visual target (including sound) or physical stimulation. As an example, there is a phenomenon in which attention is unconsciously when there is an interest in a visual target. Emotions are feelings felt by people's movements and things. Joy, sadness, anger, surprise, disgust, fear. Here, emotion (E) and emotion (F: Feeling) are collectively defined as emotion (EF).

Moreover, the technique described in the above-mentioned patent document evaluates the degree of attention to video content such as a television program, and is not assumed to be applied to other scenes.
Further, although the pupil diameter is determined as the attention level as it is, the pupil diameter greatly reacts to the brightness (illuminance and luminance) of the viewing target, and therefore means for removing the pupil diameter response corresponding to the brightness of the viewing target. In addition, the pupil diameter response with an accurate attention degree cannot be obtained.

  Therefore, in order to solve the above-described problems, the object of the present invention is to grasp the physiological reaction of the viewer more precisely and finely and to evaluate the object and the viewer's own ability more accurately and finely in various situations. It is another object of the present invention to provide a viewer feeling determination device for visual recognition scenes (including auditory, taste, tactile, olfactory stimuli, and image recall).

In order to solve the above-mentioned problem, the invention according to claim 1 is the viewer feeling determination device for determining the feeling of the specific object when the viewer looks at the scene including the specific object. An eyeball photographing device for photographing an image of the eyeball of a viewer, a visual scene photographing device for photographing a scene including the specific object as a visual image of the viewer, and a face photographing for photographing an image of the face of the viewer Device, a brightness measuring device that measures brightness information of a visual field of view by the viewer, an eyeball image captured by the eyeball imaging device, and a visual image captured by the visual scene photographing device, While calculating the viewpoint position, calculating the acceleration of the viewpoint along with the movement of the viewpoint, obtaining the movement of the viewpoint, calculating the pupil diameter from the eyeball image, and calculating the pupil diameter Based on the brightness information measured by the brightness measuring device and the relationship information between the preset brightness information and the pupil diameter, a new pupil diameter that eliminates the influence of the brightness of the visual field of view is obtained. Calculating the degree of attention from the new pupil diameter , the position of the predetermined part of the face image captured by the face photographing apparatus, the position of the predetermined part in the normal time of the viewer, and the predetermined Calculated from the visual image and eyeball image when the viewer is looking at an object that is less interested and not of interest based on the position at which the part moved to the maximum, and an analysis unit that calculates the degree of emotion viewpoint position and viewpoint movement of the viewpoint, including acceleration, the degree of the determined emotion from the face image, and with the attention level calculated from the eyeball image is previously stored as normal value, noting high interest Movement of the viewpoint when the object the viewer is watching, a storage unit for degree and attention emotion is previously stored as impressed value, the movement of the viewpoint obtained by the analysis unit, the degree of emotion and A diagnostic unit that compares the degree of attention with the viewpoint movement , the degree of emotion, and the degree of attention in the normal value and the emotional value accumulated in advance by the accumulating unit, and determines the feeling for the specific object, It is characterized by that.

Further, the invention of claim 2 is the viewer feeling determination apparatus according to claim 1, wherein the analysis unit is based on an eyeball image photographed by the eyeball photographing device and a visual image photographed by the visual scene photographing device. , Calculating the viewpoint position in the visual image, and approaching speed and approaching acceleration when the viewpoint approaches the specific object as the viewpoint moves, or the viewpoint leaves the specific object. And the diagnosis unit determines that the greater the approach speed and the approach acceleration calculated by the analysis unit, the more interested and focused on the specific object. It is determined that the smaller the approach speed and the approach acceleration are, the less interested the specific object is and attention is not paid to it. It is determined that the object is less interested and not focused, or the smaller the separation speed and the separation acceleration, the more interesting and focused the specific object is determined. To do.

Further, the invention of claim 3 is the viewer feeling determination device according to claim 2, wherein the analysis unit is further mounted on the scene photographing device with a predetermined position in the visual image as a base point. The gyroscope calculates the movement size, movement speed, and movement acceleration of the viewer's head, calculates the viewpoint position in the visual image from the visual image and eyeball image, and changes the speed of the viewpoint position and A change acceleration is calculated, and the viewpoint position including the movement of the head, the change speed and change of the viewpoint position, from the magnitude, movement speed and movement acceleration of the head, and the viewpoint position, change speed and change acceleration. An acceleration is calculated, and an approach speed and an approach acceleration when the viewpoint approaches the specific object as the viewpoint moves including the movement of the head, and the viewpoint is the specific pair. And calculating the dissociation rate and disengagement acceleration when leaving from the object.

According to a fourth aspect of the present invention, in the viewer feeling determination device according to the first aspect, the brightness measuring device is a luminance meter that measures the luminance of the visual field of view, and a new pupil is formed by the analysis unit. The relationship information between the brightness information and the pupil diameter used when calculating the diameter is the luminance and the pupil diameter when the pupil diameter of the viewer is calculated by changing the luminance, and the analysis unit By calculating the pupil diameter from the eyeball image, specifying the pupil diameter corresponding to the luminance as the brightness information from the relationship information, and subtracting the specified pupil diameter from the calculated pupil diameter, the new pupil The diameter is calculated.

Further, the invention of claim 5 is the viewer feeling determination device according to claim 4, wherein when the object to be viewed by the viewer is a video display, the brightness measuring device measures the brightness of the video display. And changing the brightness of the video display and the brightness of a predetermined pixel on the video display using the relation information between the brightness information and the pupil diameter used when the analysis unit calculates a new pupil diameter. Then, the luminance of the video display when calculating the pupil diameter of the viewer, the luminance of the predetermined pixel, and the pupil diameter, the analysis unit calculates the pupil diameter from the eyeball image, and as the brightness information The pupil diameter corresponding to the brightness of the video display and the brightness of the pixel at the viewpoint position is specified from the relation information, and the pupil diameter is calculated from the calculated pupil diameter. By subtracting the boss was pupil diameter, and calculates the new pupil diameter.

The invention of claim 6 is the viewer emotion determination apparatus according to claim 4 or 5, wherein the analysis unit comprises a result of subtracting the minimum value from the maximum value of the pupil diameter in the relationship information, and the calculated The ratio between the new pupil diameter is obtained as the degree of attention.

Further, according to a seventh aspect of the present invention, in the viewer feeling determination apparatus according to the first aspect, the analysis unit calculates the attention degree expansion speed and expansion acceleration or the attention degree reduction speed from the obtained attention degree. The reduced acceleration is calculated, and the diagnosis unit determines that the greater the degree of attention obtained by the analysis unit is, the more interested and focused the specific object is, and the analysis unit obtains the reduction acceleration. The smaller the degree of attention is, the less the interest of the specific object is determined and the attention is not focused, and the greater the expansion speed and expansion acceleration of the attention calculated by the analysis unit are, Determining that there is no interest in and attention to the specific object as the reduction speed and reduction acceleration of the degree of attention calculated by the analysis unit increases. Feature To.

Further, according to an eighth aspect of the present invention, in the viewer feeling determination apparatus according to any one of the first to seventh aspects, the viewer touches or operates the specific object with a hand, and The diagnosis unit determines the viewer's feelings about the specific object while listening to the sound of the object, smelling the object, or eating the predetermined object and feeling the taste. It is characterized by.

The invention of claim 9 is the viewer feeling determination device according to any one of claims 1 to 8 , wherein the specific object is a product, and the diagnostic unit is the viewer for the product. It is characterized by determining the willingness to purchase as feelings.

The invention according to claim 10 is the viewer feeling determination device according to any one of claims 1 to 8 , wherein the specific object is a learning object of a teacher, a teaching material, or a blackboard, and the diagnosis unit However, the learning motivation degree of the viewer with respect to the learning object is determined as emotion.

The invention of claim 11 is the viewer feeling determination device according to any one of claims 1 to 8 , wherein the specific object is a content video produced by a video content producer, The diagnosis unit determines the viewer's emotion for each scene of the content video, and the visual recognition for the content video is based on the preset emotion for each scene and the determined emotion It is characterized in that the degree of achievement of the expectation level of the person is determined.

The invention according to claim 12 is the viewer feeling determination device according to any one of claims 1 to 8 , wherein the specific object is selected from a specific screen of a video, a still image, a poster, and a magazine. The content is any one of the pages, and when the diagnostic unit displays the content on the screen, the attention degree obtained by the analyzing unit is displayed as a distribution at the viewpoint position calculated by the analyzing unit. It is characterized by.

Further, the invention of claim 13 is the viewer feeling determination device according to any one of claims 1 to 8 , further comprising a transmission unit that transmits information of the emotion determined by the diagnosis unit, The specific object is a caller who is having a conversation while watching a video of the other party using a videophone communication line, and the diagnosis unit determines an emotion from one caller to the other caller. The transmitting unit transmits information about feelings from one caller determined by the diagnosis unit to the other caller for presentation to the other caller.

Further, the invention of claim 14 is the viewer feeling determination device according to any one of claims 1 to 8 , wherein the viewer is a person or an animal that is difficult to speak a language. The diagnostic unit determines the viewer's feelings about the specific object.

The invention of claim 15 is the viewer feeling determination device according to any one of claims 1 to 8 , wherein the viewer feeling determination device is incorporated in a robot, and further, the diagnosis unit It is characterized by comprising a selection unit for selecting the speech and movement of the robot and the movement of the robot based on the determined emotion.

Further, the invention of claim 16 is the viewer feeling determination device according to any one of claims 1 to 8 , wherein the viewer is a criminal suspect, and the specific object is a crime scene, It is a criminal tool, an accomplice, or a victim, and the diagnosis unit determines the feeling of the criminal suspect with respect to the specific object.

Further, the invention of claim 17 is the viewer feeling determination device according to any one of claims 1 to 8 , wherein the viewer is a driver of a transportation facility, and the specific object is The video content includes a dangerous spot that can be seen from the driver's seat of a transportation facility, and further includes a warning unit that notifies the driver of a warning, and the diagnosis unit is configured to move the viewpoint , the degree of emotion, Dangerous driving by comparing the attention level with the viewpoint movement , emotion level, and attention level in the normal value and the impression value stored in advance by the storage unit, and determining whether or not attention is paid to the dangerous part It is determined whether or not the vehicle is in a state, and the warning unit notifies the driver of a warning when the diagnosis unit determines that the vehicle is in a dangerous driving state.

Further, the invention of claim 18 is the viewer feeling determination device according to any one of claims 1 to 8 , wherein the viewer is a worker, and the specific object includes a dangerous part. As a video content of the operation device, further comprising a response device operated by the worker, the diagnosis unit stores in advance the viewpoint movement , the degree of emotion and the degree of attention obtained by the analysis unit , and the storage unit. The movement of the viewpoint , the degree of emotion, and the degree of attention are compared with each of the normal value and the emotional value, and it is determined whether the worker is paying attention to the dangerous place, Based on this, it is determined whether or not an appropriate operation has been performed.

Further, the invention of claim 19 is the viewer feeling determination device according to any one of claims 1 to 8 , wherein the viewer is a worker, and the specific object includes a dangerous part. the video content of the operation device, wherein the diagnosis unit, the movement of the viewpoint obtained by the analysis unit, and the degree and attention emotion, movement of the viewpoint in the prestored normal value and excitement value by the accumulation unit, emotion And the degree of attention are compared with each other, and the physical condition or suitability of the worker with respect to the operating device is determined as emotion.

The invention of claim 20 is the viewer feeling determination device according to any one of claims 1 to 8 , wherein the viewer is an athlete, and the athlete exercises the specific object. and the object to see when the diagnosis section, the movement of the viewpoint obtained by the analysis unit, and the degree and attention emotion, movement of the viewpoint in the prestored normal value and excitement value by the accumulation unit, emotion And the degree of attention are compared with each other, and the skill level or appropriateness level of the athlete with respect to the object is determined as emotion.

Further, the invention of claim 21 is the viewer feeling determination device according to any one of claims 1 to 8 , wherein the viewer is a game player, the specific object is a game machine, A body temperature measuring device for measuring the body temperature of the game player, and the diagnostic unit is a normal movement stored in advance by the storage unit, and the viewpoint movement , the degree of emotion and the attention level obtained by the analysis unit. Emotions including the degree of excitement of the game player by the game player based on the body temperature measured by the body temperature measurement device It is characterized by determining.

The invention according to claim 22 is the viewer feeling determination device according to any one of claims 1 to 8 , wherein the specific object is an object arranged in a space, and the diagnosis is performed. each comparison section is, movement of the viewpoint obtained by the analysis unit, and the degree and attention of emotion, the storage unit by prestored normal value and excitement value at the viewpoint of the motion, and a degree and attention emotion And determining the feeling of the specific object, and determining the arrangement and appropriateness of the specific object.

Further, the invention of claim 23 is the viewer feeling determination device according to any one of claims 1 to 8 , wherein the viewer is a customer, and the specific object is a clerk for the customer or and receptionist, the diagnostic unit, the movement of the viewpoint obtained by the analysis unit, and the degree and attention emotion, movement of the viewpoint in the prestored normal value and excitement value by the storage unit, the degree of emotion and The degree of attention is compared, the customer's feelings with respect to the store clerk or the receptionist are determined, and the quality of the store clerk or the receptionist is determined.

According to a twenty-fourth aspect of the present invention, in the viewer feeling determination device according to any one of the first to eighth aspects, the viewer is a TV viewer, and the specific object is broadcast, networked Alternatively, the content is displayed on a display connected to the video storage device, and the diagnosis unit performs normal viewpoints stored in advance by the storage unit, viewpoint movement , emotion level, and attention level obtained by the analysis unit. The movement of the viewpoint , the degree of emotion, and the attention level in the value and the emotion value are respectively compared, the emotion of the TV viewer with respect to the content is determined, and the field of the content that is highly interested by the TV viewer is determined. The predetermined content belonging to the same field is presented to other TV viewers who are highly interested in the content belonging to the same field. To.

According to a 25th aspect of the present invention, in the viewer feeling determination apparatus according to any one of the 1st to 8th aspects, the viewer is a TV viewer, and the specific object is broadcast, networked Alternatively, the content includes a predetermined product or a person displayed on a display connected to the video storage device, and the diagnosis unit obtains the viewpoint movement , the degree of emotion, and the degree of attention obtained by the analysis unit , and the storage. Comparing the viewpoint movement , the degree of emotion, and the attention level in the normal value and the emotional value accumulated in advance by the unit to determine the emotion of the TV viewer with respect to the content, and to the content displayed on the display Among the included products or persons, a mark is given to the product or person at the viewpoint position with a high degree of attention in the threshold judgment, and the product with the mark attached Other information about the person, characterized in that it presented to the TV audience.

As described above, according to the present invention, the pupil diameter in which the influence of the brightness of the visual field of view is eliminated is calculated, and the feeling of the object is determined based on the physiological response data including the calculated pupil diameter. . Thereby, the physiological reaction of the viewer can be grasped more accurately, and the target object and the viewer's own ability can be more accurately evaluated in various situations.

It is a block diagram which shows the structure of the viewer emotion determination system by embodiment of this invention. It is a block diagram which shows the structure of the viewer feeling determination apparatus by embodiment of this invention. It is a figure explaining calculating the position of the viewpoint by embodiment of this invention. It is a figure for demonstrating the relationship of the motion of a viewer's head and visual image by embodiment of this invention. It is an example of the coordinate axis at the time of analyzing the visual recognition image by embodiment of this invention. It is a figure explaining movement of a head when a viewer's head moves to a horizontal direction by an embodiment of the present invention. When the head from the frame screen F ₀ according to an embodiment of the present invention to the next frame screen F ₀ +1 has moved a diagram illustrating the movement of the base point of the stationary. It is a figure which shows the movement of the viewpoint in the visual recognition image by embodiment of this invention. Is a diagram illustrating the movement of the viewpoint at the time of moving from the frame screen F ₀ to the frame screen F ₀ +1 according to an embodiment of the present invention. It is a figure which shows the positional relationship of the target object and viewpoint in the visual recognition image by embodiment of this invention. It is a flowchart which shows the calculation method of the approach speed, approach acceleration, detachment speed, and detachment acceleration by embodiment of this invention. 5 is a flowchart illustrating a method for calculating viewpoint tracking time according to an embodiment of the present invention. It is a figure of the part measurement of the face by the face analysis method for emotion judgment of the present invention . It is an example of the average attention level distribution of viewers in specific video content (the minimum attention level is 0 and the maximum attention level is 100). It is a figure which shows the average attention degree concentration degree (%) of the viewer in specific video content . It is a block diagram which shows the structure of the viewer feeling determination system used for compatibility diagnosis by embodiment of this invention. It is a block diagram which shows the structure of the viewer feeling determination apparatus used for compatibility diagnosis by embodiment of this invention. It is a block diagram which shows the structure of the viewer feeling determination apparatus used for the reception diagnosis of a robot by embodiment of this invention. It is a block diagram which shows the structure of the viewer feeling determination apparatus used for driver drowsiness detection diagnosis by embodiment of this invention.

The best mode for carrying out the present invention will be described below in detail with reference to the drawings.
[Viewer feeling judgment system]
First, the configuration of the viewer feeling determination system and the processing of each component will be described. FIG. 1 is a block diagram showing a configuration of a viewer feeling determination system according to an embodiment of the present invention. This viewer feeling determination system includes an eye camera 3, a transmission device 7, a storage device 8, a body motion photographing device 9 for photographing a body motion including the face of the viewer, a body temperature measuring device 10, a brightness measuring device 11, A response device 14 and a viewer feeling determination device 100 are provided.

  First, the viewer 1 equips the head with the eye camera 3 and directs his / her line of sight to the viewing scene including the object 2. When the visual target is content of a video display, a device corresponding to an eye camera may be fixed near the display. Here, the visual recognition scene may be anything as long as it is visible to the viewer 1, for example, video data displayed on a display of a computer, a mobile phone, a television, etc., in addition to a store, a plaza, a human being, and a townscape. Web page data, computer game data, data output by a computer program, and the like are also included. In the data displayed on the display, response prompting information for causing the viewer 1 to react, such as a character string, a figure, a symbol, a picture, a photo, or a video, is displayed, and the character string, the figure, the symbol, the picture, Reaction target information, such as a photo or a moving image, to which the viewer 1 reacts is included.

  The eye camera 3 includes an eyeball photographing device 4, a visual scene photographing device 5, and a microphone 6. In FIG. 1, the eye camera 3 is connected to the storage device 8 and the transmission device 7, and the viewer 1 also wears the storage device 8 and the transmission device 7. In the case where the visual recognition target is a display such as a computer, a mobile phone, or a television, the eye camera may be installed on the side of the display without mounting the eye camera on the head.

  The eyeball photographing device 4 photographs the eyeball of the viewer 1. The eye movement image b photographed by the eyeball photographing device 4 is accumulated in the accumulating device 8. Note that the eye movement video b may be accumulated as a moving image or may be accumulated as a still image periodically shot.

  The visual scene photographing device 5 photographs a visual scene including the object 2 in which the viewer 1 is looking. The visual image a photographed by the visual scene photographing device 5 is stored in the storage device 8. Note that the visual image a may be accumulated as a moving image or may be accumulated as a still image that is periodically taken.

  The brightness measuring device 11 has a function of measuring the brightness g of the visual recognition target. In this case, when the visual recognition target is a video display, the luminance of the video display is measured. The brightness data g is stored in the storage device 8.

  The microphone 6 captures the viewer 1 and surrounding sounds. The voice data c captured by the microphone 6 is stored in the storage device 8.

  The response device 14 acquires a response signal by the operation of the viewer 1. As an operation of the viewer 1, for example, a button press operation, a keyboard operation, a mouse operation, a touch panel operation, a remote control operation, an operation of a controller attached to a game device, an operation of a machine, etc. A body action is mentioned. The response data d acquired by the response device 14 is stored in the storage device 8.

  The storage device 8 stores the visual image a from the visual scene photographing device 5, the eye movement video b from the eyeball photographing device 4, the audio data c from the microphone 6, and the response data d from the response device 14. The human body motion image e, the body temperature data f, and the brightness data g including the above images are stored in time series as synchronized data.

  The transmission device 7 includes a visual image a, an eye movement video b, audio data c and response data d accumulated by the accumulation device 8, a viewer motion video e including a video of the viewer's face, body temperature data f, and brightness data. g is transmitted to the viewer feeling determination apparatus 100 by wireless or wired. At this time, the transmission apparatus 7 may transmit each data to the viewer feeling determination apparatus 100 at a predetermined time interval, and each data is transmitted to the viewer feeling determination apparatus 100 according to an instruction from the viewer feeling determination apparatus 100. May be sent to.

The body motion photographing device 9 photographs the motion of each part of the body including the face of the viewer 1. Moreover, the body motion imaging device 9 transmits the captured body motion image e to the viewer feeling determination device 100 by wireless or wired.
In this case, a face photographing device for photographing only the face of the viewer may be prepared separately, and for example, the face can be photographed by being attached to the tip of the eaves of a hat-type eye camera.

  The body temperature measuring device 10 measures the temperature of the body of the viewer 1. In addition, the body temperature measurement device 10 transmits the body temperature data f to the viewer feeling determination device 100 by wireless or wired. In FIG. 1, the temperature of the body of the viewer 1 is remotely measured by the body temperature measuring device 10, but may be directly measured by a sensor attached to the viewer 1. In this case, the measured body temperature data f is accumulated in the accumulation device 8 and transmitted to the viewer feeling determination device 100 by the transmission device 7.

  Here, it is appropriate that the imaging frequency of the video frame in the eyeball imaging device 4, the visual scene imaging device 5 and the body motion imaging device 9 is as high as 240 Hz or higher. Further, it is desirable that the shooting frequencies of these video frames are the same.

  In FIG. 1, the eye camera 3 and the response device 14 are connected to the storage device 8 and transmit each data to the storage device 8 by wire, but each data is transmitted to the storage device 8 wirelessly. You can send it. Furthermore, in the case of wireless, the visual scene photographing device 5, the eyeball photographing device 4, the brightness measuring device 11, the microphone 6, and the response device 14 may directly transmit each data to the viewer feeling determination device 100.

[Viewer feeling judgment device]
Next, the configuration of the viewer feeling determination apparatus 100 shown in FIG. 1 and the processing of each component will be described. FIG. 2 is a block diagram illustrating a configuration of the viewer feeling determination apparatus 100 according to the embodiment of the present invention. The viewer feeling determination apparatus 100 includes a receiving unit 101, an analysis unit 102, a storage unit 103, a diagnosis unit 104, and a display unit 105.

[Receiver]
The receiving unit 101 receives a visual image a, an eye movement video b, audio data c, brightness data g, and response data d from the transmission device 7. In addition, the receiving unit 101 receives the body motion image e from the body motion imaging device 9. Further, the receiving unit 101 receives body temperature data f from the body temperature measuring device 10. Then, the receiving unit 101 synchronizes all received data and outputs each data to the analyzing unit 102 at a predetermined time interval or in accordance with an instruction from the analyzing unit 102. In order to synchronize the received data, a method of including the time of the clock as a synchronization signal in each data, and a synchronization signal from the viewer feeling determination device 100, the transmission device 7, the body motion imaging device 9, and the body temperature measurement device 10 Visual image a from the transmitting device 7, eye movement image b, audio data c and response data d, body motion image e from the body motion photographing device 9, body temperature data f from the body temperature measuring device 10, brightness There is a method of including the brightness data g from the measuring device 11 as synchronization signal data. Thereby, all data can be processed synchronously.

[Analysis Department]
Analyzing unit 102 receives visual image a, eye movement video b, audio data c, response data d, body motion video e, body temperature from receiving unit 101 at predetermined time intervals or by instructing receiving unit 101. Data f and brightness data g are acquired. Then, the analysis unit 102 attaches synchronization signal data to each of the data a to g and stores the data in the storage unit 103. Moreover, the analysis part 102 reads each data ag accumulated in the accumulation | storage part 103 as needed, and uses each read data ag for analysis. The analysis unit 102 calculates the position of the viewpoint in the viewing image a from the viewing image a and the eye movement image b, and calculates the change in the physiological response data of the viewer and the accompanying acceleration.

  FIG. 3 is a diagram for explaining the calculation of the position of the viewpoint using the viewing image a and the eye movement image b. As shown in FIG. 3, the analysis unit 102 obtains a line of sight from the eye movement image b, and calculates the position of the viewpoint by combining the line of sight with the visual image a. Here, there are various gaze calculation methods. For example, there is a method of irradiating a near-infrared ray of a point light source to an eyeball called a cornea reflection method and using a reflection image (hereinafter referred to as a Purkinje image) on the cornea surface. The line-of-sight measurement method by the corneal reflection method is roughly divided into two types: a method for obtaining the line of sight from the distance between the pupil center and the Purkinje image, and a line connecting the center of the corneal curvature and the pupil center obtained from the Purkinje image as the virtual line of sight. There is a method of obtaining a line of sight by performing error correction.

  FIG. 4 is a diagram for explaining the relationship between the movement of the viewer 1's head (including the movement of the face) and the visual image a. As illustrated in FIG. 4, the analysis unit 102 analyzes the movement of the head of the viewer 1 based on the visual image a from the visual scene photographing device 5. If the frame frequency at the time of photographing by the visual scene photographing device 5 is S (/ second), the frame interval F is F = 1 / S (second). In FIG. 4, the movement of the head can be measured only with the visual image a captured by the visual scene photographing device 5 worn by the viewer 1 on the head. However, since the measurement is based on the visual image a, it is two-dimensional. Here, the viewing angle (horizontal θh, vertical θv) of the visual image a is set in advance by the visual scene photographing device 5. The analysis unit 102 can also measure the head movement using the body motion image e. Alternatively, the head motion is measured by specifying the head from the body motion image e by image processing and tracking the head. As a completely different technique, it is also possible to measure the movement of the head using a gyroscope attached to the head of the viewer 1 or the visual scene photographing device 5.

FIG. 5 is an example of coordinate axes when the visual image a is analyzed. FIG. 6 is a diagram illustrating the movement of the head when the viewer 1 moves in the horizontal direction. Here, the entire field of view around the head of the viewer 1 is compared to a spherical surface, and the movement of the head is represented by the horizontal direction h and the vertical direction by v. The coordinate axes shown in FIG. 5 may be represented by only numbers (h, v), or by radians (rad) or angles (°). For example, each round may be represented by 1 (−0.5 to +0.5), 2πrad (−π to + π), or 360 ° (−180 ° to + 180 °). In FIG. 6, the first frame picture as F _0, represents the next frame screen after 1 / S seconds as F ₀ +1.

Further, the analysis unit 102 calculates the moving distance, moving speed, and moving acceleration of the head. FIG. 7 is a diagram illustrating the movement of the base point of the stationary object when the head moves from the frame screen F ₀ of the visual image a to the next frame screen F ₀ +1. The visual image a is represented by coordinates (x, y) with the lower left corner as the origin, and the point of the non-moving object (stationary object) “A” in the first frame screen F ₀ is defined as the base point. Is (x0, y0). Here, the stationary object “A” and its point (coordinates) may be determined by an operator who operates the viewer feeling determination apparatus 100 by looking at the visual image “a”, or may be determined in advance. The feature of the still object “A” may be captured by the visual image a and automatically tracked. Note that image processing for identifying and tracking the stationary object “A” is a known technique, and thus detailed description thereof is omitted here. By analyzing the position (x1, y1) of the stationary object “A” in the next frame screen F ₀ +1 after 1 / S seconds, the analysis unit 102 calculates the following data.
Horizontal movement distance of head: h0 =-(x1-x0)
Vertical movement distance of head: v0 =-(y1-y0)
(Because the movement of the viewer's head is opposite to the movement of the base point "A", it is in the negative (-) direction)
If you phv the moving distance of the head, the movement distance from the F ₀ of the head to the _{F 0 +1: phv0 = (h0} 2 + v0 2) 1/2
The analysis unit 102 calculates the following data by combining all the moving frames (up to n).
Total moving horizontal distance: phn = Σh = h0 + h1 +... + Hn
Total moving vertical distance: pvn = Σv = v0 + v1 +... + Vn
Overall travel distance: phvn = (phn ² + pvn ² ) ^1/2
Here, 0, 1, 2,..., N are frame numbers up to n when the first frame number is 0. The analysis unit 102 can calculate the head moving speed qhv≈d (phv) / dt, and the head moving speed qhv0 in the frame screen F ₀ is
It is calculated as qhv0≈d (phv) / dt≈ (phv1-phv0) / (1 / S).
However, t represents time . The analysis unit 102 can calculate the head movement acceleration khv≈d (qhv) / (1 / S), and the head movement acceleration khv0 in the frame screen F ₀ is
It is calculated as khv0≈d (qhv) / dt≈ (qhv1-qhv0) / (1 / S).

As a method for calculating the moving distance, moving speed, and moving acceleration of the head, there is a gyroscope (unit: angular velocity: radians or degrees / second). A general method is to calculate the movement angle by integrating the angular velocity with the elapsed time. The angular movement speed of the head in the L frame can be obtained by measuring the moving speed (qhv _L ) at the frame L time with a gyroscope.
Horizontal movement speed of head at frame L = qh _L
Vertical movement speed of head at frame L = qv _l
As the horizontal movement distance of the head until the N frame is as follows.

The vertical movement distance of the head up to N frames is as follows.

It becomes. The moving distance from the head F ₀ to F _N is
phvN = (phN ² + pvN ² ) ^1/2
Further, since this analysis method analyzes video in units of frames, the analysis unit 102 calculates the following data by combining all moving video frames.
The total travel distance is
phvN = (phN ² + pvN ² ) ^1/2
Further, the analysis unit 102 calculates the head moving speed qhv0 at the time of the frame screen F ₀
qhv0≈ (phv1-phv0) / (1 / S).
Further, since the analysis unit 102 can calculate the head movement acceleration as khv≈d (qhv) / (1 / S), the head movement acceleration khv0 in the frame screen F ₀
It is calculated as khv0≈d (qhv) / dt≈ (qhv1-qhv0) / (1 / S).

Further, the analysis unit 102 calculates the movement of the viewpoint in the visual image a. FIG. 8 is a diagram illustrating movement of the viewpoint B in the visual image a. In FIG. 8, the position of the viewpoint in the visual image a on the frame screen F ₀ is displayed as (H0, V0). Further, the position of the viewpoint on the next frame screen F ₀ +1 is displayed as (H1, V1). By measuring the position (H1, V1) of the viewpoint B in the next frame screen F ₀ +1 after 1 / S seconds, the analysis unit 102 calculates the horizontal movement distance pH0 of the viewpoint.
pH0 = H1-H0
Calculate as
Similarly, the vertical movement distance pV0 is
Calculated as pV0 = V1-V0. In addition, the moving distance pHV0 from F ₀ to F ₀ +1 of the viewpoint is
Calculated as pHV0 = (pH0 ² + pV0 ² ) ^1/2 .
In addition, the horizontal movement distance pH totaling all the movement frames F ₀ to Fn is calculated as follows:
Calculate as pHn = ΣpH = pH0 + pH1 + pH2 +... + pHn.
Similarly, the vertical movement distance pV is
pVn = ΣpV = pV0 + pV1 + pV2 +... + pVn.
Therefore, the total moving distance pHVn from F ₀ to Fn is
pHVn = (pHn ² + pVn ² ) ^1/2
Calculate as
Further, the horizontal speed qH0 of the viewpoint in the visual image a on the frame screen F ₀ is set as follows:
qH0≈d (pH) / dt≈ (pH1-pH0) / (1 / S)
Similarly, the vertical velocity qV0 is
It is calculated as qV0≈d (pV) / dt≈ (pV1−pV0) / (1 / S).
The viewpoint moving speed qHV0 on the frame screen F ₀ is
It is calculated as qHV0 = (qH0 ² + qV0 ² ) ^1/2 .
Further, the horizontal acceleration kH0 of the viewpoint in the visual image a on the frame screen F ₀ is expressed as follows:
kH0≈d (qH) / dt≈ (qH1−qH0) / (1 / S).
Similarly, the vertical acceleration kV0 is
kV0≈d (qV) / dt≈ (qV1−qV0) / (1 / S).
Further, the viewpoint moving acceleration kHV0 on the frame screen F ₀ is set as follows:
It is calculated as kHV0 = (kH0 ² + kV0 ² ) ^1/2 .

Further, the analysis unit 102 calculates a viewpoint movement including a head movement. FIG. 9 is a diagram illustrating the movement of the viewpoint when moving from the frame F ₀ to the frame F ₀ +1. Assuming that the horizontal movement distance combining the head and the viewpoint is Dh, in FIG. 9, the analysis unit 102 calculates the horizontal movement distance combining the head and the viewpoint in the frame F ₀ as
Calculated as Dh0 = ph0 + pH0.
Similarly, when the vertical movement distance of the head and the viewpoint is Dv, the vertical movement distance of the head and the viewpoint in the frame F ₀ is
Calculate as Dv0 = pv0 + pV0.
Therefore, the moving distance D0 between the head and the viewpoint from F ₀ to F ₀ +1 is
Calculated as D0 = (Dh0 ² + Dv0 ² ) ^1/2 .
Also, the total movement distance TD between the total head and viewpoint from the frame screens F ₀ to Fn is
Calculated as TD = D0 + D1 +... + Dn.
Further, the moving speed of the viewpoint including the head in the frame F ₀ is calculated as follows.
Horizontal speed of viewpoint movement: Qh0 = qh0 + qH0
Vertical speed of viewpoint movement: Qv0 = qv0 + qV0
Therefore, the viewpoint movement speed Q0 in the frame F ₀ is
Q0 = (Qh0 ² + Qv0 ² ) ^1/2 is calculated.
Similarly, the movement acceleration of the viewpoint including the head in the frame F ₀ is calculated as follows.
Horizontal acceleration: Kh0 = kh0 + kH0
Vertical acceleration: Kv0 = kv0 + KV0
Therefore, acceleration K0 of viewpoint movement, including the head of the frame F ₀ is,
Calculated as K0 = (Kh0 ² + Kv0 ² ) ^1/2 .

  Further, the analysis unit 102 calculates the movement of the object in the visual image a by the same method as the movement of the viewpoint.

  Using the above information, the analysis unit 102 analyzes whether or not the viewpoint is tracking the object. FIG. 10 is a diagram illustrating a positional relationship between the object and the viewpoint in the visual image a. In FIG. 10, when the viewpoint B is included in the video of the object C in the visual image a, it is determined that the viewer's viewpoint tracks the target object. Note that the contour of the object can be specified by image processing and can be dealt with by a known technique, and thus detailed description thereof is omitted here.

  The analysis unit 102 also determines the distance between the position of the viewpoint B (H, V) and the position of the object C (X, Y), the approach speed (Bv) when the viewpoint approaches the object, and the approach acceleration (Bk). ), A separation speed (Bs) and a separation acceleration (Br) when the viewpoint is separated from the object are calculated. In addition, when the target object C deviates from the visual image a or when the predetermined condition such that the viewpoint B is separated from the target object C and a predetermined time elapses is satisfied, it is clearly determined that the viewpoint B has left the target object. .

FIG. 11 is a flowchart illustrating a method of calculating the approach speed (Bv), the approach acceleration (Bk), the separation speed (Bs), and the separation acceleration (Br). A method of calculating the approach speed (Bv), the approach acceleration (Bk), the separation speed (Bs), and the separation acceleration (Br) will be described with reference to FIG. First, in step 101, the analysis unit 102 determines whether or not the viewpoint is within the range of the object. Whether or not the object is within the range is obtained from the distance between the position of the viewpoint B (H, V) and the position of the object C (X, Y). First, the horizontal deviation distance Zh0 in the frame F ₀ is
Calculated as Zh0 = X0−H0.
Similarly, the vertical deviation distance Wv0 is
Calculate as Wv0 = Y0−V0.
Also, the divergence distance U0 is
^{U0 = (Z h0 2 + Wv0} 2) is calculated as ^1/2.
The deviation distance in the frame F ₀ U0, the divergence distance in frames F ₀ +1 and U1.

When the position (H, V) of the viewpoint B is not included in the range of the position (X, Y) of the object C (step 101: N), that is, when the viewpoint B exists outside the object C, The following processes A to C are performed.
A. In step 103, when U1-U0 <0, it is determined that the viewpoint B is approaching the object C, the process proceeds to step 104, and the approach speed and acceleration are calculated.
Assuming that the approach speed of the viewpoint B to the object C is Bv, the approach speed (Bv 0 ) of the viewpoint B to the object C in the frame F ₀ is as follows.
Bv0 =-(U1-U0) / (1 / S)
Further, the approach acceleration (Bk) of the viewpoint B to the object C is as follows.
Bk0 =-(Bv1-Bv0) / (1 / S)
Then, the process returns to step 101.
B. In step 103, when U1-U0> 0, it is determined that the viewpoint B is being detached from the object C, the process proceeds to step 105, and the separation speed and acceleration are calculated.
In the frame F ₀ , the departure speed (Bs) of the viewpoint B from the object C is as follows.
Bs0 = (U1-U0) / (1 / S)
Further, the separation acceleration (Br) of the viewpoint B from the object C is as follows.
Br0 = (Bs1-Bs0) / (1 / S)
Then, the process returns to step 101.
C. In step 103, when U1-U0 = 0, there is no movement of the viewpoint at that time.
Therefore, the same operation is repeated until the subsequent U1-U0 result becomes plus or minus. Therefore, the process returns to step 101.

  When the position (H, V) of the viewpoint B is included in the range of the position (X, Y) of the object C (step 101: Y), that is, when the viewpoint B exists inside the object C, the viewpoint Is determined to be tracking the object, the process proceeds to step 102, the number of frames being tracked (N) is counted, and, as will be described later with reference to FIG. 12, viewpoint tracking time (Bt) = ( (1 / S) × N)} is calculated.

  Further, the analysis unit 102 calculates the viewpoint tracking time. FIG. 12 is a flowchart illustrating a method for calculating the viewpoint tracking time. The viewpoint tracking time calculation method shown in FIG. 12 corresponds to step 102 shown in FIG. For each frame of each visual image a, first, in step 201, the analysis unit 102 determines whether or not the viewpoint is within the range of the target object. The method for determining whether or not the viewpoint is within the range of the object is the same as the method described in FIG. If the viewpoint is within the range of the object, the process proceeds to step 202 and the time is counted. Next, in step 203, it is determined whether or not the viewpoint is within the range of the object. If the viewpoint is within the range, the process returns to step 202 and the time is counted. On the other hand, if the viewpoint is not within the range of the object, the process proceeds to step 204, the counted time is set as the viewpoint tracking time, and the process returns to step 101 in FIG.

In case the physiological response data of the change in pupil diameter, the pupil diameter is affected by the brightness of the visual target (brightness and illumination, etc.). Therefore, it is necessary to erase the pupil diameter change portion corresponding to the brightness of the visual target (brightness and illumination, etc.). When the visual target is “video display” , the brightness of the display is slowly changed from lowest (black screen) to highest (white screen), conversely from highest to lowest, and from lowest to highest. The viewer's pupil diameter during the last movement is measured. This is referred to as “basic luminance and pupil diameter relationship data”. The reason for adopting the measured pupil diameter for the last change in brightness is that the viewer will be interested in the "changes in the beginning" and will be interested in the changes in brightness. This is because the pupil diameter due to the influence is included. It is assumed that the viewer evaluates the actual evaluation content with the room environment and the display in which the “basic luminance and pupil diameter relationship data” is measured. When measuring the actual evaluation content on a display different from the display on which the viewer measured the “basic brightness and pupil diameter relationship data”, the display that measures the actual content is displayed as “basic brightness and pupil diameter relationship data. It is assumed that the display that measures the actual content is adjusted so that it has the same brightness and hue as the same room environment as the display that measured “”. During actual content viewing, when the pupil diameter (Pt) of the viewer at a certain moment (t) is measured, the luminance (Yt) of the display is simultaneously measured using a luminance meter or the like. The analysis unit 102 calculates “relationship data between basic luminance and pupil diameter” corresponding to the luminance (Yt) when the object 2 is visually recognized from the pupil diameter (Pt) when the object 2 (in this case, the video display) is visually recognized. By subtracting the pupil diameter (Pbt) at, the pupil diameter due to the brightness of the display can be erased, and the pupil diameter (Pmt) corresponding to only the degree of attention of the viewer to the object at that moment (t) can be obtained. . Ie

In this case, when Pmt <0, the degree of attention is inferior to that in normal times, which means that the user is in a bored state.

The pupil between the average value of the pupil diameter (Pmax) when the video display has the lowest luminance and the average value of the pupil diameter (Pmin) when the video display has the highest luminance in the “relation data between basic luminance and pupil diameter”. When the diameter change range is PHPn, it is as follows.
PHPn = Pmax−Pmin
Therefore, the attention level (It) at a certain moment (t) is

Can be obtained as By doing so, it is possible to eliminate a change in pupil diameter due to individual differences among viewers, and to use a common measure of attention. If the pupil diameter is very small, Pmt <0.
It <0
It can be said that the degree of attention is lower than usual and the user is bored. Conversely, It> 1
In the case of, it can be said that it is a state of extreme attention. It≈0 is a case where the content is normal and low in stimulation, and this degree of attention is Ih. In actual measurement, the luminance (Yt) and the pupil diameter (Pt) of the viewer are measured simultaneously with the passage of time on the display of content that changes with time, and the degree of attention is calculated and analyzed to determine the degree of attention. In actual measurement, only the luminance (Yt) of the display showing the content is measured over time, and then the display of the display of the content is measured only by the pupil diameter (Pt) over time. At the time of analysis, the attention level (It) may be calculated by matching the luminance (Yt) of the video content with the time (t) of the pupil diameter (Pt).

As mentioned above, when the object to be viewed is a video display and the brightness of the viewpoint is not negligible compared to the brightness of the entire display (such as content with a sharp contrast in the screen, ), Since the luminance of the viewpoint also affects the pupil diameter, it is necessary to eliminate the pupil diameter change of the viewer corresponding to the luminance of the viewpoint. The luminance Yj of the entire screen at the scene of the frame j can be measured as described above . The influence on the luminance of red (R), green (G), and blue (B) of each pixel of the display differs depending on the manufacturer, but the influence (Cr) of green (G) on the influence of red (R) on the luminance. influence of the luminance (Cg), a blue degree of influence on the brightness of the (B) (Cb), R pixel (x) when the frame j, G, as the intensity of each of the B, of red (R) If the intensity (Srjx) , the intensity of green (G) (Sgjx) , and the intensity of blue (B) (Sbjx), the provisional luminance of the pixel (x) at frame j is defined as Bjx,
BJX = a (Srj x × Cr + Sgj x × Cg + Sbjx × Cb), the ratio of the actual luminance and the provisional luminance and e, when the actual luminance of the pixel (x) when the frame j and yjx, less become that way.
Yjx = e × Bjx
The luminance (Yj) of the entire screen of frame j is T

So

That is

It becomes. Since the luminance measurement value of the display screen at the time of actual content viewing is Yj , the average luminance (Yjh) per pixel is as follows.

The luminance ratio (Wjx) as compared with the luminance (Yjx) of the pixel x as the viewpoint and the screen average luminance (Yjh) is as follows.

That is , the luminance Yjx of the pixel x is as follows.
Yjx = Yjh × Wjx = e × Bjx
Thereby, the luminance of a certain pixel can be calculated. Since there is already a measuring device that measures the luminance of a specific pixel, the luminance of each pixel may be obtained by the measuring device. Next, the total luminance of pixels within a diameter of 17 cm centered on a certain viewpoint pixel x is Ydx. The luminance Yd of the entire display is changed stepwise, and the luminance (Ydx) of a pixel within a diameter of 17 cm centered on the pixel x from the maximum to the minimum, and further from the minimum to the maximum, for each luminance of the entire display at a certain stage. In addition, the pupil diameter is changed from the maximum to the minimum three times and the pupil diameter at the time of the last movement is measured. Since the first one pays attention to this movement, measurement is not performed because the pupil diameter due to attention is included. This table is referred to as “basic relationship data of display luminance, viewpoint luminance, and pupil diameter”. The viewer's pupil diameter (Pg) at the time of actual content viewing is measured, and the brightness (Yg) of the display at that time and the brightness (Ygx) with a diameter of 17 cm centered on the viewpoint x are measured and analyzed. (Yg) and the pupil diameter (Pgx) in the “relationship data between basic display luminance and viewpoint luminance” corresponding to the luminance (Ygx) within a diameter of 17 cm centered on the pixel x with the viewpoint as the pupil diameter of the viewer By subtracting from (Pg), the pupil diameter (Pcx) corresponding to the degree of attention at the viewing viewpoint in consideration of the influence of the luminance of the display can be obtained.

It is assumed that an intermediate value between the luminance (Yg) of the entire screen at the time of actual content viewing and the value of “related data between basic display luminance and viewpoint luminance” is obtained by an interpolation method. If the pupil diameter change range between the average pupil diameter (Pmax) when the video display has the lowest brightness and the average pupil diameter (Pmin) when the video display has the highest brightness is PHPn ,
PHPn = Pmax−Pmin
Therefore, the degree of influence on the pupil diameter by the luminance per pixel with a radius of 1.7 cm, where Sg is the number of pixels with a radius of 1.7 cm centered on the viewpoint pixel x, is as follows.

Further, assuming that the pixels of the entire screen are T, the change range of the average pupil diameter to the pupil diameter per pixel in the entire screen is as follows.

Therefore, the attention level (Ijx) of the viewpoint (x) of a certain frame (j) is

Can be obtained as In this way, the change in pupil diameter due to individual differences among viewers is absorbed, and the degree of attention of the viewpoint in the screen in consideration of the brightness of the entire display is obtained, which can be a common scale for viewers. Since there is already a measuring device that measures the luminance for each pixel, the luminance of each pixel may be obtained by the measuring device. Since Ijx corresponds to the above-mentioned It, it will be treated in the same manner thereafter. Therefore, hereinafter, It is (1) When the display is visual target, the attention It by the luminance of displays described "relationship data of the basic luminance and pupil diameter" above, also a "display luminance and the viewpoint luminance The degree of attention Ijx is adopted when the luminance of the viewpoint of the display described in “Basic relationship data of pupil diameter” is adopted, and (2) the above-mentioned “basic luminance and pupil diameter” when other than the display is set as a visual recognition target. It is assumed that the luminance of “related data” is changed to illuminance and It is adopted.

  The analysis unit 102 also analyzes the pupil diameter (P) and the number of blinks from the eye movement image b. Further, the analysis unit 102 calculates the attention level (It), the attention level expansion speed (Its), and the attention level expansion acceleration (Itk) from the pupil diameter (P) data. When Its and Itk are negative (−), the attention level reduction speed and the attention level reduction acceleration are indicated, respectively.

  As described above, the analysis unit 102 analyzes various values, and outputs the analyzed values as the analysis value A to the storage unit 103 and the diagnosis unit 104 (see FIG. 2). Therefore, the values included in the analysis value A include the attention level (It), the attention level change speed (Its), the attention level expansion acceleration (Itk), the viewpoint tracking time (Bt), the viewpoint approach speed (Bv), and the viewpoint approach. Other emotion values and excitement levels include acceleration (Bk), viewpoint departure speed (Bs), and viewpoint departure acceleration (Br).

[Storage unit]
Referring to FIG. 2 again, the storage unit 103 includes a visual image a, an eye movement video b, audio data c, response data d, body motion video e, body temperature data f, and brightness output from the analysis unit 102. Accumulate data g . Further, the storage unit 103 stores the analysis value A output from the analysis unit 102. Further, the storage unit 103 is the degree of attention (Ih), the degree of attention expansion (Ihv), and the degree of attention expansion when the viewer 1 is watching the object 2 with low interest, that is, mediocre and low stimulus. Acceleration (Ihk), viewpoint tracking time (Bht), viewpoint approach speed (Bhv), viewpoint approach acceleration (Bhk), viewpoint departure speed (Bhs), and viewpoint departure acceleration (Bhr) are stored as normal values H in advance. . The accumulation unit 103 also draws attention (Ie), attention expansion speed (Iev), attention expansion acceleration (Iek), and viewpoint tracking time when the viewer 1 is looking at the object 2 of high interest. (Bet), viewpoint approach speed (Bev), viewpoint approach acceleration (Bek), viewpoint departure speed (Bes), and viewpoint departure acceleration (Ber) are stored in advance as moving values K. In addition, the storage unit 103 stores the diagnosis result output from the diagnosis unit 104.

[Diagnostic Department]
The diagnosis unit 104 diagnoses the interest of the viewer 1 in the object 2 by comparing the analysis value A output from the analysis unit 102 with the normal value H or the emotional value K stored in the storage unit 103. The ability of the object 2 visually recognized by the viewer 1 or the viewer 1 itself is evaluated. That is, the diagnosis unit 104 sets the parameters included in the analysis value A as a, b, c,... And sets the weights for the parameters as A, B, C,. An evaluation value E (t) for the visual control is calculated.
E (t) = A · a (t) + B · b (t) + C · c (t) +.

  Specifically, the diagnosis unit 104 diagnoses that the object 2 is interested when the number of blinks within a predetermined time is less than the predetermined number. Furthermore, when the degree of attention is greater than a predetermined value, the diagnosis unit 104 diagnoses that the object 2 is interested. In addition to the above, the diagnosis unit 104 diagnoses that the object 2 is interested when the enlargement speed and the enlargement acceleration of the attention degree are larger than the predetermined value, and conversely, when the attention degree is smaller than the predetermined value, When the reduction speed and the reduction acceleration of the attention level are larger than the predetermined values, it is diagnosed that the object 2 is not interested. Further, when the viewpoint approach speed and the viewpoint approach acceleration are larger than the predetermined values, the diagnosis unit 104 diagnoses that the object 2 is interested, and conversely, when the viewpoint approach speed and the viewpoint approach acceleration are lower than the predetermined values. Diagnose that the object 2 is not interested. If the viewpoint leaving speed and the viewpoint leaving acceleration are larger than the predetermined values, it is diagnosed that the object 2 is not interested or disgust, and conversely, the viewpoint leaving speed and the viewpoint leaving acceleration are less than the predetermined values. Then, it is diagnosed that the object 2 is interested.

  Furthermore, the diagnosis unit 104 calculates the viewpoint moving speed and acceleration by combining the approaching moving speed and approach acceleration of the head direction with respect to the object 2, the viewpoint approaching speed, and the viewpoint approaching acceleration. Determination similar to “0075” is performed. If the viewer 1 is interested, the head also keeps its direction in order for the viewpoint to track the object. Although the head may continue to be directed toward the object even if not interested, in this case, the degree of attention is reduced. Further, when the head keeps moving away from the object 2 and the attention level is reduced, it is diagnosed that there is no interest.

In addition, the diagnosis unit 104, when the viewpoint of the viewer 1 on the target object 2 is tracked for a time longer than the normal value (Bht), and the attention level becomes the moving value (Ie), Diagnose that you are interested in the object. That is,
Bt> Bht
It> Ih × 1.3 or It = Ie
In this case, the diagnosis unit 104 diagnoses that the viewer 1 is interested in the object 2. In this case, when the object 2 is moving, the viewer tracks while the object 2 is visible. On the other hand, in the case of the object 2 that does not move, a time longer than the normal value (Bht) is tracked. Also, the diagnosis unit 104 once tracks the viewpoint within the range of the object 2 for a short time {<Bht}, and the degree of attention does not expand to the moving value (Ie). When the user leaves the object 2 more rapidly than the normal value (Bhs) {> Bhs} and the viewpoint does not return to the object 2 again, the viewer 1 diagnoses that the object 2 is not interested. Furthermore, the diagnostic unit 104 diagnoses that the viewer 1 feels disgust when the viewer 1 once sees the object 2 and the degree of attention expands, but the viewpoint leaves the object 2 within a predetermined time. To do. That is, when there is a viewpoint in the object,
It> Ih or It = Ie
At this time, the viewpoint leaving speed (Bs) and the viewpoint leaving acceleration (Br) from the object are larger than the normal values (Bhs, Bhr), that is, Bs> Bhs.
And
Br <Bhr
If it is, it is diagnosed that the object 2 is disgusted. In addition, in the statistics of 1000 or more cases, when the viewpoint tracking time (Bt) for the target object is Bt <0.2 seconds, it can be diagnosed that there is clearly disgust.

  Further, the diagnosis unit 104 diagnoses that the viewer 1 is excited about the object 2 when the body temperature rises using the body temperature data f as compared to the body temperature measured in advance on the day. To do. Further, the diagnosis unit 104 evaluates the capabilities of the object 2 and the viewer 1 based on the number of responses, the speed of the response operation, and the accuracy using the response data d by the operation of the response device 14. A specific method of using the response data d will be described in detail later. Further, the diagnosis unit 104 outputs the diagnosis result to the display unit 105 and the storage unit 103.

  FIG. 13 shows the viewer face image data e from the viewer motion photographing device 9. Characteristic points of each part (eye, lips, eyebrows, etc.) of the face of the viewer are marked by a face analysis method to analyze the facial expression. Although some of these feature points are shown in the figure for explanation, they are based on an actual face analysis method. Typical emotion judgments are surprise, fear, disgust, anger, joy (happiness), and sadness. Preliminarily measure and place the maximum value of each part of the face and the normal movement of the viewer, and determine the degree of emotion (F) based on the ratio to the maximum value of the amount of change at each measurement To do. Thereby, together with the emotion value (E) of the degree of attention analyzed from the change in pupil diameter or blink frequency, the viewer's feeling (EF value) for the visual target can be determined.

[Display section]
The display unit 105 displays the diagnosis result in the diagnosis unit 104 on an output device such as a display. From the diagnosis result displayed on the display unit 105, the operator can determine the capabilities of the object 2 and the viewer 1 themselves.

  As described above, according to the viewer feeling determination apparatus 100 according to the embodiment of the present invention, the object 2 or the viewer 1 himself / herself is used by using the change in physiological response data, the accompanying acceleration, and the change in the viewer's face. Evaluate ability. Examples of changes in physiological response data include attention (It), its enlarged acceleration (Itk), viewpoint approach acceleration (Bk), and viewpoint departure acceleration (Br). As the degree of attention and its enlargement speed and acceleration (Itk) increase, or as the viewpoint approach speed and acceleration (Bk) increase, it is diagnosed that the object 2 is more interested. Conversely, the degree of attention (It) is It is diagnosed that the object 2 is not interested as the negative acceleration (−Itk) increases or the viewpoint departure acceleration (Br) increases. In addition, the change of the emotion of the viewer is determined by analyzing the partial change of the face. As described above, according to the viewer feeling determination device 100 according to the embodiment of the present invention, the physiological reaction of the viewer can be grasped more accurately, and the ability of the object or the viewer himself / herself can be more accurately evaluated in various situations. can do. The application scene of the viewer feeling determination apparatus 100 according to the embodiment of the present invention will be described in detail later.

  The present invention has been described with reference to the embodiment. However, the present invention is not limited to the above-described embodiment, and various modifications can be made without departing from the technical idea thereof. For example, in the above-described embodiment, the body motion imaging device 9 and the body temperature measurement device 10 directly transmit data to the viewer feeling determination device 100, but once stored in the storage device 8, together with the visual image a and the like. Then, it may be transmitted from the transmission device 7 to the viewer feeling determination device 100. The eye camera 3 is configured to include the eyeball photographing device 4, the visual scene photographing device 5 and the microphone 6. However, the eyeball photographing device 4, the visual scene photographing device 5 and the microphone 6 are not necessarily physically connected. It is not necessary, and each may be provided as a single unit.

  In the embodiment, the corneal reflection method is used when calculating the viewpoint position. However, the method for calculating the viewpoint position is not limited to this, and various known methods can be used.

  The viewer feeling determination apparatus 100 is configured by a computer including a volatile storage medium such as a CPU and a RAM, a non-volatile storage medium such as a ROM, an interface, and the like. The functions of the receiving unit 101, the analyzing unit 102, the accumulating unit 103, the diagnosis unit 104, and the display unit 105 provided in the viewer feeling determination device 100 are realized by causing the CPU to execute a program describing these functions. The These programs can also be stored and distributed in a storage medium such as a magnetic disk (floppy (registered trademark) disk, hard disk, etc.), optical disk (CD-ROM, DVD, etc.), semiconductor memory, or the like.

  Next, a scene where the viewer feeling determination device 100 according to the embodiment of the present invention is applied will be described. Specifically, purchasing motivation diagnosis, learning motivation diagnosis, audience emotion reaction degree diagnosis with respect to the expected emotional reaction degree of video content creator, content evaluation based on distribution of attention points in content, compatibility diagnosis, baby who cannot speak language Diagnosis of communication with people with dementia, animals, etc., determination of robot response methods, criminal evidence diagnosis, driver drowsiness detection diagnosis, car driving aptitude diagnosis, occupational safety proficiency diagnosis, pre-work appropriateness diagnosis, athlete appropriateness diagnosis, game Excite degree diagnosis, spatial design diagnosis, respondent reception degree diagnosis, video content selection priority diagnosis, and automatic mark of an object or person with high interest / interest in the content will be described.

[Purchase willingness diagnosis]
In the purchase motivation degree diagnosis, a general viewer 1 looks at a certain product, and a diagnosis is performed on whether or not the viewer 1 has or does not have a purchase intention. The visual recognition method may be a method of viewing an actual product or a product displayed on a display such as an advertisement, a television, a personal computer, or a mobile terminal (including a mobile phone). For the purchasing power diagnosis, the viewer feeling determination device 100 shown in FIGS. 1 and 2 is used. Here, the object 2 is an actual product or an image of the product. The diagnosis unit 104 is a. Looking at the product, the pupil has not expanded to the emotional value (Pe) {<Pe} (indifference), b. When the viewpoint tracking time (Bt) is shorter than the normal value (Bht) (not interested), the viewer 1 diagnoses that the product has no willingness to purchase. Further, when the viewpoint withdrawal speed (Bs) and the first withdrawal acceleration (Br) from the product are larger than the normal values {Bhs, Bhr}, the aversion (DI) by the analysis of the emotion value (F) by the face analysis method is calculated. The aversion can be determined by determining. In this case, it can be said that this is clearly the opposite of the case where there is a willingness to purchase. From this, it is possible to measure the degree of diagnosis of “no purchase intention”. On the other hand, when the degree of attention (It) is expanding and the viewpoint tracking time (Bt) to the product is longer than the normal value (Bht) and reaches the impression value (Bet), it can be diagnosed as being interested. It cannot be diagnosed until there is a willingness to purchase. That is, when there is no interest, there is no willingness to purchase, but there is a case where there is no willingness to purchase even if there is interest. Therefore, the emotion determination is combined with the face analysis method. Generally, when the emotional value (F) is high in happiness (joy) (HP) and surprise (SP), it can be determined that the purchase is positive. However, when sadness (SD), anger (AG), fear (FR), and disgust (DI) of other emotion values (F) are high, it can be generally determined that the purchase is reluctant. Next, have the viewer 1 visually recognize a plurality of products, look at a specific product, A lot of attention. Viewpoint tracking time is long, c. If the viewpoint withdrawal speed and the initial acceleration are small, it can be determined that the specific product is most interesting. Furthermore, when the user feels happiness (joy) by the face analysis method, it can be determined that the specific product has a positive feeling, so the diagnosis unit 104 is more likely to purchase than other products. Is diagnosed as high. On the other hand, certain products are a. The degree of attention is small. Viewpoint dwell time is short, c. If the viewpoint withdrawal speed and the initial acceleration are large and the face analysis method does not give a sense of happiness (joy), the positive response will be small. Diagnose as smaller than the product.

[Learning motivation diagnosis]
In the learning motivation level diagnosis, the learning motivation level is diagnosed by setting the learning object (teacher, textbook, teaching material, blackboard, etc.) as the object 2. The viewer's feeling determination apparatus 100 shown in FIGS. 1 and 2 is used for the learning motivation degree diagnosis. Here, when the viewer 1 is looking at a learning target (teacher, textbook, teaching material, blackboard, etc.),
A. The degree of attention (It) has not expanded to the impression value (Ie) (indifference); When the viewpoint tracking time (Bt) is shorter than the normal value (Bht) (not interested), or when disgust is shown by the face analysis method, it can be determined that the learning object is disliked. From this, the diagnosis unit 104 diagnoses “no learning motivation”. On the other hand, even if the degree of attention (It) is expanding and the viewpoint tracking time (Bt) for the learning object is longer than the normal value (Bht), there is interest but interest is not necessarily linked to learning motivation. Therefore, when the interest shows a feeling of happiness (joy) by the emotion determination of the face analysis method, it can be determined that the visual recognition target has a positive emotion, so that it can be determined that the learning motivation level is high. In general, when the content of learning invites emotions of each purpose, the happiness (joy) degree (HP) and surprise (SP), sadness (SD), anger (AG), fear ( FR) and disgust (DI) are high, it can be determined that the learning is good. However, if you are working on a problem, if your sadness (SD), anger (AG), fear (FR), and disgust (DI) are high, the content of your study may be aimed at eliciting the emotions described above. Except for this, it can be determined that something is confusing and there is a problem with this learning. Next, a plurality of learning objects are visually recognized by the viewer 1 and a specific learning object is viewed. A lot of attention. Viewpoint tracking time is long, c. If the viewpoint withdrawal speed and the initial acceleration are small, it can be determined that the learning target is of interest. Furthermore, when the face analysis method shows a sense of happiness (joy), since it can be determined that the learning object has a positive emotion, the diagnosis unit 104 determines that the learning object is more than the other learning objects. Diagnose with high motivation to learn. On the other hand, a specific learning object is a. The degree of attention is small. The viewpoint tracking time is short. If the viewpoint withdrawal speed and the initial acceleration are large and the face analysis method does not show a sense of happiness (joy), the positive response will be small, so the diagnosis unit 104 learns the learning target. It is diagnosed that the degree of motivation is low compared to other learning subjects.

[Evaluation of viewer emotion response to expected emotion response of video content creator]
The viewer emotion reaction degree diagnosis with respect to the expected emotion reaction degree of the video content creator uses the viewer feeling determination apparatus 100 shown in FIGS. 1 and 2. When viewer 1 is watching video content on a television, a movie, etc., the happiness (joy) of the attention level (emotional level E) and emotion level (F) of viewer 1's interest ) Degree (HP), surprise degree (SP), sadness degree (SD), anger degree (AG), fear degree (FR), aversion level (DI) and excitement degree with respect to the level expected by the video content creator It is characterized by diagnosing whether it is reacting to a certain extent, and evaluating and judging video content. For all content, content creators want to be moved and cry in this scene, want to get angry here, and want to laugh here. Therefore, the producer's expected emotional response is set in the content software program. Specifically, the emotion value (EF) expected by the author on the scene (time) of the video content is set with the maximum value of emotion value (E) set to 100, and the expected value of emotion value (F) expected The maximum value is set to 100, and the happiness (joy) level (HP), surprise level (SP), sadness level (SD), anger level (AG), fear level (FR), aversion level (DI) of each basic emotion ) and to set the expectation level of excitement level. By measuring and analyzing the emotional response by the emotional analysis of the viewer and the emotional analysis by the face analysis during actual projection, how much the content reacts to the expected emotional response and emotional response expected by the producer You can judge whether you have done and evaluate the work. Note that the target content includes sound at the same time as the video, but there may be a case where only the sound is present without the video. Therefore, it can be determined that this content is a work as expected and can be highly evaluated. Another effective use is CM viewer response evaluation. In this case, it is determined that a CM with a high degree of attention to appeal points in the appeal scene and a low disgust sensitivity and low anger level is a good CM.

[ Content evaluation by distribution of attention points in content ]
In the viewer feeling determination apparatus for a viewing scene, the specific object is a content such as a television, a specific screen of a video, a still image, a poster, or a magazine page, and the viewer is the content viewer. is there. In a specific scene or still image of content, the attention level distribution of attention points of a plurality of viewers is displayed to determine the appeal point and level of the content producer. Specifically, for specific content such as video screens, still images, posters, and one page of a magazine, as shown in FIG. 14, one screen is divided into 4 × 4 = 16, and the position and attention of multiple people The average value of the attention level according to the degree is displayed in the screen, and at the same time, as shown in FIG. It is characterized by evaluating whether or not.

[Compatibility diagnosis]
In the compatibility diagnosis, as shown in FIG. 16, in a situation where a telephone call function (including a mobile phone) is used while making a call while looking at each other's face through a communication line, a compatibility diagnosis between the other parties is performed. For the compatibility diagnosis, the viewer feeling determination device 100 shown in FIGS. 16 and 17 is used. FIG. 16 is a block diagram showing a configuration of a viewer feeling determination system used for compatibility diagnosis, and FIG. 17 is a block diagram showing a configuration of a viewer feeling determination device 100 used for compatibility diagnosis. Here, the object 2 is a face image displayed on the displays 22a and 22b. First, the video phone 20a used by Mr. A and the video phone 20b used by Mr. B are connected to the viewer feeling determination apparatus 100 via the communication line 30, respectively. In addition, the storage unit 103 of the viewer feeling determination apparatus 100 shown in FIG. 17 measures and stores the personal attribute data of the viewers (here, Mr. A and Mr. B) in advance. The personal attribute data includes the normal values (“attention level”, “attention level”) when viewing an object that is mediocre and has low irritation, in addition to the age, sex, birthplace, etc. of the viewer (Mr. A, Mr. B, etc.) Magnification Speed, ”Attention Magnification Acceleration”, “Viewpoint Tracking Time”, “Viewpoint Approach Speed”, “Viewpoint Approach Acceleration”, “Viewpoint Exit Speed”, and “Viewpoint Exit Acceleration”) Impression value ("attention level", "attention level expansion speed", "attention level expansion acceleration", "start point position", "viewpoint tracking time", "viewpoint approach speed") “Viewpoint approach acceleration”, “viewpoint departure speed”, “viewpoint departure acceleration”). In addition, in order to determine the emotion of the face analysis technique, values (maximum values) when the viewer's normal time and each part of the face move to the maximum are measured and stored in advance. Furthermore, the accumulation unit 103 accumulates average values such as the normal value and the emotional value for each layer such as human age, sex, and occupation.

  The video phones 20a and 20b include face cameras 21a and 21b, displays 22a and 22b, eyeball cameras 23a and 23b, and tracking cameras 24a and 24b, respectively. The face cameras 21a and 21b capture the caller's face. The displays 22a and 22b display communication partners. The eyeball cameras 23a and 23b photograph the caller's eyeball. The tracking cameras 24a and 24b are provided on both sides of the eyeball cameras 23a and 23b, and measure the position of the caller's eyeball using the principle of triangulation. The eyeball cameras 23a and 23b track the caller's eyeball according to the measurement results obtained by the tracking cameras 24a and 24b. Note that the eyeball cameras 23a and 23b are omitted, the face cameras 21a and 21b are made highly detailed, and the analysis unit 102 in the viewer feeling determination apparatus 100 enlarges the face image of the caller and performs image analysis. A caller's eyeball image may be captured. Further, the eyeball cameras 23a and 23b are omitted, the face cameras 21a and 21b are made highly detailed, and the analysis unit 102 in the viewer feeling determination apparatus 100 enlarges the caller's face video and performs image analysis. When the caller's eyeball image is captured, other means for measuring the distance from the video phone 20a, 20b to the caller's face may be used instead of the tracking cameras 24a, 24b. In this way, the eyeball is captured and the distance is known, so that the pupil diameter can be measured.

  In the video phone 20a used by Mr. A, the face camera 21a photographs the face of Mr. A, and the eyeball camera 23a photographs the eyeball of Mr. A. Then, the video phone 20a transmits Mr. A's face data and Mr. A's eyeball data (gaze direction, pupil image, distance from the camera to the eyeball) to the viewer feeling determination apparatus 100 via the communication line 30. . Similarly, in the videophone 20b used by Mr. B, Mr. B's face is photographed by the face camera 21b, and Mr. B's eyeball is photographed by the eyeball camera 23b. Then, the video phone 20b transmits Mr. B's face data and Mr. B's eyeball data (line-of-sight direction, pupil image, distance from the camera to the eyeball) to the viewer feeling determination apparatus 100 via the communication line 30. .

  The receiving unit 101 of the viewer feeling determination apparatus 100 in FIG. 17 receives the face data and eyeball data (line-of-sight direction, pupil image, distance from the camera to the eyeball) of Mr. A and Mr. B. The analysis unit 102 of the viewer feeling determination apparatus 100 determines the viewpoint position and viewpoint movement speed (approach speed) in the image of Mr. B's face on the display 22a owned by Mr. A from the line-of-sight direction data of Mr. A's eyeball data. Bv, separation speed Bs) and acceleration (approach acceleration Bk, separation acceleration Br) are calculated. Further, the attention degree is calculated from the eyeball data of Mr. A from the distance from the eyeball camera 23a to the eyeball and the pupil image, and at the same time, the enlargement speed (Its), the enlargement acceleration (Itk) and the reduction acceleration (−) of the attention degree (It). Itk) is calculated. On the other hand, from the line-of-sight data of Mr. B's eyeball data, the position of the viewpoint, the viewpoint movement speed (approaching speed Bv, separation speed Bs) and acceleration (approaching acceleration) in Mr. A's face image shown on Mr. B's display 22b. Bk, separation acceleration Br) is calculated. Further, the attention level is calculated from the eyeball data of Mr. B from the distance from the eyeball camera 23b to the eyeball and the pupil image, and at the same time, the expansion speed (Its) and the expansion acceleration (Itk) of the attention level (It) are calculated. The analysis unit 102 outputs the analyzed value to the diagnosis unit 104. The diagnosis unit 104 tracks the face of Mr. B reflected on Mr. A's display 22a as long as the face of Mr. B is reflected, and the attention level (It) of Mr. A is stored in the storage unit. If it is expanded to the emotional value data (Ie) read from 103, Mr. A diagnoses that Mr. B is interested in Mr. B. Also, by facial analysis, emotions such as surprise, fear, disgust, anger, joy (happiness), sadness, and the like of Mr. A while viewing Mr. B's face are determined. The transmission unit 106 transmits the diagnosis result to the videophone 20b (or mediator) of Mr. B. The videophone 20b (or the intermediary videophone) that has received the diagnosis result displays a symbol (for example, a heart symbol) indicating that “Mr. A is interested / interested in B” on a specific location on the display 22b. To do. Conversely, Mr. B's time to track Mr. A's face reflected on Mr. B's display 22b is shorter than when Mr. A's face appears on Mr. B's videophone display 22b (from Mr. B's face to A 3), the attention level (It) does not increase to the impression value data (Ie) extracted from the storage unit 103, the attention level expansion speed (Its), the initial attention level expansion acceleration ( When Itk) is less than the normal value (Ihv, Ihk), the diagnosis unit 104 diagnoses that “Mr. B has no interest in Ms. A”. Further, when visually recognizing Mr. A's face by face analysis, the emotions such as surprise, fear, disgust, anger, joy (happiness), sadness and the like of Mr. B are judged. The transmission unit 106 transmits the diagnosis result to the videophone 20a (or mediator) of Mr. A. The videophone 20a (or the mediator's videophone) that has received the diagnosis result displays a symbol (for example, a x mark) indicating that {B is not interested or interested in Mr. A} and emotion on the display 22a at a specific location. To display.

  The above-described compatibility diagnosis can be applied to the following situations. (A) Used for marriage arrangements. (B) Used for interview tests. As a result, the true psychological situation of the consultee can be known, and the diagnosis result can be used as a reference for supporting the content of the statement. (C) Used for interviews with criminals. Specifically, the psychological state of the criminal suspect can be known by using it for a scene of interview with a criminal suspect in a criminal case. In these scenes, when the user is interested in the content of the story and the displayed video, the property that the degree of attention (It) is greater than the normal value (Ih) is used.

  The compatibility diagnosis has been described with reference to the block diagrams shown in FIGS. 16 and 17, but the parts that are not particularly described are the same as those in FIGS. 1 and 2 described above. In addition, by using the emotion diagnosis technology based on conversational voice that is already in the world, compatibility diagnosis with higher accuracy can be performed.

[Communication diagnosis with babies who cannot speak language, persons with dementia, animals, etc.]
In the communication diagnosis based on physiological responses of babies, persons with dementia, and animals (including pets), the intentions of babies, persons with dementia, and animals (including pets) are diagnosed. Here, the eye camera 3 is worn with the person 1 and the animal having difficulty in speaking words, such as a baby, a person with dementia, and an animal, as the viewer 1. Alternatively, there is a method in which a baby, an animal, a person with dementia or the like sits in front of a display and photographs the eyes of a baby, an animal, or a visually impaired patient with a camera under the display. The viewer feeling determination apparatus 100 shown in FIGS. 1 and 2 is used for diagnosis of communication of a baby or the like. The analysis unit 102 measures the position and attention level (It) of the viewpoints of babies, animals, persons with dementia, etc., and the diagnosis unit 104 tracks the viewpoint to the visual target, and the attention level (It) is a normal value (Ih ) When larger, diagnose that the visual target is interested in the baby, dementia, animal, etc. As a result, it is possible to grasp the interest in the visual recognition target of the baby, the person with dementia, and the animal who does not speak. On the contrary, the diagnosis unit 104 has a tracking time (Bt) of the viewpoint to the visual recognition target shorter than the normal value (Bht), the visual point departure speed (Bs) and the acceleration (Br) are faster than the normal values (Bhs, Bhr), When the degree of attention (It) is also less than the emotional value (Ie), it is diagnosed that the subject is not interested in and is not interested. This makes it possible to deal with babies, people with dementia, and animals who do not speak. For babies, facial analysis techniques can determine the emotions and levels of baby's surprise, fear, disgust, anger, joy (happiness), sadness, etc., and can be used as a reference for the response.

[Robot response method determination]
In the robot response diagnosis, the eye camera 3, the body motion photographing device 9, the body temperature measurement device 10, and the viewer feeling determination device 100 shown in FIG. The robot constantly measures the face of the person who responds, the baby who does not speak, the person with dementia, the animal, the movement of the eyeball, the body movement, the movement of the face, and the body temperature. In the response diagnosis of the robot, the viewer feeling determination system shown in FIG. 18 is incorporated using the viewer feeling determination system shown in FIG. FIG. 18 is a block diagram showing a configuration of the viewer feeling determination device 100 used for the robot diagnosis. The diagnosis unit 104 of the viewer feeling determination apparatus 100 diagnoses feelings of a person who interacts with the robot, or a baby who does not speak, a person with dementia, or an animal. Based on the diagnosis result input from the diagnosis unit 104, the behavior selection unit 107 selects a behavior corresponding to the diagnosis result from a plurality of pre-stored behaviors. As a result, the robot can grasp the feelings of the person and animal who responds according to the diagnosis result, and can issue appropriate words. Appropriate support actions can also be taken. The robot response diagnosis has been described with reference to the block diagram shown in FIG. 18, but the parts that are not particularly described are the same as those in FIG. 2 described above.

[Criminal evidence diagnosis]
In criminal evidence diagnosis, a criminal suspect is diagnosed by displaying a video image including a crime scene in the middle of an ordinary image on a video display. For the criminal evidence diagnosis, the viewer feeling determination apparatus 100 shown in FIGS. 1 and 2 is used. When a criminal suspect looks at a scene including his / her crime scene, the degree of attention (It) expands more than the other scenes, so the diagnosis unit 104 looks at when the degree of attention (It) increases. It can be used as a reference for identifying the crime scene, assuming that the location of the video that was stored is the crime scene. Furthermore, when the viewpoint tracking time (Bt) to the target image is shorter than the normal value (Bht) and the viewpoint withdrawal speed (Bs) and the initial acceleration (Br) are higher than the normal values (Bhs, Bhr), The diagnosis unit 104 diagnoses that there is disgust in the video of the scene. That is, the criminal suspect wants to avoid the scene as soon as possible. Furthermore, the facial analysis technique can determine the suspect's emotions such as fear, disgust, anger, joy (happiness), sadness, and their degree, and the diagnosis result by the diagnosis unit 104 determines that the place is a criminal place. For reference. In addition, when the suspected crime is intermingled with a video or photograph of the victim with a plurality of persons, the diagnosis unit 104 looks at the video or photograph of the victim of the crime and compares it with other persons. When the degree of attention (It) increases, it is diagnosed that the viewer may be a criminal suspect. In general, since the user does not want to keep watching his / her crime scene and victim, he / she has the same reaction as disgust, and therefore the diagnosis unit 104 has a normal point of view (Bht) as the tracking time (Bt) for the subject. If it is shorter and the speed (Bs) for diverting the viewpoint from the object is greater than the normal value (Bhs), and therefore the acceleration (Bhr) at the moment of diversion is also large, it is diagnosed that the suspect may be a criminal. Furthermore, the facial analysis technique can determine the suspect's emotions and their degree of fear, disgust, anger, joy (happiness), sadness, etc. You can confirm the feelings you want to avoid. In addition, instead of video including crime scenes, by attaching eye camera 3 to suspects of crimes and going to crime scenes, it is possible to diagnose attention to crime objects and facial expressions that are close to disgust, Can be helpful.

[Driver drowsiness detection diagnosis]
In the driver drowsiness detection diagnosis, the driver is put on the eye camera 3 and the eyeball is photographed while driving to diagnose the presence or absence of the driver's drowsiness. In the driver drowsiness detection diagnosis, the viewer feeling determination system shown in FIG. 1 is used, and the viewer feeling determination apparatus 100 shown in FIG. 19 is used. FIG. 19 is a block diagram illustrating a configuration of the viewer feeling determination device 100 used for driver drowsiness detection diagnosis. The diagnostic unit 104 of the viewer feeling determination device 100 diagnoses that the driving state is dangerous when the degree of attention (It) is less than the normal value (Ih). Further, the diagnosis unit 104 diagnoses that the driving direction is further inattentive and dangerous when the direction of the line of sight deviates from the front. By using a camera for tracking and photographing the driver's eyeball in front of the driver's seat of the automobile (above the steering wheel), the same measurement, analysis, and diagnosis as described above can be performed. The warning unit 108 uses the diagnosis result input from the diagnosis unit 104 and warns the driver when it is determined that the diagnosis result is a dangerous driving state. The warning may be an announcement or a warning sound, as long as it gives the driver a stimulus for restoring attention.

[Vehicle driving aptitude diagnosis]
In the car driving aptitude diagnosis, the viewer 1 wears the eye camera 3 and the driver 1 operates the car, an airplane simulator or an actual car, thereby diagnosing the appropriateness of driving and driving. To do. In the automobile driving aptitude diagnosis, a viewer feeling determination device 100 shown in FIGS. 1 and 2 is used. The analysis unit 102 of the viewer feeling determination device 100 analyzes the line-of-sight direction and attention level data of the eye movement data of the viewer 1, and the diagnosis unit 104 tracks the viewpoint to a point to be noted when attention is to be paid. Next, when the degree of attention (It) is larger than the normal value (Ih) and the enlargement speed (Its) and acceleration (Itk) are also larger than the normal values (Ihv, Ihk), the point of caution goes to attention Diagnose that Further, the diagnosis unit 104 diagnoses that the head direction and the line-of-sight direction are always in front. Furthermore, in order to determine whether the response device 14 is responding with the steering wheel or the brake in a timely manner, the analysis unit 102 uses the response data d to determine the response delay time and the operation method. Analyzing, the diagnosis unit 104 determines whether the operation delay time is equal to or shorter than a certain time (predetermined set time) for avoiding danger, and the operation obtained from the response data d is an appropriate operation method set in advance. And the skill level and appropriateness level of the viewer 1 are diagnosed based on the operation delay time and the operation method. Also, put the video of the car that the driver of the selfish behavior is driving into the simulator, let the viewer perform driving operation, calculate the viewer's anger degree and fear degree by face analysis, It is also possible to determine whether driving is appropriate.

[Occupational safety skill diagnosis]
In the occupational safety skill diagnosis, the worker wearing the eye camera 3 diagnoses the occupational safety skill level by operating the equipment with a simulator or an actual machine. In the occupational safety skill level diagnosis, the viewer feeling determination device 100 shown in FIGS. 1 and 2 is used. The analysis unit 102 of the viewer feeling determination apparatus 100 measures the position of the viewpoint, the degree of attention, and the degree of carelessness based on changes in the gaze direction of the worker and the degree of attention (It). In the work, the viewpoint goes to the part to be noted when it should be noted (this becomes the object), the attention level (It) increases to the moving value level (Ie), and the speed of expansion of the attention level (Its) ), When the acceleration (Itk) is also as large as the emotional value level (Iev, Iek), it is diagnosed that the portion to be noted is focused. Furthermore, the analysis unit 102 uses the response data d to analyze whether or not the response device 14 is being operated at an appropriate timing, and the response response delay time and the accuracy of the response operation (whether the correct operation has been performed or not). Data). Then, the diagnosis unit 104 diagnoses whether or not an appropriate response reaction is performed in a timely manner (no time delay) within a predetermined time to be careful. When the degree of attention is smaller than the emotional value level (Ie) and the viewpoint is not going to be seen or the body response is slow, the diagnosis unit 104 diagnoses as an inappropriate state or careless state. To do. Based on the above diagnosis, the skill level and the appropriateness level of the worker are diagnosed based on data such as the delay time and the number of correct operations among the entire operations.

[Pre-work adequacy diagnosis]
In the pre-work appropriateness diagnosis, the worker diagnoses the appropriateness of work on the day before starting work every day. In the pre-work appropriateness diagnosis, the viewer feeling determination device 100 shown in FIGS. 1 and 2 is used. The content of attention determination including points that should be noted in each part (a dangerous part if not operated carefully) is displayed on the video display to show the worker. The analysis unit 102 of the viewer feeling determination apparatus 100 analyzes the line-of-sight direction and the attention level (It) in the data from the eye camera 3, and the diagnosis unit 104 determines the viewpoint position, the attention level, and the inattention level. Measure and when the worker should be careful in the work, the viewpoint goes to the portion to be careful and tracks, the attention level (It) increases to the emotional level (Ie), the speed of attention expansion (Its), When the acceleration (Itk) is also as large as the emotional value levels (Iev, Iek), it is diagnosed that a portion to be noted is focused, and the worker's physical condition of the day is determined to be normal. However, even if a dangerous part is displayed, the approach speed (Bv) of the viewpoint is slower than the specified value, the acceleration (Bk) is also slower than the specified value, the timing is later than the specified value, and there is no appropriate response response. If any one of them applies, it is determined that the worker's physical condition on the day is not normal. Furthermore, by showing the video of family and company relations, analyzing the face at that time, judging emotions such as surprise, fear, disgust, anger, joy (happiness), sadness at that time, emotional problems of the day By analogy with the above, it is possible to determine the suitability of work on the day.

[Athlete appropriate diagnosis]
In the athlete's suitability diagnosis, in sports, it is diagnosed which part is being viewed according to the viewpoint direction at the time of competition of the athlete wearing the eye camera 3. In the athlete proper diagnosis, the viewer feeling determination device 100 shown in FIGS. 1 and 2 is used. The analysis unit 102 of the viewer feeling determination device 100 analyzes the attention level based on the magnitude of the attention level (It), the enlargement speed (Its) of the attention level, and the acceleration (Itk). And the analysis part 102 analyzes the reaction (action | movement, delay time) of the player by a body motion. Specifically, the analysis unit 102 analyzes whether the appropriate response device 14 is operated at an appropriate timing by using the response data d, so that it is timely (no time delay) to be noted. ) Analyze whether the response is appropriate. At the same time, the face analysis at that time is performed, the emotion at that time is judged, and the emotional problem is analogized. And the diagnosis part 104 diagnoses a player's skill and appropriateness compared with the average of the appropriate value of all the competitors of the same competition.

[Game excitement diagnosis]
In the game excitement level diagnosis, the viewer 1 wears the eye camera 3 and executes the game. In the excitement level diagnosis of the game, the viewer feeling determination device 100 shown in FIGS. 1 and 2 is used. The diagnosis unit 104 of the viewer feeling determination device 100 diagnoses that the interest level (It) is clearly interested in the game because the attention level (It) is expanded to the emotional value level (Ie) during the game. At the same time, the face at that time is analyzed, and emotions such as fear, disgust, anger, joy (happiness), sadness, etc. are determined to determine the compatibility of the game. Moreover, when the body temperature is higher than the normal value before the game by the viewer body temperature measurement, the diagnosis unit 104 diagnoses the degree of excitement according to the height. Conversely, when the degree of attention (It) does not expand to the emotional value level (Ie) and the body temperature does not reach the emotional value, the diagnosis unit 104 diagnoses that it is not excited. In addition, the viewer feeling determination device 100 of the present invention is incorporated in a game machine, the viewer is a game player, and the visual target is a visual content input from the diagnostic unit in the game machine as game content visually recognized by the viewer. If the game machine has a selection unit and an execution function for selecting an appropriate game content screen , sound, and game program based on the diagnosis result of the viewer, for example, the viewer's emotional value is low (interest If it is not, the game machine determines that it is clearly not interested in the visual recognition target of the viewer, and the game machine guides the game to the content of interest. In addition, according to the viewer's emotional surprise, anger, fear, sadness, disgust, happiness (joy), the game machine changes the appropriate visual target (changes game content screen, sound, game program, etc.) You can also deal with it.

[Spatial design diagnosis]
In the space design diagnosis, the viewer 1 wears the eye camera 3 and walks in the actual space, and what is the viewer's point of view in the objects arranged in the space and what are they interested in? To determine if the space being viewed is in the appropriate state of the design or arrangement as perceived. The visual recognition method may be a method of viewing an actual space or an object displayed on a display such as a television or a PC. In addition, the space refers to a space such as a cityscape, a department store, a design or display in a store, a design or display in an exhibition hall, a station passage, a station platform, a station building, or a car interior. The viewer design determination apparatus 100 shown in FIGS. 1 and 2 is used for the space design diagnosis. Note that the analysis unit 102 of the viewer feeling determination apparatus 100 determines whether the viewer 1 is moving forward, backward, or moving left and right in the actual space by analyzing the visual image a.

  Here, the object 2 is a specific object in the video (the object of appeal may be plural). The diagnosis unit 104 is a. Looking at the object, the degree of attention (It) has not expanded to the moving value (Ie) {<Ie} (indifference), b. When the tracking time (Bt) of the viewpoint is shorter than the normal value (Bht) (not interested), the viewer 1 diagnoses that the object 2 is “design is not appropriate”. Furthermore, when face analysis is performed and it is determined that there are emotions such as disgust, anger, and sadness at that time, it can be said that the behavior is clearly the opposite of the case where “design is appropriate”. From this, it is possible to determine the degree of diagnosis that “design is not appropriate”. On the other hand, when the degree of attention (It) is expanding and the viewpoint tracking time (Bt) to the object is longer than the normal value (Bht) and reaches the emotional value (Bet), it can be diagnosed that there is interest. , You can't diagnose until "design is right". Therefore, face analysis is performed and if the feeling of joy (happiness) at that time is high, it is diagnosed that the design is viewed positively. On the other hand, when sadness (SD), anger (AG), fear (FR), and disgust (DI) are high, it is determined that the design is not appropriate.

  Next, the viewer 1 visually recognizes a plurality of arrangement designs, sees the object, and from the objects in other designs, a. The degree of attention (It) is large; The viewpoint tracking time (Bt) is long. If the viewpoint departure speed (Bs) and the initial acceleration (Br) are small, the object in the design is most interesting. Furthermore, if it can be determined that the feeling of joy (happiness) at that time by face analysis is large, the diagnosis unit 104 diagnoses that “the design is more appropriate than other designs”. On the other hand, the object is a. The degree of attention (It) is small; The viewpoint retention time (Bt) is short. When the viewpoint withdrawal speed (Bs), the initial acceleration (Br) is large, and the degree of joy (happiness) by face analysis is small, the diagnosis unit 104 determines that “the object of the design is compared with the object of the other design. Is not appropriate. "

(Destination response diagnosis)
In the response level diagnosis apparatus for viewers, the target of visual recognition is a store clerk, a receptionist or the like who responds to customers, and the viewers act from the customer's standpoint. In the reception of the store clerk and the receptionist, it is characterized by diagnosing the feeling of the viewer and determining whether the reception of the store clerk and the receptionist is good or bad. If the viewer's emotional value (E) is low (not interested), it can be determined that there is clearly no particular problem with the other party and that he / she is responding normally. When the emotional value of the viewer is high (attracting interest), it can be generally determined that the response is good if the happiness (joy) degree (HP) of the emotion value (F) is high. However, when anger (AG), fear (FR), and disgust (DI) are high, it can be determined that the other party is not a preferred reception, except for the synchronous response to the conversation content.

[Video content selection priority diagnosis]
In the viewer feeling determination apparatus for a viewing scene, the viewer is a general TV viewer (recipient of video content), and the viewing target is content on a TV display connected to a broadcast, network, or video storage device. . A viewer feeling determination apparatus 100 shown in FIGS. 1 and 2 is used. Initially, the viewer feeling determination apparatus 100 selects a scene in which the viewer 1 is highly interested when the viewer 1 views the basic content in advance. However, if anger (AG), fear (FR), and disgust (DI) are high after face analysis, it is removed from the selection. Further, the content area of the viewer's high interest / interest is determined, and the content of high interest / interest viewed by other viewers showing similar interest / interest is stored in the storage unit 103 as recommended content. The recommended content is displayed when the viewer 1 actually views, and one content can be selected and viewed from the viewer's intention. On the other hand, the diagnostic unit 104 of the viewer feeling determination device 100 increases the attention level (It) to the moving value (Ie) while the video content is viewed from the next time, and the attention level expansion speed (Its). When the acceleration (Itk) is larger than the normal value (Ih, Phv) or when the body temperature is higher than the normal value and the user is excited, there is a clear interest / interest / excitement level for the scene being viewed. Diagnose. However, face analysis is performed and scenes with high anger (AG), fear (FR), and disgust (DI) are removed. Therefore, in order to select the next content to be viewed, the priority for selecting the content in that field is raised. The diagnosis result is recorded in the storage unit 103 and used as recommendation data for selecting the content at the time of viewing the video content at the next viewing time. On the contrary, when the viewer 1 determines that the interest / interest / excitement level is lower than the normal value while the viewer 1 is viewing the content, the diagnosis unit 104 is content other than that field, and the interest / interest / excitement has been previously Switch to a content that has not been viewed yet in a high-level field, and at the same time, when selecting the next content to be viewed, lower the priority for selecting the content in that field. The diagnosis result is recorded in the storage unit 103 and used as recommendation data for selecting content at the time of viewing the video content in the next viewing time.

[Automatic marking of objects or persons with high interest / interest in the content]
In the viewer feeling determination apparatus for a viewing scene, the viewer is a general TV viewer (recipient of video content), and the viewing target is content on a TV display connected to a broadcast, network, or video storage device. . A viewer feeling determination apparatus 100 shown in FIGS. 1 and 2 is used. Automatic marking of objects or people with high interest / interest in the content automatically marks the object or person with a viewpoint in the receiver at the moment of high sensitivity while watching the program. In the system in which the information on the product or person is distributed by e-mail, the product can be purchased and contacted to the person later by the viewer. However, when marking an object or person with a viewpoint in the receiver automatically, even if the emotion (interest / interest) level is high, the fear value depends on the aversion sensitivity, anger level, and conditions in the emotional value by face analysis. It is characterized by automatically removing those with a high degree.

  As described above, according to the viewer feeling determination apparatus 100 according to the embodiment of the present invention, the purchase willingness degree diagnosis, the learning willingness degree diagnosis, the viewer emotion reaction degree diagnosis with respect to the expected emotion reaction degree of the video content creator, Content evaluation by distribution of attention points, compatibility diagnosis, diagnosis of communication with non-speaking babies, people with dementia, animals, etc., robot response method determination, criminal evidence diagnosis, driver drowsiness detection diagnosis, car driving aptitude diagnosis, labor Safety proficiency diagnosis, pre-work appropriateness diagnosis, athlete appropriateness diagnosis, game excitement diagnosis, spatial design diagnosis, audience response diagnosis, video content selection priority diagnosis, high content interest / interest In various situations, such as automatic marking of a person, the ability of the object or viewer itself can be more accurately evaluated.

In this way, the first viewer emotion determination device is a viewer emotion determination device for a visual scene, which diagnoses the emotion of the viewer who looks at the scene including the specific object with respect to the specific object. , The viewer's viewing video and the brightness of the viewing video range, the viewer's face video, the viewer's body movement, eye movement video, a body temperature receiving unit, and the viewer's face video. Change of each part of the person's face, calculate the position of the viewpoint in the visual image from the visual image and the eye movement video, and change the physiological response data of the viewer and the accompanying speed and acceleration. Analyzing part to calculate, normal position (normal emotion value) of each part of the face of the viewer, changes in physiological reaction data when the viewer is looking at an object of low interest and attention, and accompanying speed And The information including the degree is accumulated as a normal emotion value, and the change value when the viewer's emotion is high such as the viewer's anger, fear, disgust, happiness (satisfaction), sadness, surprise, etc. is set as a high emotion value, Changes in physiological response data when the viewer is looking at an object of interest with high interest and information including speed and acceleration associated therewith are accumulated as high emotion values, and the viewer calculates the viewer's An accumulation unit for accumulating information including changes in physiological response data and accompanying velocity and acceleration as analysis values, and the analysis values as normal emotion values and normal emotion values (both referred to as normal values) or the high emotion values and A diagnosis unit for diagnosing the emotion of the viewer and the reaction ability of the viewer by comparing with the high emotion value (both are called emotion values), and a body temperature measuring machine if necessary A device for determining a viewer's feeling for a visual scene, characterized by comprising a normal temperature of the body and a portion for diagnosing excitement by measuring body temperature and movement at the time of excitement. To do.

The object at the time of visual recognition in this case includes the case of visual recognition while touching or operating the actual object. Furthermore, in addition to visual perception, the same reaction occurs when receiving auditory, gustatory, tactile, and olfactory stimuli. Analyze the degree of attention to these stimuli (five senses), the movement of the viewpoint, and the movement of each face part. And performing a similar emotion determination on the object.

The “normal emotion value” and the “high emotion value” are measured by visually recognizing the basic video showing the “normal emotion value” and the “high emotion value”, and the “normal emotion value” and the “high emotion value” are the faces. The position of each part of the face of “normal” (normal emotion value) and the maximum expression that shows “anger”, “fear”, “disgust”, “happiness (joy)”, “sadness”, “surprise” With the displacement value of the movement of each part of the face at that time of "anger", "fear", "disgust", "happiness (joy)", "sadness", "surprise" It is provided with measuring as "high emotion value".

Further, in the second viewer feeling determination device, the change in the physiological response data and the accompanying speed and acceleration are the approach speed and acceleration when the viewer's viewpoint approaches the specific object, or The detachment speed and acceleration when the viewer's viewpoint detaches from the specific object, and the diagnosis unit is more interested in the specific object as the approach speed and acceleration are larger. Is less interested in the specific object, the greater the separation speed and acceleration, the less interested the specific object, and the smaller the separation speed and acceleration, the more interesting the specific object. It is characterized by diagnosing it. Note that the visual recognition target may be a certain part in the target object (for example, in the case of a product, a product name, a manufacturer name, an expiration date, a note, a specification, an operation button, etc.).

Further, in the third viewer feeling determination device, the analysis unit captures a stationary object in the visual image captured by the camera when the visual image is captured by the visual scene photographing device mounted on the head of the viewer. Using the position as a base point, calculate the movement distance, movement speed, and movement acceleration of the viewer's head from the change in the position of the stationary object in the visual image, and add the change in the position of the viewpoint in the visual image. Then, the approach speed and acceleration of the viewpoint with respect to the specific object including the movement of the head or the separation speed and acceleration are calculated. An existing method such as a measurement method using a gyroscope may be used to calculate the moving direction and distance, moving speed, and moving acceleration of the viewer's head.

In the physiological response data of the first viewer feeling determination device, when the physiological response data is a change in pupil diameter, when the visual target is a video display, the visual image is content to be displayed on the display. The device and the brightness measuring device may be fixed to the video display side without being mounted on the viewer's head. In the fourth viewer feeling determination device, when the object to be viewed is a video display, the brightness of the video display is changed, the pupil diameter of the viewer corresponding thereto is obtained, and the relational data between the basic brightness and the pupil diameter is accumulated and visually recognized. When the viewer visually recognizes the target object, the pupil diameter of the viewer is measured, and at the same time, the luminance is measured with respect to the viewing range by the luminance meter, and the pupil diameter value when the object is visually recognized is accumulated. It is characterized in that the pupil diameter corresponding to only the degree of attention to the object is obtained by subtracting the value of the pupil diameter corresponding to the luminance at the time of visual recognition of the object from the relational data between the basic luminance and the pupil diameter. In the case of other visual scenes other than the video display, the eyeball photographing device, the face photographing device, the visual scene photographing device, and the brightness measuring device are integrally mounted on the viewer's head for measurement.

Further, in the fifth viewer feeling determination device, in a video where the luminance of the viewpoint cannot be ignored compared to the luminance of the video display, red (r), green (G), blue ( B) The luminance of the pixel at the viewpoint is calculated using the influence on the luminance as a parameter, and it corresponds to the luminance of the pixel at the time of visual recognition of “basic relationship data of display luminance, viewpoint luminance, and pupil diameter” from the pupil diameter of the viewer. The pupil diameter corresponding to the degree of attention of the viewer's viewpoint including the brightness of the display can be obtained by subtracting the pupil diameter (pupil diameter that reacts to brightness). Since there is already a measuring device that measures the luminance of a specific pixel, the luminance of each pixel may be obtained by the measuring device. In addition, the “basic relationship data of display brightness, viewpoint brightness, and pupil diameter” shows that the brightness of a circle of about 1.7 cm is changed from the highest (white) to the minimum (black) in the brightness of the display that changes in stages. Sometimes, the pupil diameter when the viewer visually recognizes the circle is measured, and “basic relation data of display luminance, viewpoint luminance, and pupil diameter” is calculated and accumulated.

In the sixth viewer feeling determination device, the maximum value of the pupil diameter (the lowest luminance of the video display) and the minimum value of the pupil diameter (the highest luminance of the video display) in the relational data of the basic luminance and the pupil diameter. The ratio between the change range of the pupil diameter and the pupil diameter corresponding to the degree of attention only to the object when the viewer visually recognizes the object is the degree of attention that absorbs the individual difference of the viewer To do.

Further, the seventh viewer feeling determination device determines that the interest / interest in the visual target is high when the degree of attention is high. This is an emotional reaction, and the attention level in this case is an emotional value. Furthermore, the speed and acceleration associated with the change in the physiological response data are an expansion speed and acceleration of the attention level of the viewer, or a reduction speed and acceleration of the attention level of the viewer, and the diagnosis unit The greater the degree and the larger the enlargement speed and acceleration, the more the diagnosis is made. The less the degree of attention and the greater the reduction speed and acceleration, the greater the particular object. It is characterized by diagnosing that there is no interest.

Further, in the eighth viewer feeling determination device, the movement of each part (nose, eyes, mouth, eyebrows, etc.) of the face image of the viewer is measured, the amount of change is measured, and each changed part is measured. The size of the emotion value based on the viewer's facial expression is determined based on the size. The emotion of the viewer with respect to the visual target object is determined by combining this determination with the magnitude of the emotion value based on the degree of attention due to the eye movement during the visual recognition of the visual target. The basic six elements of feeling (Feling: F) are “happiness (joy) = Happiness (Joy)”, “sadness = Sadness”, “anger = Anger”, “fear = Fear”, “disgust = Disgust (Dislike)” , "Surprise = Surprise" degree
"Happiness (joy)" = HP
"Degree of sadness" = SD
"Anger" = AG
“Fear” = FR
“Aversion” = DI
"Surprising degree" = SP
As described above, the size of emotion (= Emotion (E)) and emotion (= Feeling (F) 6 elements) is combined and defined as an EF value. Generally, when the emotional value (E) is low, the subject is bored or sleepy or tired. In that case, each emotion value (F) is low. Therefore, it can be determined that the evaluation is negative with respect to the visual recognition target. When the emotional value (E) is normal, the subject has little special interest / interest and little attention. In this case, each emotion (F) generally does not increase. Therefore, it cannot be said that emotion can be determined accurately. When the emotion value (E) is high and each emotion is high, it can be determined that the happiness (HP) is positive and effective. However, if there is a sadness (SD) expression, it can be determined that the subject is inactive. Moreover, if anger, fear, or disgust appear, it is a negative reaction. If you are surprised, you will be surprised and interested (interested), and depending on the subject, it is generally an effective response. In addition, when the object to be viewed is content such as a movie and the content intentionally invites each emotion, the happiness (joy) degree (HP) and surprise (SP), sadness (SD), anger of the emotion value (F) When (AG), fear (FR), and disgust (DI) are high as intended, the content to be visually recognized has an emotional reaction as intended and can be determined to be effective. Hereinafter, a specific determination method will be described for each target.

In the ninth viewer feeling determination device, the object at the time of visual recognition is smelled when the actual object is visually recognized while touching or operating with the hand, or when only the sound is heard. The attention of the viewer and the movement of the viewpoint when the user feels the taste of eating, when a part of the body is in contact with the object, or when there are multiple stimuli It is characterized by analyzing the movement of each part of the face and performing emotion determination for those stimuli. In particular, when the emotional value (E) is low for these stimuli, it can be determined that there is no response to the stimulus. Further, when the emotion value (E) is high, and when the happiness (joy) degree (HP) of the emotion value (F) is generally high, the stimulus can be determined to be comfortable. However, when the sadness (SD), anger (AG), fear (FR), and disgust (DI) of other emotion values (F) are high, it can be generally determined that the stimulus is not comfortable for the viewer. When attention is paid and the surprise (SP) is high, it depends on the situation, but generally, it can be determined that the user feels unexpectedness.

In the tenth viewer feeling determination device, the specific object is a product, and the diagnosis unit diagnoses the viewer's willingness to purchase the product. If the emotional value is low (not interested and not interested), it can be clearly determined that it does not lead to purchase. However, even if the emotional value is high (interesting and paying attention), it cannot be said that the interest in purchasing is high simply because the interest is high. However, when the emotion level is generally high in happiness (joy) and surprise, it can be determined that the purchase is positive. However, when sadness, anger, fear, and disgust of other emotion values are high, it can be generally determined that the purchase is not aggressive.

Further, in the eleventh viewer feeling determination device, the specific object is a learning object (lecture, reading, teacher movement, teaching material, etc.), and the diagnosis unit is provided for the learning object of the viewer. It is characterized by diagnosing learning motivation. When the emotion value is low (not interested and not interested), it can be determined that the subject is clearly not interested in the learning target and is “bored”. However, if the emotional value is high (interested and focused), it can be said that the willingness to learn is high. In general, when the content of learning is a content that invites each emotion, if the happiness (joy) degree of the emotion value and surprise, sadness, anger, fear, and disgust are high as intended, it can be determined that the learning is good. However, the emotional value is high when tackling a problem, but when sadness, anger, fear, and disgust are high, something is confusing and it can be determined that there is a problem with this learning target.

In the twelfth viewer feeling determination device, the specific object is video content such as a television or a movie, and the viewer is the video content viewer. Video content creators are interested in each scene of the video content that the viewer expects from viewers, and the happiness (joy), surprise, sadness, anger, fear, aversion sensitivity of each basic emotion And the level of excitement level are set on the time axis of the content software program, and when the video content is projected, the viewer's attention and happiness (joy) level, surprise level, sadness level, anger level, fear level, It is characterized by diagnosing how much the aversion sensitivity and excitement level have reached the level expected by the video content producer and evaluating the video content. The target content includes sound at the same time as the video, but there may be a case where only the sound is present without the video. The most effective usage methods are pre-evaluation of production movies and pre-evaluation of commercial audience response.

In the thirteenth viewer feeling determination apparatus, the specific object is video content such as a television or a movie, and the viewer is the video content viewer. Displays the distribution of attention points on the video screen according to the degree of attention of multiple viewers in a specific scene or still image of the video content, and the content appeal points compared with the appeal points on the content creator's video screen It is characterized by determining an effect.

In the fourteenth viewer feeling determination device, the receiving unit receives the caller's face, the visual image and the eye movement image from two videophones, and the diagnostic unit receives one of the calls. A transmitting unit for diagnosing the feeling of the other party from the other party, and further transmitting a result of the emotional diagnosis from the one party to the other party to a mediator such as a videophone of the other party or a diagnostic center. It is characterized by having. When the emotional value is low (not interested), it can be determined that there is clearly no interest in the other party and there is no compatibility. However, even if the emotional value is high (attention is paid), it cannot be determined that the compatibility is good simply because the interest is high. However, it can be determined that the compatibility is good if the degree of happiness (joy) of the emotion value is generally high at that time. When surprise, anger, fear, and disgust are high, it can be determined that compatibility is good in the sense of conflict in the case of a reaction based on the content of the talk. However, it is irrelevant to the content of the story, and if anger, fear, or disgust is high, it cannot be determined that the compatibility is good.

Further, in the fifteenth viewer feeling determination device, the viewer is a person or an animal who is difficult to speak a language, and their visual targets are all their visual targets. It is characterized by determining the feeling of the. When the viewer is a human (for example, a baby or a person with dementia), it can be clearly determined that he / she is not interested in the visual target if the emotion value is low (not interested). On the other hand, when the emotion value is high (attention is paid), it can be determined that the interest is high. Regardless of the emotion value, it can generally be determined that the situation is good if the happiness (joy) level of the emotion value is high. Moreover, when anger, fear, and disgust are high, it can be determined that there is dissatisfaction. When the viewer is an animal, when the emotion value is low, it can be clearly determined that the viewer is not interested in the target. Conversely, when the emotion value is high, it can be determined that the visual target is interested.

In the sixteenth viewer feeling determination device, the viewer feeling determination device is incorporated in a robot, and the viewer is a human having an emotion, is an animal, and the visual recognition target is all that is viewed by the viewer. . The robot includes a selection unit and an execution function for selecting appropriate speech and response based on a viewer's diagnosis result input from the diagnosis unit in the robot. For example, when the viewer's emotional value is low (not interested), it is clearly determined that the viewer's viewing target is not interested, and the robot behaves and moves to the interested subject. If the viewer's emotional anger, fear, or disgust is high, the robot will take appropriate measures such as uttering appropriate comfort words or changing the visual target.

In the seventeenth viewer feeling determination device, the viewer is a criminal suspect, and the specific object is a crime scene, a criminal tool, an accomplice or a victim. When the viewer's emotional value is low (not paying attention), it can be determined that the visual target is not relevant to the criminal suspect. However, even if the emotional value is high (attracting attention), it is not related to criminals simply because of high interest. In this case, if the emotional value is generally high and the emotional value is high in happiness (joy), surprise, anger, fear, or disgust, it can be determined that the crime suspect is of some relevance.

In the eighteenth viewer feeling determination device, the viewer is a driver of a transportation facility such as an automobile or an airplane, and the position of the viewpoint indicates whether or not the diagnostic unit is paying attention. And determining the pupil diameter, and if it is diagnosed that the vehicle is in a dangerous driving state, the viewer is provided with a function of performing appropriate operation assistance such as a warning or a brake. When the emotional value of the viewer (driver) is low, it can be determined that it is clearly inadvertent state or close to dozing state, so it can be said that it is a dangerous driving state. However, even if the emotional value is high (attention), it cannot be said that the driving is necessarily good. It can be determined that the driving is good when the viewpoint is always located at an appropriate position. Furthermore, it can be determined that the situation is good if the happiness (joy) level of the emotion value is high. In addition, when the feelings of anger, fear, and disgust are high, it can be determined that there is some obstacle during driving other than the case where it is influenced by the conversation in the car. For example, when stress is high.

In the nineteenth viewer feeling determination device, the viewer is a worker, the target to be viewed is a machine operation simulator or a real machine, and the analysis unit responds by the operation of the viewer response device. The data is used to analyze a response delay time and an operation method, and the diagnosis unit diagnoses the skill level or appropriateness level of the viewer. When the viewer's emotional value is low, it can be clearly determined that the person is careless or close to dozing. However, even if the emotion value is high (attention is paid), it cannot be said that the operation is necessarily good. It is possible to determine that the situation is good if the line of sight always keeps the object of operation, and if the happiness (joy) level of the emotion value is high. Further, when anger, fear, and disgust are high, it can be determined that there is some obstacle during operation. For example, anger at the operation of labor partners.

In the twentieth viewer feeling determination device, the viewer is a pre-worker worker, the visual target is operation video content or a simulator with a dangerous spot, and the diagnosis unit is configured to check the physical condition of the viewer. It is characterized by determining the suitability for employment. When the viewer's emotional value is low, it can be determined that the state is clearly inadvertent or close to dozing. For example, being distracted by other things. However, even if the emotional value is high (attention is paid), it cannot always be said that the physical condition is good. It can be determined that the situation is good if the line of sight always keeps an appropriate target in the video and the happiness (joy) level of the emotion value is high. If the feelings of anger, fear, and disgust are high, it can be determined that the viewer has some kind of disability (such as the person's physical condition, home, friend, etc.).

In the twenty-first viewer feeling determination device, the viewer is an athlete, and the specific object is an object to be seen when the athlete exercises (if a baseball batter, In the case of a tennis player, the movement of the opponent player and the ball's ball muscle, etc.), and the diagnosis unit diagnoses the condition, skill level, or appropriateness level of the viewer. When the line of sight does not go to the target to be noticed and the viewer's emotional value is low (not focused), the viewer can clearly be determined to be unskilled. However, even if the gaze goes to the target to be noticed and the emotional value is high (attention), the condition is not necessarily good, but if the emotional value is high, the situation is good it can. If anger, fear, and disgust are high (for example, dissatisfaction with the referee), it can be determined that there is some emotional problem at that time.

In the twenty-second viewer feeling determination device, the viewer feeling determination device of the present invention is incorporated in a game machine, the viewer is a game player, and the viewing target is game content visually recognized by the viewer. The game machine is provided with a selection unit and an execution function for selecting an appropriate game content screen, sound and game program based on a viewer's diagnosis result input from the diagnosis unit in the game machine. And For example, when the viewer's emotional value is low (not interested), it is clearly determined that the viewer's viewing target is not interested, and the game machine presents a content screen or the like that leads to the interested subject. In response to the viewer's emotional surprise, anger, fear, sadness, disgust, happiness (joy), the game console will take appropriate measures such as changing the viewing target (changing the game content screen, sound, and game program). Will do.

In the twenty-third viewer emotion determination device, the specific object is an object arranged in a space, and the diagnosis unit is configured to display an array of the specific object viewed from the viewer. It is characterized by diagnosing appropriateness. If the viewer's emotional value is low (not interested and low in interest), it can be clearly determined that the item on the shelf and the shelf are not noticed. However, even if the emotional value is high (attracting attention), it cannot be said that the interest in purchasing is high simply because the interest is high. However, in general, when the emotional value is high in happiness (joy) and surprise, it can be determined as favorable. However, when the feelings of sadness, anger, fear, and disgust of other emotion values are high, it can be generally determined that there is an element that leads to dissatisfaction.

Further, in the twenty-fourth viewer feeling determination device, the viewer is a visitor, and the object to be viewed is a store clerk, a receptionist or the like who responds to the customer, and the viewer's feeling is diagnosed in the reception of the store clerk or receptionist. It is characterized by determining whether the store clerk or the receptionist is good or bad. If the viewer's emotional value is low (not interested), it can be determined that there is no particular problem with the other party and that he / she is responding normally. When the viewer's emotional value is high (attracting interest), it can generally be determined that the response is good if the emotional value is high in happiness (joy). However, when anger, fear, and disgust are high, it can be determined that the response is not preferable except for the synchronous response to the conversation content.

In the twenty-fifth viewer feeling determination device, the viewer is a general television viewer (receiver of video content), and the target to be viewed is on a TV display connected to a broadcast, communication network, or video storage device. Content that is highly interested in the viewer when watching the TV, etc. when the next viewer sees the TV, etc. It is characterized by recommending.

In the twenty-six viewer feeling determination apparatus, the viewer is a general television viewer, and the viewing target is a television receiver (video content receiver) that receives via a network or radio waves. While watching a program, the object or person with a viewpoint in the receiver is automatically marked at the moment of high sensitivity, and the information on the marked object is distributed to the viewer via e-mail later. This system is characterized by being able to purchase products and contact people with the intention of the viewer thereafter. However, when automatically marking an object or person with a viewpoint in the receiver, even if the emotion (interest / interest level) is high, those with high disgust sensitivity and anger level are automatically excluded. .

DESCRIPTION OF SYMBOLS 1 Viewer 2 Object 3 Eye camera 4 Eyeball imaging device 5 Visual scene imaging device 6 Microphone 7 Transmission device 8 Accumulation device 9 Body motion imaging device 10 Body temperature measurement device 11 Brightness measurement device 14 Response device 20a, 20b Video phone 21a , 21b Face camera 22a, 22b Display 23a, 23b Eye camera 24a, 24b Tracking camera 30 Communication line 100 Viewer feeling determination device 101 Reception unit 102 Analysis unit 103 Storage unit 104 Diagnosis unit 105 Display unit 106 Transmission unit 107 Speech selection unit 108 Warning section 301 Mark 1 (feature point) in eye analysis inner side edge 302 of eyebrow Mark 2 (feature point) in eye analysis outer edge 303 Mark 3 (feature point) in face analysis Upper middle part of upper eyelid 304 Mark 4 in face analysis ( Feature) Middle lower end of jaw 401 Specific video content Appeal point of the video display on the screen of the video display screen 402 specific video content

Claims (25)

In the viewer feeling determination device for determining the feeling for the specific object when the viewer turns his gaze toward the scene including the specific object,
An eyeball photographing device for photographing an image of the eyeball of the viewer;
A visual scene photographing device for photographing a scene including the specific object as a visual image of the viewer;
A face photographing device for photographing an image of the face of the viewer;
A brightness measuring device for measuring brightness information of a visual field viewed by the viewer;
Calculating a viewpoint position in the visually recognized image from an eyeball image captured by the eyeball image capturing device and a visually recognized image captured by the visually recognized scene image capturing device, and calculating an acceleration of the viewpoint as the viewpoint moves. Then, seek the movement of the viewpoint,
Calculate the pupil diameter from the eyeball image, based on the calculated pupil diameter, the brightness information measured by the brightness measuring device, and the relationship information between the preset brightness information and the pupil diameter, Calculate a new pupil diameter that eliminates the influence of the brightness of the visual field of view, determine the degree of attention from the new pupil diameter ,
Based on the position of the predetermined part of the face image photographed by the face photographing device, the position of the predetermined part in the normal time of the viewer set in advance, and the position when the predetermined part moved to the maximum, An analysis unit for determining the degree of emotion ;
Viewpoint motion including viewpoint position and viewpoint acceleration calculated from the viewable image and eyeball image when the viewer is looking at an object of low interest and not of interest , degree of emotion determined from the face image In addition, the attention degree calculated from the eyeball image is stored in advance as a normal value, and the movement of the viewpoint , the degree of emotion, and the attention degree when the viewer looks at an object of high interest is noted. An accumulator that is accumulated in advance as a touching value;
The viewpoint movement , the degree of emotion and the attention level obtained by the analysis unit are respectively compared with the viewpoint movement , the emotion level and the attention level in the normal value and the emotional value accumulated in advance by the accumulation unit, A diagnostic unit for determining the feeling of a specific object;
A viewer feeling determination apparatus characterized by comprising: In the viewer feeling determination apparatus according to claim 1,
The analysis unit calculates a viewpoint position in the visually recognized image from an eyeball image captured by the eyeball image capturing device and a visually recognized image captured by the visually recognized scene image capturing device, and the viewpoint is moved along with the movement of the viewpoint. Calculating an approach speed and an approach acceleration when approaching the specific object, or a departure speed and an exit acceleration when the viewpoint is detached from the specific object;
The diagnostic unit further determines that the greater the approach speed and the approach acceleration calculated by the analysis unit are, the more interested and focused on the specific object, and the smaller the approach speed and the approach acceleration are. It is determined that interest is low and not focused on the specific object, and it is determined that interest is low and attention is not focused on the specific object as the separation speed and separation acceleration are large, or A viewer feeling determination apparatus characterized by determining that the specific object is more interested and focused as the separation speed and the separation acceleration are smaller. In the viewer feeling determination device according to claim 2,
The analysis unit further calculates a movement size, a movement speed, and a movement acceleration of the viewer's head based on a predetermined position in the visual image or using a gyroscope attached to the scene photographing device. And calculating the viewpoint position in the viewing image from the viewing image and the eyeball image, calculating the change speed and change acceleration of the viewpoint position, the magnitude of the movement of the head, the movement speed and the movement acceleration, From the viewpoint position, the change speed, and the change acceleration, the viewpoint position including the head movement, the change speed and the change acceleration of the viewpoint position are calculated, and the viewpoint is specified as the viewpoint including the head movement moves. Calculating an approach speed and an approach acceleration when approaching an object, and a departure speed and an exit acceleration when the viewpoint is detached from the specific object.認者 emotion determination device. In the viewer feeling determination apparatus according to claim 1,
The brightness measurement device is a luminance meter that measures the luminance of the visual field of view, and the relationship information between the brightness information and the pupil diameter used when calculating a new pupil diameter by the analysis unit, The brightness and pupil diameter when calculating the pupil diameter of the viewer by changing the brightness,
The analysis unit calculates a pupil diameter from the eyeball image, specifies a pupil diameter corresponding to luminance as the brightness information from the relation information, and subtracts the specified pupil diameter from the calculated pupil diameter. The viewer feeling determination apparatus characterized by calculating the new pupil diameter. In the viewer feeling determination device according to claim 4,
When the visual recognition target by the viewer is a video display, the brightness measurement device is a luminance meter that measures the luminance of the video display, and brightness information used when the analysis unit calculates a new pupil diameter; The relationship information between the pupil diameter, the brightness of the video display when the pupil diameter of the viewer is calculated by changing the brightness of the video display and the brightness of a predetermined pixel on the video display, the predetermined brightness Pixel brightness and pupil diameter,
The analysis unit calculates the pupil diameter from the eyeball image, specifies the pupil diameter corresponding to the luminance of the video display as the brightness information and the luminance of the pixel at the viewpoint position from the relation information, and calculates the pupil diameter The viewer feeling determination apparatus, wherein the new pupil diameter is calculated by subtracting the specified pupil diameter from the pupil diameter. In the viewer feeling determination apparatus according to claim 4 or 5,
The analysis unit obtains a ratio between a result obtained by subtracting a minimum value from a maximum value of a pupil diameter in the relationship information and the calculated new pupil diameter as the attention degree, Judgment device. In the viewer feeling determination apparatus according to claim 1,
The analysis unit calculates, from the obtained attention degree, an enlargement speed and an enlargement acceleration of the attention degree, or a reduction speed and a reduction acceleration of the attention degree,
The diagnosis unit further determines that the greater the degree of attention obtained by the analysis unit is, the more interested and focused on the specific object, and the smaller the degree of attention obtained by the analysis unit. It is determined that there is no interest in the specific object and attention is not paid, and the greater the expansion speed and expansion acceleration of the degree of attention calculated by the analysis unit, the more interest and attention to the specific object. The viewer feeling is characterized in that the greater the reduction speed and the reduction acceleration of the degree of attention calculated by the analysis unit, the more the viewer feels less interested and not interested in the specific object. Judgment device. In viewer emotion determination apparatus according to any one of claims 1 to 7,
In the state where the viewer touches or operates the specific object by hand, hears or smells the sound of the predetermined object, or feels the taste by eating the predetermined object,
The viewer feeling determination apparatus, wherein the diagnosis unit determines the viewer's feeling for the specific object. In the viewer feeling determination apparatus as described in any one of Claim 1-8 ,
The specific object is a product,
The viewer feeling determination apparatus characterized in that the diagnosis unit determines the purchase intention of the viewer for the product as feeling. In the viewer feeling determination apparatus as described in any one of Claim 1-8 ,
The specific object is a learning object of a teacher, teaching material or blackboard,
The viewer feeling determination apparatus, wherein the diagnosis unit determines the learning motivation degree of the viewer with respect to the learning target as feeling. In the viewer feeling determination apparatus as described in any one of Claim 1-8 ,
The specific object is a content video produced by a video content producer,
The diagnostic unit determines the viewer's feelings for each scene of the content video, and based on the emotion set in advance for each scene and the determined feeling, A viewer feeling determination apparatus characterized by determining a degree of achievement of a viewer's expectation. In the viewer feeling determination apparatus as described in any one of Claim 1-8 ,
The specific object is a content of one of a specific screen of a video, a still image, a poster, or a magazine,
When the content is displayed on the screen, the diagnosis unit displays the attention degree obtained by the analysis unit as a distribution at the viewpoint position calculated by the analysis unit. In the viewer feeling determination apparatus as described in any one of Claim 1-8 ,
Furthermore, a transmission unit that transmits information on emotions determined by the diagnosis unit,
The specific object is a caller having a conversation while watching a video of the other party using a videophone communication line,
The diagnosis unit determines feelings from one caller to the other caller,
The viewer's feeling determination apparatus, wherein the transmitting section transmits information about feelings from one calling party to the other calling party determined by the diagnosis section to present to the other calling party. In the viewer feeling determination apparatus as described in any one of Claim 1-8 ,
When the viewer is a person or an animal who has difficulty speaking,
The viewer feeling determination apparatus, wherein the diagnosis unit determines the viewer's feeling for the specific object. In the viewer feeling determination apparatus as described in any one of Claim 1-8 ,
The viewer feeling determination device is incorporated in the robot,
Furthermore, the viewer feeling determination apparatus characterized by including the selection part which selects the speech and behavior which the said robot utters, and the operation | movement of the said robot based on the feeling determined by the said diagnostic part. In the viewer feeling determination apparatus as described in any one of Claim 1-8 ,
The viewer is a criminal suspect;
The specific object is a crime scene, a criminal tool, an accomplice or a victim,
The viewer feeling determination apparatus, wherein the diagnosis unit determines a feeling of the criminal suspect with respect to the specific object. In the viewer feeling determination apparatus as described in any one of Claim 1-8 ,
The viewer is a transportation driver;
The specific object is a video content including a dangerous spot that can be seen from the driver's seat of the transportation facility,
Furthermore, a warning unit for notifying the driver of a warning is provided,
Said diagnostic unit, the movement of the viewpoint obtained by the analysis unit, and the degree and attention emotion, movement of the viewpoint in the prestored normal value and excitement value by the storing unit, and a degree and attention emotion Compare each of them, determine whether you are paying attention to the dangerous part, determine whether you are in a dangerous driving state,
The viewer feeling determination device, wherein the warning unit notifies the driver of a warning when the diagnosis unit determines that the vehicle is in a dangerous driving state. In the viewer feeling determination apparatus as described in any one of Claim 1-8 ,
The viewer is a worker;
The specific object is a video content of an operating device including a dangerous part,
And a response device operated by the worker,
Said diagnostic unit, the movement of the viewpoint obtained by the analysis unit, and the degree and attention emotion, movement of the viewpoint in the prestored normal value and excitement value by the storing unit, and a degree and attention emotion Each of them is compared, and it is determined whether or not attention is given to the dangerous part, and it is determined whether or not an appropriate operation has been performed based on the operation content to the response device by the worker. Viewer feeling judgment device. In the viewer feeling determination apparatus as described in any one of Claim 1-8 ,
The viewer is a worker;
The specific object is a video content of an operating device including a dangerous part,
Said diagnostic unit, the movement of the viewpoint obtained by the analysis unit, and the degree and attention emotion, movement of the viewpoint in the prestored normal value and excitement value by the storing unit, and a degree and attention emotion A viewer feeling determination device characterized by comparing each of them and determining the physical condition or suitability of the worker with respect to the operation device as feeling. In the viewer feeling determination apparatus as described in any one of Claim 1-8 ,
The viewer is an athlete;
The specific object is an object to be seen when an athlete exercises,
Said diagnostic unit, the movement of the viewpoint obtained by the analysis unit, and the degree and attention emotion, movement of the viewpoint in the prestored normal value and excitement value by the storing unit, and a degree and attention emotion A viewer feeling determination device characterized by comparing each of them and determining the skill level or appropriateness level of the athlete with respect to the object as feeling. In the viewer feeling determination apparatus as described in any one of Claim 1-8 ,
The viewer is a game player;
The specific object is a game machine,
And a body temperature measuring device for measuring the body temperature of the game player,
Said diagnostic unit, the movement of the viewpoint obtained by the analysis unit, and the degree and attention emotion, movement of the viewpoint in the prestored normal value and excitement value by the storing unit, and a degree and attention emotion A viewer feeling determination apparatus characterized by comparing each of them and determining an emotion including an excitement degree of the game player by the game player based on the body temperature measured by the body temperature measuring apparatus. In the viewer feeling determination apparatus as described in any one of Claim 1-8 ,
The specific object is an object arranged in a space,
Said diagnostic unit, the movement of the viewpoint obtained by the analysis unit, and the degree and attention emotion, movement of the viewpoint in the prestored normal value and excitement value by the storing unit, and a degree and attention emotion A viewer feeling determination apparatus characterized by comparing each of them, determining an emotion with respect to the specific object, and determining an arrangement and appropriateness of the specific object. In the viewer feeling determination apparatus as described in any one of Claim 1-8 ,
The viewer is a customer;
The specific object is a clerk or receptionist for the customer,
Said diagnostic unit, the movement of the viewpoint obtained by the analysis unit, and the degree and attention emotion, movement of the viewpoint in the prestored normal value and excitement value by the storing unit, and a degree and attention emotion A viewer feeling determination device characterized by comparing each of them, determining a feeling of the customer with respect to the store clerk or the receptionist, and determining whether the store clerk or the receptionist is good or bad. In the viewer feeling determination apparatus as described in any one of Claim 1-8 ,
The viewer is a TV viewer;
The specific object is content displayed on a display connected to a broadcast, network or video storage device,
Said diagnostic unit, the movement of the viewpoint obtained by the analysis unit, and the degree and attention emotion, movement of the viewpoint in the prestored normal value and excitement value by the storing unit, and a degree and attention emotion The television viewers are compared with each other, the emotion of the television viewer with respect to the content is determined, the field of content with high interest by the television viewer is determined, and another television with high interest with respect to content belonging to the same field is determined. A viewer feeling determination apparatus, which presents a predetermined content belonging to the same field to a viewer. In the viewer feeling determination apparatus as described in any one of Claim 1-8 ,
The viewer is a TV viewer;
The specific object is a content including a predetermined product or person displayed on a display connected to a broadcast, network or video storage device,
Said diagnostic unit, the movement of the viewpoint obtained by the analysis unit, and the degree and attention emotion, movement of the viewpoint in the prestored normal value and excitement value by the storing unit, and a degree and attention emotion Compare each of them, determine the TV viewer's feelings about the content, and out of the products or people included in the content displayed on the display, the products or people at the viewpoint position with the high degree of attention in the threshold judgment A viewer feeling determination device characterized by attaching a mark and presenting information regarding a product or a person with the mark to the television viewer.