JP4608858B2 - Emotion visualization device, emotion visualization method, and emotion visualization output - Google Patents

Description

  The present invention relates to, for example, a device and method for visualizing a person's daily emotions and emotions when viewing content, and a visualized output of emotions visualized by the devices and methods.

  For example, what kind of emotional state a person has spent in a certain period of time or at a certain time, and what emotional state a person has experienced during a certain experience, for example, in a diary, If the events and feelings at that time are written as sentences, it can be estimated by reading the diary.

  However, in the case of a diary, although it depends on a person's ability to express a sentence, it is generally difficult to objectively write the emotion at that time. Also, if you do not read the diary, you cannot know the feelings of the person at that time. In addition, since the emotion is estimated from the sentence, it is difficult to accurately recognize the emotion at that time.

  Patent Document 1 (Japanese Patent Laid-Open No. 2002-230011) discloses an emotion recognition that can discriminate an emotion expressed in a document and convert the discriminated emotion into a visual display format such as an icon for display. A system is provided.

In this Patent Document 1, an emotion expression dictionary in which emotion elements such as “joy”, “sadness”, “surprise”, “fear”, “anger”, “disgust”, etc. are quantified and assigned in advance for each word. Is prepared, the recognition target document is decomposed into words, and the emotion expressed in the recognition target document is recognized using the emotion expression dictionary. In Patent Document 1, the recognized emotions such as “joy”, “sadness”, “surprise”, “fear”, “anger”, “disgust” are converted into visual display formats such as icons, Visualization is displayed.
JP 2002-230011 A

  However, even if it is possible to discriminate the emotion expressed in the text described in the diary using the technique of the above-mentioned Patent Document 1, as described above, If emotions are not objectively and accurately described, the emotions at that time cannot be accurately reflected.

  In the case of Patent Document 1, the recognized emotion is visualized and displayed by an icon corresponding to the emotion. As this icon, for example, emotions such as “joy”, “sadness”, “surprise”, “fear”, “anger”, “hate” are expressed by facial expressions (real images, simple cartoons, etc.), for example. It is thought that things can be used.

  However, in the case of the icon as described in Patent Document 1, only the average emotion expressed for a certain period expressed in the sentence is represented. It is not possible to represent the state distribution of emotions such as how much each of various emotions such as “relaxed”, “uncomfortable”, and “comfortable” was distributed during the period.

  In view of the above points, an object of the present invention is to make it possible to accurately reflect human emotions and to visualize and express emotional changes and emotional state distributions. .

In order to solve the above problem, in the visualization device according to the invention of claim 1,
Biometric information acquisition means for acquiring human biometric information;
Analyzing the biological information from the biological information acquisition means, and obtaining emotion information representing the emotion of the person, an emotion analysis processing unit;
A conversion table information holding unit for holding conversion table information indicating the correspondence between the emotion information and the color information;
A color conversion processing unit that obtains color information corresponding to the emotion information from the emotion analysis processing unit using the conversion table information of the conversion table information holding unit;
With
Using the color information from the color conversion processing unit, the human emotion obtained by the emotion analysis processing unit is visualized by color.

  According to the first aspect of the present invention, since the emotion information representing the emotion of the person is obtained by analyzing the biological information of the person, the objective emotion information of the person can be accurately obtained. Since the emotion information is converted into color and displayed, for example, the darker “red” indicates a more comfortable emotion, and the darker “blue” indicates an unpleasant emotion. This makes it possible to visualize the emotional change as the color change and to display the emotional state distribution by the color distribution state.

The invention according to claim 2 is the emotion visualization device according to claim 1,
The emotion analysis processing unit analyzes the biological information from the plurality of biological information acquisition means by averaging each predetermined unit time, and obtains emotion information representing the average emotion for each unit time. ,
The color conversion processing unit outputs a color corresponding to emotion information representing an average emotion per unit time from the emotion analysis processing unit, using the conversion table information of the conversion table information holding unit. It is characterized by being.

  According to the second aspect of the present invention, since the average emotion per unit time is visualized as a color, the main emotion within the unit time from which fine emotional changes are removed is visualized. State judgment becomes easy.

The invention according to claim 3 is the emotion visualization device according to claim 1,
By displaying the color output based on the color information of the predetermined period from the color conversion processing unit in correspondence with the passage of time in the predetermined period, the change of emotion in the predetermined period is displayed in color. It is characterized by providing a means for visualizing emotional changes.

  According to the third aspect of the present invention, since a change in emotion in a predetermined period, for example, a specific day is visualized as a change in color in response to a change in time, It is possible to visualize whether it was spent in a state. In addition, when the predetermined period corresponds to, for example, a period in which a predetermined broadcast program or movie content is viewed, the change in emotion when the predetermined broadcast program or movie content is viewed Can be visualized.

The invention according to claim 11 is the emotion visualization apparatus according to claim 1,
Emotion distribution visualization means for visualizing the state distribution of the emotion within the predetermined period by visualizing a ratio of each color output by the color information from the color conversion processing unit for a predetermined period. Features.

  According to the invention of claim 11, the state distribution of emotions in a predetermined period, for example, a specific day is visualized as a color distribution, and the state of emotions spent on the day is visualized. Can do.

  In addition, when the predetermined period corresponds to, for example, a period during which a predetermined broadcast program or movie content is viewed, the state distribution of emotion when the predetermined broadcast program or movie content is viewed It can be visualized as a color distribution. Accordingly, it is possible to visualize in what emotional state the broadcast program or movie content is viewed.

  According to the present invention, human emotions can be accurately reflected, and emotional changes and emotional state distributions can be visualized and expressed in color in a visible manner.

  Hereinafter, embodiments of an emotion visualization device, an emotion visualization method, and an emotion visualization output product according to the present invention will be described with reference to the drawings.

  In the embodiment described below, an emotion visualization apparatus and method are applied to an imaging information recording / reproducing apparatus. The imaging information recording / reproducing apparatus as an example of the emotion visualization apparatus according to this embodiment includes an imaging unit, captures the photographer's biological information when the imaging unit captures an image, analyzes the emotion, and analyzes the emotion. The result is color-converted into visualized data, and has a function of recording the image data in association with time information.

<Hardware configuration example of visualization device (imaging information recording / reproducing device)>
FIG. 1 is a block diagram of an imaging information recording / reproducing apparatus 100 as an example of a visualization apparatus of this embodiment.

  The imaging information recording / reproducing apparatus 100 includes a computer in which a ROM (Read Only Memory) 103 and a RAM (Random Access Memory) 104 are connected to a CPU (Central Processing Unit) 101 via a system bus 102. It is configured to be equipped.

  Then, with respect to the system bus 102, an imaging unit controller 105, an imaging signal processing unit 106, a biological information input unit controller 107, an emotion analysis processing unit 108, a data storage unit controller 109, a reproduction processing unit 110, A timer unit 111, a visual data processing unit 112, a display unit controller 113, a key input interface (in the figure, the interface is described as I / F; the same applies hereinafter) 114 and a network interface 115 are connected.

  The imaging unit controller 105 supplies an imaging unit control signal to the imaging unit 121 based on control from the CPU 101 according to a program stored in the ROM 103.

  The imaging unit 121 includes a lens system and an imaging device configured by, for example, a CCD (Charge Coupled Device), receives an imaging unit control signal from the imaging unit controller 105, and starts and stops imaging recording, zoom magnification, The exposure value, auto focus, etc. are controlled.

  The captured image signal from the imaging unit 121 is converted into a digital signal by an A / D (Analog to Digital) converter 122 and then supplied to the imaging signal processing unit 106. In the imaging signal processing unit 106, processing is executed by a control instruction command from the CPU 101 according to a program stored in the ROM 103, and an imaging image signal from the A / D converter 122 is converted into, for example, an MPEG (Moving Picture Experts Group). ) Image data is compressed by the method and converted into an appropriate recording format. Further, the imaging signal processing unit 106 uses the time information from the time measuring unit 111 to perform processing for adding imaging time information to the captured image data.

  Then, the captured image data including the information on the imaging time, which has been converted into an appropriate recording format by the imaging signal processing unit 106, is transferred to the data storage unit controller 109 through the system bus 102. The data storage unit controller 109 stores the received captured image data in the data storage unit 123.

  In this example, the data storage unit 123 is composed of a hard disk drive, and has a storage capacity for storing data for one day or more, for example. The data storage unit 123 may be a device that uses a large-capacity recording type optical disc such as a DVD (Digital Versatile Disc) or a semiconductor memory as a storage medium.

  The biological information input unit controller 107 receives the biological information from the biological information input unit 124 through the A / D conversion unit 125. As will be described in detail later, the biological information input unit 124 constitutes means for acquiring biological information of the user (photographer) of the imaging information recording / reproducing apparatus 100.

  Here, the biological information in this specification refers to information on physiological, biochemical, or physical reactions or states. Biological information includes, for example, information spontaneously and evoked by humans, such as heartbeat, pulse wave, respiration, blood pressure, electrocardiogram, electroencephalogram, skin sweating, skin resistance, body movement, body position, magnetoencephalogram, myoelectricity. , Body surface temperature, pupil size, microvibration, blink (blink), biochemical reaction, and the like.

  Therefore, human biological information is not only detected through electrodes attached to humans, but also detected from information captured by a camera such as body movement and body position, and sensors such as body surface temperature. Includes those that can be detected remotely.

  The biometric information of the user acquired by the biometric information input unit 124 is converted into a digital signal by the A / D conversion unit 125, and is appropriately recorded by the biometric signal input unit controller 107, the CPU 101, the ROM 103 program, and the work area RAM 104. The data is converted into a format and stored in the data storage unit 123 by the data storage controller 109. At this time, corresponding to the captured image data, the biometric information is stored in the data storage unit 123 to which the information on the acquisition time is added.

  The biometric information acquired by the biometric information input unit 124 is also converted into a digital signal by the A / D conversion unit 125 and then supplied to the emotion analysis processing unit 108. Here, as described later, a predetermined method is used. Emotion analysis is performed accordingly. Then, information (hereinafter referred to as emotion data) representing the user's emotion obtained as a result of the emotion analysis in the emotion analysis processing unit 108 is converted into an appropriate recording format by the CPU 101 and the RAM 104 according to the program of the ROM 103. The data is converted and stored in the data storage unit 123 by the data storage controller 109. Time information is also added to the emotion data, and correspondence with captured image data and biological information is possible.

  Note that the emotion data is not generated from the biological information in real time as described above, but after the storage of the imaging information and the biological information in the data storage unit 123 is completed, the biological data stored in the data storage unit 123 is stored. It is also possible to read out information, analyze emotions, and store the analysis result in the data storage unit 123. In this case, however, the emotion display in real time as described later cannot be performed.

  In the above description, the captured image data of the subject acquired by the imaging unit 121, the user's biological information acquired through the biological information input unit 124, and the emotion obtained by analyzing the biological information by the emotion analysis processing unit 108. Although it has been described that the emotion data representing is stored and stored in the data storage unit 123, the present invention is not limited to this.

  For example, it connects to a network such as the Internet through the network interface 115, transmits it to a predetermined server device provided on the network, and causes the server device to store and hold captured image data, biological information, and information indicating an emotional state. It is also possible to do so.

  As will be described in detail later, the emotion analysis processing unit 108 analyzes various biological information acquired through the biological information input unit 124 or biological information stored in the data storage unit 123, in this example, an emotion model. To extract emotional state.

  The reproduction processing unit 110 performs image data for reproduction by performing decompression decoding processing on the compressed image data read from the data storage unit 123 through the data storage unit controller 109 during reproduction. The image data processed in the reproduction processing unit 110 is supplied to the visual data processing unit 112.

  The visual data processing unit 112 is stored in the color conversion table information storage unit 126 from the emotion data from the emotion analysis processing unit 108 or the emotion data stored in the data storage unit 123, as will be described in detail later. It has a function of generating emotion visualization data using the conversion table information and storing the generated emotion visualization data in the data storage unit 123 through the data storage unit controller 109. As will be described later, the visual data processing unit 112 also includes role functions of emotion change visualization means and emotion distribution visualization means.

  Further, at the time of reproduction, the visual data processing unit 112 reproduces the captured image data and emotion visualization data read from the data storage unit 123 in synchronization with the time axis, and through the display unit controller 113, the visual data processing unit 112 reproduces the captured image data and emotion visualization data. It has a function to be displayed on the screen. At the time of reproduction, the captured image data read from the data storage unit 123 is supplied to the visual data processing unit 112 after being decompressed and decoded by the reproduction processing unit 110 as described above.

  The display controller 113 forms a display image signal to be supplied to the display unit 127 and supplies this to the display unit 127. The display unit 127 is a display such as an LCD (Liquid Crystal Display), and displays an image corresponding to an image signal from the display unit controller 113 on the display screen.

  A key input operation unit 128 is connected to the key input interface 114. The key input operation unit 128 includes keys such as an imaging recording start key, a reproduction start key, a fast forward key, a rewind key, a jump key, and a stop key. When any key of the key input operation unit 128 is operated, the operation input information is supplied to the system bus 102 through the key input interface 114. The CPU 101 determines the operation input information, determines which key is operated, and executes processing corresponding to the operated key.

  Although not shown in FIG. 1, a microphone is attached to the imaging unit 121. During imaging, sound around the subject is picked up by the microphone, A / D converted, and data compressed as necessary. The data is stored and recorded in the data storage unit 123 together with the captured image data.

<Configuration Example of Portion of Biological Information Input Unit 124 to Biological Signal Input Unit Controller 107>
Here, a more specific configuration example of the biometric information input unit 124, the A / D conversion unit 125, and the biosignal input unit controller 107 described above will be described. FIG. 2 is a block diagram illustrating a detailed configuration example of the biometric information input unit 124, the A / D converter unit 125, and the biometric information input unit controller 107.

 Emotion data representing human emotion can be obtained using only one appropriate biological information, but in order to obtain emotion data representing human emotion more accurately, in this example, a plurality of emotion data are represented. Use biological information. In the example of FIG. 2, a case where brain waves, electrocardiograms, pulse waves (pulses), body temperature, and the like are detected through electrodes (including sensors) as user biometric information will be described as an example.

  As shown in FIG. 2, the living body information input unit 124 of this example is worn by a person and corresponds to a plurality of living body information, in this example, brain waves, electrocardiograms, pulse waves, body temperature, myoelectricity, and the like. , 1241e and a plurality of electrodes 1241a, 1241b, 1241c, 1241d,..., 1241e for collecting biological signals, and for amplifying the biological signals from the respective electrodes 1241a, 1241b, 1241c, 1241d,. The plurality of amplifiers 1242a, 1242b, 1242c, 1242d,.

  The electrodes 1241a, 1241b, 1241c, 1241d,..., 1241e have a function of measuring bioelectric phenomena including brain waves and biomagnetic phenomena including brain magnetism. Worn by the person. The amplifying units 1242a, 1242b, 1242c, 1242d,..., 1242e may perform not only amplification but also filtering processing such as dividing into several types of frequency bands as necessary.

  For example, if the biological information is an electroencephalogram, the amplifying unit 1242a amplifies the biological information signal having a potential of several microvolts obtained from the electrode 1241a and divides it into components of several frequency bands such as an alpha wave and a beta wave. In this way, the filtering process is performed.

  Each biological information detected through the electrodes 1241a, 1241b, 1241c, 1241d,..., 1241e is supplied to the corresponding amplifiers 1242a, 1242b, 1242c, 1242d,. This is supplied to the conversion unit 125. In this embodiment, the A / D converter 125 includes a plurality of A / D converters 125a, 125b, 125c, 125d,..., 125e corresponding to the supplied biological information, as shown in FIG. The biometric information (analog signal) supplied is converted into a digital signal.

  For example, the potential level of the electroencephalogram is generally about 50 μV (microvolt), and the potential level of the electrocardiogram is about 1 mV (millivolt), which varies depending on the type of biological signal. Therefore, the number of quantization bits at the time of A / D conversion in each of the A / D converters 125a, 125b, 125c, 125d,..., 125e is such that it does not fall below the potential level of the various biological signals amplified. It is made to have accuracy.

  Each biometric information after A / D conversion is supplied to the biometric information input unit controller 107. The biometric information input unit controller 107 receives the various biometric digital signals, and receives the biometric information by the emotion analysis processing unit 108, the data storage unit 123, the display unit 127, and the reproduction processing unit 110. A biological signal processing unit 1072 that performs processing for making a usable signal and a transmission unit 1073 that outputs the processed biological signal to the bus 102 are configured.

  The receiving unit 1071 receives digital biometric information having a sampling frequency at an interval at which the original biosignal is reproduced, which is output from the A / D converters 125a, 125b, 125c, 125d,. This is supplied to the biological signal processing unit 1072 in the subsequent stage.

  The biological signal processing unit 1072 receives the digital biological information supplied from the receiving unit 1071, and for each biological information, for example, filter operation processing for amplitude adjustment and band pass, time including FFT and wavelet analysis, etc. Frequency analysis or nonlinear analysis such as neural network or chaos analysis is performed to convert the signal into a format that can be used by the emotion analysis processing unit 108, the data storage unit 123, the display unit 127, and the reproduction processing unit 110. Then, the converted biological information (biological signal) is supplied to the transmission unit 1073.

  The transmission unit 1073 outputs the biological signal formed by processing as described above to the bus 102 at appropriate timing sequentially, and records it in the data storage unit 123 through the data storage unit controller 109 as described above. Or, it is supplied to the emotion analysis processing unit 108 to perform analysis of biological information.

<Example of emotion analysis in emotion analysis processing unit 108>
Next, an example of emotion analysis processing performed in the emotion analysis processing unit 108 will be described. As described above, the emotion analysis processing unit 108 performs digitization obtained by performing necessary signal processing and analysis processing through the biological information input unit 124, the A / D conversion unit 125, and the biological information input unit controller 107. The plurality of pieces of biological information that have been input are input.

  The biometric information once stored in the data storage unit 123 is read out to the emotion analysis processing unit 108 through the data storage unit controller 109, not only when biometric information is supplied from the biometric information input unit controller 107 through the bus 102. In some cases, this is supplied through the bus 102. In addition, the emotion analysis processing unit 108 can be supplied with biometric information stored in an external device or an external server through the network interface 115.

  The emotion analysis processing unit 108 receives input of a plurality of pieces of biological information, performs analysis processing based on the plurality of pieces of biological information, and outputs a plurality of emotion parameter values defined by the emotion model as emotion data. Examples of the emotion model include a 2-axis model, a 3-axis model, and a 6-axis model. An example of these models is shown in FIG.

  The two-axis model is the simplest model, and as shown in FIG. 3A, as emotions, for example, a first axis indicating a degree closer to comfort or discomfort, excitement or tension and relaxation. An emotion model is formed by two axes, the second axis indicating the degree of which is closer. Each emotion state in the two-dimensional emotion space determined by the two axes is expressed by an emotion parameter value composed of the parameter (coordinate value) in the first axis direction and the parameter (coordinate value) in the second axis direction.

  As shown in FIG. 3B, the three-axis model is closer to emotion, for example, a first axis indicating the degree of comfort or discomfort, and excitement, tension, or relaxation. An emotion model is formed by the three axes of the second axis that indicates the degree of movement and the third axis that indicates the degree of closer to tension or relaxation, and the emotion parameter is composed of coordinate values in the three-axis space. It expresses the state of emotion.

  As shown in FIG. 3C, the six-axis model includes a first axis indicating the degree of anger, a second axis indicating the degree of fear, a third axis indicating the degree of sadness, and a fourth axis indicating the degree of joy. Then, an emotion model is formed by six axes including a fifth axis indicating the degree of disgust and a sixth axis indicating the degree of surprise, and the emotional state is expressed by emotion parameters including coordinate values in the three-axis space. Is.

  The emotion analysis processing unit 108 uses, for example, any of the emotion models described above. For example, a three-layer back propagation or self-organization map (SOM: Self Using neural network techniques such as Organizing Map, the emotions of the person who is the source of the biological information are analyzed from multiple types of biological information received as input, and the analysis results are obtained. An emotion parameter value is obtained as emotion data representing emotion.

  As described above, the emotion analysis processing unit 108 of this embodiment is a provider of biological information by analyzing a plurality of types of biological information using a predetermined emotion model and a neural network technique. It becomes possible to grasp the emotional state of the user as an emotion parameter value.

  Then, as described later, the emotion parameter value of the emotion analysis result is converted into color information output by using the conversion table information in the color conversion table information storage unit 126 in the visual data processing unit 112, and corresponds to time information. And visualize. As a result, for example, it is possible to easily grasp the period in which the feeling of joy appears, the period in which the feeling of sadness appears, the period in which the feeling of anger appears, and the period in which the feeling of fear appears. , Those emotional changes can be easily grasped as color changes.

  Then, as will be described later, for example, the distribution of the emotions of the user in a predetermined period such as one day or one month, i.e., what percentage of pleasure feelings and what percentage of sadness feelings, It becomes possible to grasp by the area of the corresponding color.

  As specific examples of the analysis of the emotional state using the biological information, various types can be considered as follows. For example, for evaluation of the relaxed state, an electroencephalogram can be used as biological information. In general, it is known that the ratio of the frequency component of the α wave and the frequency component of the β wave is taken, and the more the ratio of the α wave component is, the more relaxed the state is. It is also possible to make an evaluation based on the stability of the cycle of breathing and heartbeat and the amount of increase in pulse amplitude.

  For surprising evaluation, the electrical resistance value of the skin can be used. In general, it is known that the electrical resistance value of the skin increases due to surprise, fear, or worry. By this electrical resistance value of the skin, the evaluation of surprise, which is one of emotional states, can be performed.

  When joy is evaluated, it can be evaluated using heart rate, skin temperature (body temperature), and electrocardiogram (electrocardiogram frequency). That is, it is known that when the emotion of joy is great, the heart rate decreases, the skin temperature increases, and the electrocardiogram frequency increases.

  In general, when mentally and psychologically stable, the peripheral blood vessels dilate and arterial blood flows into the peripheral blood vessels, so the heart rate and pulse change slowly, and the body temperature tends to rise It is in. On the other hand, in a high mental and psychological tension state, peripheral blood vessels contract and arterial blood flows into the peripheral blood vessels, so the heart rate and pulse rate increase, and the body temperature tends to decrease. It is possible to evaluate and grasp the emotional state according to such physiological changes.

  In addition to the physiological indices as described above, the combined use of myoelectric changes by facial expressions, facial expression recognition by images, posture recognition by acceleration sensors, tilt sensors, etc., voice recognition, etc. The emotional state may be evaluated.

  In other words, it is possible to grasp whether the person is crying, laughing, or angry by changing the myoelectricity on the human face. It is possible to know whether you are laughing or angry.

  It can also be used as information when evaluating emotional states by judging the user's posture by using an acceleration sensor or tilt sensor and detecting whether the chest is worn or bent forward. Is possible.

  Moreover, it is also possible to use the result of speech recognition as information when evaluating the emotional state. For example, when you are happy, the voice of the person is relatively high, and when you are depressed, the voice of the person tends to be relatively low. Thus, it is possible to evaluate the emotional state. It is also possible to evaluate the emotional state based on the magnitude (level) of the audio signal.

  For example, as an emotion data analysis technique, Japanese Patent Laid-Open No. 8-117199 discloses an analysis of arousal and mood by a neural network using α wave fluctuation of an electroencephalogram. This is based on the knowledge that two axes of arousal level and mood are basic indicators in psychology and psychophysiology, and quantitative analysis is performed.

  In addition, facial expression recognition has been reported in the NICOGRAPH papers pp.17-25, 1993, and in the IEICE Transactions D-II Vol.J77-D-II No.8 pp.1512-1521. Has been reported to recognize user's emotions from non-verbal information such as phonetic information. Of course, the user's emotional state may be grasped using such various analysis methods.

  As described above, the imaging information recording / reproducing apparatus 100 according to this embodiment can continuously detect user's biological information and analyze the detected biological information to grasp the user's emotion. Then, emotion data (emotion parameter value) representing the grasped user emotion can be obtained, and the visual data processing unit 112 can color-convert the emotion parameter value as described later to visualize the emotion as a color output. .

<Overview of visualization by emotion color conversion>
In this embodiment, the emotion analysis processing unit 108 adopts the above-described two-axis model as an emotion model, and detects emotion parameter values using the first axis as the X coordinate axis and the second axis as the Y coordinate axis.

  In this example, the color conversion table information storage unit 126 stores color conversion table information in the case of the biaxial model described above as the emotion model. The color conversion table information in this example will be described with reference to FIG. That is, FIG. 4 shows color output when emotion parameter values are output as color information based on this color conversion table information, but in FIG.

  In creating the color conversion table information, a color that can be perceived emotionally by appearance is used as a color that is converted and output for emotion. An example of this color conversion table information will be described below.

  That is, the color conversion table of FIG. 4 will be described. In the X-axis direction, comfort is expressed in yellow and discomfort is expressed in blue. The more comfortable, the darker yellow, and the more unpleasant, the darker blue. Accordingly, in the X-axis direction, the color changes in the order of dark blue → light blue → white → light yellow → dark yellow along the direction from uncomfortable to comfortable.

  In the Y-axis direction, excitement and tension are expressed in magenta, and relaxation is expressed in blue. And the more excited and tense, the more pure magenta, and the more relaxed, the more pure blue, and between excitement, tension and relaxation, from relaxation to excitement, along the direction of tension. From the pure blue, the ratio of the magenta color mixture gradually increases to change to pure magenta.

  The state of excitement, tension and comfortable emotion (corresponding to the upper right corner position on the color conversion table in FIG. 4) is assigned red as a result of subtractive color mixture of magenta and yellow. As the color change from the lower right corner to the upper right corner in FIG. 4, the ratio of the mixed color of red gradually increases from pure yellow to pure red. Here, the mixed color of yellow and red is orange.

  Similarly, the color change from the upper left corner to the upper right corner in FIG. 4 is as follows: dark magenta color → magenta color that gradually fades with blue → red that gradually fades with yellow → dark red It will change.

  At this time, the emotion analysis processing unit 108 obtains parameter values (coordinate values) in the X-axis direction and parameter values (coordinate values) in the Y-axis direction as emotion parameter values. The visual data processing unit 112 converts the color information determined by the parameter values in the color conversion table space of FIG. This color information is output as emotion visualization data.

  The visual data processing unit 112 does not output the color information obtained by converting the emotion parameter value using the color conversion table information as it is, but changes the emotion corresponding to the passage of time in chronological order. The output is performed in a display output manner that can be understood (hereinafter, this type of display output manner is referred to as a time-series display type). This function corresponds to the emotion change visualization means.

  Further, in this example, the visual data processing unit 112 determines how much of each emotion such as excitement, relaxation, comfort, and discomfort is included in the predetermined time. It is also possible to output in a display output mode (hereinafter, this type of display output mode is referred to as a distribution display type) represented by a color distribution corresponding to each emotion in the region. This function corresponds to the emotion distribution visualization means.

  5A, 5B, and 5C show some examples of the time series display type, and FIG. 5D shows an example of the distribution display type.

  In the time-series display type, color information corresponding to instantaneous emotions may be displayed sequentially, but this causes too many subtle emotional changes to be expressed. It may be difficult.

  Therefore, in the examples of FIGS. 5A, 5B, and 5C, the predetermined predetermined unit time is set as the averaging interval, and the emotion data obtained in the unit time interval of the averaging interval is averaged. The averaged emotion data is converted into color information using a color conversion table. Therefore, in the examples of FIGS. 5A, 5B, and 5C, the color information output corresponding to the emotion data is performed in a predetermined width unit corresponding to the unit time interval.

  FIG. 5A shows a slide bar type among time series display types, and a color display area based on color information corresponding to the emotion data for each unit time interval is one-dimensionally represented as a rectangular area. It expands and visualizes the change of the emotion for each unit time interval.

  In the example of FIG. 5A, when time elapses in the horizontal direction, color information corresponding to emotion data in each unit time interval is drawn as a vertical bar along the time elapse. It is a display mode. In this slide bar type, for example, when a captured image is reproduced, it is displayed in the lower or upper column of the display image, and the currently displayed time position is indicated by a bright spot or the like on the slide bar. For example, it is possible to easily know what emotion the user has in the scene of the screen.

  FIG. 5B shows a block type among time-series display types. A color display area based on color information corresponding to the emotion data for each unit time interval is two-dimensionally represented as a rectangular area. And the change in the emotion for each unit time interval is visualized.

  That is, this is a display mode in which a predetermined period in which emotion is to be displayed is equally divided into a plurality of divided time sections (unit time sections), and the divided time sections are displayed in a plurality of stages. This block type is convenient, for example, when the emotional change of the day is expressed, and when the horizontal bar is displayed in four rows every 6 hours and displayed for 24 hours.

  FIG. 5C shows a circle type among the time-series display types. A predetermined period in which emotion is to be displayed is assigned to one round of the annular area, and the color output corresponding to emotion is This is a display mode in which images are sequentially displayed in a predetermined width unit corresponding to the unit time interval in the circumferential direction.

  5 (E) and 5 (F) show an artistic representation or a metallic representation with a special effect on the visualized image by color conversion output in the created block type or circle type display mode. It is a thing. For example, a process of smoothing a change part for each unit time interval of emotion using blurring is performed. In addition, a metallic expression is made by adding a highlight or a shadow to a portion with a large color change corresponding to the peak point of emotion change. Further, it may be converted into an artistic expression by performing a geometrical transformation.

  FIG. 5D shows an example of a distribution display type. In this case, a map corresponding to an emotional state (expressed as an emotion) experienced within a predetermined period in which the emotion is to be displayed is created and expressed. This is created using a self-organizing map.

  According to this distribution display type display mode, it is possible to know which emotion is dominant in a predetermined period. For example, in the distribution display type display, when the portion expressed in yellow dominates and the rest is in red or magenta, it was a feeling of excitement and tension, but it was a pleasant one You can see that there was.

  FIG. 6 is a flowchart for explaining visual data creation processing in the visual data processing unit 112.

  In the example of FIG. 6, the visual data processing unit 112 converts emotion data generated by the emotion analysis processing unit 108 stored in the data storage unit 123 into emotion visual data (emotion visualized image) using color conversion table information. This is an example of generating visualization data by converting to (data).

  First, initial setting is performed (step S101). The contents of this initial setting are selection setting of display type of visualization information of emotion data to be created, setting of length of averaging interval, setting of other attribute data, and the like.

  Next, first, a unit time interval for the first averaging interval is set (step S102). Next, the emotion data within the set unit time interval is averaged, and the emotion parameter value of the average emotion within the unit time interval is calculated (step S103). Next, color conversion output information for the emotion parameter value calculated in step S103 is obtained using the color conversion table information in the color conversion table information storage unit 126 (step S104).

  Next, it is determined whether or not the type of display output mode is a time series display type (step S105), and when it is determined that it is a time series display type, one of the time series display types described above is set. In the display graphic region corresponding to the display type, image data is generated so as to draw the color based on the color conversion output information obtained in step S104 at a position corresponding to the corresponding unit time interval. Temporary storage is performed corresponding to the unit time interval (step S106).

  Then, it is determined whether or not the processing for all the emotion data to be processed has been completed (step S107). When it is determined that all the emotion data has not yet been completed, the unit for the next averaging interval is determined. A time interval is set (step S109), and the processing after step S103 is repeated. If it is determined in step S107 that the processing for all emotion data has been completed, the generated emotion visual data including the image data temporarily stored is stored together with the attribute data (step S108).

  If it is determined in step S105 that the display type is not the time series display type but the distribution display type, the color conversion output information obtained in step S104 is temporarily stored in correspondence with the unit time interval (step S110).

  Then, it is determined whether or not the processing for all the emotion data to be processed has been completed (step S111), and for all the emotion data, when it is determined that the processing has not yet been completed, a unit for the next averaging interval is determined. A time interval is set (step S109), and the processing after step S103 is repeated. If it is determined in step S111 that the processing for all emotion data has been completed, all of the color conversion output information corresponding to the temporarily stored emotion data is used, and the emotions in the target predetermined period are used. Emotional visual data indicating the state distribution is generated and displayed (step S112). Then, the generated emotion visual data (emotion visualized image data) is stored together with the attribute data (step S108). Thus, the emotion visual data generation process is completed.

<Description of recording processing operation>
Next, the flow of recording processing as the imaging information recording / reproducing apparatus 100 will be described with reference to the flowchart of FIG. Each step of the flowchart described below is executed by the CPU 101 using the RAM 104 as a work area in accordance with a program stored in the ROM 103.

  That is, when an instruction to start imaging information recording is given, initialization processing of the time information of the time measuring unit 111, the biological signal processing unit 1072, the emotion analysis processing unit 108, the display unit 127, and the like of the biological information input unit controller 107 is performed. (Step S201). Next, the setting such as designation of the display type of emotion visual data by the user is received and the setting is executed (step S202).

  The captured image data and audio data from the imaging unit 121 are sequentially added to the data storage unit 123 by the data storage unit controller 109 in a format to which time information (year / month / day, hour / minute / second) from the time measuring unit 111 is added. Store (step S203).

  In parallel with this, the biological information input unit controller 107 analyzes the biological information acquired by the biological information input unit 124 and converted into a digital signal by the A / D conversion unit 125 (step S204). Then, the biometric information from the biometric information input unit controller 107 is transferred to the emotion analysis processing unit 108 for emotion analysis, and the emotion data (emotion parameter value) as a result of the analysis is sent together with the time information by the data storage unit controller 109. The data is stored in the data storage unit 123 (step S205).

  In this step S205, emotion data (emotion parameter value) may be calculated for each piece of biological information from the biological information input unit controller 107. In this embodiment, the description has been given with reference to FIG. Similarly to the above, the emotion data is averaged at predetermined time intervals (unit time) to calculate emotion data (emotional parameter values) at the predetermined time intervals. The fixed time (unit time) in this case is a fixed time shorter than the time of the averaging interval set in FIG. Then, emotion data is stored in the data storage unit 123 in association with the time information in a certain time unit.

  In addition, the emotion data of the analysis result in the emotion analysis processing unit 108 is sent to the visual data processing unit 112, and the emotion data is converted into color information using the color conversion table information in the color conversion table information storage unit 126. Is displayed on the screen of the display unit 127 by the display unit controller 113 (step S206). In this display, the emotional state of the user at the time of recording information while imaging is displayed on the display unit 127 in real time.

  Then, it is determined whether or not a recording end operation input has been made by the user (step S207). When it is determined that a recording end operation input has not been made, the process returns to step S203, and the processing from step S203 onward is repeated.

  If it is determined in step S207 that the user has input the recording end operation, the recorded emotion data in all the recorded time intervals are read from the data storage unit 123 through the data storage unit controller 109 and shown in FIG. In this manner, the emotional visual data of the display type set in step S202 is created and stored in association with the information captured and recorded in the data storage unit 123 (step S208).

  At this time, in step S208, the small icon display image is formed by displaying the emotions in all the recorded time intervals in the designated display type, as described above, in the time series display type or the distribution display type. Information is also generated and stored in the data storage unit 123 in association with information captured and recorded.

  An icon displayed by the icon display image information (hereinafter referred to as an emotion display icon) is displayed as an index for each imaging record information on a list screen of recorded imaging record information at a later time. Can do. The user can infer the emotion of the user when the corresponding imaging record information is recorded from the emotion display icon of the index, which is convenient.

  When the data storage unit 123 uses a removable recording medium such as a DVD, for example, the emotion display icon is printed in color on a sticker or recording paper, and is attached to a DVD or in a jacket. It can be enclosed. Thus, by looking at the printed emotion display icon at a later point in time, it is possible to infer from the emotional state of the user when the imaging information recorded on the DVD is recorded, which is convenient.

  Then, management information that associates the captured image data and audio data, emotion data, emotion visual data, and the icon display image information is stored in the data storage unit 123 (step S209).

  The relationship between this management information and each data is shown in FIG. That is, in the management information, the title, the owner, the disclosure attribute, the recording start date and time, and the recording time are recorded for the entire captured and recorded information. For emotional visual data, the address ( (Location information), data attributes, and display type are recorded. For captured image data and audio data, their addresses (location information) and data attributes are recorded, and for emotion data, their addresses (location information). Information attributes, emotion models, emotion parameters, and sampling intervals are recorded.

  The title, the owner, and the like serve as identification information about the entire imaging record information. However, in addition to the title, the owner, and the like, identification information such as an identification number may be added to the management information. In this example, the management information as shown in FIG. 8 is used to store the various types of information shown in FIG. 8 in association with each other. It is not limited to a simple management method. For example, each piece of data may be stored separately, and identification information regarding the above-described imaging record information may be added to each piece of stored data so as to be associated with each other.

  In addition, in the case of the above-described example, the emotion visual data address, the captured image data and audio data addresses, and the emotion data address are addresses in the data storage unit 123. An address on the DVD is used, but when the data is stored in the server device via the network, the URL (Uniform Resource Locator) and the memory address of the server device are used.

  In the above description, the emotional visual data is always generated after the recording is completed, but is stored in the data storage unit 123 only when the user gives an instruction after the recording of the imaging information is completed. The emotion data may be read and generated.

<Reproduction of imaging information and emotional visual data>
Next, with reference to FIG. 9, a reproduction process for the imaging information and emotional visual data recorded as described above will be described. 9 is also executed by the CPU 101 using the RAM 104 as a work area in accordance with a program stored in the ROM 103.

  First, the management information stored in the data storage unit 123 is read (step S301). Next, the captured image data is read from the data storage unit 123, supplied to the display unit 127 through the reproduction processing unit 110 and the display unit controller 113, displayed on the display screen, and reproduced. Although not shown, related audio data is read from the data storage unit 123, supplied to a speaker through a reproduction processing unit and an amplifier, and is reproduced acoustically. At this time, the time information about the captured image data and audio data currently being reproduced is managed (step S302).

  Next, emotion visual data is read from the data storage unit 123 and displayed on a part of the image of the captured image data on the screen of the display unit 127. Alternatively, the screen of the display unit 127 may be divided so that the image based on the captured image data and the emotion visualization display image based on the emotion visual data data are displayed in different areas (step S303).

  At this time, in the displayed emotion visualized display image, the time position of the captured image data and audio data currently being reproduced corresponds to which position in the emotion visualized display image corresponds to the bright spot or the like. It is displayed and can be referenced.

  FIGS. 10A to 10D show emotion display images of each display type at this time, and correspond to the currently displayed captured image data depending on the color of the bright spot position indicated by CS. Shows emotional state. That is, the bright spot CS indicates an index of the emotional state corresponding to the captured image data. In the case of the time series display type shown in FIGS. 10A to 10C, the position of the bright spot CS indicates the time position together. Then, the position of the bright spot CS changes so as to move to the emotion data position at the corresponding time as the playback proceeds with time.

  During playback, when the viewing user does not perform any operation, the operations in steps S302 and S303 described above are continued until the end of the playback data, and playback processing is continuously performed. However, when the user performs an event operation for changing the playback position such as fast forward, rewind, and jump through the key input operation unit 128, the CPU 101 executes a process corresponding thereto.

  In this embodiment, the bright spot CS on the emotion visualized image is regarded as a cursor mark, and the position of the bright spot CS can be changed with the cursor key. Control is performed so that captured image data and audio data at the time axis position are reproduced.

  That is, the CPU 101 determines whether any event has occurred through the key input interface 114 (step S304). If it is determined that no event has occurred, the CPU 101 returns to step S302 and repeats the above operation.

  If it is determined in step S304 that some event has occurred, it is determined whether or not the operation is a playback position change event operation such as fast forward, rewind, or jump (step S305). When it is determined, the reproduction position of the captured image data and the audio data is changed, and the position of the bright spot CS in the emotion visualized image is changed to a position corresponding to the changed reproduction position (step S306). Then, it returns to step S302 and repeats the above-mentioned operation.

  When it is determined in step S305 that the event is not an operation for changing the reproduction position, it is determined whether or not the position change of the bright spot CS on the emotion visualized image by the cursor key is specified (step S307). When it is determined that the position change is specified, time information is searched from the position of the bright spot CS on the emotion visualization image, and the captured image data and audio data at the corresponding time position are read from the data storage unit 123 and reproduced ( Step S308).

  If it is determined in step S307 that the position change of the bright spot CS is not specified, it is determined whether or not the stop key has been operated (step S309). If it is determined that a key other than the stop key has been operated, the operation is performed. The event process corresponding to the key is performed (step S310), and the process returns to step S302. If it is determined in step S309 that the stop key has been operated, this reproduction processing routine is terminated.

<Application by storing emotion data or emotion visual data>
As described above, if the data stored in the data storage unit 123 can be accumulated over a long period of time such as one year, two years, three years, etc., what will happen to the past day, month, year? It is possible to visualize and display what was spent in a state of emotion. The example described below takes this into consideration.

  FIG. 11 is a block diagram illustrating an example of a data storage system. The data storage system of this example is configured by connecting a plurality of imaging information recording / reproducing apparatuses 100 to a server apparatus 20 composed of, for example, a personal computer through a network 10.

  The server device 20 includes a visual data processing unit 21 and a color conversion table information storage unit 22 having the same configuration as the visual data processing unit 112 and the color conversion table information storage unit 126 of the imaging information recording / reproducing apparatus 100 described above. A capacity data storage unit 23 is provided. Under the control of the CPU 24, emotion visual data generation processing using the color conversion table information in the color conversion table information storage unit 22 in the visual data processing unit 21 is performed, and data is stored in the large-capacity data storage unit 23. And the reading process is controlled. Note that the server device 20 of FIG. 11 shows only main parts necessary for the description here, and the program ROM, RAM, network interface, and other processing units are omitted.

  In this system, each imaging information recording / reproducing device 100 is configured to access the server device 20 with its own device ID (Identification). Then, each imaging information recording / reproducing apparatus 100 sends the captured image data, audio data, emotion data, and emotion visual data stored in its own data storage unit 123 to the server device 20 together with time information for associating them, It is configured so that it can be accumulated.

  In addition, each imaging information recording / reproducing device 100 can request the server device 20 to generate emotion visualized image information on a yearly, monthly, and daily basis from past accumulated data. The server device 20 is configured to generate emotion visualized image information corresponding to the request and send the generated emotion visualized image information to the requested imaging information recording / reproducing device 100.

  Then, the imaging information recording / reproducing device 100 displays the emotion visualized image based on the emotion visualized image information received from the server device 20 on the screen of the display unit 127, and on the screen, the same as the bright spot CS described above on the emotion visualized image. The reproduction position instruction means can specify the reproduction position of the image and sound. When the reproduction position is designated, the imaging information recording / reproduction device 100 is configured to send the reproduction position designation information to the server device 20.

  When the server apparatus 20 receives the reproduction position designation information, the server apparatus 20 reads out the captured image data and audio data at the reproduction time position indicated by the reproduction position designation information from the data storage unit 23, and requests the imaging information recording / reproduction apparatus 100 that has requested. The function to send to.

  FIG. 12 and FIG. 13 are flowcharts showing processing operations in the server device 20 in this system. The processing of each step in this flowchart is executed by the CPU 24 together with the program ROM and work area RAM.

  That is, first, it is determined whether or not an access from any of the imaging information recording / reproducing devices 100 has been received (step S401). When it is determined that no access has been received, other processing is performed (step S402), and the process proceeds to step S401. Return.

  If it is determined in step S401 that an access from any one of the imaging information recording / reproducing devices 100 has been received, the device ID of the accessed imaging information recording / reproducing device 100 is recognized and authenticated (step S403).

  Then, it is determined whether or not a data storage request has been received from the imaging information recording / reproducing apparatus 100 (step S404). If it is determined that the request is a data storage request, then the imaging information sent from the imaging information recording / reproducing apparatus 100 is received. Image data, audio data, emotion data, and emotion visual data are received together with time information for associating them with each other, and stored in the data storage unit 23 in association with the device ID and the time information (step S405). Then, the process returns to step S401.

  If it is determined in step S404 that it is not a data accumulation request, it is determined whether or not it is a past emotion visual data creation request (step S406). If not, it is determined whether or not reproduction position designation information has been received. (Step S407). If it is determined that the reproduction position designation information has not been received, other processing is performed (step S408), and the process returns to step S401.

  If it is determined in step S407 that the reproduction position designation information has been received, the captured image data and audio data stored in correspondence with the apparatus ID of the requested imaging information recording / reproducing apparatus from the data storage unit 23 are stored. The captured image data and audio data at the reproduction time position corresponding to the designated reproduction position are read from the inside, and sent to the requested imaging information recording / reproducing apparatus 100 (step S409). Then, the process returns to step S401.

  If it is determined in step S406 that a past emotion visual data creation request has been received, it is determined whether the request unit is year, month, or day (step S411).

  If it is determined in step S411 that the request is in units of years, the emotion data stored in correspondence with the device ID of the requested imaging information recording / reproducing device from the data storage unit 23 is stored for the designated year. In this example, the emotion data is averaged on a monthly basis, and emotion data (emotional parameters) for each month is calculated (step S412).

  Next, the calculated emotion data for each month is converted into color information using the color conversion table information (step S413). Next, emotion visualized image data capable of expressing monthly emotional changes is created for each designated year (step S414). Then, the created emotion visualization image data is transferred to the requested imaging information recording / reproducing apparatus 100 (step S415). Then, the process returns to step S401.

  FIG. 14 shows an example of an image based on yearly emotion visualized image data. This example is a slide bar type image of a time series display type, and is a case where 2000 to 2002 is designated as a year. It can be seen that the monthly emotional state of each year is visualized by color output. This allows you to see at a glance what emotional state you spent every month in each year.

  If it is determined in step S411 that the request is in units of months, designation of the year and month to be displayed is accepted (step S416). Then, the emotion data of the designated month in the designated year is selected from the emotion data stored in correspondence with the requested device ID of the imaging information recording / reproducing device from the data storage unit 23. Then, averaging is performed on a daily basis, and emotion data (emotional parameters) for each day is calculated (step S417).

  Next, the calculated emotion data for each day is converted into color information using the color conversion table information (step S418). Next, emotion visualized image data that can express a change in emotion in units of one day for a designated month in a designated year is created (step S419). Then, the created emotion visualization image data is transferred to the requested imaging information recording / reproducing apparatus 100 (step S415). Then, the process returns to step S401.

  FIG. 15 shows an example of an image based on monthly emotion visualized image data. This example is a time-series display type block type image in which 2003 is designated as the year and January to March is designated. It can be seen that the emotional state of each day of each month is visualized by color output. Thus, it is possible to understand at a glance what emotional state each day has spent in the designated fiscal year and month.

  Next, when it is determined in step S411 that the request is in units of days, designation of the year, month, and day to be displayed is accepted (step S420). Then, from the emotion data stored in correspondence with the requested device ID of the imaging information recording / reproducing device from the data storage unit 23, the specified month and the specified day of the specified year In this example, the emotion data is averaged for each hour, and emotion data (emotion parameter) for each hour is calculated (step S421).

  Next, the calculated emotion data for each hour is converted into color information using the color conversion table information (step S422). Next, emotion visualized image data capable of expressing an hourly emotion change for each designated day of a designated month in a designated year is created (step S423). Then, the created emotion visualization image data is transferred to the requested imaging information recording / reproducing apparatus 100 (step S415). Then, the process returns to step S401.

  FIG. 16 shows an example of an image based on daily emotion visualized image data. This example is a time-series display type circle type image in which 2003 is designated as the year and all days in April are designated. It can be seen that the state of emotion in hour units for each day of the specified month is visualized by color output. As a result, it is possible to see at a glance what emotional state each hour has spent on the specified day of the specified year and month. In the case of the example in FIG. 16, it is also possible to know what kind of emotion has changed on a daily basis in the designated month.

  In the above-described example, the emotion data is averaged every month for yearly designation, averaged for every day for monthly designation, and further every hour for daily designation. However, the average period or time length is not limited to this, and may be specified by the user.

  For example, when this system is configured as a home LAN system for home use, the server device can be configured as a personal computer. In this case, an instruction to generate past emotional visual data is used as imaging information. It can be performed not only from the recording / reproducing apparatus but also based on input of a keyboard or mouse of a personal computer.

  In the system of FIG. 11, since the server device 20 stores emotion data of a plurality of persons of the plurality of imaging information recording / reproducing apparatuses 100, the emotions of each of the plurality of persons and the plurality of persons You can also visualize the average emotion.

  For example, when traveling in a group such as a family or a friend, each person records emotion data and emotion visual data based on biological information while performing image recording with the imaging information recording / reproducing apparatus 100. After returning home, emotion data and emotion visual data are stored in the server device 20 through the network 10 together with the image recording data. The server device 20 can generate emotion visualized image data that displays an average emotion of a plurality of people about the trip by, for example, a distribution display type using the accumulated data.

  According to this emotion visualized image, it is possible to visualize and know what the average emotions of the traveled people about the travel were. In addition, the average emotion of the travel participant at each visited place or event during the trip is visualized by color-displaying the average value of the emotion data of the travel participant in the time zone corresponding to the visited place or event. It can be used as reference material for evaluating the quality of travel.

  In addition, since the emotional state of each of a plurality of people can be visualized, various usage forms such as comparing the emotions of other people with their own emotions can be considered.

<Other embodiments>
In the above embodiment, a recording / playback apparatus including an imaging unit acquires the photographer's biological information while recording and recording, records the photographer's emotion data, and generates emotion visual data from the emotion data. In this case, the present invention is applied, but it goes without saying that the present invention is not applied only in such a case.

  For example, when recording while viewing a broadcast program, biometric information of the viewer is collected, emotion data is analyzed from the collected biometric information, and emotion visual data is generated from the emotion data. It is also possible to record the recorded information together with the recorded information and display the emotion visualized image in accordance with the reproduced image when the recorded information is reproduced.

  In that case, after viewing the broadcast program, a time-series display type or distribution display type icon-like emotion visualization image that visualizes the emotion during the period of watching the broadcast program is printed on a recording sheet or sticker. It is also possible to output and record a sticker on the recorded DVD or its jacket, or leave the recording paper in a jacket. If you do so, you can see at a glance what emotions you have when viewing the recorded program by looking at the recording paper and stickers.

  Similarly, when a DVD or the like on which a movie or drama content is recorded is played back and viewed instead of a broadcast program, emotion visual data at the time of viewing is generated and the emotion visualized image is displayed on a sticker or recording paper. As with the broadcast program described above, you can view the recording paper or sticker by sticking a sticker on the DVD or its jacket or leaving the recording paper in the jacket. Therefore, you can see at a glance what emotions you have when watching movies and dramas.

  In addition, not only in the case of emotion visualization with image and sound viewing as described above, but also in the case of emotion visualization when listening to music alone, in the case of emotion visualization when reading a book, It can also be applied when visualizing emotions when viewing.

  Furthermore, not only in the case described above, but also in the case of applying the visualization of the average emotion of a plurality of people, for example, in the case of visualizing and displaying the average emotion of people attending a concert, a theme park, or an event. Applicable. Even in that case, the present invention can be applied to the case of comparing the feelings of oneself and those around.

  Furthermore, it goes without saying that the present invention is not applied only when information recording such as imaging data is accompanied.

  For example, it is of course possible to simply collect the biological information of the subject who wants to visualize the emotion, analyze emotion data from the collected biological information, and generate emotion visual data from the emotion data.

  In this specification, the color is determined by the hue, the saturation, and the lightness. Therefore, as described above, the color is changed by a combination thereof. Therefore, the color change in the present invention is not limited to the case where the hue changes, but also the case where the hue is the same and the saturation and brightness change.

It is a block diagram of an example of the imaging information recording / reproducing apparatus to which the embodiment of the emotion visualization method according to the present invention is applied. FIG. 2 is a block diagram illustrating a configuration example of a part of FIG. 1. It is a figure which shows the example of the emotion model used for description of embodiment of the visualization method of the emotion by this invention. It is a figure which shows an example of the color conversion table information used for embodiment of the emotion visualization method by this invention. It is a figure for demonstrating from several display types of the visualization output image by embodiment of the visualization method of the emotion by this invention. It is a figure which shows the flowchart for demonstrating embodiment of the emotion visualization method by this invention. It is a figure which shows the flowchart for demonstrating the processing operation at the time of the recording process in the imaging information recording / reproducing apparatus of FIG. FIG. 2 is a diagram for explaining a management structure of stored data in the imaging information recording / reproducing apparatus of FIG. 1. It is a figure which shows the flowchart for demonstrating the processing operation at the time of the reproduction | regeneration processing in the imaging information recording / reproducing apparatus of FIG. It is a figure which shows the example of an emotion visualized image display at the time of the reproduction | regeneration processing in the imaging information recording / reproducing apparatus of FIG. It is a figure which shows an example of the network system with which embodiment of the emotion visualization method by this invention is applied. It is a figure which shows a part of flowchart for demonstrating the application of the visualization expression using embodiment of the emotion visualization method by this invention. It is a figure which shows a part of flowchart for demonstrating the application of the visualization expression using embodiment of the emotion visualization method by this invention. It is a figure which shows the example of the visualization display of an emotion. It is a figure which shows the example of the visualization display of an emotion. It is a figure which shows the example of the visualization display of an emotion.

Claims (21)

Biometric information acquisition means for acquiring human biometric information;
Analyzing the biological information from the biological information acquisition means using a predetermined emotion model and a neural network, and obtaining an emotion parameter value indicating the state of the person's emotion;
A conversion table information holding unit for holding conversion table information indicating a correspondence relationship between the emotion parameter value and the color information;
A color conversion processing unit for obtaining color information corresponding to the emotion parameter value from the emotion analysis processing unit, using the conversion table information of the conversion table information holding unit;
By displaying the color output based on the color information of the predetermined period from the color conversion processing unit in correspondence with the passage of time in the predetermined period, the change of emotion in the predetermined period is displayed in color. Emotion change visualization means to visualize by,
An emotion visualization device characterized by comprising: The emotion analysis processing unit obtains an emotion parameter value indicating an average emotional state per unit time by averaging the emotion parameter value obtained from the biological information every predetermined unit time.
The color conversion processing unit uses the conversion table information of the conversion table information holding unit to obtain color information corresponding to an emotion parameter value indicating an average emotional state per unit time from the emotion analysis processing unit. The emotion visualization device according to claim 1, wherein the emotion visualization device outputs the emotion. The emotion change visualization means visualizes and displays the color output based on the color information for each unit time in the predetermined period from the color conversion processing unit in correspondence with the passage of time in the predetermined period, characterized in that to visualize the emotional changes per unit time in said predetermined time period as a color visualization apparatus feelings of claim 2. The biological information is acquired in correspondence with the time information acquired, and the emotion information storage means for storing the emotion parameter value from the emotion analysis processing unit corresponding to the time information,
For a predetermined period such as a predetermined one day or several days, one month or several months, one year or several years based on the time information, every predetermined time, every predetermined number of days, every predetermined number of months within the predetermined period , An emotion change visualization means for visualizing the change of the emotion as a color change by allocating and visualizing the color information from the color conversion processing unit every unit time consisting of every predetermined years, etc.,
The emotion visualization device according to claim 1, further comprising: The emotion change visualizing means expands a color display region based on color information corresponding to the emotion parameter value for each of a plurality of unit times included in the predetermined period in a one-dimensional manner into a rectangular region, and The emotion visualization apparatus according to claim 1, wherein a change in the emotion over time is visualized. The emotion change visualization means two-dimensionally expands a color display area based on color information corresponding to the emotion parameter value for each of a plurality of unit times included in the predetermined period into a rectangular area, and The emotion visualization apparatus according to claim 1, wherein a change in the emotion over time is visualized.   The emotion change visualization means assigns the predetermined period to the annular area, and displays a color display area based on color information corresponding to the emotion parameter value for each of a plurality of unit times included in the predetermined period. The emotion visualization device according to claim 1, wherein a change in the emotion per unit time is visualized by assigning to a region divided in a circumferential direction.   The emotion visualization device according to claim 1, wherein the emotion change visualization means visualizes the emotion change due to a color change by displaying it on a display screen.   The emotion visualization device according to claim 1, wherein the emotion change visualization means visualizes the emotion change due to a color change by printing on a record carrier.   Emotion distribution visualization means for visualizing the state distribution of the emotion within the predetermined period by visualizing a ratio of each color output by the color information from the color conversion processing unit for a predetermined period. The emotion visualization apparatus according to claim 1, wherein the emotion visualization apparatus is characterized.   The emotion distribution visualization means processes the color information from the color conversion processing unit for the predetermined period using a self-organizing map, and calculates the state distribution of the emotion within the predetermined period. The emotion visualization device according to claim 10, wherein the emotion visualization device is visualized by a color change.   The emotion visualization device according to claim 10, wherein the emotion distribution visualization means visualizes the distribution of the emotion due to a change in color on a display screen.   12. The emotion visualization device according to claim 10, wherein the emotion distribution visualization means visualizes the distribution of the emotion due to color change by printing on a record carrier. A biological information acquisition step of acquiring human biological information;
Analyzing the biological information acquired in the biological information acquisition step using a predetermined emotion model and a neural network, and obtaining an emotion parameter value indicating the state of the person's emotion;
A color conversion processing step of obtaining color information corresponding to the emotion parameter value from the emotion analysis processing unit using conversion table information indicating a correspondence relationship between the emotion parameter value and color information;
Emotion change visualization step of visualizing a change in emotion in the predetermined period by color by visualizing and displaying a color output based on the color information in a predetermined period corresponding to the passage of time in the predetermined period A method of visualizing emotions, comprising: In the emotion analysis processing step, the emotion parameter value obtained from the biological information is averaged every predetermined unit time, and an emotion parameter value indicating an average emotion state per unit time is obtained.
In the color conversion processing step, color information corresponding to an emotion parameter value indicating an average emotional state per unit time obtained in the emotion analysis processing step is output using the conversion table information. The emotion visualization method according to claim 14, wherein:   By displaying the color output based on the color information for each unit time in the predetermined period obtained in the color conversion processing step in correspondence with the passage of time in the predetermined period, The emotion visualization method according to claim 15, further comprising an emotion change visualization step of visualizing the change in emotion per unit time in color.   An emotion distribution visualization step of visualizing a state distribution of the emotion within the predetermined period by visualizing a ratio of each color output by the color information obtained in the color conversion processing step for a predetermined period; The emotion visualization method according to claim 14, wherein: Analyzing each of the biological information acquired from a plurality of people using a predetermined emotion model and a neural network, and obtaining an emotion parameter value indicating the state of the emotion of each of the plurality of people,
A conversion table information holding unit for holding conversion table information indicating a correspondence relationship between the emotion parameter value and the color information;
A color conversion processing unit that obtains color information corresponding to an emotion parameter value obtained by averaging the emotion parameter values of each of the plurality of persons from the emotion analysis processing unit using the conversion table information of the conversion table information holding unit. When,
Emotion visualization means for visualizing the averaged emotions of the plurality of people in correspondence with the passage of time within the predetermined period using each color output by the color information of the predetermined period from the color conversion processing unit When,
An emotion visualization device characterized by comprising: Analyzing each of biological information obtained from a plurality of people using a predetermined emotion model and a neural network, an emotion analysis processing unit for obtaining an emotion parameter value indicating the state of each of the plurality of people,
A conversion table information holding unit for holding conversion table information indicating a correspondence relationship between the emotion parameter value and the color information;
A color conversion processing unit that obtains color information corresponding to the emotion parameter values of the plurality of persons from the emotion analysis processing unit, using the conversion table information of the conversion table information holding unit;
Using each color output by the color information corresponding to the emotions of each of the plurality of persons in a predetermined period from the color conversion processing unit, the plurality of persons corresponding to the passage of time within the predetermined period. Emotion visualization means for visualizing the state distribution of emotions;
An emotion visualization device characterized by comprising: An accumulation means for accumulating emotion parameter values indicating the state of the person's emotion obtained together with time information, obtained by analyzing the biological information of the person using a predetermined emotion model and a neural network;
Conversion table information holding means for holding conversion table information indicating a correspondence relationship between the emotion parameter value and color information;
Of the emotion parameter values stored by the storage means, the emotion parameter values for a specified predetermined period are read out, and the emotions for the predetermined period are read using the conversion table information of the conversion table information holding means. Color conversion processing means for obtaining color information corresponding to each of the parameter values;
Using the color information from the color conversion processing means, emotion visualization means for generating emotion visualization image information for visualizing the human emotion in the predetermined period by color;
Emotion visualization information output means for outputting the emotion visualization image information generated by the emotion visualization means;
An emotion visualization device characterized by comprising:   The emotion visualization means is configured to perform predetermined time, predetermined number of days, predetermined number of months within the predetermined period for the predetermined period such as a predetermined day or days, a month or months, a year or years. The emotion visualized image information is generated by allocating and visualizing color information from the color conversion processing unit every unit time consisting of every predetermined number of years or the like. Emotion visualization device.

Images