JP4264717B2 - Imaging information search and playback apparatus and method, content search and playback apparatus and method, and emotion search apparatus and method - Google Patents

Description

  The present invention relates to an apparatus and a method for making it possible to search, for example, the daily experience of a person from the emotion of the person at that time, or the reproduced content from the emotion of the person who has watched the content. The present invention also relates to an apparatus and method for searching for an emotion that approximates a certain emotion that can be used in the apparatus and method.

  For example, what kind of emotional state a person has spent in a past period or at a certain time, what kind of environment the person was at that time, what kind of experience he had I want to know. For example, when the event and emotion at that time are written as a sentence in a diary, the above emotion and the event when the emotion is held can be estimated by reading the diary.

  However, in the case of a diary, although it depends on a person's ability to express sentences, in general, it is difficult to objectively and accurately write down experiences and events at that time and emotions at that time. Also, if you do not read the diary, you cannot know the experiences and events at that time and the emotions of the people at that time. In addition, since emotions are estimated from sentences, it is difficult to accurately recognize experiences and events at that time and human emotions at that time.

  Patent Document 1 (Japanese Patent Laid-Open No. 2002-230011) discloses an emotion recognition that can discriminate an emotion expressed in a document and convert the discriminated emotion into a visual display format such as an icon for display. A system is provided.

In this Patent Document 1, an emotion expression dictionary in which emotion elements such as “joy”, “sadness”, “surprise”, “fear”, “anger”, “disgust”, etc. are quantified and assigned in advance for each word. Is prepared, the recognition target document is decomposed into words, and the emotion expressed in the recognition target document is recognized using the emotion expression dictionary. In Patent Document 1, the recognized emotions such as “joy”, “sadness”, “surprise”, “fear”, “anger”, “disgust” are converted into visual display formats such as icons, Visualization is displayed.
JP 2002-230011 A

  However, even if the technique of the above-mentioned patent document 1 can discriminate the emotion expressed in the text described in the diary, the event or experience when the person has the emotion is searched. I can't. Further, even if an attempt is made to search for a certain emotion, for example, a state where the user feels relaxed and a state having the same emotion, the search cannot be performed.

  A person may want to know when, for example, when he / she had a certain emotion, it was the same emotion. If you know when you have such feelings, you can expect to find out to some extent what happened around the person at that time, based on diaries, other materials, and memories. Furthermore, it would be very convenient if there was a way to objectively know what happened at the time of having that emotion or what experience he had.

  Also, in this specification, content refers to something that can be visually or audibly appreciated, and includes emotions (for example, movies, music, novels, documents, etc.) It would be very convenient if you could search for content that would give you the feelings you want when you want to get the same feelings as if you were excited.

  An object of this invention is to provide the apparatus and method which can respond to the above requests.

In order to solve the above problems, the invention of claim 1
An imaging information storage unit in which the captured imaging information is stored corresponding to the time information of the imaging time;
Emotion information representing emotions of the photographer obtained by analyzing biological information acquired from the photographer who performs the imaging is stored in correspondence with time information of the imaging time at which the emotion information was acquired. An information storage unit;
Conversion table information holding means for holding conversion table information indicating a correspondence relationship between the emotion information and visualization information that is information for visualizing the emotion represented by the emotion information;
Conversion processing means for converting the emotion represented by the emotion information in the whole period or a part of the period during which the imaging was performed into the visualization information by the conversion table information ;
Display means for displaying the emotion visible image that will be visualized on the display screen by the visualization information generated by the conversion processing means,
Designating means for allowing a user to specify an arbitrary image positions or any image area on the display screen displayed the emotion visualization,
The visualization information corresponding to the arbitrary image position or the arbitrary image range on the emotion visualization image designated by the user through the designation means is detected, and the emotion information corresponding to the detected visualization information is detected. Emotion information detection means obtained using the conversion table information;
Search means for searching emotion information obtained by the emotion information detection means from the emotion information storage unit;
Means for reading out and reproducing the imaging information corresponding to the time position of the emotion information searched by the search means from the imaging information storage unit;
An apparatus for retrieving and reproducing imaging information is provided.

  In the image information retrieval / playback apparatus according to the first aspect of the present invention, an event or experience experienced by a person is imaged and stored in the image information storage unit, and emotion information representing a person's emotion at that time is stored. It is stored in the emotion information storage unit. In this case, the imaging information and the emotion information are stored in correspondence with each other through the time information of the imaging time.

  And in invention of Claim 1, emotion information is displayed on a display screen as an emotion visualization image using the visualization information which can be converted by conversion table information. As described above, each visualization information constituting the emotion visualization image corresponds to each emotion information. Therefore, when a certain position or range is designated on the emotion visualized image, the instruction corresponds to an instruction of the emotion or emotion range.

  Then, on the display screen, when an arbitrary image position or an arbitrary image range on the emotion visualization image is specified by the user, it corresponds to the visualization information of the specified arbitrary image position or the arbitrary image range. The emotion information to be detected is detected using the conversion table information.

  Then, the detected emotion information is searched from the emotion information storage unit, and time information corresponding to the emotion information is detected. Then, the imaging information specified by the detected time information is read from the imaging information storage unit and reproduced.

  Therefore, according to the first aspect of the present invention, by using the emotion visualization image generated by converting the emotion information into the visualization information, it is possible to specify a specific emotion or emotion change range using the emotion visualization information. It becomes possible.

  According to the first aspect of the present invention, since the imaging information is retrieved and reproduced when the specified specific emotion or the specific emotion change occurs, the specified specific emotion or emotion change is specified. However, it is easy to see what happened.

An imaging information storage unit in which the captured imaging information is stored corresponding to the time information of the imaging time;
Emotion information representing emotions of the photographer obtained by analyzing biological information acquired from the photographer who performs the imaging is stored in correspondence with time information of the imaging time at which the emotion information was acquired. An information storage unit;
Conversion table information holding means for holding conversion table information indicating a correspondence relationship between the emotion information and visualization information that is information for visualizing the emotion represented by the emotion information;
Conversion processing means for converting the emotion represented by the emotion information in the whole period or a part of the period during which the imaging was performed into the visualization information in time series using the conversion table information ;
Display means for displaying said conversion processing means display screen emotions visible image that will be visualized in time series by the Visualization of the generated the time series manner,
Designating means for allowing a user to specify an arbitrary image positions or any image area on the display screen displayed the emotion visualization,
The visualization information corresponding to the arbitrary image position or the arbitrary image range on the emotion visualization image designated by the user through the designation means is detected, and the time position in time series of the detected visualization information is determined. A search means for searching;
The imaging information corresponding to the time position information searched by the searching means, and means for reading out and reproducing from the imaging information storage unit,
An apparatus for retrieving and reproducing imaging information is provided.

  In the image information retrieval / playback apparatus according to the second aspect of the present invention, emotion information is displayed on the display screen as an emotion visualization image visualized in time series using visualization information that can be converted by conversion table information. Therefore, when a certain position or range is designated on the emotion visualized image, the instruction corresponds to time information from which emotion information is obtained.

  On the display screen, when an arbitrary image position or an arbitrary image range on the emotion visualization image is specified by the user, time information corresponding to the specified arbitrary image position or the arbitrary image range is obtained. Searched. Then, the imaging information specified by the retrieved time information is read from the imaging information storage unit and reproduced.

  Therefore, according to the invention of claim 2, by using the emotion visualization image generated by converting the emotion information into the visualization information, the emotion visualization information is used to correspond to a specific emotion or emotion change range. The time can be specified.

  According to the first aspect of the present invention, since the imaging information is retrieved and reproduced when the specified specific emotion or the specific emotion change occurs, the specified specific emotion or emotion change is specified. However, it is easy to see what happened.

According to a third aspect of the present invention, there is provided a retrieval / reproduction apparatus for imaging information according to the first or second aspect,
Before asked visualization information, a color information,
The conversion table information holding means stage as the conversion table information comprises said emotion information, the color conversion table information indicating a correspondence relationship between the color corresponding to the emotion represented by the emotion information,
The conversion processing means converts the input emotion information into color information corresponding to the emotion represented by the emotion information using the color conversion table information, and obtains the visualization information including the color information. It is characterized by that.

  According to the invention of claim 3, since the color information is used as the information for visualizing the emotion information and the emotion information is converted into a color and displayed, for example, the darker “red” is, the more comfortable it is. The emotional change is visualized as the color change, and the emotional state distribution is displayed according to the color distribution state, for example, by displaying the emotion and displaying the unpleasant emotion as “blue” is darker. It becomes possible.

The invention of claim 6
An imaging information storage unit in which the captured imaging information is stored corresponding to the time information of the imaging time;
Emotion information representing emotions of the photographer obtained by analyzing biological information acquired from the photographer who performs the imaging is stored in correspondence with time information of the imaging time at which the emotion information was acquired. An information storage unit;
Conversion table information holding means for holding conversion table information indicating a correspondence relationship between the emotion information and visualization information that is information for visualizing the emotion represented by the emotion information;
Conversion processing means for converting the emotion represented by the emotion information in the whole period or a part of the period during which the imaging was performed into the visualization information by the conversion table information ;
Display means for displaying the emotion visible image that will be visualized on the display screen by the visualization information generated by the conversion processing means,
Designating means for allowing a user to specify an arbitrary image positions or any image area on the display screen displayed the emotion visualization,
The visualization information corresponding to the arbitrary image position or the arbitrary image range on the emotion visualization image designated by the user through the designation means is detected, and the emotion information corresponding to the detected visualization information is detected. Emotion information detection means obtained using the conversion table information;
Search means for searching emotion information representing an emotion similar to the emotion represented by the emotion information obtained by the emotion information detection means from the emotion information stored in the emotion information storage unit;
Means for reading out and reproducing the imaging information corresponding to the time position of the searched emotion information from the imaging information storage unit;
Provide search playback device that an imaging information comprising a.

  In the invention of claim 6, emotion information indicating an emotion approximate to the emotion represented by the emotion information corresponding to the visualization information designated on the emotion visualization image by the user is retrieved from the emotion information storage unit. And the imaging information corresponding to the time position of the searched emotion information is reproduced.

  Therefore, according to the invention of claim 6, it is possible to search when an emotion that approximates a specific emotion is obtained, and to determine the surrounding environment such as an event when the approximate emotion is obtained. I can know.

The invention of claim 16
A content storage unit in which content data is stored corresponding to the time position information;
Emotion information storage unit in which emotion information representing the viewer's emotion obtained by analyzing the biological information of the viewer who has viewed the content is stored corresponding to the time position information from which the emotion information was acquired When,
Conversion table information holding means for holding conversion table information indicating a correspondence relationship between the emotion information and visualization information that is information for visualizing the emotion represented by the emotion information;
Conversion processing means for converting the emotion represented by the emotion information when the viewer appreciates the content into the visualization information using the conversion table information ;
Display means for displaying the emotion visible image that will be visualized on the display screen by the visualization information generated by the conversion processing means,
Designating means for allowing a user to specify an arbitrary image positions or any image area on the display screen displayed the emotion visualization,
The visualization information corresponding to the arbitrary image position or the arbitrary image range on the emotion visualization image designated by the user through the designation means is detected, and the emotion information corresponding to the detected visualization information is Emotion information detection means obtained using the conversion table information;
Search means for searching emotion information obtained by the emotion information detection means from the emotion information storage unit;
The content data corresponding to the time position of the emotion information searched by the searching means, and means for reading out and reproducing from the content storage unit,
A content search / playback apparatus comprising:

According to the sixteenth aspect of the present invention, it is possible to visualize changes in emotion when, for example, a predetermined broadcast program or movie content is viewed using the emotion visualization image. It is also possible to visualize the emotional state distribution when viewing a predetermined broadcast program or movie content.

  Then, on the display screen, when an arbitrary image position or an arbitrary image range on the emotion visualization image is specified by the user, it corresponds to the visualization information of the specified arbitrary image position or the arbitrary image range. The emotion information to be detected is detected using the conversion table information.

  Then, the detected emotion information is searched from the emotion information storage unit, and time information corresponding to the emotion information is detected. Then, the content specified by the detected time information is reproduced.

Therefore, according to the sixteenth aspect of the present invention, for example, in a movie content, it is possible to view a scene retrieved from the emotion state designated by the emotion visualization image. That is, on the emotion visualized image, for example, when a visualized image portion corresponding to an excited emotion is indicated, a scene that makes the excited state appear is reproduced.

The invention of claim 21
A content storage unit in which data of a plurality of contents are stored corresponding to the time position information;
Emotion information storage unit in which emotion information representing the viewer's emotion obtained by analyzing the biological information of the viewer who has viewed the content is stored corresponding to the time position information from which the emotion information was acquired When,
Conversion table information holding means for holding conversion table information indicating a correspondence relationship between the emotion information and visualization information that is information for visualizing the emotion represented by the emotion information;
Conversion processing means for converting the emotion represented by the emotion information when the viewer appreciates the content into the visualization information using the conversion table information ;
And emotion visible image information storage unit for storing information of visualized Ru emotion visualization by the visualization information generated by the conversion processing means,
Display means for displaying emotion visualized image information stored in the emotion visualized image information storage unit on a display screen;
A designation means for the user to designate an arbitrary emotion visualization image from among the plurality of emotion visualization images displayed on the display screen;
Emotion visualization image information stored in the emotion visualization image information storage unit is searched for emotion visualization image information that becomes an emotion visualization image that approximates the emotion visualization image specified through the specifying means, and the emotion visualization image information of the search result Search and presentation means for presenting content corresponding to
Means for reading out and reproducing the data of the presented content from the content storage unit;
Provide search reproducing apparatus Turkey content comprising a.

In the invention of claim 21 , corresponding to each content, an emotion visualization image is generated by visualizing emotion information when the viewer views the content with visualization information that can be converted by conversion table information, The emotion visualized image is stored in the emotion visualized image information storage unit.

  In this case, the emotion visualization image shows the emotional state of the viewer when viewing the corresponding content.

  A plurality of emotion visualized images stored in the emotion visualized image information storage unit are displayed on the display screen. Then, the user can designate the plurality of emotion visualized images through the designation means. When any emotion visualized image is selected and specified, emotion visualized image information that becomes an emotion visualized image that approximates the specified emotion visualized image is searched from the emotion visualized image information stored in the emotion visualized image information storage unit. Is done. Then, the content corresponding to the emotion visualized image information of the search result is presented.

  That is, when a user designates a certain emotion visualized image, content that can obtain an emotional state similar to the emotional state expressed in the emotion visualized image is retrieved and presented. Then, the user can reproduce the presented content.

Therefore, according to the twenty-first aspect of the present invention, it is possible to select content that the user wants to appreciate based on the emotional state appearing in the emotion visualization image.

The invention of claim 29 provides
Emotion information storing the emotion information representing the emotion of the person obtained by analyzing the biological information acquired from a person, corresponding to the time information acquired the emotion information;
Conversion table information holding means for holding conversion table information indicating a correspondence relationship between the emotion information and visualization information that is information for visualizing the emotion represented by the emotion information;
Conversion processing means for converting the emotion represented by the emotion information stored in the emotion information storage unit into the time-series visualization information using the conversion table information ;
Display means for displaying said conversion processing means display screen emotions visible image that will be visualized in time series by the Visualization of the generated the time series manner,
Designating means for allowing a user to specify an arbitrary image positions or any image area on the display screen displayed the emotion visualization,
Search for detecting the visualization information corresponding to the arbitrary image position or the arbitrary image range on the emotion visualization image specified through the specifying means, and searching for a time position in the time series of the detected visualization information Means,
Means for informing the user of the time position of the search result;
To provide sensitive information retrieval device Ru equipped with.

In this invention of Claim 29 , the time corresponding to the emotion information or emotion information range corresponding to the visualization information designated on the emotion visualization image by the user is searched, and the time information is notified to the user.

Therefore, according to the invention of claim 29 , it is possible to search for the date and time when a specific emotion is obtained.

The invention of claim 30
Emotion information storing the emotion information representing the emotion of the person obtained by analyzing the biological information acquired from a person, corresponding to the time information acquired the emotion information;
Conversion table information holding means for holding conversion table information indicating a correspondence relationship between the emotion information and visualization information that is information for visualizing the emotion represented by the emotion information;
Conversion processing means for converting the emotion represented by the emotion information stored in the emotion information storage unit into the visualization information using the conversion table information ;
Display means for displaying the emotion visible image that will be visualized on the display screen by the visualization information generated by the conversion processing means,
Designating means for allowing a user to specify an arbitrary image positions or any image area on the display screen displayed the emotion visualization,
The visualization information corresponding to the arbitrary image position or the arbitrary image range on the emotion visualization image designated through the designation means is detected, and the emotion information corresponding to the detected visualization information is converted to the conversion table. Means for detecting using information;
Search means for searching emotion information stored in the emotion information storage unit for emotion information similar to the detected emotion information;
Means for informing the user so that the time corresponding to the emotion information or emotion information range of the search result is known;
To provide sensitive information retrieval device Ru equipped with.

In the invention of claim 30 , emotion information indicating an emotion approximate to the emotion represented by the emotion information corresponding to the visualization information designated on the emotion visualization image by the user is retrieved from the emotion information storage unit, The user is informed so that the time corresponding to the searched emotion information or emotion information range is known.

Therefore, according to the invention of claim 30 , it is possible to search for the date and time when an emotion approximate to a specific emotion is obtained.

  According to the present invention, it is possible to display emotions and changes in emotions held by a person as emotion visualized images, and to search and instruct predetermined emotions through the emotion visualized images. According to the first aspect of the present invention, it is possible to retrieve and reproduce the imaging information obtained when the emotion visualized on the emotion visualized image is designated.

  According to the sixth aspect of the present invention, it is possible to search for a point in time or a time range having an emotion similar to the emotion designated on the emotion visualization image. Then, when an emotion at a specific time point or time range is specified from the search results, the corresponding imaging information can be searched and reproduced. Therefore, it is possible to search when an emotion that approximates a specific emotion is obtained, and to know the surrounding environment such as an event when the approximate emotion is obtained.

According to the sixteenth aspect of the present invention, it is possible to visualize emotional changes and emotional state distributions when, for example, a predetermined broadcast program or movie content is viewed using the emotional visualization image.

  Then, on this display screen, when an arbitrary image position or an arbitrary image range on the emotion visualization image is specified by the user, an emotion or an emotion range corresponding to the specified image position or image range is searched, The corresponding content scene can be viewed.

According to the twenty-first aspect of the present invention, it is possible to select content that the user wants to appreciate based on the emotional state appearing in the emotion visualization image.

According to the invention of claim 29 , it is possible to search for the date and time when a specific emotion is obtained. Furthermore, according to the invention of claim 30 , it is possible to search for a date and time when an emotion approximate to a specific emotion is obtained.

[First Embodiment]
The first embodiment described below is a case where the imaging information searching / reproducing apparatus and method according to the present invention are applied to an imaging information recording / reproducing apparatus. The imaging information recording / reproducing apparatus according to the first embodiment includes an imaging unit, captures the photographer's biological information when the imaging unit is imaging, performs emotion analysis, and performs color conversion on the emotion analysis result. Visualization data (color information) is obtained, and emotion visualized image information generated based on the visualization data is recorded in association with time information together with imaging information.

  In this embodiment, color information is used as the visualization data. That is, in this embodiment, the emotion information of the emotion analysis result is converted into color information and visualized. Here, since the color is determined by the hue, saturation, and brightness, the color change is made by a combination thereof. Accordingly, not only those whose hue changes but also those whose hue is the same and whose saturation and brightness change.

<Example of hardware configuration of imaging information recording / reproducing apparatus>
FIG. 1 is a block diagram of an imaging information recording / reproducing apparatus 100 of this embodiment.

  The imaging information recording / reproducing apparatus 100 includes a computer in which a ROM (Read Only Memory) 103 and a RAM (Random Access Memory) 104 are connected to a CPU (Central Processing Unit) 101 via a system bus 102. It is configured to be equipped.

  Then, with respect to the system bus 102, an imaging unit controller 105, an imaging signal processing unit 106, a biological information input unit controller 107, an emotion analysis processing unit 108, a data storage unit controller 109, a reproduction processing unit 110, Timekeeping unit 111, visual data processing unit 112, display unit controller 113, key input interface (in the figure, the interface is described as I / F. The same applies hereinafter) 114, network interface 115, printer controller 116, search A processing unit 117 is connected.

  The imaging unit controller 105 supplies an imaging unit control signal to the imaging unit 121 based on control from the CPU 101 according to a program stored in the ROM 103.

  The imaging unit 121 includes a lens system and an imaging device configured by, for example, a CCD (Charge Coupled Device), receives an imaging unit control signal from the imaging unit controller 105, and starts and stops imaging recording, zoom magnification, The exposure value, auto focus, etc. are controlled.

  The captured image signal from the imaging unit 121 is converted into a digital signal by an A / D (Analog to Digital) converter 122 and then supplied to the imaging signal processing unit 106. In the imaging signal processing unit 106, processing is executed by a control instruction command from the CPU 101 according to a program stored in the ROM 103, and an imaging image signal from the A / D converter 122 is converted into, for example, an MPEG (Moving Picture Experts Group). ) Image data is compressed by the method and converted into an appropriate recording format. Further, the imaging signal processing unit 106 uses the time information from the time measuring unit 111 to perform processing for adding imaging time information to the captured image data.

  Then, the captured image data including the information on the imaging time, which has been converted into an appropriate recording format by the imaging signal processing unit 106, is transferred to the data storage unit controller 109 through the system bus 102. The data storage unit controller 109 stores the received captured image data in the data storage unit 123.

  In this example, the data storage unit 123 is composed of a hard disk drive, and has a storage capacity for storing data for one day or more, for example. The data storage unit 123 may be a device that uses a large-capacity recording type optical disc such as a DVD (Digital Versatile Disc) or a semiconductor memory as a storage medium.

  The biological information input unit controller 107 receives the biological information from the biological information input unit 124 through the A / D conversion unit 125. As will be described in detail later, the biological information input unit 124 constitutes means for acquiring biological information of the user (photographer) of the imaging information recording / reproducing apparatus 100.

  Here, the biological information in this specification refers to information on physiological, biochemical, or physical reactions or states. Biological information includes, for example, information spontaneously and evoked by humans, such as heartbeat, pulse wave, respiration, blood pressure, electrocardiogram, electroencephalogram, skin sweating, skin resistance, body movement, body position, magnetoencephalogram, myoelectricity. , Body surface temperature, pupil size, microvibration, blink (blink), biochemical reaction, and the like.

  Therefore, human biological information is not only detected through electrodes attached to humans, but also detected from information captured by a camera such as body movement and body position, and sensors such as body surface temperature. Includes those that can be detected remotely.

  The biometric information of the user acquired by the biometric information input unit 124 is converted into a digital signal by the A / D conversion unit 125, and is appropriately recorded by the biometric signal input unit controller 107, the CPU 101, the ROM 103 program, and the work area RAM 104. The data is converted into a format and stored in the data storage unit 123 by the data storage controller 109. At this time, corresponding to the captured image data, the biometric information is stored in the data storage unit 123 to which the information on the acquisition time is added.

  The biometric information acquired by the biometric information input unit 124 is also converted into a digital signal by the A / D conversion unit 125 and then supplied to the emotion analysis processing unit 108. Here, as described later, a predetermined method is used. Emotion analysis is performed accordingly. Then, information (hereinafter referred to as emotion data) representing the user's emotion obtained as a result of the emotion analysis in the emotion analysis processing unit 108 is converted into an appropriate recording format by the CPU 101 and the RAM 104 according to the program of the ROM 103. The data is converted and stored in the data storage unit 123 by the data storage controller 109. Time information is also added to the emotion data, and correspondence with captured image data and biological information is possible.

  Note that the emotion data is not generated from the biological information in real time as described above, but after the storage of the imaging information and the biological information in the data storage unit 123 is completed, the biological data stored in the data storage unit 123 is stored. It is also possible to read out information, analyze emotions, and store the analysis result in the data storage unit 123. In this case, however, the emotion display in real time as described later cannot be performed.

  In the above description, the captured image data of the subject acquired by the imaging unit 121, the user's biological information acquired through the biological information input unit 124, and the emotion obtained by analyzing the biological information by the emotion analysis processing unit 108. Although it has been described that the emotion data representing is stored and stored in the data storage unit 123, the present invention is not limited to this.

  For example, it connects to a network such as the Internet through the network interface 115, transmits it to a predetermined server device provided on the network, and causes the server device to store and hold captured image data, biological information, and information indicating an emotional state. It is also possible to do so.

  As will be described in detail later, the emotion analysis processing unit 108 analyzes various biological information acquired through the biological information input unit 124 or biological information stored in the data storage unit 123, in this example, an emotion model. To extract emotional state.

  The reproduction processing unit 110 performs image data for reproduction by performing decompression decoding processing on the compressed image data read from the data storage unit 123 through the data storage unit controller 109 during reproduction. The image data processed in the reproduction processing unit 110 is supplied to the visual data processing unit 112.

  The timer unit 111 is a clock circuit with a calendar function, and sends out date and time information to the system bus 102.

  The visual data processing unit 112 is stored in the color conversion table information storage unit 126 from the emotion data from the emotion analysis processing unit 108 or the emotion data stored in the data storage unit 123, as will be described in detail later. Emotion visualization data (in this example, color information) is obtained using the conversion table information, emotion visualization image information using the visualization data is generated, and the generated emotion visualization image information is transmitted through the data storage controller 109. A function of storing in the data storage unit 123 is provided. As will be described later, the visual data processing unit 112 also includes role functions of emotion change visualization means and emotion distribution visualization means.

  Further, the visual data processing unit 112 reproduces the captured image data and the emotion visualized image information read from the data storage unit 123 at the time of reproduction in synchronization with the time axis, and through the display unit controller 113, the display unit 127 It has a function to be displayed on the screen. At the time of reproduction, the captured image data read from the data storage unit 123 is supplied to the visual data processing unit 112 after being decompressed and decoded by the reproduction processing unit 110 as described above.

  The display controller 113 forms a display image signal to be supplied to the display unit 127 and supplies this to the display unit 127. The display unit 127 is a display such as an LCD (Liquid Crystal Display), and displays an image corresponding to an image signal from the display unit controller 113 on the display screen.

  A key input operation unit 128 is connected to the key input interface 114. The key input operation unit 128 includes keys such as an imaging recording start key, a reproduction start key, a fast forward key, a rewind key, a jump key, and a stop key. When any key of the key input operation unit 128 is operated, the operation input information is supplied to the system bus 102 through the key input interface 114. The CPU 101 determines the operation input information, determines which key is operated, and executes processing corresponding to the operated key.

  In this example, a color printer 129 is connected to the printer controller 116. As will be described later, an output image based on emotion visualization data obtained from the visual data processing unit 112 is printed out in color by the color printer 129. It is configured.

  Although not shown in FIG. 1, a microphone is attached to the imaging unit 121. During imaging, sound around the subject is picked up by the microphone, A / D converted, and data compressed as necessary. The data is stored and recorded in the data storage unit 123 together with the captured image data.

  The search processing unit 117 executes search processing as described below according to the emotion or emotion range specified through the output image (emotion visualized image) based on the emotion visualized image information. That is, in this embodiment, a simple search mode and an approximate search mode are provided as search modes.

  The simple search mode searches the time position or time range corresponding to the specified emotion or emotion range through the output image (emotion visualized image) based on the emotion visualized image information, and corresponds from the data storage unit 123 based on the search result. This is a mode for retrieving imaging information of a time position or a time range to be performed. This simple search mode is a mode for searching and reproducing imaging information corresponding to a specified emotion.

  The approximate search mode searches other emotions or emotion ranges that approximate the specified emotion or emotion range through the output image (emotional visualization image) based on emotion visualized image information, and the emotion or emotion range time or time of the search result In this mode, the range is notified to the user. In this approximate search mode, when the time or time range of the search result is instructed by the user, the imaging information corresponding to the time or time range can be read and reproduced.

  Detailed processing operations in the simple search mode and approximate search mode performed by the search processing unit 117 will be described in detail later.

<Configuration Example of Portion of Biological Information Input Unit 124 to Biological Signal Input Unit Controller 107>
Here, a more specific configuration example of the biometric information input unit 124, the A / D conversion unit 125, and the biosignal input unit controller 107 described above will be described. FIG. 2 is a block diagram illustrating a detailed configuration example of the biometric information input unit 124, the A / D converter unit 125, and the biometric information input unit controller 107.

 Emotion data representing human emotion can be obtained using only one appropriate biological information, but in order to obtain emotion data representing human emotion more accurately, in this example, a plurality of emotion data are represented. Use biological information. In the example of FIG. 2, a case where brain waves, electrocardiograms, pulse waves (pulses), body temperature, and the like are detected through electrodes (including sensors) as user biometric information will be described as an example.

  As shown in FIG. 2, the living body information input unit 124 of this example is worn by a person and corresponds to a plurality of living body information, in this example, brain waves, electrocardiograms, pulse waves, body temperature, myoelectricity, and the like. , 1241e and a plurality of electrodes 1241a, 1241b, 1241c, 1241d,..., 1241e for collecting biological signals, and for amplifying the biological signals from the respective electrodes 1241a, 1241b, 1241c, 1241d,. The plurality of amplifiers 1242a, 1242b, 1242c, 1242d,.

  The electrodes 1241a, 1241b, 1241c, 1241d,..., 1241e have a function of measuring bioelectric phenomena including brain waves and biomagnetic phenomena including brain magnetism. Worn by the person. The amplifying units 1242a, 1242b, 1242c, 1242d,..., 1242e may perform not only amplification but also filtering processing such as dividing into several types of frequency bands as necessary.

  For example, if the biological information is an electroencephalogram, the amplifying unit 1242a amplifies the biological information signal having a potential of several microvolts obtained from the electrode 1241a and divides it into components of several frequency bands such as an alpha wave and a beta wave. In this way, the filtering process is performed.

  Each biological information detected through the electrodes 1241a, 1241b, 1241c, 1241d,..., 1241e is supplied to the corresponding amplifiers 1242a, 1242b, 1242c, 1242d,. This is supplied to the conversion unit 125. In this embodiment, the A / D converter 125 includes a plurality of A / D converters 125a, 125b, 125c, 125d,..., 125e corresponding to the supplied biological information, as shown in FIG. The biometric information (analog signal) supplied is converted into a digital signal.

  For example, the potential level of the electroencephalogram is generally about 50 μV (microvolt), and the potential level of the electrocardiogram is about 1 mV (millivolt), which varies depending on the type of biological signal. Therefore, the number of quantization bits at the time of A / D conversion in each of the A / D converters 125a, 125b, 125c, 125d,..., 125e is such that it does not fall below the potential level of the various biological signals amplified. It is made to have accuracy.

  Each biometric information after A / D conversion is supplied to the biometric information input unit controller 107. The biometric information input unit controller 107 receives the various biometric digital signals, and receives the biometric information by the emotion analysis processing unit 108, the data storage unit 123, the display unit 127, and the reproduction processing unit 110. A biological signal processing unit 1072 that performs processing for making a usable signal and a transmission unit 1073 that outputs the processed biological signal to the bus 102 are configured.

  The receiving unit 1071 receives digital biometric information having a sampling frequency at an interval at which the original biosignal is reproduced, which is output from the A / D converters 125a, 125b, 125c, 125d,. This is supplied to the biological signal processing unit 1072 in the subsequent stage.

  The biological signal processing unit 1072 receives the digital biological information supplied from the receiving unit 1071, and for each biological information, for example, filter operation processing for amplitude adjustment and band pass, time including FFT and wavelet analysis, etc. Frequency analysis or nonlinear analysis such as neural network or chaos analysis is performed to convert the signal into a format that can be used by the emotion analysis processing unit 108, the data storage unit 123, the display unit 127, and the reproduction processing unit 110. Then, the converted biological information (biological signal) is supplied to the transmission unit 1073.

  The transmission unit 1073 outputs the biological signal formed by processing as described above to the bus 102 at appropriate timing sequentially, and records it in the data storage unit 123 through the data storage unit controller 109 as described above. Or, it is supplied to the emotion analysis processing unit 108 to perform analysis of biological information.

<Example of emotion analysis in emotion analysis processing unit 108>
Next, an example of emotion analysis processing performed in the emotion analysis processing unit 108 will be described. As described above, the emotion analysis processing unit 108 performs digitization obtained by performing necessary signal processing and analysis processing through the biological information input unit 124, the A / D conversion unit 125, and the biological information input unit controller 107. The plurality of pieces of biological information that have been input are input.

  The biometric information once stored in the data storage unit 123 is read out to the emotion analysis processing unit 108 through the data storage unit controller 109, not only when biometric information is supplied from the biometric information input unit controller 107 through the bus 102. In some cases, this is supplied through the bus 102. In addition, the emotion analysis processing unit 108 can be supplied with biometric information stored in an external device or an external server through the network interface 115.

  The emotion analysis processing unit 108 receives input of a plurality of pieces of biological information, performs analysis processing based on the plurality of pieces of biological information, and outputs a plurality of emotion parameter values defined by the emotion model as emotion data. Examples of the emotion model include a 2-axis model, a 3-axis model, and a 6-axis model. An example of these models is shown in FIG.

  The two-axis model is the simplest model, and as shown in FIG. 3A, as emotions, for example, a first axis indicating a degree closer to comfort or discomfort, excitement or tension and relaxation. An emotion model is formed by two axes, the second axis indicating the degree of which is closer. Each emotion state in the two-dimensional emotion space determined by the two axes is expressed by an emotion parameter value composed of the parameter (coordinate value) in the first axis direction and the parameter (coordinate value) in the second axis direction.

  As shown in FIG. 3B, the three-axis model is closer to emotion, for example, a first axis indicating the degree of comfort or discomfort, and excitement, tension, or relaxation. An emotion model is formed by the three axes of the second axis that indicates the degree of movement and the third axis that indicates the degree of closer to tension or relaxation, and the emotion parameter is composed of coordinate values in the three-axis space. It expresses the state of emotion.

  As shown in FIG. 3C, the six-axis model includes a first axis indicating the degree of anger, a second axis indicating the degree of fear, a third axis indicating the degree of sadness, and a fourth axis indicating the degree of joy. Then, an emotion model is formed by six axes including a fifth axis indicating the degree of disgust and a sixth axis indicating the degree of surprise, and the emotional state is expressed by emotion parameters including coordinate values in the three-axis space. Is.

  The emotion analysis processing unit 108 uses, for example, any of the emotion models described above. For example, a three-layer back propagation or self-organization map (SOM: Self Using neural network techniques such as Organizing Map, the emotions of the person who is the source of the biological information are analyzed from multiple types of biological information received as input, and the analysis results are obtained. An emotion parameter value is obtained as emotion data representing emotion.

  As described above, the emotion analysis processing unit 108 of this embodiment is a provider of biological information by analyzing a plurality of types of biological information using a predetermined emotion model and a neural network technique. It becomes possible to grasp the emotional state of the user as an emotion parameter value.

  Then, as described later, the emotion parameter value of the emotion analysis result is converted into color information output by using the conversion table information in the color conversion table information storage unit 126 in the visual data processing unit 112, and corresponds to time information. And visualize. Thus, for example, it is possible to easily grasp the period in which the feeling of joy appears, the period in which the feeling of sadness appears, the period in which the feeling of anger appears, and the period in which the feeling of fear appears. , Those emotional changes can be easily grasped as color changes.

  Then, as will be described later, for example, the distribution of the emotions of the user in a predetermined period such as one day or one month, i.e., what percentage of pleasure feelings and what percentage of sadness feelings, It becomes possible to grasp by the area of the corresponding color.

  As specific examples of the analysis of the emotional state using the biological information, various types can be considered as follows. For example, for evaluation of the relaxed state, an electroencephalogram can be used as biological information. In general, it is known that the ratio of the frequency component of the α wave and the frequency component of the β wave is taken, and the more the ratio of the α wave component is, the more relaxed the state is. It is also possible to make an evaluation based on the stability of the cycle of breathing and heartbeat and the amount of increase in pulse amplitude.

  For surprising evaluation, the electrical resistance value of the skin can be used. In general, it is known that the electrical resistance value of the skin increases due to surprise, fear, or worry. By this electrical resistance value of the skin, the evaluation of surprise, which is one of emotional states, can be performed.

  When joy is evaluated, it can be evaluated using heart rate, skin temperature (body temperature), and electrocardiogram (electrocardiogram frequency). That is, it is known that when the emotion of joy is great, the heart rate decreases, the skin temperature increases, and the electrocardiogram frequency increases.

  In general, when mentally and psychologically stable, the peripheral blood vessels dilate and arterial blood flows into the peripheral blood vessels, so the heart rate and pulse change slowly, and the body temperature tends to rise It is in. On the other hand, in a high mental and psychological tension state, peripheral blood vessels contract and arterial blood flows into the peripheral blood vessels, so the heart rate and pulse rate increase, and the body temperature tends to decrease. It is possible to evaluate and grasp the emotional state according to such physiological changes.

  In addition to the physiological indices as described above, the combined use of myoelectric changes by facial expressions, facial expression recognition by images, posture recognition by acceleration sensors, tilt sensors, etc., voice recognition, etc. The emotional state may be evaluated.

  In other words, it is possible to grasp whether the person is crying, laughing, or angry by changing the myoelectricity on the human face. It is possible to know whether you are laughing or angry.

  It can also be used as information when evaluating emotional states by judging the user's posture by using an acceleration sensor or tilt sensor and detecting whether the chest is worn or bent forward. Is possible.

  Moreover, it is also possible to use the result of speech recognition as information when evaluating the emotional state. For example, when you are happy, the voice of the person is relatively high, and when you are depressed, the voice of the person tends to be relatively low. Thus, it is possible to evaluate the emotional state. It is also possible to evaluate the emotional state based on the magnitude (level) of the audio signal.

  For example, as an emotion data analysis technique, Japanese Patent Laid-Open No. 8-117199 discloses an analysis of arousal and mood by a neural network using α wave fluctuation of an electroencephalogram. This is based on the knowledge that two axes of arousal level and mood are basic indicators in psychology and psychophysiology, and quantitative analysis is performed.

  In addition, facial expression recognition has been reported in the NICOGRAPH papers pp.17-25, 1993, and in the IEICE Transactions D-II Vol.J77-D-II No.8 pp.1512-1521. Has been reported to recognize user's emotions from non-verbal information such as phonetic information. Of course, the user's emotional state may be grasped using such various analysis methods.

  As described above, the imaging information recording / reproducing apparatus 100 according to this embodiment can continuously detect user's biological information and analyze the detected biological information to grasp the user's emotion. Then, emotion data (emotion parameter value) representing the grasped user emotion can be obtained, and the visual data processing unit 112 can color-convert the emotion parameter value as described later to visualize the emotion as a color output. .

In this embodiment, in the emotion analysis processing unit 108 employs a two-axis model described above as the emotion model, the first axis is X axis, to detect the emotion parameter value and the second axis as Y axis.

  In this example, the color conversion table information storage unit 126 stores color conversion table information in the case of the biaxial model described above as the emotion model. The color conversion table information in this example will be described with reference to FIG. That is, FIG. 4 shows color output when emotion parameter values are output as color information based on this color conversion table information, but in FIG.

  In creating the color conversion table information, a color that can be perceived emotionally by appearance is used as a color that is converted and output for emotion. An example of this color conversion table information will be described below.

  That is, the color conversion table of FIG. 4 will be described. In the X-axis direction, comfort is expressed in yellow and discomfort is expressed in blue. The more comfortable, the darker yellow, and the more unpleasant, the darker blue. Accordingly, in the X-axis direction, the color changes in the order of dark blue → light blue → white → light yellow → dark yellow along the direction from uncomfortable to comfortable.

  In the Y-axis direction, excitement and tension are expressed in magenta, and relaxation is expressed in blue. And the more excited and tense, the more pure magenta, and the more relaxed, the more pure blue, and between excitement, tension and relaxation, from relaxation to excitement, along the direction of tension. From the pure blue, the ratio of the magenta color mixture gradually increases to change to pure magenta.

  The state of excitement, tension and comfortable emotion (corresponding to the upper right corner position on the color conversion table in FIG. 4) is assigned red as a result of subtractive color mixture of magenta and yellow. As the color change from the lower right corner to the upper right corner in FIG. 4, the ratio of the mixed color of red gradually increases from pure yellow to pure red. Here, the mixed color of yellow and red is orange.

  Similarly, the color change from the upper left corner to the upper right corner in FIG. 4 is as follows: dark magenta color → magenta color that gradually fades with blue → red that gradually fades with yellow → dark red It will change.

  At this time, the emotion analysis processing unit 108 obtains parameter values (coordinate values) in the X-axis direction and parameter values (coordinate values) in the Y-axis direction as emotion parameter values. The visual data processing unit 112 converts the color information determined by the parameter values in the color conversion table space of FIG. This color information is output as emotion visualization data.

  The visual data processing unit 112 does not output the color information obtained by converting the emotion parameter value using the color conversion table information as it is, but changes the emotion corresponding to the passage of time in chronological order. The output is performed in a display output manner that can be understood (hereinafter, this type of display output manner is referred to as a time-series display type). This function corresponds to the emotion change visualization means. This time-series display type is used when emotion visualized images are displayed corresponding to the image data captured by the imaging unit 121 in time-series order.

  Each image position of this time-series display type emotion visualized image corresponds to time information from which emotion data of the image position is obtained. In this case, time information may be stored corresponding to each image position, or the start and end times of a time range in which a time series display type emotion visualization image is generated may be held. For example, time information corresponding to each image position can be easily obtained by assigning, for example, a serial number to each image position.

  Further, in this example, the visual data processing unit 112 determines how much of each emotion such as excitement, relaxation, comfort, and discomfort is included in the predetermined time. It is also possible to output in a display output mode (hereinafter, this type of display output mode is referred to as a distribution display type) represented by a color distribution corresponding to each emotion in the region. This function corresponds to the emotion distribution visualization means. This distribution display type is used when displaying what kind of emotional state distribution was made for a predetermined period of time taken by the imaging unit 121. For example, it is possible to display that the user feels comfortable as a whole, but that there is a partly tense period.

  Each image position of this distribution display type emotion visualized image may be associated with the time information from which emotion data at the image position is obtained. For example, the beginning and end time of a time range in which a distribution display type emotion visualized image is generated are held, and for example, continuous in time series direction (time direction) corresponding to each image position of the emotion visualized image If a number is assigned, time information corresponding to each image position can be easily obtained.

  5A, 5B, and 5C show some examples of the time series display type, and FIG. 5D shows an example of the distribution display type.

  In the time-series display type, color information corresponding to instantaneous emotions may be displayed sequentially, but this causes too many subtle emotional changes to be expressed. It may be difficult.

  Therefore, in the examples of FIGS. 5A, 5B, and 5C, the predetermined predetermined unit time is set as the averaging interval, and the emotion data obtained in the unit time interval of the averaging interval is averaged. The averaged emotion data is converted into color information using a color conversion table. Therefore, in the examples of FIGS. 5A, 5B, and 5C, the color information output corresponding to the emotion data is performed in a predetermined width unit corresponding to the unit time interval. In this case, the time information is unit time interval unit information. As described above, in the case of this example, the time information stores the start and end times of the predetermined period in which the emotion is to be displayed, and the number in the time direction of each unit time interval. Of course, the time information for each unit time interval may be stored one by one.

  FIG. 5A shows a slide bar type among time series display types, and a color display area based on color information corresponding to the emotion data for each unit time interval is one-dimensionally represented as a rectangular area. It expands and visualizes the change of the emotion for each unit time interval.

  In the example of FIG. 5A, when time elapses in the horizontal direction, color information corresponding to emotion data in each unit time interval is drawn as a vertical bar along the time elapse. It is a display mode. In this slide bar type, for example, when a captured image is reproduced, it is displayed in the lower or upper column of the display image, and the currently displayed time position is indicated by a bright spot or the like on the slide bar. For example, it is possible to easily know what emotion the user has in the scene of the screen.

  FIG. 5B shows a block type among time-series display types. A color display area based on color information corresponding to the emotion data for each unit time interval is two-dimensionally represented as a rectangular area. And the change in the emotion for each unit time interval is visualized.

  That is, this is a display mode in which a predetermined period in which emotion is to be displayed is equally divided into a plurality of divided time sections (unit time sections), and the divided time sections are displayed in a plurality of stages. This block type is convenient, for example, when the emotional change of the day is expressed, and when the horizontal bar is displayed in four rows every 6 hours and displayed for 24 hours.

  FIG. 5C shows a circle type among the time-series display types. A predetermined period in which emotion is to be displayed is assigned to one round of the annular area, and the color output corresponding to emotion is This is a display mode in which images are sequentially displayed in a predetermined width unit corresponding to the unit time interval in the circumferential direction.

  5 (E) and 5 (F) show the created block type or the circle type display mode with a special effect on the visualized image by the color conversion output, and can be expressed in an artistic manner. It is a thing. For example, a process of smoothing a change part for each unit time interval of emotion using blurring is performed. In addition, a metallic expression is made by adding a highlight or a shadow to a portion with a large color change corresponding to the peak point of emotion change. Further, it may be converted into an artistic expression by performing a geometrical transformation.

  FIG. 5D shows an example of a distribution display type. In this case, a map corresponding to an emotional state (expressed as an emotion) experienced within a predetermined period in which the emotion is to be displayed is created and expressed. This is created using a self-organizing map. In this case, the time information is information for each emotion data unit. As described above, in the case of this example, the time information stores the beginning and end times of a predetermined period in which emotions are to be displayed, and the number in the time direction of each emotion data unit. Of course, time information for each emotion data may be stored one by one in correspondence with each image position on each emotion visualized image.

  According to this distribution display type display mode, it is possible to know which emotion is dominant in a predetermined period. For example, in the distribution display type display, when the portion expressed in yellow dominates and the rest is in red or magenta, it was a feeling of excitement and tension, but it was a pleasant one You can see that there was.

  FIG. 6 is a flowchart for explaining visual data creation processing in the visual data processing unit 112.

  In the example of FIG. 6, the visual data processing unit 112 converts emotion data generated by the emotion analysis processing unit 108 stored in the data storage unit 123 into visualization data including color information using color conversion table information. It is an example in the case of converting and producing | generating emotion visual data (emotional visualization image information).

  First, initial setting is performed (step S101). The contents of this initial setting are selection setting of display type of visualization information of emotion data to be created, setting of length of averaging interval, setting of other attribute data, and the like.

  Next, first, a unit time interval for the first averaging interval is set (step S102). Next, the emotion data within the set unit time interval is averaged, and the emotion parameter value of the average emotion within the unit time interval is calculated (step S103). Next, color conversion output information for the emotion parameter value calculated in step S103 is obtained using the color conversion table information in the color conversion table information storage unit 126 (step S104).

  Next, it is determined whether or not the type of display output mode is a time series display type (step S105), and when it is determined that it is a time series display type, one of the time series display types described above is set. In the display graphic region corresponding to the display type, image data is generated so as to draw the color based on the color conversion output information obtained in step S104 at a position corresponding to the corresponding unit time interval. Temporary storage is performed corresponding to the unit time interval (step S106).

  Then, it is determined whether or not the processing for all the emotion data to be processed has been completed (step S107). When it is determined that all the emotion data has not yet been completed, the unit for the next averaging interval is determined. A time interval is set (step S109), and the processing after step S103 is repeated. If it is determined in step S107 that the processing for all emotion data has been completed, the generated emotion visual data including the image data temporarily stored is stored together with the attribute data (step S108).

  If it is determined in step S105 that the display type is not the time series display type but the distribution display type, the color conversion output information obtained in step S104 is temporarily stored in correspondence with the unit time interval (step S110).

  Then, it is determined whether or not the processing for all the emotion data to be processed has been completed (step S111), and for all the emotion data, when it is determined that the processing has not yet been completed, a unit for the next averaging interval is determined. A time interval is set (step S109), and the processing after step S103 is repeated. If it is determined in step S111 that the processing for all emotion data has been completed, all of the color conversion output information corresponding to the temporarily stored emotion data is used, and the emotions in the target predetermined period are used. Emotional visual data indicating the state distribution is generated and displayed (step S112). Then, the generated emotion visual data (emotion visualized image data) is stored together with the attribute data (step S108). Thus, the emotion visual data generation process is completed.

<Description of recording processing operation>
Next, the flow of recording processing as the imaging information recording / reproducing apparatus 100 will be described with reference to the flowchart of FIG. Each step of the flowchart described below is executed by the CPU 101 using the RAM 104 as a work area in accordance with a program stored in the ROM 103.

  That is, when an instruction to start imaging information recording is given, initialization processing of the time information of the time measuring unit 111, the biological signal processing unit 1072, the emotion analysis processing unit 108, the display unit 127, and the like of the biological information input unit controller 107 is performed. (Step S201). Next, the setting such as designation of the display type of emotion visual data by the user is received and the setting is executed (step S202).

  The captured image data and audio data from the imaging unit 121 are sequentially added to the data storage unit 123 by the data storage unit controller 109 in a format to which time information (year / month / day, hour / minute / second) from the time measuring unit 111 is added. Store (step S203).

  In parallel with this, the biological information input unit controller 107 analyzes the biological information acquired by the biological information input unit 124 and converted into a digital signal by the A / D conversion unit 125 (step S204). Then, the biometric information from the biometric information input unit controller 107 is transferred to the emotion analysis processing unit 108 for emotion analysis, and the emotion data (emotion parameter value) as a result of the analysis is sent together with the time information by the data storage unit controller 109. The data is stored in the data storage unit 123 (step S205).

  In this step S205, emotion data (emotion parameter value) may be calculated for each piece of biological information from the biological information input unit controller 107. In this embodiment, the description has been given with reference to FIG. Similarly to the above, the emotion data is averaged at predetermined time intervals (unit time) to calculate emotion data (emotional parameter values) at the predetermined time intervals. The fixed time (unit time) in this case is a fixed time shorter than the time of the averaging interval set in FIG. Then, emotion data is stored in the data storage unit 123 in association with the time information in a certain time unit. In this embodiment, the emotion data and the captured image data can be associated with each other based on the time information.

  In addition, the emotion data of the analysis result in the emotion analysis processing unit 108 is sent to the visual data processing unit 112, and the emotion data is converted into color information using the color conversion table information in the color conversion table information storage unit 126. Is displayed on the screen of the display unit 127 by the display unit controller 113 (step S206). In this display, the emotional state of the user at the time of recording information while imaging is displayed on the display unit 127 in real time.

  Then, it is determined whether or not a recording end operation input has been made by the user (step S207). When it is determined that a recording end operation input has not been made, the process returns to step S203, and the processing from step S203 onward is repeated.

  If it is determined in step S207 that the user has input the recording end operation, the recorded emotion data in all the recorded time intervals are read from the data storage unit 123 through the data storage unit controller 109 and shown in FIG. In this manner, the emotional visual data of the display type set in step S202 is created and stored in association with the information captured and recorded in the data storage unit 123 (step S208).

  At this time, in step S208, the small icon display image is formed by displaying the emotions in all the recorded time intervals in the designated display type, as described above, in the time series display type or the distribution display type. Information is also generated and stored in the data storage unit 123 in association with information captured and recorded.

  An icon displayed by the icon display image information (hereinafter referred to as an emotion display icon) is displayed as an index for each imaging record information on a list screen of recorded imaging record information at a later time. Can do. The user can infer the emotion of the user when the corresponding imaging record information is recorded from the emotion display icon of the index, which is convenient.

  When the data storage unit 123 uses a removable recording medium such as a DVD, for example, the emotion display icon information is output to the color printer 129 through the printer controller 116 and printed on a sticker or recording paper. To do. Therefore, this color-printed sticker or recording paper can be attached to a DVD or enclosed in a jacket. Thus, by looking at the printed emotion display icon at a later point in time, it is possible to infer from the emotional state of the user when the imaging information recorded on the DVD is recorded, which is convenient.

  Then, management information that associates the captured image data and audio data, emotion data, emotion visual data, and the icon display image information is stored in the data storage unit 123 (step S209).

  The relationship between this management information and each data is shown in FIG. That is, in the management information, the title, the owner, the disclosure attribute, the recording start date and time, and the recording time are recorded for the entire captured and recorded information. For emotional visual data, the address ( (Location information), data attributes, and display type are recorded. For captured image data and audio data, their addresses (location information) and data attributes are recorded, and for emotion data, their addresses (location information). Information attributes, emotion models, emotion parameters, and sampling intervals are recorded.

  The title, the owner, and the like serve as identification information about the entire imaging record information. However, in addition to the title, the owner, and the like, identification information such as an identification number may be added to the management information. In this example, the management information as shown in FIG. 8 is used to store the various types of information shown in FIG. 8 in association with each other. It is not limited to a simple management method. For example, each piece of data may be stored separately, and identification information regarding the above-described imaging record information may be added to each piece of stored data so as to be associated with each other.

  In addition, in the case of the above-described example, the emotion visual data address, the captured image data and audio data addresses, and the emotion data address are addresses in the data storage unit 123. An address on the DVD is used, but when the data is stored in the server device via the network, the URL (Uniform Resource Locator) and the memory address of the server device are used.

  In the above description, the emotional visual data is always generated after the recording is completed, but is stored in the data storage unit 123 only when the user gives an instruction after the recording of the imaging information is completed. The emotion data may be read and generated.

<Reproduction of imaging information and emotional visual data>
In this embodiment, general reproduction in which the recorded imaging information is sequentially reproduced, and reproduction is performed according to the search result corresponding to the emotion designated through the emotion visualization image by the search processing unit 117. There is search and playback.

  FIG. 9 is a flowchart for explaining general reproduction. With reference to FIG. 9, the reproduction process for the imaging information and emotion visual data during general reproduction will be described. 9 is also executed by the CPU 101 using the RAM 104 as a work area in accordance with a program stored in the ROM 103.

  First, the management information stored in the data storage unit 123 is read (step S301). Next, the captured image data is read from the data storage unit 123, supplied to the display unit 127 through the reproduction processing unit 110 and the display unit controller 113, displayed on the display screen, and reproduced. Although not shown, related audio data is read from the data storage unit 123, supplied to a speaker through a reproduction processing unit and an amplifier, and is reproduced acoustically. At this time, the time information about the captured image data and audio data currently being reproduced is managed (step S302).

  Next, emotion visual data is read from the data storage unit 123 and displayed on a part of the image of the captured image data on the screen of the display unit 127. Alternatively, the screen of the display unit 127 may be divided so that the image based on the captured image data and the emotion visualization display image based on the emotion visual data data are displayed in different areas (step S303).

  At this time, in the displayed emotion visualized display image, the time position of the captured image data and audio data currently being reproduced corresponds to which position in the emotion visualized display image corresponds to the bright spot or the like. It is displayed and can be referenced.

  FIGS. 10A to 10E show emotion display images of each display type at this time, and correspond to the currently displayed captured image data by the color of the bright spot position indicated by CS, respectively. Shows emotional state. That is, the bright spot CS indicates an index of the emotional state corresponding to the captured image data. In the case of the time series display type shown in FIGS. 10A to 10C, the position of the bright spot CS indicates the time position together. Then, the position of the bright spot CS changes so as to move to the emotion data position at the corresponding time as the playback proceeds with time.

  During playback, when the viewing user does not perform any operation, the operations in steps S302 and S303 described above are continued until the end of the playback data, and playback processing is continuously performed. However, when the user performs an event operation for changing the playback position such as fast forward, rewind, and jump through the key input operation unit 128, the CPU 101 executes a process corresponding thereto.

  In this embodiment, the bright spot CS on the emotion visualized image is regarded as a cursor mark, and the position of the bright spot CS can be changed with the cursor key. Control is performed so that captured image data and audio data at the time axis position are reproduced.

  That is, the CPU 101 determines whether any event has occurred through the key input interface 114 (step S304). If it is determined that no event has occurred, the CPU 101 returns to step S302 and repeats the above operation.

  If it is determined in step S304 that some event has occurred, it is determined whether or not the operation is a playback position change event operation such as fast forward, rewind, or jump (step S305). When it is determined, the reproduction position of the captured image data and the audio data is changed, and the position of the bright spot CS in the emotion visualized image is changed to a position corresponding to the changed reproduction position (step S306). Then, it returns to step S302 and repeats the above-mentioned operation.

  If it is determined in step S305 that the operation is not an operation for changing the reproduction position, it is determined whether or not the position change of the bright spot CS on the emotion visualized image is specified by the cursor key (step S307). When it is determined that the position change is specified, the time information is searched from the position of the bright spot CS on the emotion visualized image, and the captured image data and audio data at the corresponding time position are read from the data storage unit 123 and reproduced ( Step S308).

  If it is determined in step S307 that the position change of the bright spot CS is not specified, it is determined whether or not the stop key has been operated (step S309). If it is determined that a key other than the stop key has been operated, the operation is performed. The event process corresponding to the key is performed (step S310), and the process returns to step S302. If it is determined in step S309 that the stop key has been operated, this reproduction processing routine is terminated.

<Description of search processing operation>
Next, processing during search reproduction will be described. FIG. 11 and FIG. 12 are flowcharts for explaining the processing operation during the search reproduction. This search process is performed using the search processing unit 117 and the visual data processing unit 112 under the control of the CPU 101. This search process can also be configured by software based on a program executed by the CPU 101 without providing the search processing unit 117.

  In this search reproduction, the position of the bright spot CS in the emotion visualized image shown in FIG. 10 displayed on the screen of the display unit 227 is the key input operation unit 228 by the user or the cursor of the remote control transmitter not shown. This is done by instructing the setting using the key. It is also possible to shift to the search reproduction from the above-described general reproduction state.

  As a method of instructing the setting of the position of the bright spot CS in the search reproduction, in the case of instructing one spot as a spot (spot search) and in the case of the time series display type of FIGS. 10 (A) to 10 (C), There are cases where the time range is designated and set by designating two points along the time. In the case of the distribution display type shown in FIG. 10D, only spot search is possible in the approximate search for search reproduction.

  That is, when the user points to an arbitrary point on the emotion display image of the distribution display type, the search processing unit 117 obtains emotion data from the color information corresponding to the designated position, and approximates the emotion data. A time range having an approximate emotion value is searched. Then, the CPU 101 reproduces an image corresponding to the searched time range.

  Furthermore, as a modification, a display format and a search method as shown in FIG. 10E can be performed. The distribution display type emotion visualized image shown in FIG. 10 (E) is the same as the emotion visualized image in FIG. 10 (D), but in the case of FIG. 10 (E), it slides below the emotion visualized image. A bar 30 is also displayed. This slide bar 30 is displayed corresponding to the time interval TG from the start point to the end point of the image data in a predetermined period in which the emotion is visualized by the emotion visualized image of FIG. Is associated with the start point time of the time interval TG, and the right end is associated with the end point time.

  On the slide bar 30, an operation knob 31 that moves on the slide bar 30 is displayed. The operation knob 31 is configured to move on the slide bar 30 by a predetermined key operation of the user's key input operation unit 128.

  Then, on the emotion visualized image, as shown in FIG. 10E, the emotion data portion corresponding to the position of the operation knob 31 on the slide bar 30 is displayed by the bright spot CS. In addition, a trajectory 32 representing the transition of emotion data from the start point time to the end point time of the time section TG in time series is drawn. Then, by moving the operation knob 31 displayed on the slide bar 30, the bright spot CS is moved on the locus 32.

  That is, when the operation knob 31 is at the left end of the slide bar 30, the bright spot CS is at the position of the start point ST of the locus 32. As the operation knob 31 is gradually moved to the right side, the bright spot CS is on the locus 32. When the operation knob 31 moves to the right end of the slide bar 30, the bright spot CS moves to the position of the end point ED of the locus 32.

  When the display format shown in FIG. 10E is employed, both simple search and approximate search can be performed. That is, in the case of simple search, the operation knob 31 of the slide bar 30 is moved by a predetermined interval to obtain time interval information or color information group corresponding to this interval, and the corresponding image is used as a search key. Search for data. Further, in the case of an approximate search, a search can be performed in the same manner as in FIG. 10D by pointing and setting one point on the trajectory 32 or emotion visualization image.

  The means for operating the time position from the start point to the end point of the time interval corresponding to the distribution display type emotion visualization image is not limited to the slide bar as shown in FIG. It may be something like this. Further, the bright spot CS may be moved directly on the locus 32.

  As described above, since search reproduction includes simple search and approximate search, the user also gives an instruction to select which search. In the case of an approximate search, search setting inputs such as a time range for performing an approximate search and a maximum value of the number of items to be searched for output are also made.

  The position of the bright spot CS is designated and set on the emotion visualized image by any one of the above instruction setting methods, and either a simple search or approximate search is selected and a search setting is input, and the search instruction is issued. When done, the flowcharts of FIGS. 11 and 12 start. This search process is performed by the search processing unit 117 under the control of the CPU 101. Note that the search processing unit 117 can also be configured by software based on a program executed by the CPU 101.

  As shown in FIG. 11, first, the search processing unit 117 accepts a position or range designated on the emotion visualization image, a selection instruction for simple search or approximate search, and a search setting input (step S311). .

  Next, it is determined whether or not the search is a simple search (step S312). When it is determined that the search is simple, the search processing unit 117 detects color information in the image position or range designated on the emotion visualized image, and transfers the color information to the visual data processing unit 112. The visual data processing unit 112 converts this color information into emotion data using the color conversion table information stored in the color conversion table information storage unit 126, and searches the emotion data corresponding to the color information with the search processing unit 117. Send to. The search processing unit 117 receives emotion data corresponding to the color information (step S313).

  Next, the search processing unit 117 searches the emotion data stored in the data storage unit 123 through the data storage unit controller 109, and detects time information in which the received emotion data is recorded. Then, the captured image data and audio data associated with the detected time information are read from the data storage unit 123 and transferred to the reproduction processing unit 110, and the captured image and audio are reproduced as described above (step S314). ).

  At this time, if the image position is designated on the emotion visualized image, the captured image data and the audio data are reproduced starting from the emotion position corresponding to the image position designated. Played sequentially. Further, when the emotion visualized image is a time-series display type and the instruction set on the emotion visualized image is an image range, the reproduction of the captured image data and the sound data is performed in the image range set by the instruction. In the corresponding time range.

  When the emotion visualized image is a distribution display type, only the instruction setting of the image position can be performed on the emotion visualized image in this simple search. This is because in the distribution display type emotion visualization image, the image range designated and set in the rectangular area is discontinuous data in terms of time.

  If it is determined in step S312 that the search is not a simple search, the search processing unit 117 determines that the search is an approximate search, detects the image position or image range designated on the emotion visualized image, and also searches the search range (this In the example, the setting input of the maximum value of the number of cases to be searched and output is detected (step S315).

  Then, the search processing unit 117 determines whether or not the designated image position is a spot-like image position (step S316). And the color information is transferred to the visual data processing unit 112. Then, emotion data corresponding to the color information is received from the visual data processing unit 112, and the emotion data designated by the instruction is detected (step S317).

  Next, the search processing unit 117 searches the emotion data stored in the data storage unit 123 through the data storage unit controller 109 within the set search range for emotion data that approximates the emotion data detected in step S317. Search for. Then, the emotion data detected as an approximation as a result of the search is listed for the maximum number of set search results. Here, when the number of emotion data detected as approximate is larger than the maximum value, the emotion data is listed in order from the latest (step S318).

  Here, retrieval of emotion data that approximates the instructed emotion data can be performed by calculating the following emotion vector distance. That is, since emotion data can be expressed by a plurality of parameters as described above, emotion data can be handled by an n-th order term.

Therefore, the emotion data instructed by the search source
E = E ₀ , E ₁ , E ₂ ,..., E _n−1
E = e ₀ , e ₁ , e ₂ ,..., E _n−1.
The emotion vector distance D between the two emotion data is
D = {k ₀ (E ₀ −e ₀ ) ² + k ₁ (E ₁ −e ₁ ) ² +...
+ Kn _-1 (En _-1 -en _-1 ) ² } ^1/2
Can be obtained.

  The emotion data e having a smaller emotion vector distance D can be detected as being closer to the emotion data E instructed by the search source. Therefore, in step S318, the emotion data can be listed as approximate emotion data in order from the smallest emotion vector distance D.

  The search processing unit 117 displays the emotion data searched as described above on the display unit 127 in order to notify the user (step S319). As one method of displaying the search result, for example, a method of displaying the emotion data portion of the search result with an indicator such as an arrow on the emotion visualization image displayed on the display screen of the display unit 127, for example. There is. As another method, there is a method of sequentially listing times corresponding to searched emotion data on a display screen. Needless to say, the display method of the search result is not limited to this.

  The user selects and designates what he / she wants to reproduce from the search results displayed in step S319. Therefore, the search processing unit 117 waits for a selection input for the search result from the list (step S320), and when it is determined that the selection input has been received, detects the time information corresponding to the selected emotion data, The captured image data and audio data corresponding to the time information are read from the data storage unit 123, and reproduction is executed in the same manner as described above (step S321). Then, this search processing routine ends.

Further, in step S316, the specified is when it is determined that the predetermined image range, the search processing unit 117, instructions specified image range (because it is time-series display type, the time range) The emotion data corresponding to each of the color information is detected using the visual data processing unit 112 and the color conversion table information storage unit 126 in the same manner as in step S313 and step S317 described above (step S331).

  Next, the search processing unit 117 stores, in the data storage unit 123, an emotion data range that changes in the time range set by the search setting input and approximates the detected emotion data change in the designated range. Search from emotion data. This search is also performed by using the emotion vector distance D described above. For example, the emotion data in the data storage unit 123 is extracted in the unit of the emotion data range having the same time width as the search source emotion data range, and is compared with the emotion data in the search source range. Then, an emotion vector distance D between emotion data at the same time position within the time range is obtained. Then, as a result of the search, a list corresponding to the maximum value of the number of set search results is listed as a closer approximation in order from the smallest emotion vector distance D (step S332).

  In the above description, the emotion data range that changes to approximate the emotion data change in the designated range is searched. However, even if the emotion data changes in different ways, the designated range is included. A certain time range including emotion data and similar emotion data may be searched from the emotion data stored in the data storage unit 123.

  Next, the search processing unit 117 displays the emotion data range searched as described above on the display unit 127 in order to notify the user (step S333). As one of the search result display methods in step S333, for example, on the emotion visualization image displayed on the display screen of the display unit 127, the emotion data range of the search result is displayed, for example, with different brightness. There is a way to do that. As another method, there is a method of sequentially listing the time range corresponding to the searched emotion data range on the display screen. Needless to say, the display method of the search result is not limited to this.

  The user selects and designates what he / she wants to reproduce from the search results displayed in step S333. Therefore, the search processing unit 117 waits for a selection input for the search result from the list (step S334), and when determining that the selection input has been accepted, the search processing unit 117 detects a time range corresponding to the selected emotion data range. Then, the captured image data and audio data corresponding to the time range are read from the data storage unit 123, and reproduction is executed in the same manner as described above (step S335). Then, this search processing routine ends.

  In steps S313, S317, and S331 in the above-described example, when the emotion visualized image is a time-series display type, color information corresponding to the position of the bright spot CS on the image is acquired, and the emotion information is used as emotion data. However, this is not a limitation.

  For example, when a certain position on the emotion visualized image is designated and set by the bright spot CS, time information corresponding to the position is detected, and the corresponding emotion data is read from the data storage unit 123 using this time information. May be used as In particular, in the case of a simple search, it may be further simplified to detect time information corresponding to the position of the designated bright spot CS and use the time information as a search key.

  When searching using emotion data converted from color information as a search key as in the above example, if there are multiple images having the same emotion data, it is also necessary to select which image to reproduce from among them. However, when the corresponding emotion data is specified from the time information corresponding to the position of the bright spot CS, it is possible to directly reproduce the image without such trouble.

  Further, in the case of the approximate search, when the time information is detected from the position of the designated bright spot CS, the position of the designated bright spot CS can be specified. The emotion data at the position of the bright spot CS can be excluded from the data or the list of approximate emotion data changes.

  In the search processing shown in the flowcharts of FIGS. 11 and 12, the search mode is set in step S311, but the following modifications may be considered.

  That is, when a time-series display type emotion visualization image is displayed, the user sets the bright spot CS at an arbitrary position. It is assumed that the simple search mode is started at this point. The reproduction of the image data / audio data in this mode is performed in the same manner as in the above embodiment. During this reproduction, the bright spot CS on the emotion visualized image is also displayed with its position changed according to the time information of the reproduction data.

  During this reproduction, the user presses an operation key for instructing approximate search. Then, the device shifts to the recent search mode, detects the corresponding time information or emotion data from the position of the bright spot CS when the operation key is pressed, and uses the detected time information or emotion data as the search key. Search for approximate emotion data.

<Application by storing emotion data or emotion visual data>
As described above, if the data stored in the data storage unit 123 can be accumulated over a long period of time such as one year, two years, three years, etc., what will happen to the past day, month, year? It is possible to visualize and display what was spent in a state of emotion. For this purpose, the capacity of the data storage unit 123 needs to have such a large capacity. However, the example described below proposes a data storage system that can satisfy the above-described requirements even when the capacity of the data storage unit 123 is small.

  FIG. 13 is a block diagram illustrating an example of a data storage system. The data storage system of this example is configured by connecting a plurality of imaging information recording / reproducing apparatuses 100 to a server apparatus 20 composed of, for example, a personal computer through a network 10.

  The server device 20 includes a visual data processing unit 21 and a color conversion table information storage unit 22 having the same configuration as the visual data processing unit 112 and the color conversion table information storage unit 126 of the imaging information recording / reproducing apparatus 100 described above. A capacity data storage unit 23 is provided. Under the control of the CPU 24, emotion visual data generation processing using the color conversion table information in the color conversion table information storage unit 22 in the visual data processing unit 21 is performed, and data is stored in the large-capacity data storage unit 23. And the reading process is controlled. Note that the server device 20 of FIG. 13 shows only main parts necessary for the description here, and the program ROM, RAM, network interface, and other processing units are omitted.

  In this system, each imaging information recording / reproducing device 100 is configured to access the server device 20 with its own device ID (Identification). Then, each imaging information recording / reproducing apparatus 100 sends the captured image data, audio data, emotion data, and emotion visual data stored in its own data storage unit 123 to the server device 20 together with time information for associating them, It is configured so that it can be accumulated.

  In addition, each imaging information recording / reproducing device 100 can request the server device 20 to generate emotion visualized image information on a yearly, monthly, and daily basis from past accumulated data. The server device 20 is configured to generate emotion visualized image information corresponding to the request and send the generated emotion visualized image information to the requested imaging information recording / reproducing device 100.

  Then, the imaging information recording / reproducing device 100 displays the emotion visualized image based on the emotion visualized image information received from the server device 20 on the screen of the display unit 127, and on the screen, the same as the bright spot CS described above on the emotion visualized image. The reproduction position instruction means can specify the reproduction position of the image and sound. When the reproduction position is designated, the imaging information recording / reproduction device 100 is configured to send the reproduction position designation information to the server device 20.

  When the server apparatus 20 receives the reproduction position designation information, the server apparatus 20 reads out the captured image data and audio data at the reproduction time position indicated by the reproduction position designation information from the data storage unit 23, and requests the imaging information recording / reproduction apparatus 100 that has requested. The function to send to.

  14 and 15 are flowcharts showing processing operations in the server device 20 in this system. The processing of each step in this flowchart is executed by the CPU 24 together with the program ROM and work area RAM.

  That is, first, it is determined whether or not an access from any of the imaging information recording / reproducing devices 100 has been received (step S401). When it is determined that no access has been received, other processing is performed (step S402), and the process proceeds to step S401. Return.

  If it is determined in step S401 that an access from any one of the imaging information recording / reproducing devices 100 has been received, the device ID of the accessed imaging information recording / reproducing device 100 is recognized and authenticated (step S403).

  Then, it is determined whether or not a data storage request has been received from the imaging information recording / reproducing apparatus 100 (step S404). If it is determined that the request is a data storage request, then the imaging information sent from the imaging information recording / reproducing apparatus 100 is received. Image data, audio data, emotion data, and emotion visual data are received together with time information for associating them with each other, and stored in the data storage unit 23 in association with the device ID and the time information (step S405). Then, the process returns to step S401.

  If it is determined in step S404 that it is not a data accumulation request, it is determined whether or not it is a past emotion visual data creation request (step S406). If not, it is determined whether or not reproduction position designation information has been received. (Step S407). If it is determined that the reproduction position designation information has not been received, other processing is performed (step S408), and the process returns to step S401.

  If it is determined in step S407 that the reproduction position designation information has been received, the captured image data and audio data stored in correspondence with the apparatus ID of the requested imaging information recording / reproducing apparatus from the data storage unit 23 are stored. The captured image data and audio data at the reproduction time position corresponding to the designated reproduction position are read from the inside, and sent to the requested imaging information recording / reproducing apparatus 100 (step S409). Then, the process returns to step S401.

  If it is determined in step S406 that a past emotion visual data creation request has been received, it is determined whether the request unit is year, month, or day (step S411).

  If it is determined in step S411 that the request is in units of years, the emotion data stored in correspondence with the device ID of the requested imaging information recording / reproducing device from the data storage unit 23 is stored for the designated year. In this example, the emotion data is averaged on a monthly basis, and emotion data (emotional parameters) for each month is calculated (step S412).

  Next, the calculated emotion data for each month is converted into color information using the color conversion table information (step S413). Next, emotion visualized image data capable of expressing monthly emotional changes is created for each designated year (step S414). Then, the created emotion visualization image data is transferred to the requested imaging information recording / reproducing apparatus 100 (step S415). Then, the process returns to step S401.

  FIG. 16 shows an example of an image based on yearly emotion visualized image data. This example is a slide bar type image of a time series display type, and is a case where 2000 to 2002 is designated as a year. It can be seen that the monthly emotional state of each year is visualized by color output. This allows you to see at a glance what emotional state you spent every month in each year.

  If it is determined in step S411 that the request is in units of months, designation of the year and month to be displayed is accepted (step S416). Then, the emotion data of the designated month in the designated year is selected from the emotion data stored in correspondence with the requested device ID of the imaging information recording / reproducing device from the data storage unit 23. Then, averaging is performed on a daily basis, and emotion data (emotional parameters) for each day is calculated (step S417).

  Next, the calculated emotion data for each day is converted into color information using the color conversion table information (step S418). Next, emotion visualized image data that can express a change in emotion in units of one day for a designated month in a designated year is created (step S419). Then, the created emotion visualization image data is transferred to the requested imaging information recording / reproducing apparatus 100 (step S415). Then, the process returns to step S401.

  FIG. 17 shows an example of an image based on monthly emotion visualized image data. This example is a time-series display type block type image in which 2003 is designated as the year and January to March is designated. It can be seen that the emotional state of each day of each month is visualized by color output. Thus, it is possible to understand at a glance what emotional state each day has spent in the designated fiscal year and month.

  Next, when it is determined in step S411 that the request is in units of days, designation of the year, month, and day to be displayed is accepted (step S420). Then, from the emotion data stored in correspondence with the requested device ID of the imaging information recording / reproducing device from the data storage unit 23, the specified month and the specified day of the specified year In this example, the emotion data is averaged for each hour, and emotion data (emotion parameter) for each hour is calculated (step S421).

  Next, the calculated emotion data for each hour is converted into color information using the color conversion table information (step S422). Next, emotion visualized image data capable of expressing an hourly emotion change for each designated day of a designated month in a designated year is created (step S423). Then, the created emotion visualization image data is transferred to the requested imaging information recording / reproducing apparatus 100 (step S415). Then, the process returns to step S401.

  FIG. 18 shows an example of an image based on daily emotion visualized image data. This example is a time-series display type circle type image in which 2003 is designated as the year and all days in April are designated. It can be seen that the state of emotion in hour units for each day of the specified month is visualized by color output. As a result, it is possible to see at a glance what emotional state each hour has spent on the specified day of the specified year and month. In the case of the example in FIG. 18, it is possible to know what kind of emotion has changed on a daily basis in the designated month.

  For example, in the emotion visualized image display as in the example of FIG. 18, the user performs a specific time zone on a specific day on the display screen of the imaging information recording / reproducing apparatus 100 through an instruction unit such as a cursor key. When the position of the cursor mark composed of the above-mentioned bright spot is designated, the reproduction position designation information indicating the designated position is generated and sent to the server device 20.

  As described above, the server device 20 confirms reception of the reproduction position designation information in step S407, and requests the captured image data and audio data at the time position designated by the reproduction position designation information in step S409. To the imaging information recording / reproducing apparatus 100.

  In the above-described example, the emotion data is averaged every month for yearly designation, averaged for every day for monthly designation, and further every hour for daily designation. However, the average period or time length is not limited to this, and may be specified by the user.

  For example, when this system is configured as a home LAN system for home use, the server device can be configured as a personal computer. In this case, an instruction to generate past emotional visual data is used as imaging information. It can be performed not only from the recording / reproducing apparatus but also based on input of a keyboard or mouse of a personal computer.

  In the system of FIG. 13, since the server device 20 stores emotion data of a plurality of persons of the plurality of imaging information recording / reproducing apparatuses 100, the emotions of each of the plurality of persons and the plurality of persons You can also visualize the average emotion.

  For example, when traveling in a group such as a family or a friend, each person records emotion data and emotion visual data based on biological information while performing image recording with the imaging information recording / reproducing apparatus 100. After returning home, emotion data and emotion visual data are stored in the server device 20 through the network 10 together with the image recording data. The server device 20 can generate emotion visualized image data that displays an average emotion of a plurality of people about the trip by, for example, a distribution display type using the accumulated data.

  According to this emotion visualized image, it is possible to visualize and know what the average emotions of the traveled people about the travel were. In addition, the average emotion of the travel participant at each visited place or event during the trip is visualized by color-displaying the average value of the emotion data of the travel participant in the time zone corresponding to the visited place or event. It can be used as reference material for evaluating the quality of travel.

  In addition, since the emotional state of each of a plurality of people can be visualized, various usage forms such as comparing the emotions of other people with their own emotions can be considered.

  In the system of FIG. 13 as well, captured image data, audio data, emotion data, and emotion visualized image data are stored in the data storage unit 23 of the server device 20 in association with time information. In the emotion visualized images in the cases of FIGS. 16 to 18, the user can perform the above-described simple search and approximate search by specifying and instructing as a cursor index such as the bright spot CS.

[Second Embodiment]
The second embodiment is a case where the present invention is applied to a recording / reproducing apparatus having a function of receiving distribution data from a broadcast receiver unit and a server apparatus of a network such as the Internet. In this recording / playback apparatus, at the time of playback of recorded broadcast program content or content downloaded from a server device, it captures the biological information of the viewer of the content, performs emotion analysis, and color-converts the emotion analysis result to visualize the emotion A function of recording information as an image in association with time information together with imaging information is provided.

  Also in this embodiment, the visualization method uses a method of visualizing emotion information as a result of emotion analysis by converting it into color information. In this specification, the content means what can be viewed visually or audibly.

<Hardware configuration example of recording / playback device>
FIG. 19 is a block diagram of the recording / playback apparatus 200 of this embodiment. The recording / reproducing apparatus 200 includes a computer in which a ROM 203 and a RAM 204 are connected to a CPU 201 via a system bus 202.

  Then, with respect to the system bus 202, a broadcast receiver unit 205, a recording encoding / decoding unit 206, a biological information input unit controller 207, an emotion analysis processing unit 208, a data storage unit controller 209, a time measuring unit 211, A visual data processing unit 212, a display unit controller 213, a key input interface 214, a network interface 215, a printer controller 216, an AV (Audio Visual) interface 217, and a search processing unit 218 are connected.

  A biological information input unit 224 is connected to the biological information input unit controller 207 via an A / D conversion unit 225. The configurations of the biometric information input unit 224, A / D conversion unit 225, and biometric information input unit controller 207 are the same as those of the biometric information input unit 124, A / D conversion unit 125, and biometric information input unit controller in the first embodiment described above. The configuration is exactly the same as 107.

  Also, the emotion analysis processing unit 208 has the same configuration as the emotion analysis processing unit 108 of the first embodiment described above.

  The data storage unit 209 is connected to the data storage unit controller 209, for example, a large-capacity optical disk device such as a hard disk drive or a DVD recorder. This configuration is also the same as that of the data storage controller 109 and the data storage unit 123 of the first embodiment described above.

  In this example, in the case of content data of a broadcast program, the data storage unit 223 corresponds to time information of the broadcast time, and the time position from the top of the content is stored in this example. Stored as you can see. In this case, when the content data is compressed and stored by the recording encoding / decoding unit 206 by the MPEG method, the time position information is sequentially inserted into the compressed data in the same manner as described above. . Similarly, in the case of download content, in this example, the content is compressed and stored by the MPEG method. Therefore, time position information is inserted into the compressed content data.

  The timekeeping unit 211 has a configuration similar to that of the timekeeping unit 111 of the first embodiment, and is a clock circuit with a calendar function, and sends date and time information to the system bus 202. In the second embodiment, it also serves as a monitoring timer for timer recording of a broadcast program described later.

  The visual data processing unit 212 has the same configuration as the visual data processing unit 112 of the first embodiment. From the emotion data from the emotion analysis processing unit 208 or the emotion data stored in the data storage unit 223, Similar to the first embodiment, color information as emotion visualization data is obtained using the color conversion table information stored in the color conversion table information storage unit 226, and based on the color information as the visualization data. The emotion visualization image information is generated, and the generated emotion visualization image information is stored in the data storage unit 223 through the data storage controller 209. Similar to the first embodiment, the visual data processing unit 212 also includes role functions of emotion change visualization means and emotion distribution visualization means.

  Further, the visual data processing unit 212 reproduces the content data and emotion visualized image information read from the data storage unit 223 in synchronization with the time axis, and displays them on the screen of the display unit 227 through the display unit controller 213. It has a function to do so. During the reproduction, the content data read from the data storage unit 223 is decoded by the decoding unit of the recording encoding / decoding unit 206 and then supplied to the visual data processing unit 212.

  The display unit controller 213 forms a display image signal to be supplied to the display unit 227 and supplies it to the display unit 227. The display unit 227 is a display such as an LCD (Liquid Crystal Display), and displays an image corresponding to an image signal from the display unit controller 213 on the display screen.

  A key input operation unit 228 is connected to the key input interface 214. The key input operation unit 228 includes a broadcast program selection key in addition to a recording key, a recording reservation key, a reproduction key, a fast forward key, a rewind key, a jump key, a stop key, and the like. When any key of the key input operation unit 228 is operated, the operation input information is supplied to the system bus 202 through the key input interface 214. The CPU 201 determines this operation input information, determines which key has been operated, and executes processing corresponding to the operated key.

  Of course, it is possible to connect a remote control receiver to the system bus 202 so that an input operation using the above-described operation keys can be performed through the remote control transmitter.

  In this example, a color printer 229 is connected to the printer controller 216. As will be described later, an output image based on emotion visualized image information obtained from the visual data processing unit 212 is printed out in color by the color printer 229. It is configured.

  The AV output interface 217 has a function of supplying audio data and / or image data obtained by being decoded by the recording encoding / decoding unit 206 to the monitor device 230 as an analog signal during reproduction. Further, when emotion visual data is stored in the data storage unit 223 in association with audio data and / or image data to be reproduced as described later, an emotion visualized image based on the emotion visual data is also reproduced. In addition, a function is also provided in which the image is displayed on the screen of the monitor device 230 while being superimposed on the screen or in a state where the reproduced image is divided into screens.

  Similar to the first embodiment described above, the search processing unit 218 has a function of executing search processing according to an emotion or emotion range specified through an output image (emotional visualization image) based on emotion visualizing image information. In the second embodiment, the search range can be specified as “only within one content”, “a range of content of a specific genre”, “a range of all content”, and the like. The search mode includes a simple search mode and an approximate search mode, as in the first embodiment described above.

  In this simple search mode and approximate search mode, the data corresponding to emotion data is captured image data or audio data in the case of the first embodiment, but is changed to content data in the second embodiment. Since the rest is the same, detailed description thereof is omitted here.

  The search processing unit 218 of the second embodiment further includes a content search / playback mode capable of searching for content that approximates the emotion after being viewed as a whole. That is, in the second embodiment, an emotion display icon obtained by converting an emotion visualized image into an icon is stored in the data storage unit 223 corresponding to each content, and is displayed in a content list at the time of content playback. When the user designates and designates any emotion display icon, the search processing unit 218 searches for content that can obtain an emotion display icon whose emotion state is similar to the designated emotion display icon.

  In this content search / playback mode, for example, the user can easily find other content for which he / she wants to obtain the same emotion as the content storing emotions when viewed in the past.

  For example, when the user wants to find another content B that wants to experience the same excitement as the very excited content A, the user designates an emotion display icon corresponding to the content A. Then, the search processing unit 218 searches for a candidate for the content B and displays it as a list, so that the user can select a favorite content from the list display. Details of the content search reproduction will be described later.

  In this example, the broadcast receiver unit 205 has a function of receiving digital television broadcasts. The broadcast receiver unit 205 is selected by a broadcast program selection key in the key input operation unit 228 connected to the key input interface 214 or the remote control transmission unit. A channel selection signal according to the station operation is supplied, and a broadcast program desired by the user is received.

  The recording / playback apparatus 200 of this example is configured so that the broadcast program content received by the broadcast receiver unit 205 can be stored in the content storage unit 223 and recorded. Further, as described above, the recording / reproducing apparatus 200 has a timer function by the time measuring unit 211 and a function capable of performing scheduled recording of broadcast program content.

  In addition, the recording / playback apparatus 200 of this example downloads content including music content, image content, and text data such as an electronic novel (some of which may include image information) via the network interface 215, A function of recording in the data storage unit 223 is also provided. The recording encoding unit of the recording encoding / decoding unit 206 performs necessary recording encoding processing when the broadcast program content and the download content are stored in the data storage unit 223 through the data storage unit controller 209.

  Then, when the stored data is read from the data storage unit 223 through the data storage unit controller 209, the decoding unit of the recording encoding / decoding unit 206 performs a decoding process on the read data, As described above, the decoding result data is transferred to the AV output interface 217.

  In the content storage unit 223, actual data of broadcast program content (hereinafter simply referred to as broadcast program content data) received by the broadcast receiver unit 202, downloaded content data, and information related to these contents are stored together. Has been. As information related to this content, in relation to content identification information (content ID (Identification)), in the case of broadcast program content, information such as broadcast program name (title) broadcast date and time is stored. In the case of download content, the title name, download date, data capacity, etc. of the content are stored.

  Further, the content data is stored in the data storage unit 223 with time information (time position information) based on the time information of the broadcast time and / or the reception time added. With this time information (time position information), it is possible to associate emotion data with content data. Therefore, in this embodiment, this time information needs to be recorded at least in units of time for obtaining emotion data.

  When a user (a viewer of content) views the content at the time of recording the content, that is, when the content data is stored in the data storage unit 223, the biological information of the user who is viewing at that time Can be acquired based on the biometric information, and the emotion data of the viewer can be acquired, and the emotion data can be stored in the data storage unit 223 in correspondence with, for example, the content ID and the time information at the time of recording. .

  The recording operation of the emotion data at the time of recording the content data is the same as the recording operation of the emotion data at the time of recording the imaging information in the first embodiment described above, and the emotion representing the emotion when viewing the content Data can be stored in the data storage unit 223 in association with the time information of the content, and can be displayed in real time together with an image of the content being viewed. Then, at the end of recording the content (at the end of viewing), emotion visual data during the viewing period of the content is generated and stored in the data storage unit 223, and the image data of the emotion display icon made up of the emotion visual data is stored. The image data of the generated emotion display icon is also stored in the data storage unit 223.

  However, at the time of scheduled recording or content download, the user generally does not appreciate the broadcast program content or download content. Therefore, in this embodiment, at the time of reproduction, the biological information of the user who is viewing the content is collected, the emotion data of the viewer is acquired based on the biological information, and the emotion data is, for example, the content ID and The data is stored in the data storage unit 223 in correspondence with the time information at the time of recording.

  The image information of the emotion display icon stored in the data storage unit 223 displays a list of reproducible contents stored in the data storage unit 223 prior to playback, as in the case of the first embodiment described above. When doing so, it is displayed on the screen corresponding to each content. As a result, the emotional state when each content has been viewed before can be understood and used as reference data for selecting the content to be viewed.

<Description of normal playback and search playback>
FIG. 20 and FIG. 21 are flowcharts for explaining the processing operation when the content stored in the data storage unit 223 in the recording / playback apparatus 200 is played back. 20 and 21 are executed by the CPU 201 using the RAM 204 as a work area according to a program stored in the ROM 203 and executed by the search processing unit 218. Content playback and content search playback mode in accordance with instruction designation through emotion display icons. As described above, the simple search and approximate search performed through the emotion visualization image are complicated in description, and thus the description thereof is omitted in FIGS. 20 and 21.

  Of course, the search processing unit 218 may be configured by software, similar to the search processing unit 117 of the first embodiment described above.

  Prior to the start of the processing of FIG. 20 and FIG. 21, a designation input for normal reproduction or content search reproduction is accepted. Then, when there is such a designation input, the processing starts from the start of FIG.

  First, the search processing unit 218 determines whether or not the instruction input is content search reproduction (step S501). If it is determined that the instruction search is not content search reproduction, the search processing unit 218 determines that normal reproduction is instructed. The search processing unit 218 displays a list of reproducible contents stored in the data storage unit 223 (step S511 in FIG. 21).

  In this list display, the image of the emotion display icon corresponds to the content that the user has already viewed once and the emotion data, emotion visual data, and emotion display icon data are stored in the data storage unit 223. It is displayed next to the display of the content name.

  Next, the CPU 201 waits for the content to be reproduced to be selected from the displayed content list (step S512), and when determining that the content has been selected, the selected content is the content with the emotion display icon. It is discriminate | determined whether it is (step S513). When it is determined that the selected content is content with an emotion display icon, since the viewer's emotion data has already been accumulated, in this example, the emotion data is not re-accumulated and is selected from the data storage unit 223. The content data read out is read and played back (step S514). However, it is of course possible to discard the previously recorded emotion data and record emotion data and emotion visual data when a new viewing (viewing here) is performed again. Furthermore, it is also possible to record them separately and view their changes, or perform an averaging process.

  At the time of reproducing the content in step S514, an emotion visualized image based on emotion visual data is also displayed on the screen of the monitor device 230 or the screen of the display unit 227, for example. Then, as in the first embodiment described above, the position of the emotion data corresponding to the part currently being viewed in the content being played back in the emotion visualized image is the bright spot CS in FIG. Are displayed so that the correspondence can be understood. If the emotion visualized image is of the time series type, it is possible to know which playback time position the emotion is in the content.

  Next, the CPU 201 continues the playback process until the stop key is operated and a playback stop instruction is issued. When the CPU 201 determines that the playback stop instruction is issued (step S515), the CPU 201 stores the content data from the data storage unit 223. Reading is stopped (step S516). Then, it is determined whether or not the playback mode end operation has been performed (step S517). When it is determined that the playback mode end operation has not been performed, the process returns to step S511, and it is determined that the playback mode end operation has been performed. If so, the reproduction processing routine is terminated.

  Further, when the emotion visualized image is displayed on the screen at the time of reproduction or at the time of reproduction stop, the simple search or approximate search can be designated through the emotion visualized image. When the search processing unit 218 determines that a simple search or approximate search instruction is specified, the search processing unit 218 performs a simple search or a search on the content stored in the data storage unit 223 in the same manner as in the first embodiment described above. Perform an approximate search. Then, the part of the content designated by the user is reproduced from the list of results of the simple search or approximate search.

  The simple search or approximate search may not be performed in the middle of playback as described above, but may be performed through the emotion visualized image by displaying an emotion visualized image of a certain content on the screen. .

  Next, when it is determined in step S513 that the emotion display icon is not included, the CPU 201 waits for an instruction input as to whether or not to analyze the emotion of the user, and determines whether or not to perform emotion analysis (step S518). When it is determined that emotion analysis is not performed, the process proceeds to step S514, and the processes after step S514 described above are executed.

  If it is determined in step S518 that emotion analysis is to be performed, the CPU 201 reproduces content while acquiring biometric information (step S519). At this time, when the user configures the biological information input unit 224, for example, by detecting that the output data is not attached to an electrode, the user is informed that the biological information cannot be acquired. Notification is made to prompt the user to install the electrode, and after confirming the installation of the electrode, the content is played back. In this reproduction, the CPU 201 monitors the time information recorded together with the content data in the data storage unit 223 and also manages the reproduction time position.

  Next, the CPU 201 causes the biometric information input unit controller 207 to analyze the biometric information of the content viewer acquired by the biometric information input unit 224 and converted into a digital signal by the A / D conversion unit 225 (Step S201). S520). Then, the biological information from the biological information input unit controller 207 is transferred to the emotion analysis processing unit 208 for emotion analysis, and the emotion data (emotion parameter value) of the analysis result corresponds to the managed playback time position. Along with the time information, the data storage unit controller 209 stores the data in the data storage unit 223 (step S521).

  In step S521, emotion data (emotion parameter value) may be calculated for each piece of biometric information from the biometric information input unit controller 207. In this embodiment, the description has been given with reference to FIG. Similarly to the above, the emotion data is averaged at predetermined time intervals (unit time) to calculate emotion data (emotional parameter values) at the predetermined time intervals. The fixed time (unit time) in this case is a fixed time shorter than the time of the averaging interval set in FIG. Then, emotion data is stored in the data storage unit 223 in association with the time information in a fixed time unit. In this embodiment, emotion data and content data can be associated with each other based on time information.

  In addition, the emotion data of the analysis result in the emotion analysis processing unit 208 is sent to the visual data processing unit 212, and the emotion data is converted into color information using the color conversion table information in the color conversion table information storage unit 226. Is sent to the monitor device 230 through the AV output interface 217 and displayed on the display screen. Or it is made to display on the screen of the display part 227 by the display part controller 213 (step S522). In this display, the user's emotional state at the time of content playback is displayed in real time on the content playback screen of the monitor device 230 or on the display unit 227.

  Then, it is determined whether or not the reproduction stop operation input is made by the user and the reproduction is completed, or whether or not all the target content data is read and the reproduction is completed (step S523), and the reproduction is terminated. If not, the CPU 201 returns to step S519 and repeats the processing after step S519.

  When it is determined in step S523 that the reproduction has ended, the CPU 201 reads emotion data in all the recorded time intervals from the data storage unit 223 through the data storage unit controller 209, and as shown in FIG. The emotional visual data of the set display type is created and stored in association with the content data stored in the data storage unit 223 (step S524).

  At this time, in step S524, for the playback content, the emotions in all the sections of the watched time period are displayed with the designated display type, as described above, with the time series display type and the distribution display type, and the small emotions. A display icon is also generated and stored in the data storage unit 223 in association with the reproduced content.

  Then, the management information (see FIG. 8) associated with the content data, emotion data, emotion visual data, and image information for display of the emotion display icon stored in the data storage unit 223 is updated. Is performed (step S525). Then, the process proceeds to step S517, where it is determined whether or not a playback mode end operation has been performed. If it is determined that the playback mode end operation has not been performed, the process returns to step S511, and the playback mode end operation has been performed. When it is determined, the reproduction processing routine is terminated.

  Next, when it is determined in step S501 that the content search / playback mode has been designated by the user, the search processing unit 218 is a reproducible content stored in the data storage unit 223 and the emotion display icon data is stored. A list of contents stored together is displayed (step S502). In this content list display, an image of a corresponding emotion display icon is displayed beside the display of each content name.

  Next, the search processing unit 218 waits for the emotion display icon to be searched from the displayed content list to be selected (step S503), and when it is determined that the emotion display icon has been selected, the selected emotion is displayed. Emotional visual data corresponding to the display icon is detected from the data storage unit 223 (step S504).

  Next, the search processing unit 218 searches the data storage unit 223 for the detected emotion visual data and the approximate emotion visual data (step S505). This search can be performed using the emotion vector distance D described above. That is, for example, on the image of the emotion display icon, all the emotion vector distances between the emotion data at the positions corresponding to each other are obtained, and it is determined that the approximation is made when all the emotion vector distances are shorter than a certain distance. Then, it is determined that the smaller the total distance difference is, the closer the approximation is.

  In addition, even if it does not use the emotion data of emotion visual data, it can also be discriminate | determined whether it is approximate on those images using the color information of an emotion display icon or emotion visual data. .

  When the approximate emotional visual data is detected as described above, all the contents corresponding to the emotional visual data are listed and displayed together with the emotion display icon on the screen of the display unit 227 (step S506).

  Then, the search processing unit 228 waits for a selection input regarding the search result from the list (step S507), and when determining that the selection input has been accepted, the search processing unit 228 detects the selected content and stores the data of the content as data. The data is read from the storage unit 223, and reproduction is executed in the same manner as described above (step S514 in FIG. 21).

  As described above, the user can search for content that provides a similar emotional atmosphere by specifying and designating the emotion display icon of the emotional atmosphere that the user wants to view. Then, the user can reproduce the content by selecting the desired content from the searched contents, which is very convenient.

  Note that the image of the emotion display icon described above or the emotion visualized image based on emotion visual data can be output to the color printer 229 through the printer controller 216 and printed in color on a sticker or recording paper. It is also possible to print out the color print output together with the content ID, content title, and the like.

  Therefore, when the data storage unit 223 is composed of a device that uses a removable recording medium such as a DVD, for example, this color-printed sticker or recording paper is attached to the DVD or enclosed in a jacket. Can do. This allows you to see at a glance what emotions you have when viewing the content stored on the DVD by looking at the emotion display icons printed on the stickers and recording paper at a later time. is there.

  In the example of FIG. 17, broadcast program content and content downloaded from a network such as the Internet are stored in the data storage unit, but an AV input terminal is provided, for example, audio data and image data from a DVD player or the like. In response, the data may be stored in the data storage unit 223.

  When the user of the recording / playback apparatus 200 is a family user, for example, emotion data for each content may be recorded for each family member. In that case, a user ID is assigned to each family member, and emotion data is stored in the data storage unit 223 corresponding to the user ID.

  However, in the content list display, all the emotion display icons of the family members may be displayed, or only the emotion display icon of the user who is requesting the reproduction may be displayed. Furthermore, average emotion data of the family may be generated, and a visualized image based on the average emotion data may be displayed as an emotion display icon.

  In addition, the recording / reproducing apparatus 200 is not limited to the example of FIG. 19, and the data storage unit 223 is configured by, for example, a hard disk drive, and in addition to this, a DVD drive is provided to enable recording / reproduction of a DVD. It can also be. In that case, emotion visual data and emotion display icons may be recorded on each DVD. In addition, instead of recording on each DVD, the data storage unit 223 is associated with the identifier of each DVD, and is associated with the reproduction time position information of the content recorded on the DVD, so that the content of the DVD Emotion data, emotion visual data, and emotion display icons can be recorded.

  If the second embodiment is focused only on the playback unit, it goes without saying that the present invention can be applied not only to a recording / playback apparatus but also to a content playback apparatus such as a DVD player.

<Other embodiments>
The present invention is not limited to the visualization of emotions accompanied by viewing images and sounds as described above. In the case of visualization of emotions when listening only to music, the visualization of emotions when a book (electronic document) is read. In this case, the present invention can also be applied to the case of visualizing feelings when viewing an image of a work of art. In these cases, music content (audio information), electronic document content (image information), and artwork content (image information) are recorded in the data storage unit in correspondence with emotion data, as in the above example. It is.

  In addition, the present invention is not limited to the above-described case where the visualization of the average emotions of a plurality of people is applied, and for example, the average emotions of people gathered in a concert, a theme park, or an event are visualized and displayed. It is also applicable to cases. In that case, the image and sound information of the concert, the image and sound information at the theme park, the image and sound information at the event, and the average emotion data of the plurality of persons are recorded in association with each other.

  In the above-described embodiment, color information is used as emotion visualization information. However, for example, it is expressed by a facial expression (real image, simple cartoon, etc.) or by a symbol. There may be. In that case, it is possible to prepare a conversion table of emotion data and facial expressions and symbols, and perform the same process as the color conversion process described above. It is also possible to express emotional changes in light and dark.

1 is a block diagram of an example of an imaging information recording / reproducing apparatus to which an embodiment of a method for searching and reproducing imaging information according to the present invention is applied. FIG. FIG. 2 is a block diagram illustrating a configuration example of a part of FIG. 1. It is a figure which shows the example of the emotion model used for description of embodiment of the search reproduction | regeneration method of the imaging information by this invention. It is a figure which shows an example of the color conversion table information used for embodiment of the search reproduction | regeneration method of the imaging information by this invention. It is a figure for demonstrating from several display types of the visualization output image by embodiment of the search reproduction | regeneration method of the imaging information by this invention. It is a figure which shows the flowchart for describing embodiment of the search reproduction | regeneration method of the imaging information by this invention. It is a figure which shows the flowchart for demonstrating the processing operation at the time of the recording process in the imaging information recording / reproducing apparatus of FIG. FIG. 2 is a diagram for explaining a management structure of stored data in the imaging information recording / reproducing apparatus of FIG. 1. It is a figure which shows the flowchart for demonstrating the processing operation at the time of the reproduction | regeneration processing in the imaging information recording / reproducing apparatus of FIG. It is a figure which shows the example of an emotion visualized image display at the time of the reproduction | regeneration processing in the imaging information recording / reproducing apparatus of FIG. 5 is a part of a flowchart for explaining a search processing operation in the first embodiment of the method for searching and reproducing imaging information according to the present invention. 5 is a part of a flowchart for explaining a search processing operation in the first embodiment of the method for searching and reproducing imaging information according to the present invention. It is a figure which shows an example of the network system with which embodiment of the search reproduction | regeneration method of the imaging information by this invention is applied. It is a figure which shows a part of flowchart for demonstrating the application of the visualization expression using embodiment of the search reproduction | regeneration method of the imaging information by this invention. It is a figure which shows a part of flowchart for demonstrating the application of the visualization expression using embodiment of the search reproduction | regeneration method of the imaging information by this invention. It is a figure which shows the example of the visualization display of an emotion. It is a figure which shows the example of the visualization display of an emotion. It is a figure which shows the example of the visualization display of an emotion. 1 is a block diagram of an example of a recording / playback apparatus to which an embodiment of a content search / playback method according to the present invention is applied. FIG. It is a figure which shows a part of flowchart for demonstrating the reproduction | regeneration processing in the video recording / reproducing apparatus of the example of FIG. It is a figure which shows a part of flowchart for demonstrating the reproduction | regeneration processing in the video recording / reproducing apparatus of the example of FIG.

Claims (37)

An imaging information storage unit in which the captured imaging information is stored corresponding to the time information of the imaging time;
Emotion information representing emotions of the photographer obtained by analyzing biological information acquired from the photographer who performs the imaging is stored in correspondence with time information of the imaging time at which the emotion information was acquired. An information storage unit;
Conversion table information holding means for holding conversion table information indicating a correspondence relationship between the emotion information and visualization information that is information for visualizing the emotion represented by the emotion information;
Conversion processing means for converting the emotion represented by the emotion information in the whole period or a part of the period during which the imaging was performed into the visualization information by the conversion table information ;
Display means for displaying the emotion visible image that will be visualized on the display screen by the visualization information generated by the conversion processing means,
Designating means for allowing a user to specify an arbitrary image positions or any image area on the display screen displayed the emotion visualization,
The visualization information corresponding to the arbitrary image position or the arbitrary image range on the emotion visualization image designated by the user through the designation means is detected, and the emotion information corresponding to the detected visualization information is detected. Emotion information detection means obtained using the conversion table information;
Search means for searching emotion information obtained by the emotion information detection means from the emotion information storage unit;
Means for reading out and reproducing the imaging information corresponding to the time position of the emotion information searched by the search means from the imaging information storage unit;
An apparatus for retrieving and reproducing imaging information comprising: An imaging information storage unit in which the captured imaging information is stored corresponding to the time information of the imaging time;
Emotion information representing emotions of the photographer obtained by analyzing biological information acquired from the photographer who performs the imaging is stored in correspondence with time information of the imaging time at which the emotion information was acquired. An information storage unit;
Conversion table information holding means for holding conversion table information indicating a correspondence relationship between the emotion information and visualization information that is information for visualizing the emotion represented by the emotion information;
Conversion processing means for converting the emotion represented by the emotion information in the whole period or a part of the period during which the imaging was performed into the visualization information in time series using the conversion table information ;
Display means for displaying said conversion processing means display screen emotions visible image that will be visualized in time series by the Visualization of the generated the time series manner,
Designating means for allowing a user to specify an arbitrary image positions or any image area on the display screen displayed the emotion visualization,
The visualization information corresponding to the arbitrary image position or the arbitrary image range on the emotion visualization image designated by the user through the designation means is detected, and the time position in time series of the detected visualization information is determined. A search means for searching;
The imaging information corresponding to the retrieved the time position in the searching means, and means for reading out and reproducing from the imaging information storage unit,
An apparatus for retrieving and reproducing imaging information comprising: The apparatus for searching and reproducing imaging information according to claim 1 or 2,
Before asked visualization information, a color information,
The conversion table information holding means stage as the conversion table information comprises said emotion information, the color conversion table information indicating a correspondence relationship between the color corresponding to the emotion represented by the emotion information,
The conversion processing means converts the input emotion information into color information corresponding to the emotion represented by the emotion information using the color conversion table information, and obtains the visualization information including the color information.
Search and reproducing apparatus of an imaging information. The apparatus for searching and reproducing imaging information according to claim 3,
The emotion visualized image visualizes the color output based on the color information constituting the visualization information from the conversion processing means in correspondence with the passage of time, so that the emotion in the period in which the imaging information is recorded is displayed. Change is visualized by color
Search and reproducing apparatus of an imaging information. The apparatus for searching and reproducing imaging information according to claim 3,
In the emotion visualized image, the imaging information is recorded by visualizing the ratio of the color output based on the color information constituting the visualization information from the conversion processing unit for the period in which the imaging information is recorded. Visualize the state distribution of the emotion within a certain period
Search and reproducing apparatus of an imaging information. An imaging information storage unit in which the captured imaging information is stored corresponding to the time information of the imaging time;
Emotion information representing emotions of the photographer obtained by analyzing biological information acquired from the photographer who performs the imaging is stored in correspondence with time information of the imaging time at which the emotion information was acquired. An information storage unit;
Conversion table information holding means for holding conversion table information indicating a correspondence relationship between the emotion information and visualization information that is information for visualizing the emotion represented by the emotion information;
Conversion processing means for converting the emotion represented by the emotion information in the whole period or a part of the period during which the imaging was performed into the visualization information by the conversion table information ;
Display means for displaying the emotion visible image that will be visualized on the display screen by the visualization information generated by the conversion processing means,
Designating means for allowing a user to specify an arbitrary image positions or any image area on the display screen displayed the emotion visualization,
The visualization information corresponding to the arbitrary image position or the arbitrary image range on the emotion visualization image designated by the user through the designation means is detected, and the emotion information corresponding to the detected visualization information is detected. Emotion information detection means obtained using the conversion table information;
Search means for searching emotion information representing an emotion similar to the emotion represented by the emotion information obtained by the emotion information detection means from the emotion information stored in the emotion information storage unit;
Means for reading out and reproducing the imaging information corresponding to the time position of the searched emotion information from the imaging information storage unit;
Find reproducing apparatus that an imaging information comprising a. An imaging information storage unit in which the captured imaging information is stored corresponding to the time information of the imaging time;
Emotion information representing emotions of the photographer obtained by analyzing biological information acquired from the photographer who performs the imaging is stored in correspondence with time information of the imaging time at which the emotion information was acquired. An information storage unit;
Conversion table information holding means for holding conversion table information indicating a correspondence relationship between the emotion information and visualization information that is information for visualizing the emotion represented by the emotion information;
Conversion processing means for converting the emotion represented by the emotion information in the whole period or a part of the period during which the imaging was performed into the visualization information in time series using the conversion table information ;
Display means for displaying said conversion processing means display screen emotions visible image that will be visualized in time series by the Visualization of the generated the time series manner,
A designation means for the user to designate any image position or any image range on the emotion visualization image displayed on the display screen;
The visualization information corresponding to the arbitrary image position or the arbitrary image range on the emotion visualization image designated by the user through the designation means is detected, and the time position in time series of the detected visualization information is determined. Obtaining time position detecting means;
Retrieval means for retrieving emotion information representing emotions similar to emotions represented by emotion information corresponding to the time position obtained by the time position information detection means from emotion information stored in the emotion information storage unit;
Means for reading out and reproducing the imaging information corresponding to the time position of the searched emotion information from the imaging information storage unit;
Find reproducing apparatus that an imaging information comprising a. The apparatus for searching and reproducing imaging information according to claim 6 or 7,
More when the emotion information is retrieved, the search reproduction apparatus further comprising Ru IMAGING information selection means for selecting one of the emotion information of which at the searching means. The apparatus for searching and reproducing imaging information according to claim 6 or 7,
Before asked visualization information, a color information,
The conversion table information holding means stage as the conversion table information comprises said emotion information, the color conversion table information indicating a correspondence relationship between the color corresponding to the emotion represented by the emotion information,
The conversion processing means converts the input emotion information into color information corresponding to the emotion represented by the emotion information using the color conversion table information, and obtains the visualization information including the color information.
Search and reproducing apparatus of an imaging information. The imaging information retrieval / playback apparatus according to claim 9,
The emotion visualized image visualizes the color output based on the color information constituting the visualization information from the conversion processing means in correspondence with the passage of time, so that the emotion in the period in which the imaging information is recorded is displayed. Change is visualized by color
Search and reproducing apparatus of an imaging information. The imaging information retrieval / playback apparatus according to claim 9,
In the emotion visualized image, the imaging information is recorded by visualizing the ratio of the color output based on the color information constituting the visualization information from the conversion processing unit for the period in which the imaging information is recorded. Visualize the state distribution of the emotion within a certain period
Search and reproducing apparatus of an imaging information. An imaging information storage unit in which captured imaging information is stored in association with time information of an imaging time;
Emotion information representing the photographer's emotion obtained by analyzing biological information acquired from the photographer who performs the imaging is stored in correspondence with the time information of the imaging time at which the emotion information was acquired. An information storage unit;
Conversion table information holding means for holding conversion table information indicating a correspondence relationship between the emotion information and visualization information that is information for visualizing the emotion represented by the emotion information;
Conversion processing means for converting the emotion represented by the emotion information in the whole period or a part of the period during which the imaging was performed into the visualization information by the conversion table information ;
Display means for displaying the emotion visible image that will be visualized on the display screen by the visualization information generated by the conversion processing means,
The apparatus having a, a method of searching for the imaging information corresponding to the emotion that is specified through the emotion visualization,
In the display the on emotion visible image displayed on the screen, a designation accepting step of detecting any image position or any image range specified by the user,
The visualization information corresponding to the arbitrary image position or the arbitrary image range on the emotion visualization image detected by the designation reception step is detected, and the emotion information corresponding to the detected visualization information is converted to the conversion information. An emotion information detection process obtained using table information;
A search step for searching the emotion information obtained in the emotion information detection step from the emotion information storage unit;
Reading and reproducing the imaging information corresponding to the time position of the emotion information searched in the search step from the imaging information storage unit;
A method for retrieving and reproducing imaging information comprising: An imaging information storage unit in which the captured imaging information is stored in correspondence with the time information of the imaging time;
Emotion information representing the photographer's emotion obtained by analyzing biological information acquired from the photographer who performs the imaging is stored in correspondence with the time information of the imaging time at which the emotion information was acquired. An information storage unit;
Conversion table information holding means for holding conversion table information indicating a correspondence relationship between the emotion information and visualization information that is information for visualizing the emotion represented by the emotion information;
Conversion processing means for converting the emotion represented by the emotion information in the whole period or a part of the period when the imaging was performed into the visualization information in time series using the conversion table information ;
Display means for displaying said conversion processing means display screen emotions visible image that will be visualized in time series by the Visualization of the generated the time series manner,
The apparatus having a, a method of searching for the imaging information corresponding to the emotion that is specified through the emotion visualization,
In the display the on emotion visible image displayed on the screen, a designation accepting step of detecting any image position or any image range specified by the user,
The visualization information corresponding to the arbitrary image position or the arbitrary image range on the emotion visualization image detected by the designation receiving step is detected, and a time position in the time series of the detected visualization information is searched. Search process;
The imaging information corresponding to the said time position search in the search step, a step of reading out and reproducing from the imaging information storage unit,
A method for retrieving and reproducing imaging information comprising: An imaging information storage unit in which captured imaging information is stored in association with time information of an imaging time;
Emotion information representing the photographer's emotion obtained by analyzing biological information acquired from the photographer who performs the imaging is stored in correspondence with the time information of the imaging time at which the emotion information was acquired. An information storage unit;
Conversion table information holding means for holding conversion table information indicating a correspondence relationship between the emotion information and visualization information that is information for visualizing the emotion represented by the emotion information;
Conversion processing means for converting the emotion represented by the emotion information in the whole period or a part of the period during which the imaging was performed into the visualization information in time series using the conversion table information ;
Display means for displaying said conversion processing means display screen emotions visible image that will be visualized in time series by the Visualization of the generated the time series manner,
A method for searching for the imaging information corresponding to emotion information designated through the emotion visualization image, comprising:
In the display the on emotion visible image displayed on the screen, a designation accepting step of detecting any image position or any image range specified by the user,
The visualization information corresponding to the arbitrary image position or the arbitrary image range on the emotion visualization image detected by the designation receiving step is detected, and the emotion information corresponding to the detected visualization information is An emotion information detection step obtained using the conversion table information;
Searching the emotion information stored in the emotion information storage unit for emotion information that approximates the emotion information detected in the emotion information detection step;
Reading and reproducing the imaging information corresponding to the time position of the searched emotion information from the imaging information storage unit;
A method for retrieving and reproducing imaging information comprising: An imaging information storage unit in which captured imaging information is stored in association with time information of an imaging time;
Emotion information representing the photographer's emotion obtained by analyzing biological information acquired from the photographer who performs the imaging is stored in correspondence with the time information of the imaging time at which the emotion information was acquired. An information storage unit;
Conversion table information holding means for holding conversion table information indicating a correspondence relationship between the emotion information and visualization information that is information for visualizing the emotion represented by the emotion information;
Conversion processing means for converting the emotion represented by the emotion information in the whole period or a part of the period during which the imaging was performed into the visualization information in time series using the conversion table information ;
Display means for displaying said conversion processing means display screen emotions visible image that will be visualized in time series by the Visualization of the generated the time series manner,
A method for searching for the imaging information corresponding to emotion information designated through the emotion visualization image, comprising:
In the display the on emotion visible image displayed on the screen, a designation accepting step of detecting any image position or any image range specified by the user,
Time for detecting the visualization information corresponding to the arbitrary image position or the arbitrary image range on the emotion visualization image detected by the designation receiving step, and obtaining a time position in the time series of the detected visualization information A time position detecting step for obtaining a position ;
Searching emotion information stored in the emotion information storage unit for emotion information similar to emotion information corresponding to the time position detected in the time position detection step;
Reading and reproducing the imaging information corresponding to the time position of the searched emotion information from the imaging information storage unit;
A method for retrieving and reproducing imaging information comprising: A content storage unit in which content data is stored corresponding to the time position information;
Emotion information storage unit in which emotion information representing the viewer's emotion obtained by analyzing the biological information of the viewer who has viewed the content is stored corresponding to the time position information from which the emotion information was acquired When,
Conversion table information holding means for holding conversion table information indicating a correspondence relationship between the emotion information and visualization information that is information for visualizing the emotion represented by the emotion information;
Conversion processing means for converting the emotion represented by the emotion information when the viewer appreciates the content into the visualization information using the conversion table information ;
Display means for displaying the emotion visible image that will be visualized on the display screen by the visualization information generated by the conversion processing means,
Designating means for allowing a user to specify an arbitrary image positions or any image area on the display screen displayed the emotion visualization,
The visualization information corresponding to the arbitrary image position or the arbitrary image range on the emotion visualization image designated by the user through the designation means is detected, and the emotion information corresponding to the detected visualization information is Emotion information detection means obtained using the conversion table information;
Search means for searching emotion information obtained by the emotion information detection means from the emotion information storage unit;
The content data corresponding to the time position of the emotion information searched by the searching means, and means for reading out and reproducing from the content storage unit,
A content search / playback apparatus comprising: A content storage unit in which content data is stored corresponding to the time position from the beginning of the content;
Emotion information representing the viewer's emotion obtained by analyzing the biological information of the viewer who has viewed the content is stored corresponding to the time information from the beginning of the content from which the emotion information was acquired. An emotion information storage unit;
Conversion table information holding means for holding conversion table information indicating a correspondence relationship between the emotion information and visualization information that is information for visualizing the emotion represented by the emotion information;
Conversion processing means for converting emotion expressed by the emotion information when the viewer appreciates the content into the visualization information in time series using the conversion table information ;
Display means for displaying said conversion processing means display screen emotions visible image that will be visualized in time series by the Visualization of the generated the time series manner,
Designating means for allowing a user to specify an arbitrary image positions or any image area on the display screen displayed the emotion visualization,
The visualization information corresponding to the arbitrary image position or the arbitrary image range on the emotion visualization image designated by the user through the designation means is detected, and the time position in time series of the detected visualization information is determined. A search means for searching;
The content data corresponding to the retrieved the time position in the searching means, and means for reading out and reproducing from the content storage unit,
A content search / playback apparatus comprising: The content search / playback apparatus according to claim 16 or 17,
Before asked visualization information, a color information,
The conversion table information holding means stage as the conversion table information comprises said emotion information, the color conversion table information indicating a correspondence relationship between the color corresponding to the emotion represented by the emotion information,
The conversion processing means converts the input emotion information into color information corresponding to the emotion represented by the emotion information using the color conversion table information, and obtains the visualization information including the color information.
Search and reproducing apparatus of the content. The content search / playback apparatus according to claim 18,
The emotion visualized image is a color output based on the color information that constitutes the visualization information from the conversion processing means, and is visualized in correspondence with the passage of time, whereby the change in emotion during the viewing period of the content is color-coded. Is visualized by
Search and reproducing apparatus of the content. The content search / playback apparatus according to claim 18,
The emotion visualized image is visualized within a viewing period of the content by visualizing a ratio of a color output based on color information constituting the visualization information from the conversion processing unit for a period in which the imaging information is recorded. Visualizing the state distribution of emotions
Search and reproducing apparatus of the content. A content storage unit in which data of a plurality of contents are stored corresponding to the time position information;
Emotion information storage unit in which emotion information representing the viewer's emotion obtained by analyzing the biological information of the viewer who has viewed the content is stored corresponding to the time position information from which the emotion information was acquired When,
Conversion table information holding means for holding conversion table information indicating a correspondence relationship between the emotion information and visualization information that is information for visualizing the emotion represented by the emotion information;
Conversion processing means for converting the emotion represented by the emotion information when the viewer appreciates the content into the visualization information using the conversion table information ;
And emotion visible image information storage unit for storing information of visualized Ru emotion visualization by the visualization information generated by the conversion processing means,
Display means for displaying emotion visualized image information stored in the emotion visualized image information storage unit on a display screen;
A designation means for the user to designate an arbitrary emotion visualization image from among the plurality of emotion visualization images displayed on the display screen;
Emotion visualization image information stored in the emotion visualization image information storage unit is searched for emotion visualization image information that becomes an emotion visualization image that approximates the emotion visualization image specified through the specifying means, and the emotion visualization image information of the search result Search and presentation means for presenting content corresponding to
Means for reading out and reproducing the data of the presented content from the content storage unit;
Search and reproducing apparatus of Turkey content with a. The content search / playback apparatus according to claim 21,
When said plurality of contents in a search presenting means presented, the search reproduction apparatus further comprising Turkey content selection means for selecting one content of them. The content search / playback apparatus according to claim 21,
Before asked visualization information, a color information,
The conversion table information holding means stage as the conversion table information comprises said emotion information, the color conversion table information indicating a correspondence relationship between the color corresponding to the emotion represented by the emotion information,
The conversion processing means converts the input emotion information into color information corresponding to the emotion represented by the emotion information using the color conversion table information, and obtains the visualization information including the color information.
Search and reproducing apparatus of the content. The content search / playback apparatus according to claim 23,
The emotion visualized image is a color output based on the color information that constitutes the visualization information from the conversion processing means, and is visualized in correspondence with the passage of time, whereby the change in emotion during the viewing period of the content is color-coded. Is visualized by
Search and reproducing apparatus of the content. The content search / playback apparatus according to claim 23,
The emotion visualized image is visualized within a viewing period of the content by visualizing a ratio of a color output based on color information constituting the visualization information from the conversion processing unit for a period in which the imaging information is recorded. Visualizing the state distribution of emotions
Search and reproducing apparatus of the content. A content storage unit in which content data is stored corresponding to the time position information;
Emotion information storage unit in which emotion information representing the viewer's emotion obtained by analyzing the biological information of the viewer who has viewed the content is stored corresponding to the time position information from which the emotion information was acquired When,
Conversion table information holding means for holding conversion table information indicating a correspondence relationship between the emotion information and visualization information that is information for visualizing the emotion represented by the emotion information;
Conversion processing means for converting the emotion represented by the emotion information when the viewer appreciates the content into the visualization information using the conversion table information ;
Display means for displaying the emotion visible image that will be visualized on the display screen by the visualization information generated by the conversion processing means,
The content playback device comprising a, a method of searching a portion or the contents of the content corresponding to the emotion that is specified through the emotion visualization,
In the display the on emotion visible image displayed on the screen, a designation accepting step of detecting any image position or any image range specified by the user,
Detection obtained by detecting the visualization information corresponding to the arbitrary image position or the arbitrary image range detected by the designation receiving step, and using the emotion information corresponding to the detected visualization information and the conversion table information Process,
A search step for searching the emotion information detected in the detection step from the emotion information storage unit;
Reading the content data corresponding to the time position of the emotion information searched in the search step from the content storage unit and playing it back;
A method for retrieving and reproducing content comprising: A content storage unit in which content data is stored corresponding to the time position information;
Emotion information storage unit in which emotion information representing the viewer's emotion obtained by analyzing the biological information of the viewer who has viewed the content is stored corresponding to the time position information from which the emotion information was acquired When,
Conversion table information holding means for holding conversion table information indicating a correspondence relationship between the emotion information and visualization information that is information for visualizing the emotion represented by the emotion information;
Conversion processing means for converting emotion expressed by the emotion information when the viewer appreciates the content into the visualization information in time series using the conversion table information ;
Display means for displaying said conversion processing means display screen emotions visible image that will be visualized in time series by the Visualization of the generated the time series manner,
In the content reproduction apparatus comprising: a method for searching a part of the content or the content corresponding to emotion information designated through the emotion visualization image,
In the display the on emotion visible image displayed on the screen, a designation accepting step of detecting any image position or any image range specified by the user,
The visualization information corresponding to the arbitrary image position or the arbitrary image range on the emotion visualization image detected by the designation receiving step is detected, and a time position in the time series of the detected visualization information is searched. Search process;
The content data corresponding to the said time position search in the search step, a step of reading out and reproducing from the content storage unit,
A method for retrieving and reproducing content comprising: A content storage unit in which content data is stored corresponding to the time position information;
Emotion information storage unit in which emotion information representing the viewer's emotion obtained by analyzing the biological information of the viewer who has viewed the content is stored corresponding to the time position information from which the emotion information was acquired When,
Conversion table information holding means for holding conversion table information indicating a correspondence relationship between the emotion information and visualization information that is information for visualizing the emotion represented by the emotion information;
Conversion processing means for converting emotion expressed by the emotion information when the viewer appreciates the content into the visualization information in time series using the conversion table information ;
Display means for displaying said conversion processing means display screen emotions visible image that will be visualized in time series by the Visualization of the generated the time series manner,
In the content reproduction apparatus comprising: a method for searching a part of the content or the content corresponding to emotion information designated through the emotion visualization image,
A designation receiving step for accepting designation of an arbitrary emotion visualization image selected from the plurality of emotion visualization images displayed on the display screen;
The emotion visualized image information stored in the emotion visualized image information storage unit is searched for emotion visualized image information that becomes an emotion visualized image that approximates the emotion visualized image designated by the designation receiving step, and the emotion visualized image as a search result A search and presentation process for presenting content corresponding to information;
Reading and presenting the data of the presented content from the content storage unit;
Search reproduction method of Turkey content with a. Emotion information storing the emotion information representing the emotion of the person obtained by analyzing the biological information acquired from a person, corresponding to the time information acquired the emotion information;
Conversion table information holding means for holding conversion table information indicating a correspondence relationship between the emotion information and visualization information that is information for visualizing the emotion represented by the emotion information;
Conversion processing means for converting the emotion represented by the emotion information stored in the emotion information storage unit into the time-series visualization information using the conversion table information ;
Display means for displaying said conversion processing means display screen emotions visible image that will be visualized in time series by the Visualization of the generated the time series manner,
Designating means for allowing a user to specify an arbitrary image positions or any image area on the display screen displayed the emotion visualization,
Search for detecting the visualization information corresponding to the arbitrary image position or the arbitrary image range on the emotion visualization image specified through the specifying means, and searching for a time position in the time series of the detected visualization information Means,
Means for informing the user of the time position of the search result;
Feeling Ru with the information retrieval apparatus. Emotion information storing the emotion information representing the emotion of the person obtained by analyzing the biological information acquired from a person, corresponding to the time information acquired the emotion information;
Conversion table information holding means for holding conversion table information indicating a correspondence relationship between the emotion information and visualization information that is information for visualizing the emotion represented by the emotion information;
Conversion processing means for converting the emotion represented by the emotion information stored in the emotion information storage unit into the visualization information using the conversion table information ;
Display means for displaying the emotion visible image that will be visualized on the display screen by the visualization information generated by the conversion processing means,
Designating means for allowing a user to specify an arbitrary image positions or any image area on the display screen displayed the emotion visualization,
The visualization information corresponding to the arbitrary image position or the arbitrary image range on the emotion visualization image designated through the designation means is detected, and the emotion information corresponding to the detected visualization information is converted to the conversion table. Means for detecting using information;
Search means for searching emotion information stored in the emotion information storage unit for emotion information similar to the detected emotion information;
Means for informing the user so that the time corresponding to the emotion information or emotion information range of the search result is known;
Feeling Ru with the information retrieval apparatus. The emotion retrieval apparatus according to claim 29 or claim 30 , wherein
Before asked visualization information, a color information,
The conversion table information holding means stage as the conversion table information comprises said emotion information, the color conversion table information indicating a correspondence relationship between the color corresponding to the emotion represented by the emotion information,
The conversion processing means converts the input emotion information into color information corresponding to the emotion represented by the emotion information using the color conversion table information, and obtains the visualization information including the color information.
Sensitive information retrieval apparatus. The emotion retrieval apparatus according to claim 31, wherein
The emotion visualization image visualizes a change in emotion by color by visualizing a color output based on color information constituting the visualization information from the conversion processing unit in correspondence with the passage of time.
Sensitive information retrieval apparatus. The emotion retrieval apparatus according to claim 31, wherein
The emotion visualized image is a state distribution of the emotion within the predetermined period by visualizing a ratio of a color output based on color information constituting the visualization information from the conversion processing unit for a predetermined period. Is to visualize
Sensitive information retrieval apparatus. The emotion retrieval apparatus according to claim 29 or claim 30 , wherein
The emotion information is acquired every unit time within the period for each period of arbitrary length, and is stored in the emotion information storage unit.
The emotion visualized image displays, for each period including the emotion information, color information corresponding to the emotion information in the period corresponding to the passage of time, thereby changing the emotion in the period. To visualize,
The search means has a state distribution that approximates the change of the emotion in the designated emotion visualized image when any of the emotion visualized images displayed on the display screen is designated by the designation means. Search for emotion visualization images
Sensitive information retrieval apparatus. The emotion retrieval apparatus according to claim 29 or claim 30 , wherein
The emotion information is acquired every unit time within the period for each period of arbitrary length, and is stored in the emotion information storage unit.
The emotion visualization image visualizes the state distribution of the emotion within the period for each period including the emotion information,
The search means, when any of the emotion visualized images displayed on the display screen is designated by the designation means, a state distribution that approximates the state distribution of the emotion in the designated emotion visualized image. To search for emotion visualization images
Sensitive information retrieval apparatus. Emotion information storing the emotion information representing the emotion of the person obtained by analyzing the biological information acquired from a person, corresponding to the time information acquired the emotion information;
Conversion table information holding means for holding conversion table information indicating a correspondence relationship between the emotion information and visualization information that is information for visualizing the emotion represented by the emotion information;
Conversion processing means for converting the emotion represented by the emotion information stored in the emotion information storage unit into the time-series visualization information using the conversion table information ;
Display means for displaying said conversion processing means display screen emotions visible image that will be visualized in time series by the Visualization of the generated the time series manner,
An emotion search method in an apparatus comprising:
A designation receiving step for accepting designation by a user of an arbitrary image position or an arbitrary image range on the emotion visualization image displayed on the display screen;
A search step of searching for a time position corresponding to the visualization information of the arbitrary image position or the arbitrary image range specified in the specification reception step;
Informing the user of the time position of the search result in the search step;
An emotion retrieval method characterized by comprising: Emotion information storing the emotion information representing the emotion of the person obtained by analyzing the biological information acquired from a person, corresponding to the time information acquired the emotion information;
Conversion table information holding means for holding conversion table information indicating a correspondence relationship between the emotion information and visualization information that is information for visualizing the emotion represented by the emotion information;
Conversion processing means for converting the emotion represented by the emotion information stored in the emotion information storage unit into the time-series visualization information using the conversion table information ;
Display means for displaying said conversion processing means display screen emotions visible image that will be visualized in time series by the Visualization of the generated the time series manner,
An emotion search method in an apparatus comprising:
A designation accepting step for accepting a designation by the user of any image position or any image range on the display screen displayed the emotion visualization,
The visualization information corresponding to the arbitrary image position or the arbitrary image range on the emotion visualization image designated in the designation receiving step is detected, and the emotion information corresponding to the detected visualization information is converted to the conversion information. A detection step of detecting using table information;
A search step for searching emotion information stored in the emotion information storage unit for emotion information approximate to the emotion information detected in the detection step;
A step of notifying the user of the time corresponding to the emotion information or emotion information range of the search result in the search step;
An emotion retrieval method characterized by comprising:

Images