JP2007299255A - Media expression document information generating system - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide an emotional expression document information generating system for easily generating emotional expression document which reflects emotion of a speaker. <P>SOLUTION: A context and emotion information acquiring device 23 in the system acquires context information "it is fine today", an emotional vector (smile, anger, sadness) when image information and voice information are acquired from a camera 25 and a microphone 27. A media expression document information generating device 21 determines a main word as "weather" from the context information. Furthermore, an emotional direction vector (θ, ϕ), and strength of the emotion are acquired from the emotional vector. In addition, corresponding emotional reflection parameter and emotional reflection logic are acquired. Furthermore, media expression information of acquired media expression is adjusted with respect to the emotional reflection logic by using the emotional direction vector (θ, ϕ), the strength of the emotion, and a set parameter ID. Then, the media expression document is generated based on the adjusted media expression, context information "it is fine today", and the main word information "weather". <P>COPYRIGHT: (C)2008,JPO&INPIT

Description

  The present invention relates to an emotion expression document information generation system that generates an emotion expression document that reflects the emotion of a speaker, and particularly relates to a system that reflects an emotion when a speaker speaks.

  Conventionally, there are the following methods for expressing human emotions on a computer. For example, when generating an e-mail, auxiliary information such as a still image, a moving image, ASCII art, and voice (hereinafter, referred to as a still image, a moving image, an ASCII art, and a voice) in order to express the sender's emotion together with a sentence that the e-mail sender wants to convey , And media expression information.) May be supplemented to supplement the text and promote clear communication of intention. Media expression information includes emoticons ((^ _ ^), m (_ _) m, etc.), ASCII art (Kita ━━━━━━━ (゜ ∀ ゜) ━━━━━━━ !! etc.) There is ASCII art.

  In the field of voice recognition technology, there is an invention that recognizes emotion from voice information and outputs information corresponding to the emotion. For example, in Japanese Patent Application Laid-Open No. 2005-352892 “Information Processing Device and Information Processing Program”, an emotion of a speaker is expressed by estimating the emotion of the speaker from input speech and changing the face image to be output.

JP-A-2005-352892

  The sentence creation method having the media expression information described above has the following problems. There is a problem that a person who intends to create a text such as an e-mail takes time to generate media expression information such as ASCII art that expresses his / her emotion each time.

  Moreover, in the above-mentioned Japanese Patent Application Laid-Open No. 2005-352892, the face image to be output is not associated with the context where the speaker speaks. Therefore, there is a problem that it is not possible to supplement the context in which the speaker speaks and promote clear communication of will.

  An object of the present invention is to provide an emotion expression document information generation system that easily generates an emotion expression document that reflects an emotion when a speaker speaks.

  Means for solving the problems relating to the present invention and effects of the present invention will be described below.

  In the emotion expression document information generation system, the emotion expression document information generation device, the emotion expression document information generation program, and the emotion expression document information generation method according to the present invention, the content of the context in the context information is represented from the acquired context information. A word is determined as main word information, and emotion information representing emotion, word information representing a word, and emotion expression information expressing at least emotion in emotion information based on the acquired emotion information and the determined main word information Corresponding emotion expression information is acquired from the emotion expression information database registered in association, and emotion expression document information is generated based on the acquired context information and the acquired emotion expression information.

  Thereby, the emotion expression document expressing the emotion can be easily generated.

  In the emotion expression document information generation system, the emotion expression document information generation device, and the emotion expression document information generation program according to the present invention, when the emotion expression information corresponding to the emotion information and the main word information exists, Emotion expression information is acquired from the emotion expression information database, and when there is no corresponding emotion expression information, emotion expression information closest to the emotion information and the main word information is acquired from the emotion expression information database.

  Thereby, even when emotion expression information does not exist, emotion expression document information can be generated.

  In the emotion expression document information generation system, the emotion expression document information generation apparatus, and the emotion expression document information generation program according to the present invention, the emotion expression information acquired when the corresponding emotion expression information does not exist, the acquired emotion information and As corresponding to the acquired main word information, they are associated with each other and registered in the emotion expression information database.

  Thereby, new emotion expression information can be automatically registered. Since the number of registrations in the emotion expression information database can be automatically increased, emotion expression document information desired by the user can be generated every time the user uses it.

  In the emotion expression document information generation system, the emotion expression document information generation device, and the emotion expression document information generation program according to the present invention, the main word information used when generating the emotion expression document is changed to another main word information And the corresponding emotion expression information is acquired from the emotion expression information database based on the emotion information and the changed main word information, and the emotion expression information is changed before the main word information is changed. The emotion expression information related to the main word information before changing the main word information registered in the database is deleted, and the emotion expression information newly corresponds to the emotion information and the changed main word information. It associates and registers in the emotion expression information database.

  Thereby, only emotion expression information desired by the user can be registered in the emotion expression information database.

  In the emotion expression document information generation system, the emotion expression document information generation apparatus, and the emotion expression document information generation program according to the present invention, any word set for each word is preferentially determined as the main word information. Priority information indicating whether the main word is selected is used.

  Thereby, a main word can be determined easily.

  In the emotion expression document information generation system, the emotion expression document information generation device, and the emotion expression document information generation program according to the present invention, the main word information used when generating the emotion expression document is changed to another main word information When the main word change information is acquired, in the main word database in which the priority information and the corresponding word are stored in association with each other, the value of the priority information of the word corresponding to the main word information before the change is lowered, and after the change The priority information value of the word corresponding to the main word information is increased.

  Thereby, the accuracy at the time of determining the main word can be increased every time the user uses it.

  In the emotion expression document information generation system, the emotion expression document information generation apparatus, and the emotion expression document information generation program according to the present invention, the emotion expression information is information regarding anger, sadness, and laughter.

  Thereby, emotion expression information can be expressed easily.

  In the emotion expression document information generation system, the emotion expression document information generation device, and the emotion expression document information generation program according to the present invention, the emotion expression information is information expressed using polar coordinates.

  Thereby, emotion expression information can be expressed easily.

  In the database update method according to the present invention, in a database in which the first information and the second information are associated and registered, search information for performing a search for the first information is acquired, and the database is stored in the database. When the first information corresponding to the search information exists, the second information associated with the first information is acquired from the database, and the database corresponds to the search information. If the first information does not exist, the second information associated with the first information closest to the search information is acquired from the database, and the first information corresponding to the search information is stored in the database. The second information acquired when there is no existing information is registered in the database in association with each other as corresponding to the acquired search information. To.

  As a result, the number of registrations in the database can be automatically increased each time search information is given, so that it is possible to easily construct a database that enables highly accurate searches in future searches.

  Here, the correspondence relationship between the elements described in the claims and the elements in the embodiment is shown. The emotion expression information generation system is the media expression document information generation system 1, the emotion expression information generation apparatus is the media expression document information generation apparatus 21, the speech recognition apparatus is the context / emotion information acquisition apparatus 23, and the emotion recognition apparatus is the context / emotion. Each corresponds to the information acquisition device 23.

  The emotion expression information database and the main word database are stored in the hard disk 213, the main word determination unit is stored in the CPU 211 and the memory 212, the emotion expression information acquisition unit is stored in the CPU 211 and the memory 212, and the emotion expression document information generation unit is stored in the CPU 211 and the memory 212. The expression information database update means corresponds to the CPU 211 and the memory 212, the main word change information acquisition means corresponds to the CPU 211 and the memory 212, and the main word database update means corresponds to the CPU 211 and the memory 212, respectively.

  The emotional expression information database is the word media expression related database and the media expression master database, the main word database is the word master database, the image information is the video information, the emotion information is the emotional information about the person shown in the video information, the voice Information is voice information corresponding to video information, context information is context information obtained by executing voice recognition processing on voice information, emotion expression document information is media expression document information, word information is word master database Is the value of the word priority column C505 of the word master database (see FIG. 5) among all the words acquired by the wording process. The main word change button B3 (see FIG. 19) is selected as the main word change information. The information indicating that it has been, the priority information is the value of the word Priority column C505 word master database, the corresponding.

  The first information corresponds to the values of θ and φ of the word media expression related database, the second information corresponds to the media expression ID of the word media expression related database, and the search information corresponds to the emotion direction vector.

  Examples of the portable terminal in the present invention will be described below.

1. Overview FIG. 1 shows a functional block diagram according to Embodiment 1 of an emotion expression information generating system according to the present invention. The emotion expression information generation system M1 includes an emotion expression document information generation device M11, a context information acquisition device M13, an emotion information acquisition device M15, an emotion expression information database M17, and a main word database M19.

  The context information acquisition device M13 acquires voice information, performs voice recognition processing on the acquired voice information, and acquires context information corresponding to the voice information. The emotion information acquisition device M15 acquires image information, and acquires emotion information representing the emotion of the person shown in the image information based on the acquired image information.

  The emotion expression information database M17 stores an emotion expression information database in which emotion information, word information representing a word, and emotion expression information representing an emotion in the emotion information and a word in the word information are registered in association with each other. To do. In the main word database M19, the priority information and the corresponding word are stored in association with each other.

  The emotion expression document information generation device M11 includes a main word determination unit M101, an emotion expression information acquisition unit M105, an emotion expression document information generation unit M107, an emotion expression information database update unit M111, a main word change information acquisition unit M113, and a main word database. Update means M115 is provided.

  The main word determining means M101 determines, from the acquired context information, a word representing the context content in the context information as main word information. Further, when determining main word information, the main word determination means M101 uses priority information that is set for each word and indicates which word is preferentially selected as the main word.

  The emotion expression information acquisition means M105 acquires the corresponding emotion expression information from the emotion expression information database M17 based on the acquired emotion information and the determined main word information. Further, when there is emotion expression information corresponding to the emotion information and main word information, the emotion expression information acquisition unit M105 acquires the emotion expression information from the emotion expression information database M17, and the corresponding emotion expression information exists. If not, the emotion expression information closest to the emotion information and the main word information is acquired from the emotion expression information database M17. Furthermore, the emotion expression information acquisition unit M105 acquires the corresponding emotion expression information from the emotion expression information database M17 based on the emotion information and the changed main word information.

  The emotion expression document information generation unit M107 generates emotion expression document information based on the acquired context information and the acquired emotion expression information.

  The emotion expression information database updating means M111 associates the emotion expression information acquired when the corresponding emotion expression information does not exist with the acquired emotion information and the acquired main word information in association with each other as the emotion expression information database M17. Register with. Also, the emotion expression information database updating unit M111 deletes the emotion expression information related to the main word information before changing the main word information registered in the emotion expression information database M17 before changing the main word information, and newly The emotion expression information is registered in the emotion expression information database M17 in association with each other as corresponding to the emotion information and the changed main word information.

  The main word change information acquisition unit M113 acquires main word change information indicating that the main word information used when generating the emotion expression document is changed to another main word information.

  When the main word database update unit M115 acquires the main word change information indicating that the main word information used when generating the emotion expression document is changed to another main word information, the priority word and the corresponding word are associated with each other. In the stored main word database M19, the value of the word priority information corresponding to the main word information before the change is lowered, and the value of the word priority information corresponding to the changed main word information is raised.

  Emotion expression information is information about anger, sadness, and laughter. The emotion expression information is information expressed using polar coordinates.

2. Hardware configuration
2.1. Media expression document information generation system 1
A hardware configuration of the media expression document information generation system 1 will be described with reference to FIG. The media expression document information generation system 1 includes a media expression document information generation device 21, a context / emotion information acquisition device 23, a camera 25, and a microphone 27.

  The context / emotion information acquisition device 23 acquires video information from the camera 25 and audio information corresponding to the video information from the microphone 27. The context / emotion information acquisition device 23 acquires emotion information related to the person shown in the video information based on the acquired video information and audio information. In addition, the context / emotion information acquisition device 23 performs a voice recognition process on the acquired voice information, and acquires context information corresponding to the voice information.

  The emotion expression document information generation device 21 acquires emotion information and context information from the context / emotion information acquisition device 23, and generates media expression document information based on the emotion information and context information.

2.2. Emotional expression document information generation device 21
The hardware configuration of the emotion expression document information generation device 21 will be described with reference to FIG. The emotional expression document information generation device 21 includes a CPU 211, a memory 212, a hard disk 213, and a communication circuit 218. In addition, a keyboard 214, a mouse 215, a display 216, and a CD-ROM drive 217 are connected to the emotional expression document information generation device 21.

  The CPU 211 performs processing based on other applications such as an operating system (OS) and an emotion expression document information generation program recorded in the hard disk 213. The memory 212 provides a work area for the CPU 211. The hard disk 213 records and holds other applications such as an operating system (OS) and an emotion expression document information generation program. A word master database (hereinafter referred to as a word master DB) recorded on the hard disk 213, a word media expression related database (hereinafter referred to as a word media expression related DB), a media expression master database (hereinafter referred to as a media expression master DB). ), The emotion reflection parameter database (hereinafter referred to as emotion reflection parameter DB), and the media expression type master database (hereinafter referred to as media expression type master DB) will be described later.

  The CD-ROM drive 217 reads the emotion expression document information generation program from the CD-ROM 210 in which the emotion expression document information generation program is recorded, and reads other application programs from other CD-ROMs. Read data from.

  The keyboard 214 and the mouse 215 accept external commands. The display 216 displays the generated emotion expression document, other user interfaces, and the like.

2.3. Context / Emotion Information Acquisition Device 23
The hardware configuration of the context / emotion information acquisition apparatus 23 will be described with reference to FIG. The context / emotion information acquisition device 23 includes a CPU 231, a memory 232, a hard disk 233, a communication circuit 238-1, a camera interface circuit 238-2, and a microphone interface circuit 238-3. In addition, a keyboard 234, a mouse 235, a display 236, and a CD-ROM drive 237 are connected to the context / emotion information acquisition device 23.

  The CPU 231 performs processing based on other applications such as an operating system (OS) and a context / emotion information acquisition program recorded on the hard disk 233. The memory 232 provides a work area for the CPU 231. The hard disk 233 records and holds other applications such as an operating system (OS), a context / emotion information acquisition program, and various necessary data. The communication circuit 238-1 exchanges data with other computers. The camera interface circuit 238-2 acquires data from the camera 25. The microphone interface circuit 238-3 acquires data from the microphone 27.

  The CD-ROM drive 237 reads a context / emotion information acquisition program from a CD-ROM 230 in which a context / emotion information acquisition program is recorded, and reads other application programs from other CD-ROMs. Read data from.

  The keyboard 234 and the mouse 235 accept external commands. The display 236 displays an image such as a user interface.

3. Data Each database recorded in the hard disk 213 of the emotional expression document information generating device 21 will be described.

3.1. Word Master DB
The word master DB is a database that shows, for each word, how much a certain word can represent the context in a certain context.

  The word master DB will be described with reference to FIG. The word master DB includes a [word ID] column C501, a [word name] column C503, and a [word priority] column C505. [Word ID] column C501 describes an identification number for uniquely identifying each word.

  A specific word name is described in the [Word Name] column C503. Note that words do not ask parts of speech such as nouns, verbs, adjectives, adverbs. For words related to parts of speech with case changes, basic forms are described.

  In the [word priority] column C505, a value indicating how much the word described in the [word name] column C503 can represent the context in a certain context is described as the priority. Is done. As for the priority, when each word is first registered in the word master DB, a predetermined priority is given. The priority can be updated by a word master DB update process (FIG. 18: step S1815) described later. In this way, by updating the priority, it is possible to improve the accuracy of selecting a word representing a certain context.

  Taking the word master DB shown in FIG. 5 as an example, if the word “weather” has the word ID “T0001” and its priority is “30”, the “word ID” column C501 has “T0001”. However, the word name “weather” is described in the corresponding [word name] column C503, and the priority “30” is described in the [word priority] column C505.

  In addition, since it is assumed that no word is registered in the word master DB for the words included in the context information, default information is registered in the word master DB in advance for such a case. deep.

3.2. DB related to word media expression
The word media expression related DB is, when a certain word is used in a certain emotion, the word, the emotion, and a media expression used for expressing the emotion in a document using the word, It is a database registered in association with each other.

  The data structure of the word media expression related DB is shown in FIG. The word media expression related DB includes a [word ID] column C601, a [θ] column C603, a [φ] column C605, a [media expression ID] column C607, and a [last use date] column C609. [Word ID] column C601 describes an identification number for uniquely identifying each word. In the [θ] column C603 and the [φ] column C605, values “θ” and “φ” that uniquely specify emotions are described, respectively. “Θ” and “φ” will be described later. [Media expression ID] column C607 describes a media expression ID for uniquely specifying the media expression information arranged in the media expression document. Here, the media expression information refers to information such as a still image, a moving image, ASCII art, and voice related to the context in a context where a certain word is used. [Last used date and time] column C609 describes the last date and time when each record of the DB was used.

  Taking the word media expression-related DB of FIG. 6 as an example, the context in which the word with the word ID “T0001” is used in the emotion represented by “θ = 80, φ = 140” is represented by the media with the media expression ID “MED0002”. When a media expression document is generated at “March 11, 2006, 9:20:00” using the expression information, “T0001” is displayed in the [Word ID] column C601, and “θ” is stored in the [θ] column C603. 80 ”,“ 140 ”in the [φ] column C605,“ MED0002 ”in the [Media Expression ID] column C607, and“ 2006-03-11 09:20: 00 ”is described respectively.

  Since it is assumed that none of the words included in the context information is registered in the word media expression related DB, the default information is previously stored in the word media expression related DB for such a case. Register with.

3.3. Media Expression Master DB
The media expression master DB is a database that specifically describes media expressions used as media expression information.

  The data structure of the media expression master DB is shown in FIG. The media expression master DB includes a [media expression ID] column C701, a [media expression name] column C703, a [media expression type ID] column C705, and a [media expression] column C707.

  [Media expression ID] column C701 describes an identification number for uniquely identifying the media expression to be used. [Media expression name] column C703 describes the name of the media expression to be used. [Media expression type ID] column C705 describes the type to which the media expression to be used belongs. [Media expression] column C707 specifically describes the contents of the media expression.

3.4. Emotion reflection parameter DB
The emotion reflection parameter DB is a database that describes how emotions in a media expression document are reflected in the media expression when a media expression is used in the media expression document.

  The data structure of the emotion reflection parameter DB is shown in FIG. The emotion reflection parameter DB includes a [media expression ID] column C801, a [setting parameter ID] column C803, and a [setting value] column C805. [Media expression ID] column C801 describes an identification number for uniquely identifying the media expression. In the [set parameter ID] column C803, an identification display for uniquely specifying a parameter set when emotion is reflected in the media expression is described. [Set value] column C805 describes specific values of parameters set when emotion is reflected in the media expression. In the present embodiment, the set value is a proportional value with respect to emotion strength (described later).

  Taking the emotion reflection parameter DB in FIG. 8 as an example, when trying to set “image height”, “image width”, and “GBR values” for the media expression with the media expression ID “MED0002”, In the [Media Expression ID] column C801, “MED0002” is displayed. In the [Setting Parameter ID] column C803, “WidthPerR”, “HeightPerR”, “ColorRedPerR”, “ColorGreenPerR”, and “ColorBluePerR” are set as [Setting Parameter ID]. “0.1”, “0.1”, “0.3”, “0.3”, and “0.3” are respectively set for each value in the [setting value] column C805 corresponding to the column C803. Described.

3.5. Media expression type master DB
The media expression type master DB is a database in which the data type and logic of media expression information are registered in association with each other. The logic describes a display method (logic) indicating how the media expression information is displayed when the media expression information is displayed on the display 216.

FIG. 9 shows the data structure of the media expression type master DB. The media expression type master DB has a [media expression type ID] column C901, a [media expression type name] column C903, and an [emotion reflection logic] column C905. [Media expression type ID] column C901 describes the data type of the media expression information. For example, “AA” is described for ASCII art, and “JPEG” is described for JPEG images. [Media expression type name] column C903 describes a general name of the data type of the media expression information. Note that the value in the [Media expression type name] column C903 is automatically input in conjunction with the [Media expression type ID] column C901. [Emotion reflection logic] column C905 describes a logic number for specifying a logic to be applied to each piece of media expression information. The logic describes how the emotion strength “r” and emotion expression parameters are used. For example, in the media expression type master DB in FIG. 9 in which the emotion strength “r”, a calculation formula using emotion expression parameters, fixed parameters used in the calculation formula, and the like are described, the data type is “ When the logic of the logic number “LOG001” is applied to the media expression information “ASCII art”, “AA” is displayed in the “Media expression type ID” column C901, and “ASCII art” is displayed in the “Media expression type name” column C903. “LOGO LOGIC LOGIC” column C905 describes “LOG001”.

  In addition, it is possible to change the application method of the variation parameter flexibly by separately defining the variation parameter such as emotion strength and emotion expression parameter and the logic that is the application method of the variation parameter. It becomes possible. For example, even if the user wants to change the way the media expression information is displayed in the media expression document, it is only necessary to change the logic, so the media expression document can be easily changed as the user desires. There are advantages.

  Since it is assumed that media expression information having a data type that is not registered is newly registered, default information is registered in advance in the media expression type master DB for such a case.

4. Operation of Media Representation Document Information Generation System 1 FIG. 10 shows an outline of operation in the media expression document information generation system 1. The camera 25 acquires video information (S1), and transmits the acquired video information to the context / emotion information acquisition device 23 (S3). The microphone 27 acquires voice information (S5), and transmits the acquired voice information to the context / emotion information acquisition device 23 (S7).

  The context / emotion information acquisition device 23 acquires video information and audio information, and acquires emotion information representing emotions of a person shown in the video information based on the acquired video information and audio information. Is executed (S11). Further, the context / emotion information acquisition device 23 acquires voice information, executes voice recognition processing on the acquired voice information, and executes context information acquisition processing for acquiring context information corresponding to the voice information (S13). .

  Based on the acquired emotion information and context information, the media expression document information generation device 21 performs main word determination processing (S21), emotion information analysis processing (S23), media expression information acquisition processing (S25), media expression document information generation. Processing (S27) and media expression editing processing (S29) are executed.

  In the following, emotion information acquisition processing (S11), context information acquisition processing (S13) main word determination processing (S21), emotion information analysis processing (S23), media expression information acquisition processing (S25), media expression document information generation processing ( S27) and the media expression editing process (S29) will be described.

4.1. Operation of Context / Emotion Information Acquisition Device 23 Emotion information acquisition processing (S11: see FIG. 10) and context information acquisition processing (S13: see FIG. 10) performed by the context / emotion information acquisition device 23 will be described below.

4.1.1 Emotion Information Acquisition Processing Emotion information acquisition processing performed by the CPU 231 of the context / emotion information acquisition device 23 will be described with reference to the flowchart shown in FIG. The CPU 231 acquires audio information from the microphone 27 (S1101), acquires video information from the camera 25 (S1103), and performs emotion analysis processing (S1105).

  For the emotion analysis process, for example, the technique disclosed in Japanese Patent Laid-Open No. 10-228295 is used. By using this technology, the emotion of the person projected in the video information can be expressed as values of laughter, sadness, and anger. In this embodiment, “happiness” in JP-A-10-228295 is “laughing”. In Japanese Patent Laid-Open No. 10-228295, emotions are expressed using six elements of anger, happiness (laughter), sadness, surprise, disgust, and fear. In this embodiment, anger, happiness (laughter) ), Expressing emotions using only the three elements of sadness.

  In analyzing emotions based on video information, one frame with video information may be extracted, and the emotion of the person shown in the video information may be determined from the image in that frame. Moreover, you may judge using the emotion regarding all the frames of video information. For example, the emotion value in each frame may be calculated for each emotion, and the sum may be used as the emotion value. In addition, a value obtained by adding an average process, a weighted average process, or the like to the calculated sum may be used as an emotion value.

  The CPU 231 outputs the values of laughter, sadness, and anger obtained by the emotion analysis process as emotion information (laughter, sadness, anger) (S1107).

4.1.2 Context Information Acquisition Processing Context information acquisition processing performed by the CPU 231 of the context / emotion information acquisition device 23 will be described with reference to the flowchart shown in FIG. When the CPU 231 receives the voice information from the microphone 27 (S1101), the CPU 231 performs a context analysis process for characterizing the context related to the voice information (S1111). Note that the kana-kanji conversion processing technology based on general voice input is used for the context analysis processing. The CPU 231 outputs the textual context information obtained by the context analysis process (S1113).

4.2. Operation of Media Expression Document Information Generation Device 21 Main word determination processing (S21: see FIG. 10), emotion information analysis processing (S23: see FIG. 10), media expression information acquisition processing (S21: see FIG. 10) S25: See FIG. 10), media expression document information generation processing (S27: see FIG. 10) and media expression editing processing (S29: see FIG. 10) will be described below.

4.2.1 Main Word Determination Processing The main word determination processing performed by the CPU 211 of the media expression document information generation device 21 will be described using the flowchart shown in FIG.

  When the CPU 211 determines that the context information has been acquired (S1201), the CPU 211 executes a wording process for dividing the context represented by the context information into words (S1203). For this wording process, for example, the technique described in JP-A-2002-1117025 is used.

  The CPU 211 acquires the value of the corresponding [word priority] column C505 from the word master DB (see FIG. 5) for all words acquired by the wording process (S1205). The CPU 211 selects the highest value from the acquired values in the [word priority] column C505, and selects the word in the [word name] column C503 corresponding to the selected value in the [word priority] column C505. The word is determined (S1207).

4.2.2 Emotion Information Analysis Processing Emotion information analysis processing performed by the CPU 211 of the media expression document information generation device 21 will be described using the flowchart shown in FIG.

  If the CPU 211 determines that an emotion vector (laughter, sadness, anger) as emotion information has been acquired (S1301), the CPU 211 calculates the emotion strength “r” based on the acquired emotion vector (S1303). The calculated emotion strength “r” is temporarily stored in the memory 212.

  The emotion strength “r” is calculated using the following equation (1).

  Further, the CPU 211 calculates an emotion direction vector based on the acquired emotion vector (S1305).

  Here, the emotion direction vector will be described with reference to FIG. An emotion direction vector is a vector that specifies the direction of the emotion vector when emotion information is projected onto a spherical coordinate system. FIG. 14 shows a case where a spherical coordinate system for projecting emotion vectors is expressed by three axes of “laughter”, “anger”, and “sadness”. By using the spherical coordinate system, the above-mentioned “emotion strength” can be defined using the distance “r” from the origin O. The emotion direction vector can be defined using an angle “θ” in the direction from the laughing axis to the sadness axis and an angle “φ” in the direction from the anger axis to the laughing axis. The emotion direction vector (θ, φ) is defined using the angle “θ” and the angle “φ”. Even when the variables constituting the emotion vector increase or decrease, the emotion strength and the emotion direction vector can be defined by the same method.

  The value of the emotion direction vector (θ, φ) is calculated using the following equation (2).

4.2.3 Media Expression Information Acquisition Processing Media expression information acquisition processing performed by the CPU 211 of the media expression document information generation device 21 will be described with reference to the flowcharts shown in FIGS. 15 and 15a. As shown in FIG. 15, the CPU 211 uses the main word information determined in the main word determination process and the emotion direction vector (θ, φ) calculated in the emotion information analysis process to use the word media expression related DB ( The optimum media expression information is acquired from FIG.

  The CPU 211 acquires a word ID corresponding to the main word information determined in the main word determination process from the word master DB (S1501). The CPU 211 determines whether or not the value of the word ID of the acquired main word information is registered in the [word ID] column C601 of the word media expression related DB (S1503).

  If the CPU 211 determines that the value of the word ID of the main word information is registered in the [word ID] column C601, the CPU 211 extracts a record in which the value of the [word ID] column C601 is equal to the acquired word ID value (S1505). ). Whether the CPU 211 has the extracted records that match the values of the emotion direction vector (θ, φ) for the values of the [θ] column C603 and the [φ] column C605 of the record. It is determined whether or not (S1507).

  If the CPU 211 determines that there is an object whose values in the [θ] column C603 and the [φ] column C605 of the extracted record coincide with the values (θ, φ) of the emotion direction vector, the CPU 211 acquires a corresponding record ( S1511).

  Moving to FIG. 15a, the CPU 211 updates the value of the [last used date and time] column C609 of the acquired record to the current time (S1513), and overwrites the original record with the updated record in the media representation related DB. Registration is performed (S1515). The CPU 211 sets a record overwrite flag indicating that the original record is overwritten when the updated record is overwritten and registered. Further, the CPU 211 temporarily stores the value of the [last use date / time] column C609 of the original record in the memory 212 as pre-update last use date / time information before overwriting registration. Further, the CPU 211 temporarily stores in the memory 212 update record specifying information for specifying a record after registration. As the update record specifying information, a record ID or the like given for uniquely specifying each record is used.

  For example, when the emotion direction vector (θ, φ) = (0, 40), the CPU 211 acquires a record R601 of the word media expression related DB shown in FIG. Then, the CPU 211 updates the value “2006-03-11 09:00:00” in the “last used date” column C609 of the record R601 to the current time, for example, “2006-03-11 10:00:00”. The updated record is overwritten on the original record and registered in the media expression related DB. FIG. 21A shows an example of the media expression related DB updated by overwriting the record R601 on the original record.

  Returning to step S1507 in FIG. 15, the CPU 211 determines that there is no one in which the values of the [θ] column C603 and the [φ] column C605 of the extracted record match the values of the emotion direction vector (θ, φ). If it is determined, a record having “θ” and “φ” closest to the emotion direction vector is acquired (S1521). In calculating a value close to the emotion direction vector, the determination is made based on the distance between emotion direction vectors having a unit length (unit emotion direction vector). Further, when there are a plurality of items having the same value between the unit emotion direction vectors, a record having the closest value of “θ” is acquired. The determination may be based on the value of “φ” instead of the value of “θ”. Moreover, you may make it judge by another element. For example, another index may be generated using “θ” and “φ”.

  Moving to FIG. 15a, the CPU 211 updates the values of the [θ] column C603 and the [φ] column C605 of the extracted records to the values of the emotion direction vector (θ, φ) (S1523). Further, the CPU 211 updates the value of the [last used date and time] column C609 of the acquired record to the current time (S1525), and registers the updated record as a new record in the media representation related DB (S1515). When registering a new record, the CPU 211 sets a record generation flag indicating that a new record has been generated. Further, the CPU 211 temporarily stores the update record specifying information in the memory 212.

  For example, if the emotion direction vector (θ, φ) = (0, 60), the CPU 211 acquires a record R601 of the word media expression related DB shown in FIG. Then, the CPU 211 updates the value “0” in the [θ] column C603 and the value “40” in the [φ] column C605 of the code of the record R601 to the value (θ, φ) = (0, 60) of the emotion direction vector. To do. In addition, the CPU 211 updates the value “2006-03-11 09:00:00” in the [last used date and time] column C609 to the current time, for example, “2006-03-11 10:00:00”, and updates the value. The record is registered in the media expression related DB as a new record R603. An example of a media expression related DB in which a new record R603 is registered is shown in FIG. 22A.

  Returning to step S1503 in FIG. 15, when the CPU 211 determines that the value of the word ID of the main word information is not registered in the [word ID] column C601, the value of C601 in the [word ID] column is “DEFAULT”. A record (see FIG. 6) is acquired (S1531).

  Moving to FIG. 15A, the CPU 211 updates the values of the [θ] column C603 and the [φ] column C605 of the extracted “DEFAULT” record to the values of the emotion direction vector (θ, φ) (S1533). Further, the CPU 211 updates the value of the “last used date” column C609 of the acquired “DEFAULT” record to the current time (S1535). Further, the CPU 211 updates the value of the [word ID] column C601 of the acquired “DEFAULT” record to a word ID corresponding to the main word information (S1537). Then, the CPU 211 registers the updated “DEFAULT” record as a new record in the media expression related DB (S1515). When registering a new record, the CPU 211 sets a record generation flag indicating that a new record has been generated. Further, the CPU 211 temporarily stores the update record specifying information in the memory 212.

  Moving to FIG. 15A, the CPU 211 acquires the value of the [media expression ID] column C607 of the registered record (S1517). Based on the value of the media expression ID acquired in step S1517, the CPU 211 stores, in the [media expression] column C707, a record having the media expression ID in the [media expression ID] column C701 from the media expression master DB (see FIG. 7). Is acquired (S1519). Thereby, media expression information reflecting the direction of emotion can be acquired.

4.2.4 Media Expression Document Information Generation Processing Media expression document information generation processing performed by the CPU 211 of the media expression document information generation apparatus 21 will be described with reference to the flowchart shown in FIG. Based on the media expression ID acquired in step S1517 (see FIG. 15a), the CPU 211 determines from the emotion expression parameter DB (see FIG. 8) the media expression ID obtained by the value of the [media expression ID] column C801 of the DB. The value of the [Setting parameter ID] column C803 and the value of the [Setting value] column C805 of the matching record are acquired (S1601).

  Based on the value of the media expression ID acquired in step S1517 (see FIG. 15a), the CPU 211 stores the media expression ID from the media expression master DB (see FIG. 7) in the [media expression ID] column C701 [ The value of the media expression type ID] column C705 is acquired (S1605). Then, the CPU 211 acquires the logic described in the [emotional logic] column C905 corresponding to the acquired value of the [media expression type ID] column C705 from the media expression type master DB (see FIG. 9) (S1607). The logic describes a display method that indicates how the media expression information is displayed using the emotion strength “r” and the emotion expression parameter when the media expression information is displayed on the display 216. . For example, in the present embodiment, the logic number “LOG001” applied to ASCII art is a value obtained by multiplying the value of the emotion strength “r” and the value of the emotion expression parameter as a proportional value for scaling. It is described. Thereby, media expression information reflecting the strength of emotion can be generated.

  The CPU 211 acquires the value of the emotion strength “r” calculated in step S1303 (see FIG. 13) from the memory 212 (S1609). The CPU 211 uses the value of the [setting value] column C805 acquired in step S1601 and the value of the emotion strength “r” acquired in step S1609 for the media expression information acquired in step S1519 (see FIG. 15A). Then, media expression information adjustment processing based on the logic acquired in step S1607 is executed (S1611).

  The CPU 211 generates media expression document information based on the context information acquired in step S1201 (see FIG. 12) and the media expression information subjected to the media expression information adjustment process in step S1611 (S1613). FIG. 17 shows a display screen example of the media expression document information generated by the CPU 211. In the displayed media expression document information, the word related to the main word information is highlighted. In addition, an edit button B1 that is selected when editing the media expression document information is arranged on the display screen. Note that the user of the media expression document information generation device 21 selects the edit button B1 with the mouse 215 or the like.

4.2.5 Media Expression Editing Process The media expression editing process performed by the CPU 211 of the media expression document information generating apparatus 21 will be described with reference to the flowchart shown in FIG. When the CPU 211 determines that the edit button B1 (see FIG. 19) for editing the generated media expression document information is selected (S1801), the CPU 211 switches to the edit mode (S1803).

  Here, FIG. 19 shows an example of an edit screen in the edit mode. On the editing screen, a main word change button B3, a media expression change button B5, an emotion reflection change button B7, and a return button B9 to the display screen are arranged. The user of the media expression document information generation device 21 selects the buttons B3 to B9 with the mouse 215 or the like.

  Returning to FIG. 18, when the CPU 211 determines that the main word change button B3 (see FIG. 19) has been selected (S1805), the word to be changed is selected from the context of the media expression document information displayed on the display. It is determined whether or not (S1811). When the CPU 211 determines that a word has been selected, the CPU 211 highlights the word (S1813).

  In the word master DB, the CPU 211 increases the value of the [Priority] column C505 corresponding to the word after the main word change by a predetermined amount, and decreases the value of the [Priority] column C505 corresponding to the word before the change by a predetermined amount. The word master DB update process is executed (S1815).

  FIG. 20 shows an example of the word master DB in which the value of the [Priority] column is changed. FIG. 20 shows the word master DB when the main word of the media expression document information shown in FIG. 19 is changed from “weather” to “good”. In this case, the value of the [word priority] column C505 of the record R501 corresponding to the word “weather” is changed from “30” to “20”, and the value of the [word priority] column C505 of the record R503 corresponding to the word “good”. The value is set from “20” to “30”.

  Returning to FIG. 18, the CPU 211 executes a media expression related DB update process (1) for updating the media expression related DB (see FIG. 6) in accordance with the change of the main word (S1817).

  The media expression related DB update process (1) will be described with reference to the flowchart shown in FIG. The CPU 211 determines whether a record overwrite flag is set (S2301). If the CPU 211 determines that the record overwrite flag is set, the CPU 211 returns the value in the [last used date / time] column C609 of the record corresponding to the update record specifying information to the value before the change based on the pre-change use date / time information. (S2303).

  On the other hand, when determining in step S2101 that the record overwrite flag is not set, the CPU 211 determines whether or not the record generation flag is set (S2305). If the CPU 211 determines that the record generation flag is set, the CPU 211 deletes the record corresponding to the update record specifying information from the media representation related DB (S2307). If the CPU 211 determines in step S2105 that the record overwrite flag is not set, it executes error processing (S2309).

  Returning to FIG. 18, when the media expression related DB update process (1) (S1817) ends, the CPU 211 uses the selected word as the main word information again to obtain the media expression information acquisition process (S25: see FIG. 10). , Media expression document information generation processing (S27: see FIG. 10)), and if necessary, media expression editing processing (S29: see FIG. 10)).

  FIG. 21B and FIG. 22B show an example of the word media expression related DB updated by the media expression related DB update process (1). FIG. 21B shows a case where records that match each of the acquired emotion direction vector values (θ, φ) exist in the word media representation related DB, and FIG. 22B shows the acquired emotion direction vector values (θ, φ). ) Shows a case where a record matching each of the above does not exist in the word media expression related DB.

  In FIG. 21B, from the state where the record R601 is overwritten and registered based on the emotion direction vector (θ, φ) = (0, 40) (see FIG. 21A), the main word is changed from the word with the word ID “T0001” to the word ID “ A case where the word is changed to “T0002” is shown. When the main word is changed from the word with the word ID “T0001” to the word with the word ID “T0002” from the state of FIG. 21A, the CPU 211 determines the value “2006-03-” in the [Last Use Date / Time] column C609 of the record R601. 11 “10:00:00” is returned to the original value “2006-03-11 09:10:00” (see FIG. 6). Then, the CPU 211 acquires a record R605 closest to the emotion direction vector (θ, φ) = (0, 40) with the word ID “T0002” as the main word. The CPU 211 sets the values of the [θ] column C603 and the [φ] column C605 of the acquired record R605 as the value of the emotion direction vector (θ, φ) = (0, 40), and further, in the [last used date and time] column C609. A record R607 whose value is the current value “2006-03-11 10:10:00” is newly generated. The CPU 211 registers the newly generated record R607 in the word media expression related DB as shown in FIG. 21B.

  In FIG. 22B, from the state where the record R603 is registered as a new record based on the emotion direction vector (θ, φ) = (0, 60) (see FIG. 22A), the main word is from the word with the word ID “T0001”. The case where it is changed to the word of the word ID “T0002” is shown. When the main word is changed from the word with the word ID “T0001” to the word with the word ID “T0002” from the state of FIG. 22A, the CPU 211 deletes the record R603 from the word media expression related DB. Then, the CPU 211 acquires a record R605 closest to the emotion direction vector (θ, φ) = (0, 60) with the word ID “T0002” as the main word. The CPU 211 sets the values of the [θ] column C603 and the [φ] column C605 of the acquired record R605 as the value of the emotion direction vector (θ, φ) = (0, 60), and further, in the [last used date and time] column C609. A record R609 whose value is the current value “2006-03-11 10:10:00” is newly generated. The CPU 211 registers the newly generated record R609 in the word media expression related DB as shown in FIG. 22B.

  If it is determined in step S1805 in FIG. 18 that the media expression change button B5 (see FIG. 19) has been selected, the CPU 211 displays a dialog for newly specifying media expression information (S1821). When the CPU 211 determines that an image or the like as media expression information is newly selected in the dialog (S1823), the CPU 211 executes a media expression master DB update process (S1825).

  The media expression master DB update process will be described with reference to the flowchart shown in FIG. The CPU 211 determines whether or not the designated media expression information is already registered in the media expression master DB (S2401). When the CPU 211 determines that the designated media expression information is already registered in the media expression master DB, the CPU 211 sets the value of the [media expression ID] column 701 of the corresponding media expression information from the media expression master DB (see FIG. 7). Obtain (S2403).

  On the other hand, if the CPU 211 determines in step S2401 that the designated media expression information is not yet registered in the media expression master DB, the CPU 211 newly assigns a media expression ID (S2411). In addition, the CPU 211 sets the value of DEFAULT based on the data format of the media expression information as the value of the [Type] column C705 (S2413). For example, if the media expression information is ASCII art, “AA” is set as the value in the [Type] column C705, and “JPEG” is set as the value in the [Type] column C705 if it is a JPEG image. The CPU 211 registers a newly generated record in the media expression master DB based on the assigned “media expression ID” and “type” (S2415). Note that the value of the [media expression name] column C703 may be acquired from the user at this stage or may be acquired later.

  Returning to FIG. 18, next, the CPU 211 executes media expression related DB update processing (2) for updating the media expression related DB (see FIG. 6) in accordance with the change of the media expression information (S1827).

  The media expression related DB update process (2) will be described with reference to the flowchart shown in FIG. In the media expression related DB (see FIG. 6), the CPU 211 sets the value of the [media expression ID] column C701 of the corresponding record to the value of the media expression ID acquired in step S2403 or the value of the media expression ID assigned in step S2411. Overwriting is performed (S2501). In addition, the CPU 211 updates the value of the “last used date” column C609 of the corresponding record to the current date (S2503).

  When the media expression related DB update process is completed, the CPU 211 changes the media expression ID acquisition process (S1517: see FIG. 15a) and the media expression document information generation process (S27: see FIG. 10) again based on the media expression information. If necessary, the media expression editing process (S29: see FIG. 10) is executed.

  21C and 22C show an example of the word media expression related DB updated by the media expression related DB update process (2). FIG. 21C shows a case where a record that matches each of the acquired emotion direction vector values (θ, φ) exists in the word media representation related DB, and FIG. 22C shows the acquired emotion direction vector values (θ, φ). ) Shows a case where a record matching each of the above does not exist in the word media expression related DB.

  In FIG. 21C, from the state where the record R601 is overwritten and registered based on the emotion direction vector (θ, φ) = (0, 40) (see FIG. 21A), the media expression information is from the media expression of the media expression ID “MED001”. A case where the media expression is changed to the media expression of the media expression ID “MED002” is shown. When the media representation information is changed from the media representation with the media representation ID “MED001” to the media representation with the media representation ID “MED002” from the state of FIG. 21A, the CPU 211 changes the value of the [Media representation ID] column C607 in the record R601. “MED001” is set to “MED002”, and the value “2006-03-11 10:00:00” in the [last used date and time] column C609 is set to the current value “2006-03-11 10:10:00”. . The CPU 211 overwrites and registers the changed record R601 in the word media expression related DB as shown in FIG. 21C.

  In FIG. 22C, from the state where the record R603 is registered as a new record based on the emotion direction vector (θ, φ) = (0, 60) (see FIG. 22A), the media expression information has the media expression ID “MED001”. A case where the media expression is changed to the media expression of the media expression ID “MED002” is shown. When the media representation information is changed from the media representation with the media representation ID “MED001” to the media representation with the media representation ID “MED002” from the state of FIG. 22A, the CPU 211 changes the value in the [Media representation ID] column C607 of the record R603. “MED001” is set to “MED002”, and the value “2006-03-11 10:00:00” in the [last used date and time] column C609 is set to the current value “2006-03-11 10:10:00”. . The CPU 211 overwrites and registers the changed record R603 in the word media expression related DB as shown in FIG. 22C.

  If it is determined in step S1805 in FIG. 18 that the emotion reflection change button B7 (see FIG. 19) is selected, the CPU 211 displays a dialog for newly specifying a setting value for reflecting the emotion (S1831). The type of set value is determined by the emotion expression parameter set in each media expression information. For example, if emotion expression parameters are set for the font size and the display color of the screen, a dialog for changing the emotion expression parameters is displayed.

  When the CPU 211 determines that a new emotion expression parameter is set in the dialog (S1833), the CPU 211 newly sets the emotion expression parameter of the corresponding media expression information based on the emotion expression parameter, and sets the emotion expression parameter DB. Update (S1835).

  FIG. 26 shows an example of the emotion expression parameter DB after the emotion expression parameters are changed. FIG. 26 shows a case where the font size included in the media expression information of the media expression ID “MED001” is enlarged. In this case, when the enlargement ratio desired by the user is given to the media expression document information generation apparatus 21, the CPU 211 automatically calculates the enlargement ratio that matches the system and registers it in the [set value] column C805. In FIG. 26, the value “0.8” in the “Installation Value” column C805 of the record R801 in FIG. 8 is set to “1.2”.

  Returning to FIG. 18, when the emotion expression parameter DB update process (S1835) is completed, the CPU 211 again needs the media expression document information generation process (S27: see FIG. 10) based on the set emotion expression parameters. If there is, the media expression editing process (S29: see FIG. 10) is executed.

  If it is determined in step S1805 in FIG. 18 that the return button B9 (see FIG. 19) has been selected, the CPU 211 exits the edit mode and returns to the emotion-expression document information display screen (S1841).

4.3. Specific Example A specific operation of the media expression document information generation system 1 will be described with reference to FIGS. In the following, a case is considered where media expression document information is generated based on an emotion when a certain person says “It is a good weather today”.

  The camera 25 and the microphone 27 acquire video information showing a state in which a certain person is speaking and sound information in which a certain person's remarks are recorded. When the video / audio information is acquired from the camera 25 and the microphone 27, the context / emotion information acquisition device 23 executes a context analysis process (see FIG. 11: S1111) and acquires the context information “It ’s good weather today”. To do. Also, emotion analysis processing (see FIG. 11: S1105) is executed to obtain an emotion vector (laughter, anger, sadness) = (10√3, 10, 0).

  Moving to FIG. 28, when the CPU 211 of the media expression document information generation device 21 acquires the context information “It is a good weather today”, it executes a main word determination process for determining a main word from the acquired context information. When the main word determination process (see FIG. 12) is executed based on the word master DB shown in FIG. 28, the CPU 211 selects the word “weather (word ID: word ID) having the highest priority among the words included in the acquired context information. T001) "is determined as the main word.

  Moving to FIG. 29, when the CPU 211 acquires the emotion vector (laughter, anger, sadness) = (10√3, 10, 0), the emotion information analysis process (FIG. 13) acquires the emotion direction vector from the acquired emotion vector. ). In the emotion information analysis process, the emotion vector is subjected to polar coordinate conversion, and the CPU 211 acquires, for example, an emotion direction vector (θ, φ) = (0, 60) and an emotion strength “r” = 20.

  Referring to FIG. 30, the CPU 211 obtains media expression information acquisition processing based on the main word information “weather (word ID: T001)” and the emotion direction vector (θ, φ) = (0, 60) (see FIG. 15: S1521). ), And from the word media expression related DB shown in FIG. 30, (θ, φ) = (0, 40) closest to the emotion direction vector (θ, φ) = (0, 60) for “word ID: T001”. ) To get a record with the value of

  Moving to FIG. 31, the CPU 211 acquires the media expression ID “MED0001” from the acquired record. Then, the CPU 211 executes a media expression information acquisition process (see FIG. 15A: S1519), and acquires the corresponding media expression and media expression type ID “AA” from the media expression master DB shown in FIG.

  32, the CPU 211 further executes a media expression document information generation process (see FIG. 16: S1601) based on the acquired media expression ID “MED0001”. The CPU 211 acquires the setting parameter ID and the value corresponding to the media expression ID “MED0001” from the emotion reflection parameter DB illustrated in FIG. 32.

  Moving to FIG. 33, the CPU 211 executes a media expression document information generation process (see FIG. 16: S1607) based on the acquired media expression type ID “AA”. The CPU 211 acquires the emotion reflection logic “LOG0001” corresponding to the media expression type ID “AA” from the media expression type DB shown in FIG.

  Moving to FIG. 34, for the acquired media expression, the CPU 211 determines the emotion direction vector (θ, φ) = (0, 60), the emotion strength “r”, the setting parameter ID, and its value. Then, media expression information adjustment processing (see FIG. 16: S1611) is executed using the emotion reflection logic “LOG0001”.

Moving on to FIG. 35, the CPU 211 executes the media expression information adjustment processing, the context information “Today is good weather”, and the main word information “weather (word ID: T001)”. A media expression document shown in FIG.

[Other Embodiments]
(1) Constituent Elements of Emotion Information In the first embodiment described above, laughter, sadness, and anger are exemplified as the emotions constituting the emotion information, but the constituent elements of emotion information are not limited to those illustrated. For example, other emotions such as surprise, disgust, and fear may be digitized as components.

  Further, the method of calculating the value of emotion information is not limited to the exemplified Japanese Patent Laid-Open No. 10-228295. Any method can be used as long as emotion can be quantified for one or more elements.

(2) Acquisition of Emotion Information / Context Information In the first embodiment described above, emotion information is acquired from audio information and video information, but it may be acquired only from video information. In the first embodiment, the context information is acquired from the audio information. However, the context information may be acquired from the video information and the audio information.

(3) Polar coordinates In the above-described first embodiment, the emotion vector is subjected to the polar coordinate conversion, and the emotion direction vector and the emotion strength are acquired. However, the present invention is not limited to the example as long as the emotion information can be processed. . For example, the emotion direction vector and the strength of emotion may be calculated using the emotion vector as it is.

(4) Acquisition of video information / audio information In the first embodiment, the video information and the audio information are acquired from the camera 25 and the microphone 27, respectively, but any method can be used as long as it can acquire the video information / audio information. It is not limited to. For example, it may be acquired from video information / audio information already stored in a database or the like.

(5) Acquisition of Emotion Direction Vector In the first embodiment, when a record having “θ” and “φ” closest to the emotion direction vector (θ, φ) is acquired from the word media related DB (see FIG. 6). In addition, “record having“ θ ”and“ φ ”closest to emotion direction vector (θ, φ)” is defined as “record having the closest distance between unit emotion direction vectors”, but is not limited to the example. .

(6) Update of Word Master DB In the above-described first embodiment, when updating the word master DB, the priority for the word before update is lowered and the priority for the word after update is raised. However, you may make it raise only the priority about the word after an update. Moreover, you may make it reduce only the priority about the word before an update.

  Further, the width for increasing the priority for the word after the change and the width for decreasing the priority for the word before the change may not be the same value. Furthermore, the range for increasing or decreasing the priority may be changed according to the number of times a word is changed.

(7) Order of processing in flowchart In the first embodiment described above, each processing is realized based on each flowchart shown in the figure. However, as long as each process can be realized, the order of the processes in each flowchart is not limited to the example.

(8) Processing Order In the first embodiment, the operation of the media expression document information generation system 1 is realized by the flowcharts shown in the figure. However, as long as the functions in the functional block diagram of the first embodiment (see FIG. 1) can be realized, the processing is not limited to the processing according to the exemplary flowchart. In particular, the order of processing is not limited.

(9) Selection of Function In the first embodiment, the media expression document information generation system 1 that realizes the functional block diagram shown in FIG. 1 is shown. However, the functions of the media expression document information generation system are not limited to the examples. For example, some functions may not be selectively provided.

It is the figure which showed the functional block diagram in Example 1 of the emotion expression information generation system in this invention. It is the figure which showed the hardware constitutions of the media expression document information generation system. It is the figure which showed the hardware constitutions of the emotion expression document information generation apparatus. It is the figure which showed the hardware constitutions of the context and emotion information acquisition apparatus. It is the figure which showed the data structure of word master DB. It is the figure which showed the data structure of word media expression relevant DB. It is the figure which showed the data structure of media expression master DB. It is the figure which showed the data structure of emotion reflection parameter DB. It is the figure which showed the data structure of media expression classification master DB. 5 is a flowchart showing an outline of an operation in the media expression document information generation system 1. It is the flowchart which showed the emotion information acquisition process and the context information acquisition process. It is the flowchart which showed the main word determination process. It is the flowchart which showed the emotion information analysis process. It is a figure for demonstrating the relationship between an emotion direction vector and a spherical coordinate. It is the flowchart which showed media expression information acquisition processing. It is the flowchart which showed media expression information acquisition processing. It is the flowchart which showed media expression document information generation processing. It is the figure which showed the example of a display screen of media expression document information. It is the flowchart which showed media expression edit processing. It is the figure which showed the example of an edit screen in edit mode. It is the figure which showed an example of the word master DB which changed the value of the [priority] column. It is the figure which showed an example of the updated word media expression related DB. It is the figure which showed an example of the updated word media expression related DB. It is the flowchart which showed media expression related DB update processing (1). It is the flowchart which showed media expression master DB update processing. It is the flowchart which showed media expression related DB update processing (2). It is the figure which showed an example of emotion expression parameter DB after an emotion expression parameter was changed. It is a figure for demonstrating the operation | movement of the specific media expression document information generation system 1. FIG. It is a figure for demonstrating the operation | movement of the specific media expression document information generation system 1. FIG. It is a figure for demonstrating the operation | movement of the specific media expression document information generation system 1. FIG. It is a figure for demonstrating the operation | movement of the specific media expression document information generation system 1. FIG. It is a figure for demonstrating the operation | movement of the specific media expression document information generation system 1. FIG. It is a figure for demonstrating the operation | movement of the specific media expression document information generation system 1. FIG. It is a figure for demonstrating the operation | movement of the specific media expression document information generation system 1. FIG. It is a figure for demonstrating the operation | movement of the specific media expression document information generation system 1. FIG. It is a figure for demonstrating the operation | movement of the specific media expression document information generation system 1. FIG.

Explanation of symbols

M1 ... Emotion expression information generation system M11 ... Emotion expression document information generation apparatus M13 ... Context information acquisition apparatus M15 ... Emotion information acquisition apparatus
M17 ... Emotion expression information database M19 ... Main word database M101 ... Main word determination means M105 ... Emotion expression information acquisition means M107 ... Emotion expression document information Generation means M111 ... Emotion expression information database update means M113 ... Main word change information acquisition means M115 ... Main word database update means 1 ... Media expression document information generation system 21・ ・ ・ ・ ・ Media expression document information generator 23 Context / emotion information acquisition device 25 ・ ・ ・ ・ ・ Camera 27 ・ ・ ・ ・ ・ Microphone

Claims (12)

An emotion information acquisition device that acquires image information and acquires emotion information representing the emotion of the person shown in the image information based on the acquired image information;
Context information acquisition device that acquires voice information, executes voice recognition processing on the acquired voice information, and acquires context information corresponding to the voice information;
An emotion expression document information generating device for generating emotion expression document information based on the emotion information and the context information;
An emotional expression document information generation system having
The emotion expression document information generation device includes:
Emotion expression information database storage for storing the emotion expression information database in which the emotion information, the word information representing a certain word, and the emotion expression information representing the emotion in the emotion information and the word in the word information are registered in association with each other. means,
Main word determination means for determining, as main word information, a word representing the context content in the context information from the acquired context information;
Emotion expression information acquisition means for acquiring corresponding emotion expression information from the emotion expression information database based on the acquired emotion information and the determined main word information;
Emotion expression document information generating means for generating emotion expression document information based on the acquired context information and acquired emotion expression information;
Having
Emotion expression document information generation system characterized by Main word determination means for determining, as main word information, a word representing the context content in the context information from the acquired context information;
Based on the acquired emotion information and the determined main word information, emotion information representing emotion, word information representing a certain word, and emotion expression information registered in association with emotion expression information representing at least emotion in emotion information Emotion expression information acquisition means for acquiring corresponding emotion expression information from a database;
Emotion expression document information generating means for generating emotion expression document information based on the acquired context information and acquired emotion expression information;
An emotional expression document information generation device having An emotion expression document document information generation program for generating emotion expression document information based on the emotion information and the context information,
The emotion expression document information generation program includes:
Computer
Main word determination means for determining, as main word information, a word representing the context content in the context information from the acquired context information;
Based on the acquired emotion information and the determined main word information, emotion information representing emotion, word information representing a certain word, and emotion expression information registered in association with emotion expression information representing at least emotion in emotion information Emotion expression information acquisition means for acquiring corresponding emotion expression information from a database;
Emotion expression document information generating means for generating emotion expression document information based on the acquired context information and acquired emotion expression information;
To function as a
Emotion expression document information generation program characterized by In any of the emotion expression document information generation system according to claim 1, the emotion expression document information property apparatus according to claim 2, or the emotion expression document information generation program according to claim 3,
The emotion expression information acquisition means includes:
When the emotion expression information corresponding to the emotion information and the main word information exists, the emotion expression information is acquired from the emotion expression information database, and when there is no corresponding emotion expression information, the emotion Obtaining the emotion expression information closest to the information and the main word information from the emotion expression information database;
It is characterized by. In any of the emotion expression document information generation system, the emotion expression document information generation device, or the emotion expression document information generation program according to claim 4,
The emotion expression document information generating device further includes:
Emotion expression information database update that associates the emotion expression information acquired when there is no corresponding emotion expression information with the acquired emotion information and the acquired main word information and registers them in the emotion expression information database. means,
Have In any of the emotion expression document information generation system, the emotion expression document information generation device, or the emotion expression document information generation program according to claim 4,
The emotion expression document information generating device further includes:
Main word change information acquisition means for acquiring main word change information indicating that main word information used in generating the emotion expression document is changed to another main word information;
Have
The emotion expression information acquisition means includes:
Based on the emotion information and the changed main word information, the corresponding emotion expression information is acquired from the emotion expression information database,
The emotion expression information database update means includes:
The emotion expression information related to the main word information before changing the main word information registered in the emotion expression information database before changing the main word information is deleted, and the emotion expression information is newly changed to the emotion information and the change. Registered in the emotion expression information database in association with each other as corresponding to the main word information
It is characterized by. In any one of the emotion expression document information generation system, the emotion expression document information generation device, or the emotion expression document information generation program according to claim 1,
The main word determining means further includes
In determining as the main word information, using priority information that is set for each word and indicates which word is preferentially selected as the main word,
It is characterized by. In any of the emotion expression document information generation system, the emotion expression document information generation apparatus, or the emotion expression document information generation program according to claim 7,
When the main word change information indicating that the main word information used when generating the emotion expression document is changed to another main word information is acquired, the priority word is stored in association with the corresponding word. In the database, a main word database changing means that lowers the value of the priority information of the word corresponding to the main word information before the change and increases the value of the priority information of the word corresponding to the main word information after the change,
Having
It is characterized by. In any one of the emotion expression document information generation system, the emotion expression document information generation apparatus, or the emotion expression document information generation program according to claim 1,
The emotional expression information is information on anger, sadness, and laughter;
It is characterized by. In any one of the emotion expression document information generation system, the emotion expression document information generation device, or the emotion expression document information generation program according to claim 1,
The emotion expression information is information expressed using polar coordinates;
It is characterized by. From the acquired context information, a word representing the context content in the context information is determined as main word information,
Based on the acquired emotion information and the determined main word information, emotion information representing emotion, word information representing a certain word, and emotion expression information registered in association with emotion expression information representing at least emotion in emotion information Get the corresponding emotional expression information from the database,
An emotion expression document information generation method for generating emotion expression document information based on acquired context information and acquired emotion expression information. In a database in which the first information and the second information are associated and registered, search information for performing a search for the first information is acquired,
When the first information corresponding to the search information exists in the database, the second information associated with the first information is acquired from the database,
If the first information corresponding to the search information does not exist in the database, the second information associated with the first information closest to the search information is acquired from the database,
A database update method in which the second information acquired when the first information corresponding to the search information does not exist in the database is registered in the database in association with each other as corresponding to the acquired search information.

Images