JP4965322B2 - User support method, user support device, and user support program - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide a user support device capable of timely associating users mutually with high reliability. <P>SOLUTION: The user support device is provided with a feeling estimation part F100 extracting an attribute or a value of the attribute of content by using one or more of sound information and image information of the content, a database 400 storing identification information of the content, the attribute estimated as to the content, and the attribute information in association mutually, a user information storage part F200 storing contents published by a user, contents viewed by the user, and information identifying at least one among viewed contents as user taste information for each of users, and a user association part F300 associating a first user as an association target with one or more users except the user as the association target based on the user taste information. <P>COPYRIGHT: (C)2009,JPO&amp;INPIT

Description

  The present invention relates to a user support method, a user support apparatus, and a user support program for associating a plurality of users.

  Currently, not only broadcasting, but also viewing of content is increasing on websites and personal PCs, and the types of content are very diverse, such as movies, dramas, home videos, news, documentaries, music, etc. It has become. With the cooperation of communication and broadcasting, it is easily expected that the number of users who enjoy viewing content will increase further in the future.

  Recently, it has become common to use content sharing sites where users can share content on the Web and communicate with each other.

  In the content sharing site, the content is accumulated by each user uploading the content that the user wants to view to the server. Since the accumulated content is disclosed in principle, each user can select and receive the content he / she wishes to view and can view it freely.

  In addition, each user can write comments such as impressions and opinions on the viewed content. Since these comments are also released in principle, enjoy communication with users who have published the content, other users who have watched the same content, and community formation among users who like the same content. Can be done.

  In particular, in a community-sharing type content sharing site that has become the mainstream recently, so-called SNS (social networking service) type content sharing sites, users can communicate with each other or form a community among users via content. It has become the biggest purpose.

  In order to take active and meaningful communication, it is important that the sensibilities of users are similar and that the topic to be shared is common. In the case of communication on a content sharing site, the similarities of sensibility, similar tastes to content, impressions and emotions when viewing content, and the commonality of topics, Conditions such as a relationship between a user who has published a certain content and a user who has viewed the content, a content having the same or similar content being disclosed or viewed, and so on.

  Based on these points of view, realization of user support technology that mediates between users with similar tastes and emotions and provides an opportunity for communication is more convenient for users and new value. It brings great benefits in terms of creation.

  2. Description of the Related Art Conventionally, as a user support technology for supporting communication, a technology has been invented that mediates between users based on the similarity of comments given by users. As one of such techniques, there is a technique disclosed in Patent Document 1.

  In the technique disclosed in Patent Document 1, the user's comments are derived from the knowledge that the user is interested, and the users who have similar comments have similar sensibility. On the basis of the hypothesis that they are similar, a method and apparatus for providing a function for giving a comment from a user in units of scenes for each content, and further mediating a plurality of users having similar scenes to which the comment is given. It is disclosed.

As related techniques of the present invention, a parameter estimation method for a probability model for calculating a speech feature amount appearance probability (see, for example, Non-Patent Documents 1 and 2), a fundamental frequency and power extraction method (for example, see Non-Patent Document 3). ), Extraction method of time variation characteristics of speech rate (see, for example, Non-Patent Document 4 and Patent Document 2), method for obtaining a generalized state space model (see, for example, Non-Patent Document 5), emotion, and emotion level A method (for example, see Patent Documents 3, 4, 5, and 6), a method for extracting color information and editing information from video information (for example, see Non-Patent Document 6), and a method for extracting motion vector information (for example, Non-Patent Document 7). (See, for example, Non-Patent Documents 8 and 9 and Patent Document 7) and a method for extracting facial expression information (see, for example, Patent Documents 8 and 9).
Japanese Patent No. 3622711 Japanese Patent Laying-Open No. 2005-345496 (paragraphs [0011] to [0014] and the like). JP 2005-275348 A Japanese Patent No. 3372532 Japanese Patent Laid-Open No. 2006-113717 Japanese Patent Laid-Open No. 5-73317 Japanese Patent Laying-Open No. 2005-077855 JP 2005-157911 A Japanese Patent No. 3098276 Kenichiro Ishii, Noriyoshi Ueda, Eisaku Maeda, Hiroshi Murase, “Intuitive Pattern Recognition”, Ohmsha, 1st edition, August 1998, pp. 52-54. Kanayoshi Tsuji, Shuzuka Tezuka, Nobuo Ueda, Masaaki Taguri, “Computational Statistics I: A New Method of Stochastic Calculations, Frontier of Statistical Science 11 Chapter 3 3EM Method, Four Variational Bayes Method”, Iwanami Shoten, June 2003, pp. 157-186. Sadahiro Furui, “Digital Audio Processing, Chapter 4, 4.9 Pitch Extraction”, Tokai University Press, September 1985, pp. 57-59. Shigeki Hiyama and Fumitada Itakura, “Personality information included in the dynamic scale of speech”, Proceedings of the Acoustical Society of Japan 1979 Spring Research Presentation, 3-2-7, 1979, pp. 589-590. Kitagawa, G .; , "Non-Gaussian state-space modeling of non- sternary time series", Journal of the American Statistical Association, December 1987, pp. 10-28. 1032-1063. “Research on structured video handling mechanism and video interface based on video feature indexing, Chapter 3 Video indexing based on image processing”, Yoshinobu Tonomura, Ph.D. 15-23, 2006 "Computer image processing", edited by Hideyuki Tamura, Ohmsha, pp. 242-247, December 2002 “Computational Statistics I Chapter 3 3EM Method 4 Variation Bayes Method”, Nobuyoshi Ueda, Iwanami Shoten, pp. 157-186, June 2003 "Japanese vocabulary system", supervised by NTT Communication Science Laboratories, Satoru Ikehara, Masahiro Miyazaki, Satoshi Shirai, Akio Yokoo, Hiroaki Nakaiwa, Kentaro Ogura, Yoshifumi Oyama, Yoshihiko Hayashi, Iwanami Shoten, 1997

  However, in such a conventional user support method and apparatus, in principle, there is no means for mediating between users unless two or more users actually view content and write comments. It was.

  Therefore, for users who have never written a comment, or for users who have watched content that other users have never watched, information about which users have the same preferences and feelings as their own. There was a problem that I could not get it.

  On the other hand, user preferences and emotions change from moment to moment. For this reason, it is necessary to provide user support that reflects the preference and emotion at the time of communication in a timely manner.

  On the other hand, the conventional user support method and apparatus share preference and emotion based on comments. Comments, unlike chat, etc., have low real-time characteristics as a means of communication. For example, when a user has given a comment in the past, the preferences and feelings at the time when the user communicates with other users It tends to happen that preferences and feelings are different. For this reason, there has been a problem that it is difficult to implement user support reflecting tastes and feelings in a timely manner.

  The present invention solves the above-described problems, and an object of the present invention is to provide a user support method, a user support apparatus, and a user support program capable of realizing a highly reliable and timely association between users. .

  The present invention analyzes a content attribute that reflects a user's preference and an attribute value based on information included in the content, and discloses and views similar content based on the analyzed attribute and attribute value It is a user support method, a device, and a program that judge that sensibilities such as preferences are similar to each other and perform association.

  More specifically, attributes and attribute values included in the content using at least one of the content published by each user, the content being viewed, the audio information such as the content viewed, and the video information. To extract the emotional level. Furthermore, if necessary, the activity information of the user when viewing the content is acquired. These pieces of information are used as indices reflecting user preferences and feelings, and two or more users are associated with each other based on the indices.

  This method allows users to communicate based on publicly available content, watched content, and watched content without requiring users to take special actions such as writing comments. In addition to making it possible to give users, it is possible to support users through simple and timely reflection of preferences and emotions that are important in communication and community formation.

  In the present invention, preference includes interest and interest, and emotion includes emotion, atmosphere, impression, and the like. In addition, content refers to video / audio content, and audio includes not only human speech but also singing audio, music, environmental sound, and the like.

The method according to the first invention (Claim 1) and the apparatus (Claim 4) are user support methods and apparatuses for associating a plurality of users, and are characterized by any one or more of content audio information and video information An estimation step / means for extracting an amount, estimating an emotion level of each of a plurality of emotions of the content based on the extracted feature amount and a pre-configured statistical model, identification information of the content, A content attribute storage step / means for storing the emotion estimated for the content in association with the emotion level, content published by the user terminal, content being viewed, viewing The information for identifying the content and the information for identifying the user terminal are associated with each other for at least one of the obtained contents, and the user preference A user information storing step / means for storing as, based on the user preference information, the subject content associated with the respective user terminal, for each of the plurality of emotion, the weighted average of the emotion of the emotional category A user association step for obtaining a preference value for each user terminal and associating one or more user terminals other than the first user terminal with the first user terminal based on the similarity of the obtained preference values of the user terminals. / Means.

The method according to the second invention (claim 2), the apparatus (claim 5) The method according to the first invention, in the apparatus, said user information storing step / means, for each user terminal, the user terminal Information for identifying at least one of the published content, the viewed content, and the viewed content, and the user terminal when the content is viewed or viewed on the user terminal Activity information consisting of at least one of user's operation information, voice information, and biological information , and storing this as user preference information of the user terminal . The user terminal is associated based on the preference value of the user terminal and the similarity calculated from the activity information .

  In the method (invention 3) and the apparatus (invention 6) according to the third invention, in the method and apparatus according to the first or second invention, the user information storing step / means further includes a time change of the user preference information. It is characterized by memorizing.

  According to a fourth aspect of the present invention, there is provided a program for executing the steps according to any one of the first to third aspects.

(1) According to the first to seventh aspects of the present invention, based on the information included in the content, it is possible to estimate the content attributes and attribute values (emotion, emotion level) reflecting the user's preference. . Further, based on the attribute and the value of the attribute, the user's preference can be immediately analyzed from the content that the user has made public / viewed or viewed, and the association can be performed based on this.

According to this invention, it is not simply associated with users who have viewed the same content, or with users who have added comments, but by associating with content attributes and attribute values that reflect user preferences. Reliable and timely association can be realized. Moreover, according to this invention, in order to implement association, it is not necessary for the user to take an action such as giving a comment, and there is an advantage that the use barrier is low for the user.
(2) According to the second and fifth aspects of the present invention, user preferences and emotions can be analyzed based on user preference information and user activity information during content viewing. According to the present invention, more accurate association between users can be realized.
(3) According to the inventions described in claims 3 and 6, by storing the time change of the user preference information, it is possible to obtain an index for the dynamic change of the preference and emotion of each user. According to this invention, it is possible to further improve the timeliness and accuracy of association between users by performing association including changes in user preferences and emotions.
(4) According to the invention as set forth in claim 7, it is possible to execute all the above inventions on a computer.

  Hereinafter, embodiments of the present invention will be described with reference to the drawings, but the present invention is not limited to the following embodiments. There are various content attributes that reflect user preferences and emotions, such as content genre, subject, shooting location, shooting time, time, content creator, etc. Select an attribute that reflects the impression that the user receives when viewing.

As an example of such an attribute, in the embodiment of the present invention, an emotion included in content is assumed, an emotion level is estimated as an attribute value, and users are associated with each other based on the emotion and the emotion level. An example of the case will be described. Hereinafter, three examples of the embodiment of the present invention will be described with reference to FIGS.
[First example of embodiment]
In the first example of the embodiment of the present invention, when user support is executed, emotions and emotion levels are extracted from audio information of contents, user preference information is obtained based on this information, and association between users is executed. It is about the case.

  A user support method and a user support apparatus according to a first example of an embodiment of the present invention will be described. FIG. 1 is a flowchart showing a process for explaining a user support method according to an embodiment of the present invention, and FIG. 2 is a block diagram for explaining a user support apparatus 100 according to the embodiment of the present invention.

  The user support apparatus 100 in the first example of this embodiment includes at least user terminals 200a, 200b,..., An information control unit 300, and a database 400 (content attribute storage means of the present invention). These may be connected within the same terminal, or may be connected via a predetermined communication means that can communicate with each other, such as a LAN (Local Area Network) and / or a wide area communication network such as the Internet.

  The information control unit 300 and the database 400 may be stored in the same device, for example, because the database 400 is included in the information control unit 300.

  FIG. 3 is a block diagram illustrating the configuration of each user terminal 200a, 200b,. The user terminals 200a, 200b,... Are, for example, a keyboard 211, a pointing device 212 represented by a mouse, etc., and data input for inputting audio information (audio signal data) and video information (video signal data) of content. An input unit 210 having means (not shown), a CPU (Central Processing Unit) 221, a ROM (Read Only Memory) 222, a RAM (Random Access Memory) 223, a control unit 220, and an HDD (Hard Disc Drive) 231. The display unit 240 includes a storage unit 230 including a monitor screen 241 such as a liquid crystal screen, and displays information output from the control unit 220 in accordance with an operation of the input unit 210.

  The information control unit 300 is configured by connecting a CPU 301, a ROM 302, a RAM 303, an HDD 304, and the like. Various processes in the present invention are all performed by the information control unit 300, and all programs and data for realizing the various processes are stored in a memory device such as the ROM 302 and the HDD 304, and are appropriately read out to the RAM 303. Processing is executed in the CPU 301.

  The information control unit 300 includes, for each function unit, an emotion estimation unit F100 (estimation unit of the present invention), a user information storage unit F200 (user information storage unit of the present invention), a user association unit F300, and a notification if necessary. It is assumed that a part F400 is provided.

  As step S100, in the user support apparatus 100 in this embodiment, when receiving a content release request from the user terminals 200a, 200b,... And storing the content in the database 400, as shown in FIG. The content, the identification information unique to the partial content obtained by dividing the content into predetermined partial sections, and the emotion and emotion level of the content estimated by the emotion estimation unit F100 are stored in association with each other. Further, as user preference information of the user terminal that published the content, the identification information of the user terminal that published the content, the identification information of the published content, the date of publication of the content, etc. are stored in the user information storage unit F200 as user preference information. To do.

  In step S200, content publishing from user terminals 200a, 200b,... Or content publishing or distribution is executed in response to a content distribution request, and at the same time, identification information of each user terminal and content identification information, User preference information associated with delivery date and time is stored in the user information storage unit F200.

  Next, in step S300, the user association unit F300 associates the user terminals based on the stored user preference information.

  Finally, in step S400, the notification unit F400 transmits information related to the association result to each of the user terminals 200a, 200b,.

  Hereinafter, the flow of processing in one example of the embodiment of the present invention will be described in detail with reference to FIG.

  First, when receiving a content release request, the emotion estimation unit F100 extracts in advance the emotion and emotion level of the content based on the audio information included in the content.

  The emotion estimation unit F100 calculates the emotion and the emotion level of the content based on any one or more of the prosodic information and the text information obtained as a result of speech recognition among the audio information included in the content. The process for calculating the emotion and the emotion level of the content by the emotion estimation unit F100 is a pre-process to be performed before the user support is actually executed according to the present invention.

  An example of a method for estimating emotion and emotion level from prosodic information included in speech will be described.

  FIG. 5 is a flowchart for explaining an emotion detection method in the present embodiment.

  Note that the data in the description of the present embodiment is stored in and accessed by a general-purpose storage unit (for example, a memory or a hard disk device) or storage means.

  First, step S110 (statistical model construction processing step) is a step that is performed in advance before actually determining the emotional state by the emotion detection method according to the present embodiment, and calculates the emotional state probability. This is a step of constructing a statistical model used for this purpose. The entity of the statistical model is a function in which the statistics are calculated and parameters such as statistics are described as a program. In addition, although the code | symbol showing the parameter and function type | mold of the said statistical model will be stored in a memory | storage part, the memory capacity for them is comparatively small.

  Next, in step S120 (speech feature amount extraction processing step), a desired speech feature amount is calculated as a vector for each analysis frame (hereinafter simply referred to as a frame) and extracted from the captured content speech signal data. . The speech feature vector is a vector composed of one or more elements of a fundamental frequency, a time variation characteristic sequence of the fundamental frequency, a power, a time variation characteristic series of power, and a time variation characteristic of speech rate. is there. The audio signal data is input by an audio signal data input unit (for example, the input unit 210 in FIG. 3). The extracted voice feature amount is stored in the storage unit. Steps S120 to S150 are processes for calculating the emotional state probability.

  Next, in step S130 (speech feature amount appearance probability calculation processing step), a speech feature amount vector corresponding to the emotional state is determined for each frame based on the speech feature amount vector calculated in step S120 and stored in the storage unit. The probability of appearance is calculated by a statistical model configured in advance in step S110, and the calculation result is regarded as the speech feature amount appearance probability.

  Next, in step S140 (emotional state transition probability calculation processing step), for each frame, in the time direction of one or more state variables corresponding to the emotional state, using the statistical model previously configured in step S110. The transition probability is calculated, and the calculation result is regarded as the emotional state transition probability.

  Next, in step S150 (emotional state probability calculation processing step), the emotional state probability is calculated for each frame based on the speech feature amount appearance probability calculated in step S130 and the emotional state transition probability calculated in S140. To do.

  In step S160 (emotional state determination processing step), the emotional state of the frame is determined and output for each frame based on the emotional state probability calculated in step S150.

  In step S170 (emotion, emotion level estimation processing step), one or more of the emotional state probability calculated in step S150, the determined emotional state, continuous speech and non-speech time, continuous speech, and continuous non-speech time Based on the above, a section composed of one or more frames is formed, and the emotion and the emotion level are estimated using this section as a unit.

  Hereinafter, each step of the emotion detection method will be described in detail.

  First, details of the processing in step S110 constituting the statistical model will be described with reference to FIG. It is assumed that the statistical model is acquired by learning from learning speech signal data.

  First, in step S111, learning speech signal data is input. Note that the learning speech signal data may be input from speech signal data input means, or may be input from input means dedicated to learning speech signal data (learning speech signal data input means).

  Next, in step S112, the learning voice signal data is given the emotional state e in the frame, which is actually viewed and judged by a human, for each frame over the entire learning voice signal data. Here, in this emotional state e, what is determined by a human is referred to as a label, and the determination action is referred to as labeling.

  Strictly speaking, even if the emotional state e label is not given for each frame, if it can be converted so that it is labeled in units of frames, it can be used by converting to this. Also good. As an example of labeling, there is a case where a label of emotional state e is given in a certain section. In this case, it is possible to label each frame by giving the same label as the label given to the section to the frame included in the section. More specifically, when the time t1 to t2 with the sound is labeled as the emotional state e, all the frames in the section are configured as e.

  Next, in step S113, the speech feature quantity vector x is extracted for each frame, as in step S120. Hereinafter, the voice feature vector xt and the emotional state et in the frame Ft of the frame number t are represented.

  Next, in step S114, a statistical model for calculating the speech feature amount appearance probability and a statistical model for calculating the emotional state transition probability are configured by learning.

  First, an example of a statistical model learning method for calculating the speech feature amount appearance probability will be described.

The statistical model for calculating the speech feature amount appearance probability is a probability distribution in the speech feature vector vector x space given for each frame. For example, in the frame Ft, a certain (n−1) frame before that emotional state et for the interval _{= {e t, e t-} 1, ···, e t-n + 1} depending on represents the probability that xt appears, the conditional probability distribution p ^a (xt | et) Create as. For example, n may be about 2 to 3.

The conditional probability distribution p ^A (xt | et) may be configured using a probability model such as a normal distribution, a mixed normal distribution, or the like for each possible value of et. Further, it may be configured using a probability model such as a normal distribution, a mixed normal distribution, and a multinomial distribution for each type of speech feature. The parameters of these probability models are estimated based on the learning speech signal data.

  As the parameter estimation method, for example, a maximum likelihood estimation method, an EM algorithm, a variational Bayes method, or the like (for example, see Non-Patent Document 1, Non-Patent Document 2, etc.) can be used. .

  Next, a statistical model for calculating the emotional state transition probability will be described.

  The statistical model for calculating the emotional state transition probability is acquired by performing learning from the speech signal data for learning, similarly to the statistical model for calculating the speech feature amount appearance probability.

  In steps S111 to S113, the learning speech signal data includes the extracted speech feature vector x for each frame over all the learning speech signal data, and the frame actually determined by human viewing and viewing. An example of the step of performing learning in step S114 will be described below on the assumption that the emotional state e is given. The emotional state at frame number t is represented by et.

Statistical model for the emotional state transition probability calculating is a probability distribution on series et spatial emotional state in the t-th frame Ft, emotional state sequence in Ft previous (n-1) frame e _{t -1 = {e t-1,} e t-2, ···, e tn} , depending on the conditional probability distribution p ^B representing the probability that et appears | created as (e _t e _t-1) To do.

Note that et is a variable representing an emotional state such as joy, anger, and sadness, and is a discrete variable. Therefore, the conditional probability distribution p ^B (e _t | e _t-1 ) is, for example, It may be configured by creating a bi-gram type histogram. In this case, based on the learning audio signal data, when the e _t-1 are given, series e _t of each emotional state can configure this by counting how many times to emerge.

  The above is the detailed processing of step S110.

  Next, in step S120, a desired audio feature vector is extracted for each frame from the audio signal data of the captured content. In addition, the audio | voice in this invention shall include not only the conversation audio | voice by a human but singing audio | voice, or music.

  Below, an example of the audio | voice feature-value vector extraction method is demonstrated.

  First, the audio feature amount will be described. As the voice feature amount in the present embodiment, it is preferable that the voice feature amount is obtained stably even in a noisy environment and is less dependent on the speaker profile in determining the emotional state as compared with the voice spectrum, cepstrum, and the like. .

As speech feature quantities satisfying the above-described conditions, the fundamental frequency f0, the fundamental frequency time variation characteristic sequence {rf0 ⁱ }, the power p, the power temporal variation characteristic sequence {rp ⁱ }, and the speech rate temporal variation characteristic. Etc. are extracted. Note that i is an index of time variation characteristics.

  In the present embodiment, a series is defined as a set having one or more elements. Examples of time fluctuation characteristics include, for example, regression coefficient, average value, maximum value, minimum value of change amount in analysis frame, cumulative sum of absolute value of change amount in analysis frame, range, etc. To choose. In particular, in the case of regression coefficients, the index can be associated with the order. The regression coefficient may be used up to any order, but may be about 1 to 3 for example. In the following example, a case where only the regression coefficient is employed as the time variation characteristic will be described. As the power p, an amplitude value of a sound signal waveform may be used, or an absolute value, a smooth value, or an rms value may be used. Further, an average value of power in a certain frequency band, for example, a region such as 500 Hz (hertz) to 3 KHz (kilohertz) that is easy for humans to hear may be used.

  There are various methods for extracting the fundamental frequency f0 and the power p to be extracted. These extraction methods are publicly known, and for details, refer to the method described in Non-Patent Document 3, for example.

  The time variation characteristic of the speech rate to be extracted is extracted as the time variation characteristic rm of the dynamic measure m by a known method (see, for example, Non-Patent Document 4 and Patent Document 2).

  For example, a method may be used in which the utterance speed is detected by detecting the peak of the dynamic scale and counting the number, and the average value and the variance of the peak interval corresponding to the utterance rhythm are calculated to calculate the utterance speed. A method of detecting the time variation characteristic may be adopted.

  In the following description, the time variation characteristic rm of the dynamic scale using the peak interval average value of the dynamic scale will be described.

An example of a method for extracting a regression coefficient as the sequence {rf0 ⁱ } of time variation characteristics of the fundamental frequency to be extracted and the sequence {rp ⁱ } of power time variation characteristics will be described.

When the time to analyze is t, the relationship between the fundamental frequency f0 _t (for example, the graph of the symbol δ in FIG. 7) or p _t extracted at time _t and {rf0 ⁱ _t } or {rp ⁱ _t } Is represented by the following approximate expression.

Here, I represents the maximum order of the regression function. It suffices to determine {rf0 ⁱ _t } and {rp ⁱ _t } so that this approximation error becomes small in the vicinity of _t . As a method for realizing this, for example, the least square method may be used.

I may be an arbitrary value, but here, as an example, a case of obtaining rf0 ¹ t when I = 1 will be described. The same calculation can be performed for rp ¹ _j . If the time to analyze is t, the time variation characteristic rf0 ¹ _t of the fundamental frequency at _t is

  It can be asked. Here, d is the number of sampling points used for calculation before and after time t, and corresponds to the vicinity of t. For example, d = 2.

Hereinafter, for example, a case will be described in which only rf0 ¹ _t and rp ¹ _j obtained in the case of I = 1 are treated as the time variation characteristic rf0 of the fundamental frequency and the time variation characteristic rp of the power, respectively.

  An example of a method for calculating a speech feature amount for each frame will be described. Assume that the length of one frame (hereinafter referred to as the frame length) is 100 ms (milliseconds), and the next frame is formed by a time shift of 50 ms with respect to the current frame.

  For each frame, the average value of each voice feature amount in each frame, that is, the average fundamental frequency f0 ′, the average time variation characteristic rf0 ′ of the fundamental frequency, the average power p ′, and the average time variation characteristic rp ′ of the power. The average peak interval average value rm ′ of the dynamic scale is calculated. Alternatively, not only the average value but also the maximum value, the minimum value, or the fluctuation range of each voice feature amount in the frame may be calculated and used. Below, the case where only an average value is used is demonstrated.

  Each voice feature is preferably normalized in advance. The normalization may be performed, for example, for f0 ′ by, for example, subtracting the average fundamental frequency over the entire audio signal data to be processed or dividing by the average fundamental frequency. It may be. It is conceivable that the other audio feature amounts are also performed in the same manner.

  In the determination of the emotional state, it is necessary to capture the temporal behavior of the voice feature amount. In the present embodiment, a speech feature vector is calculated from speech features calculated for each frame using speech feature values of one or more frames, and the temporal behavior of the speech feature is captured. Note that a section in which the speech feature vector is extracted is referred to as a speech feature vector extraction section (for example, a section indicated by a symbol W in FIG. 8).

  Hereinafter, an example of a method for capturing the temporal behavior of the voice feature amount will be described with reference to FIG.

For the current frame F, its frame number is set as j and expressed as Fj. The speech feature values for each frame included in Fj are respectively the fundamental frequency f0 ′ _j , the fundamental frequency time variation characteristic rf0 ′ _j , the power p ′ _j , the power temporal variation characteristic rp ′ _j , and the peak interval average of the dynamic scale. It shall be represented as a value rm ′ _j .

  As a method for constructing a speech feature vector based on the speech feature obtained as described above, for example, a method of constructing by embedding each speech feature in the delay coordinate space is conceivable. That is, the speech feature amount included from the current frame to the past frames of S frames is represented by a vector.

For example, in the case of the fundamental frequency, if the current frame number is t, the speech feature vector f0 of the fundamental frequency is _{expressed as} f0 = {f0 ′ _t , f0 ′ _t−1 ,..., F0 ′ _tS } ^T. Let it be what you want. Frame in Figure 8, the frame F _tS indicated at w1, frame F _t-1 indicated by reference numeral w2, a frame F _t indicated at w3.

  Alternatively, a method is also conceivable in which the interframe difference amount from the current frame to the past frame of S frames is calculated for each audio feature amount, and this is expressed as a vector.

  Here, the value of S is, for example, S = 5. Similarly, the time variation characteristic rf0 of the fundamental frequency, the power p, the time variation characteristic rp of the power, and the time variation characteristic rm of the dynamic scale are calculated.

Assume that x is a list of all voice feature vectors determined to be used for determining an emotional state in advance. For example, when all the extracted speech feature values are used, x = {f0 ^T , rf0 ^T , p ^T , rp ^T , rm ^T } ^T. Then, when using the time variation characteristic rf0 of the fundamental frequency, the time variation characteristic rp of the power, and the peak interval average value rm of the dynamic scale, x = {rf0 ^T , rp ^T , rm ^T } ^T.

  In the present embodiment, the emotional state can be determined by using one or more of the audio feature quantities. However, in an utterance that appears characteristically in an emotional state, it is often difficult to extract the fundamental frequency f0 itself, which is often lost. For this reason, it is preferable to include the time variation characteristic rf0 of the fundamental frequency that can easily obtain the effect of complementing such a defect. Furthermore, it is preferable to further include a time variation characteristic rp of power in order to increase the determination accuracy while keeping speaker dependency low.

  As described above, it is possible to obtain the audio feature vector in all frames by performing the calculation processing of the audio feature value and the audio feature vector performed for each frame on the frame over the entire content. .

  The above is the detailed processing of step S120.

  Next, step S130 uses the speech feature vector of each frame extracted in step S120 and the statistical model pre-configured in step S110, and the appearance probability of the speech feature vector in each emotional state ( (Speech feature appearance probability) is calculated.

  Below, an example of the process performed by step S130 is demonstrated.

  An example of a method for calculating the appearance probability of a speech feature vector based on the statistical model created in step S110 will be described.

Since the statistical model is a conditional probability distribution p ^A (x _t | e _t ) in the speech feature vector x _t space given for each frame, the input speech feature vector x _t is preliminarily obtained in step S110. The likelihood is calculated based on the created statistical model p ^A (x _t | e _t ). This calculated likelihood is regarded as a speech feature amount appearance probability at which x _t appears in each emotional state.

  By performing the above processing over all the frames, it is possible to calculate the speech feature amount appearance probability for all the frames.

  The above is the detailed processing of step S130.

  Next, in step S140, using the statistical model, transition to each emotional state in the current frame depends on the emotional state of one or more previous frames immediately before the frame for determining the emotional state (current frame). Transition probability (ie, emotional state transition probability) is calculated.

  Below, an example of a process for performing step S140 is demonstrated.

  First, an example of a method for calculating the emotional state transition probability based on the statistical model created in step S110 will be described.

The statistical model is a conditional probability distribution p ^B (e _t | e _t-1 ) on a sequence _et space of emotional states. Therefore, in step S140, if e _t-1 has already been determined and determined, each emotional state is based on the statistical model p ^B (e _t | e _t-1 ) created in advance by the method in step S110 or the like. Compute the probability of a sequence e _t of. The calculated probability is regarded as an emotional state transition probability.

The determination of emotional state, since we performed sequentially along the time proceeding direction of the audio signal data, by the frame number t is a monotonically increasing with respect to the time axis, step of determining e _t The state in which _et-1 is already determined can be configured.

  By performing the above processing over all the frames, the emotional state transition probability can be calculated for all the frames.

  The above is the detailed processing of step S140.

  Next, in step S150, the emotional state probability is calculated based on the speech feature amount appearance probability and the emotional state transition probability calculated in steps S130 and S140.

  Below, an example of the process which calculates the emotional state probability performed by step S150 is demonstrated.

^{A set of the} statistical model p ^A (xt | et) and p ^B (e _t | e _t-1 ) has a structure called a generalized state space model, and is often used for speech recognition and the like. Left-to-Right type HMM (Hidden Markov Models), such as the same causal structure (e.g., a emotional state e _t1, e _t represented by reference numeral S _t1 in FIG. 9, the audio, it indicated at S _t2 Characteristic quantities x _t-1 and x _t ).

Generalized state space model, if obtaining the observation sequence {x _t} up to time t, the probability distribution of the internal state e _t at time _{t p (e t | {x} t}) is p ^A (x _{Based on t} | e _t ) and p ^B (e _t | e _t-1 ), the following equation can be calculated recursively (for example, see Non-Patent Document 5).

However, E is a set of all values that e _t can take.

Generally, in the generalized state space model, since the value of E is a very large value, it is difficult to obtain the value of p (e _t | {x _t }) by directly calculating the above equation.

In this embodiment, since all values that et can take are emotional states to be handled, that is, joy, anger, sadness, fear, surprise, impatience, etc., if this number is | e |, then e _t = { Considering all possible combinations of e _t , e _t−1 ,..., e _{t−n + 1} }, the size of E is | e | n.

  For example, in the present embodiment, the number of emotional states to be handled | e | is assumed to be about 10 for example, assuming joy, anger, sadness, fear, surprise, impatience, calmness, and the like. At this time, for example, if n = 3, the size order of E is about 10 3, and even the currently popular general-purpose computer is a scale that can sufficiently directly calculate the expression. .

Therefore, in this embodiment, since the value of the probability distribution p (e _t | {x _t }) of this expression can be directly calculated, the emotional state probability is calculated by direct calculation. To do.

Further, let e ′ _t = {e _t−1 ,..., E _{t−n + 1} } excluding et from the elements of et, and a set E ′ of all values that e′t can take. When expressed, emotional state probability of the emotional state e _t for each frame,

  Can be obtained by calculating. FIG. 10 shows an example of emotional state probabilities when emotional states of joy, sadness, and calm are handled. In other words, the curve indicated by the symbol L1 in FIG. 10 is the emotional state probability of pleasure, the curve indicated by the symbol L2 is the calm emotional state probability, and the curve indicated by the symbol L3 is the sad emotional state probability.

  By performing the above processing over all frames, it is possible to calculate the emotional state probability for all the frames.

  The above is the detailed processing of step S150.

  In step S160, the emotional state probability calculated in step S150 is taken into the emotion determination means, and the emotional state is determined based on the emotional state probability.

Hereinafter, an example of the process of step S160 for determining the emotional state will be described. In the following description, the categories of emotional states to be handled are indexed in order as e ¹ , e ² ,..., E ^{| e |} . For example, when dealing with emotional states of joy, anger, sadness, fear, surprise, impatience, calmness, e ¹ : joy, e ² : anger, e ³ : sadness, e ⁴ : fear, e ⁵ : surprise, e ⁶ : impatience, e ⁷ : calmness, etc. In this case, | e | = 7.

In step S150, the emotional state emotional state probability is e ^k p ^k _t = p in the frame Ft of the frame number t | because it calculates the ^{_{(e t = e k {x}} t}), the simplest Can determine the emotional state corresponding to e ^k having the highest probability p ^k _{t as} the emotional state in Ft. Alternatively, one or more e ^k having a high value of p ^k _t may be selected in descending order, and the determination result may be obtained by having these plural emotional states.

Or, depending on the emotional state, there are things that are easily recollected at the same time and things that are difficult to recall. For example, it is easy to imagine that joy and sorrow are difficult to recall at the same time. This phenomenon in view, the emotional state probability p ^k _t corresponding to a certain emotional state e ^k, other emotional states ^{{e 1, e 2, ···} , e k-1, e k ^+1, ···, ^{e |} e ^| emotional state probability {p ¹ t corresponding ^{to}, p 2 t, ···,} p k-1 t, p k + 1 t, ···, p | ^{e |} t} is a convex combination,

May be compared as a new p ^k t obtained by normalization.

Or, simply, determined in advance certain threshold, the emotional state e ^k corresponding to the value of the large p ^k t than this, it may be determined that emotional state.

  By performing the above processing over all the frames, it is possible to determine the emotional state for each frame.

  The detailed processing in step S160 has been described above.

  Through the above steps S110 to S160, it is possible to determine the emotional state for each frame with respect to content including arbitrary audio signal data.

  In step S170, the emotion and emotion level are estimated and output based on the emotional state probability calculated in step S150 and the emotional state determined in step S160.

  Hereinafter, an example of processing for estimating emotion and emotion level will be described. In the present embodiment, a set of utterance sections that are considered to be continuous utterances is processed as one section. Hereinafter, a set of utterance sections composed of continuous utterances is referred to as an audio sub-paragraph.

  Here, an example of a method for generating a small audio paragraph will be described.

  First, a section that is considered to be an utterance section is extracted. As an example of a method of extracting such a section, a method of extracting a corresponding section by using a periodicity of a speech section in a speech waveform and considering a section having a high autocorrelation function as a speech section. is there.

  Actually, a section showing an autocorrelation function value higher than a certain threshold is regarded as an utterance section. This threshold value may be given as a constant in advance, or after calculating the autocorrelation function value of the entire content, the speech time (or voiced time) and non-speech time (or unvoiced time) in the general case As a reference, the threshold may be determined so that the speech time ratio is close to this reference.

  In the present embodiment, the unit constituting the speech section is a frame unit. That is, if the fundamental frequency is included in the speech feature vector, a section higher than a certain threshold value may be regarded as an utterance section (that is, an utterance frame).

  Through the above processing, it is possible to separate speech frames and non-speech frames from content in units of frames.

  Next, among the extracted utterance frames, an utterance frame set considered to be continuous utterances is collected as one section, thereby generating a small audio paragraph.

  An example of a method for generating such a small audio paragraph will be described.

Assume that a set of utterance frames F ′ in the content is {F ′ ₁ , F ′ ₂ ,..., F ^′ _N } in order from the earliest time. Here, N is the total number of speech frames.

Next, the speech frame F _'i, F' adjacent on the time axis _{i + 1} of the time interval, i.e., a _iend 'end time F of _i' F, which is the next frame F _{'i + 1} start time F 'For _{i + 1start} , the time difference F _{i + 1start} -F _iend is calculated.

  Next, if the calculation result is smaller than a predetermined threshold, Fi and Fi + 1 are considered to be continuous speech frames, and form one audio sub-paragraph.

  Then, by repeating this process over all frames, frames that are considered to be continuous speech can be grouped into audio sub-paragraphs.

  As another example of the method for generating the audio sub-paragraph, the following can be considered.

  First, a start time and an end time are obtained for each utterance frame over the entire content, and these are collectively regarded as a two-dimensional vector.

  For this vector, the bottom-up clustering method is applied so that the ratio of the speech time to the non-speech time of the entire content is approximately the same as the ratio of the speech time to the non-speech time in the general case. Generate. However, when the genre or the like of the target content is known in advance from the content attribute information or the like, the target ratio may be changed accordingly.

  In the other example described above, unlike the method in which the threshold value is determined in advance, the difference in the speech rate between contents can be absorbed and the audio sub-paragraph can be configured adaptively.

  Through the above processing, each audio sub-paragraph necessarily includes one or a set of continuous frames, and the entire speech frame included in the content can be combined into several audio sub-paragraphs.

  Next, the emotional level, which is the emotional state of each constructed audio sub-paragraph, is calculated. Hereinafter, an example of a method for calculating the emotion level will be described with reference to FIG.

Assume that a set of audio sub-paragraphs S ′ in the content is {S ₁ , S ₂ ,..., S _NS } in order from the earliest time. For example, in FIG. 11, the audio sub-paragraph S _i-1 indicated by the reference symbol v1, the audio sub-paragraph Si indicated by the reference symbol v2, and the audio sub-paragraph S _{i + 1} indicated by the reference symbol v3.

Here, NS is the total number of audio sub-paragraphs. Also, an utterance frame included in a certain audio sub-paragraph Si is set as {f ₁ , f ₂ ,..., F _NFi }. NFi is the number of speech frames included in the audio sub-paragraph Si.

Each speech frame ft is given an emotional state probability p (et | {xt}) by the emotional state probability calculation means. The emotional level p _Si (e = e ^k ) in which the emotional state e of the audio sub-paragraph Si is e ^k can be calculated by, for example, the following expression representing an average value.

  Further, for example, it is conceivable to calculate by the following expression representing the maximum value.

  In addition to these, for example, there are various methods such as calculating the emotion level after windowing in the audio sub-paragraph, but the emotion level may be compared between the audio sub-paragraphs, so the emotion level is It is preferable that the value falls within a certain value range, for example, between 0 and 1.

  In addition, the emotional degree in FIG. 11 is an emotional degree shown with the code | symbol H1-H3.

  It is possible to perform the calculation as described above over all the audio sub-paragraphs and to give emotion levels of all emotional states to all the audio sub-paragraphs.

  As a method for extracting emotion and emotion level from text information obtained as a result of speech recognition, for example, first, text information is obtained by performing speech recognition from speech information using the technique of Patent Document 3, for example. Next, from the obtained text information, for example, a word or the like corresponding to the emotional expression may be extracted using the technique of Patent Document 4.

  As for the emotion level, for example, “word:“ can laugh ”, emotion:“ fun ”, emotion level: 0.8”, “word:“ muddy ”→ emotion:“ anger ”, emotion level: 0.7, etc. It can be extracted by holding a dictionary in which words corresponding to emotional expressions are associated with emotions and feelings, and referring to this dictionary. For the feelings and feelings of the words not registered in the dictionary, for example, by calculating the similarity between the words registered in the dictionary and the unregistered words by the method described in Patent Document 5, the similarity is high. The average value of the emotion level of one or more registered words may be set as the emotion level of the unregistered word.

  For example, for words that are difficult to be attributed to one emotion category, such as “odd”, “word:“ odd ”” → emotion: scary, emotion level: 0.3 AND emotion: mysterious , Emotion level: 0.3 ”or the like, the process may be executed by belonging to a plurality of emotion levels and emotion levels.

  In addition, relative expressions such as “very” can be quantified by using the technique of Patent Document 6 and the like, and the emotion level may be appropriately adjusted based on this. The emotion level is calculated as a numerical value, and preferably falls within a certain range such as an upper limit and a lower limit, for example, 1 and 0.

  Even if the emotion level obtained for each word is calculated over the entire content or part of the content, the sum, average value, maximum value, etc. are calculated to define the emotion level of the entire content or part. Good.

  The dictionary used here may be designed in advance by the designer, or may be constructed using learning to take into account the actual subjective feeling of the user.

  For emotion categories extracted using voice information, for example, “fun”, “sad”, “scary”, “severe”, “cool”, “cute”, “exciting”, “passionate”, “ The index set may be “romantic”, “violent”, “calm”, “healed”, “warm”, “cold”, “creepy”, etc.

  The range for extracting emotions and emotion levels may be the entire content or a partial content unit obtained by dividing the content into predetermined intervals.

  There are cases where a plurality of emotion levels are extracted for one emotion. In this case, by calculating the sum, average value, maximum value, etc. of the emotion level for each emotion, do it.

  The content emotion and emotion level estimated in this way are stored in the database 400 in association with the content identification information. The database 400 includes a predetermined storage device and content information including content, emotion, and emotion level. This predetermined storage device is, for example, a predetermined server connected to a user terminal by a HDD in a user terminal, an HDD in an HDD recorder, or a LAN in the case of a relatively small use range such as an individual or home. In the example of the embodiment of the present invention, a plurality of user terminals connected by a wide-area communication network such as the Internet may be used. It deals with those who share the feelings of those who have a relatively large scope of use. In this case, for example, a storage device with a server device connected by a wide area communication network or the like may be used.

  At the same time as the estimation of emotion and emotion level of the published content, the information control unit 300 uses the user preference information of the user terminal that published the content as the user preference information of the user terminal that published the content, the identification information of the published content, As shown in FIG. 12, the date and time of content release are stored in the user information storage unit F200 as user preference information.

  Regarding the identification information of the user terminal, for example, when the user performs user registration in advance at the content sharing site or the like, the identification information can be assigned for each user. It may be information.

  Even when user registration or the like is not performed, in the case of use through communication such as the Internet, if identification information assigned to each terminal for communication, for example, an IP (Internet Protocol) address, etc. is acquired as much as possible. Good.

  In the following, unless otherwise mentioned, these are not distinguished and referred to as user terminal identification information.

  Next, in step S200, when a content distribution request is received from the user terminal, the identification information of each user terminal is associated with the identification information of the content, the distribution date and time of the content, and the user information is stored as user preference information. Store in part F200.

  A content distribution request input from each of the user terminals 200a, 200b,... Is transmitted to the information control unit 300 by a predetermined communication unit. At this time, each of the user terminals 200a, 200b,... Is associated with identification information such as an ID by the information control unit 300, and is identified as different from each other.

  When receiving the content distribution request, the information control unit 300 searches the content corresponding to the request from the content stored in the database 400, and distributes the content to each user terminal 200a, 200b,. At this time, as shown in FIG. 12, the identification information of the user terminal that is the distribution destination of the content, the identification information of the distributed content, the content distribution date and time, etc. are stored as user preference information in the user information storage unit F200.

  At this time, not only the content delivery date but also various time information such as viewing start time and viewing end time, and temporal changes in the preference information and / or operation information are stored in association with the time information. By doing so, it is possible to take into account dynamic changes in the user's emotion when associating the user terminals described later.

  Next, the user association unit F300 associates each user terminal based on the stored user preference information. In the first example of the embodiment of the present invention, this association is calculated based on the similarity of the emotion and emotion level of the content included in the user preference information as the similarity, and the association is performed based on the similarity.

  As a timing of association, for example, in the case of a content sharing site or the like, among user terminals accessing the site, at least a predetermined percentage of user terminals have received content distribution at least once, and The calculation may be performed when a predetermined time has elapsed since the previous association process. Further, it may be performed at the time when execution of association is requested from the user terminal.

  Hereinafter, it is assumed that the association described below targets only user terminals that have published content at least once or are receiving distribution. That is, only a user terminal in which one or more user preference information composed of user terminal identification information, content publication / distribution date and time, and content identification information is stored is targeted.

  A flowchart illustrating an example of the flow of processing in step S300 for performing user association processing executed by the user association unit F300 will be described with reference to FIG.

  First, in step S310, a preference value as a reference value of user preference information is calculated for each user terminal.

Hereinafter, an example of a method for calculating the preference value will be described. For example, the user terminal 200a, the viewed content i-th, the emotion of the k-th category e ^k represents the ri (e ^k), represents the number of viewed content in the N. Each ri (e ^k ) is preferably normalized to a numerical value between 0 and 1, for example. At this time, for example, the preference value Ra (e ^k ) of the category e ^k of the ^kth emotion of the user terminal 200a is

  It is conceivable to calculate by

  Here, wi is a weight. For example, the more recent the time when viewing the content, the larger the value can be considered. Also, a forgetting model incorporating psychological knowledge such as the forgetting curve of Ebbing House may be introduced, and the weight may be determined by associating it with the viewing time of past content.

  By such weighting processing, it is possible to calculate a preference value taking into account a change in user preference. For example, the user terminal 200a previously liked the content of the “fun” impression, but recently likes the content of the “crying” impression, and therefore associates with the user terminal having a similar tendency, etc. , More timely and accurate associations can be performed.

Similarly, the user terminal 200b, 200c, respectively, for even ..., preference value Rb for each emotional category ^{e k (e k), Rc} (e k), calculated as ....

  Next, in step S320, the similarity between the user terminals is calculated based on the preference value of each user terminal 200a, 200b,. The similarity indicates, for example, how similar the preference values of the other user terminals 200b, 200c,... Are with respect to the preference value of the user terminal 200a. Hereinafter, an example of a method for calculating the similarity between the user terminal 200a and the user terminal 200b will be described.

  For example, the similarity fs (a, b) between the user terminal 200a and the user terminal 200b is expressed as follows: if the emotion category index set is K and the number is # (K),

  Can be calculated by: The calculation formula of the similarity is not limited to the above formula 9, and for example, the denominator may be replaced by the p-order average norm of the preference value between target users.

  Furthermore, the past preference value may be stored, and the preference value may be calculated by adding correction based on this value. By taking into account past preference values, it is possible to capture invariance and large changes in user preferences.

  Similarly, the similarity between the user terminal 200a and the user terminals 200c, 200d,... Is calculated as fs (a, c), fs (a, d),. ... Also calculates the similarity fs (b, c), fs (b, d),... With other terminals. By such processing, the similarity between all target two user terminals can be calculated.

  The above preference value calculation and similarity calculation need not necessarily use all of the user preference information. For example, the preference value may be calculated only for the content published by the user, or the preference value may be calculated only for the content viewed by the user.

  For example, when the preference value is calculated using only the information viewed by the user terminal 200a, the preference value becomes a preference index (viewing preference value) for viewing of the user terminal 200a, and is also disclosed to the user terminal 200b. When the preference value is calculated using only the information on the content that has been made, the preference value becomes an index (public preference value) of the tendency of the content preference published by the user terminal 200b.

  Using these values, for example, when the similarity between the viewing preference value of the user terminal 200a and the public preference value of the user terminal 200b is high, the user terminal 200a meets the preference in the notification process described later. It is also possible to notify the information of the user terminal 200b as a user who has published the content.

  In addition, for example, when calculating the sum of similarities between a certain user and one or more other users, this value is an index indicating how much users and how strongly they are similar (total Degree of similarity). It becomes.

  For example, for the user terminal 200a, the total similarity between the public preference value and the viewing preference value of the other user terminals 200b, 200c,... Is higher than the total similarity calculated in the other user terminals in the same manner. And Based on this, the user terminal 200a can notify other users of information, etc., assuming that content that is strongly preferred by many users is disclosed.

  Finally, the notification unit F400 transmits the result based on the similarity to each user terminal 200a, 200b,.

  The notification unit F400 notifies each user terminal of a result based on the similarity calculated by the user association unit F300. Hereinafter, step S400 for executing this process will be described. As a notification method, information on a user terminal having a high degree of similarity may be notified to a certain user terminal. As a result of notification, for example, in a content sharing site or the like, when each user performs user registration in advance, the user is permitted to release the profile information input by each user at the time of registration. Profile information and user information such as the preference value of the user terminal.

  As other information to be notified, attribute information of content distributed to a user (similar user) terminal with high similarity, information on content such as emotion and emotion level, thumbnail, summary, and the like can be considered.

  The number of contents to be notified may be determined as a constant for each similar user terminal in advance, for example, one or two in order from the most recently distributed, or the number may be determined according to the ranking of similarity. You may change it.

  As the content attribute information, content title, producer, keyword, outline, creation date / time, format, related content attribute information, and the like can be considered, for example, description by XML (eXtensible Markup Language) such as MPEG7. If it conforms to the format, it can be easily given.

  Hereinafter, an example of a method for creating summary content will be described.

  The summary content is generated by generating an audio paragraph composed of audio small paragraphs as a unit. As the audio paragraph, for example, a target time of an audio paragraph such as 5 s (second), 10 s or the like is set, and audio sub-paragraphs are gathered so as to approach this.

  Alternatively, if the unvoiced interval between audio sub-paragraphs is equal to or less than a certain threshold, it is conceivable to compose an audio paragraph by collecting them.

  Since the emotion level is given in units of small voice paragraphs, each voice paragraph includes one or more emotion levels. For example, the audio paragraph VC1 in FIG. 14 includes audio subparagraphs v11, v12, and v13, and the audio subparagraph v11 includes emotion levels H11 to H13.

  Actually, the cumulative emotion level of the voice paragraph is calculated based on these emotion levels. As this calculation method, for example, an average of the emotion level of the audio sub-paragraph, a weighted average, or a maximum value can be considered.

  The summary content is created by ranking the descending order from the speech paragraph with the highest cumulative emotion level, and extracting the speech paragraphs in order from the top so that the compression ratio, which is the ratio to all content, approaches a certain target value. Also good.

  The target value may be input by the viewer as desired. Furthermore, the viewer may input a reference value corresponding to the cumulative emotion level, and extract the audio paragraph portion that matches the reference to create a summary.

  The summary content can be reproduced by, for example, an operation in which the user points the area on the monitor 241 where the summary is presented using the pointing device 212 or the like.

  As for thumbnails, there is a method of extracting and presenting a predetermined time portion of the above-mentioned summary, for example, the top image, the center image, etc. of the summary content as a still image. In addition, in the emotion determined to be most dominant in the content, an image of a frame having the highest emotional state probability may be used as a thumbnail.

  Content that is determined to suit each user's chances of communication with their users, who are likely to be similar to their preferences and emotions, and their preferences and emotions based on notification of results based on such similarity Information can be obtained.

  Further, by storing the history of actions taken by the user by the notification of this result, secondary user support that does not depend only on the similarity can be executed.

For example, as a result of the previous similarity calculation, it is assumed that the user terminal 200b is introduced to the user terminal 200a and the user terminal 200c is introduced to the user terminal 200b as a user terminal having a high similarity. At this time, if it is stored that the user terminal 200a and the user terminal 200b have communicated more than a certain number of times, it is determined that the preferences and emotions of the user terminal 200a and the user terminal 200b are more similar. The user terminal 200c is newly introduced to the user terminal 200a.
[Second Example of Embodiment]
In the second example of the embodiment of the present invention, when user support is executed, emotions and emotion levels are extracted from audio information and video information of content, user preference information is obtained based on this information, and association between users is obtained. It is about the case of executing.

  As long as the flow of processing according to the second example of the embodiment of the present invention and one example of the specific configuration of the apparatus are shown in the flowchart of FIG. 1 and the block diagram of FIG. It may be the same as the case of the first example of the form.

  The difference from the first example of the embodiment is that the emotion estimation unit F100 obtains user preference information using not only audio information but also video information. Hereinafter, an example of the emotion / feeling level estimation method in step S <b> 100 executed by the emotion estimation unit F <b> 100 will be described. Other processing flows and the specific configuration of the apparatus may all be the same as in the first example of the embodiment.

  The emotion estimation unit F100 includes one or more information of prosodic information, text information obtained as a result of speech recognition, and color information, editing information, motion, among video information, and audio information included in content. The emotion of the content and the emotion level are calculated based on any one or more of the vector information. The audio information may be exactly the same as that described in the first example of the embodiment of the present invention.

  Among the video information, what is included in the color information is, for example, a hue value calculated for each region constituted by a single pixel and / or one or more pixels and this temporal variation characteristic, a single pixel and / or There are a luminance value calculated for each region constituted by one or more pixels, a temporal variation characteristic thereof, a hue histogram, a temporal variation characteristic of the hue histogram, a luminance histogram, a temporal variation characteristic of the luminance histogram, and the like. As a method for extracting such information, for example, a method shown in Non-Patent Document 6 or the like can be used.

  Examples of the editing information include cut point frequency, shot length, and connection effect information between shots such as wipes and dissolves. As a method for extracting such information, for example, the method shown in Non-Patent Document 6, etc. can be used.

  Examples of the motion vector information include camera work information and operation information of an imaging target. For example, the motion vector information may be extracted from the entire image and / or a specific region using the method described in Non-Patent Document 7, for example.

  As an example of a method for extracting emotion and emotion level from these pieces of information, a statistical model constructed by learning can be used. In this case, it is assumed that the learning video information is provided with a label corresponding to the emotion category by the extracted video information and manually.

  By associating emotion with a statistical model, a statistical model for calculating a probability for each emotion is obtained. As the statistical model, for example, a normal distribution, a mixed normal distribution, a hidden Markov model, a generalized state space model, or the like may be used. Preferably, a time series model such as a hidden Markov model or a generalized state space model that can model the temporal transition of emotion is employed.

  As a method for estimating the parameters of these video models, for example, a maximum likelihood estimation method, an EM algorithm, a variational Bayes method, and the like are known, and these can be used. For details, refer to Non-Patent Document 8, Non-Patent Document 9, and the like. For the emotion level, for example, the calculated probability of each emotion or the average value or maximum value of the probabilities included in a predetermined section may be regarded as the emotion level.

  In addition to this, for example, information on the result of recognizing the imaging target using the technique of Patent Document 7 may be considered. In particular, when the imaging target is a person or the like, facial expression information or the like may be taken into consideration.

  The emotion and the emotion level in this case are, for example, “imaging target:“ stuffed toy ”→ emotion:“ cute ”, emotion level: 0.6”, “imaging target:“ flower ”emotion: It is possible to extract by referring to a dictionary in which an imaging target that can be associated with an emotion, such as “beautiful”, emotion level: 0.4 ”, and the emotion and emotion level are associated with each other. This dictionary may be constructed using learning to take into account the actual subjective feeling of the user.

  For facial expression information, for example, an area determined to be a face in a video is detected by a method disclosed in Patent Document 8, and the facial expression information is detected by a method disclosed in Patent Document 9. Information can be extracted.

  For the emotion level, for example, if the result of recognizing the facial expression is determined to be “fun”, the emotion level corresponding to the emotion of the recognition result, such as increasing the emotion level for the “fun” emotion by a predetermined value, etc. It is possible to take a method such as increasing the emotion level of a particular emotion and / or decreasing the emotion level of emotions corresponding to other emotions by a predetermined value.

  The emotion category extracted using the video information may be the same as the category extracted using the audio information. The range for extracting emotions and emotion levels may be the entire content or a partial content unit obtained by dividing the content into predetermined intervals.

  Regarding emotion level, there are cases where multiple emotion levels are extracted for one emotion. In this case, by calculating the average value, maximum value, etc. of the emotion level for each emotion, The emotion level should be used.

  The content emotion and emotion level estimated in this way are stored in the database 400 in association with the content identification information.

Henceforth, the flow of the process after step S200 should just be performed similarly to the 1st example of embodiment of this invention.
[Third example of embodiment]
The third example of the embodiment of the present invention is obtained by sensing the user's activity information at the time of content viewing and storing the user preference information obtained and stored in the first example and the second example of the embodiment. This is the case.

  The flow of processing according to the third example of the embodiment of the present invention is shown in the flowchart of FIG. 15, and the specific configuration of the apparatus is shown in the block diagram of FIG.

  The user support apparatus 100 in the third example of this embodiment is such that at least the user terminals 200a, 200b,..., The information control unit 300, and the database 400 are connected via a predetermined communication means that can communicate with each other. Composed.

  FIG. 17 is a block diagram illustrating the configuration of each user terminal 200a, 200b,. In addition to the configurations of the first example and the second example of the embodiment, the user terminals 200a, 200b,... Further include a body sensor 250 as means for acquiring user operation information. Examples of the body sensor include an imaging device, a sound collection device, a pressure sensor, an acceleration sensor, a temperature sensor, a pulse sensor, a myoelectric sensor, a sweat sensor, and an electroencephalogram sensor.

  The information control unit 300 includes an emotion estimation unit F100, a user information storage unit F200, a user association unit F300, a notification unit F400, and an activity information recognition unit F500 for each function.

  When the emotion estimation unit F100 extracts the emotion and the emotion level of the content and distributes the content to the user terminal, the identification information of each user terminal is associated with the identification information of the content, and the user preference information is used as the user preference information. All the processes stored in the information storage unit F200 may be the same as the processes in the first example or the second example of the embodiment.

  In the present embodiment, the activity information recognition unit F500 further recognizes user activity information when viewing the content after distributing the content to the user terminal. The activity information obtained as a result is stored in the user information storage unit F200 in addition to the user preference information acquired by analyzing the content.

  Hereinafter, an example of the flow of processing of the third example of the embodiment of the present invention will be described in detail with reference to the flowchart of FIG.

  In step S100B and step S200B, the same processing as step S100 and step S200 of the first example or the second example of the embodiment may be performed.

  In step S300B, in the activity information recognition unit F500, after distributing the content to each user terminal, the activity information at the time of viewing the content is acquired by the body sensor 250 in the user terminal 200. This activity information can be configured by at least one of, for example, motion information, audio information, and biological information.

  Examples of the operation information include facial expression information, posture information, operation frequency information of the keyboard 211, pointing device 212, and operation amount information.

  Of these, facial expression information is, for example, detected from the image information acquired by the imaging device by an area determined to be a face in a video by a method disclosed in Patent Document 6, and the like. 7 can be extracted numerically as a feature vector that characterizes facial expressions.

  The posture information can be acquired by, for example, an imaging device, an acceleration sensor, a myoelectric sensor, or the like.

  When the image is acquired by the imaging device, for example, a person area in the video can be detected and extracted as a numerical value based on the inclination from the vertical axis, the size of the person area, and the like.

  In addition, it is conceivable to measure the movement of the hand. For example, when the user wears an acceleration sensor on the wrist, the amount of movement per unit time can be calculated by integral calculation. , And can be extracted numerically as the norm, which is the speed.

  The operation frequency and the operation amount information can be respectively obtained as numerical values such as the number of times the key of the keyboard 211 is pressed and pushed up per unit time, the amount of pointer movement by the pointing device 212, the number of clicks, and the like.

  The speech information includes prosodic information of the user's uttered speech, text information obtained as a result of speech recognition of the utterance, and these are the emotion estimation unit of the content in step S100 of the first example of the embodiment of the present invention. It can be quantified as the emotion level of each emotion by the same method as executed by F100.

  Examples of the biological information include body temperature information, pulse information, sweat information, and electroencephalogram information.

  These can be acquired as numerical values using various biological sensors such as a temperature sensor, a pulse sensor, a myoelectric sensor, a sweat sensor, and an electroencephalogram sensor.

  As the timing for extracting such activity information, for example, the content information may be extracted at regular intervals after the content is distributed to the user terminal until the reproduction of the content is finished in the user terminal.

  Moreover, even if extraction is performed at a timing when a portion having a relatively high emotion level is reproduced based on the information on the reproduction location (time) of the distributed content and the emotion level information estimated in step S100B. Good.

  Each activity information obtained as described above is stored in the user information storage unit F200 in a format to be added to the user preference information as shown in FIG.

  Next, the user association unit F300 associates each user terminal based on the stored user preference information. In the third example of the embodiment of the present invention, this association is performed by calculating the similarity degree of the user preference information as the similarity degree and performing the association based on the similarity degree.

  As a timing of association, for example, in the case of a content sharing site or the like, among user terminals accessing the site, at least a predetermined percentage of user terminals have received content distribution at least once, and The calculation may be performed when a predetermined time has elapsed since the previous association process. Further, it may be performed at the time when execution of association is requested from the user terminal.

  Hereinafter, the association described below receives at least one content distribution, that is, a user terminal in which one or more distribution records composed of user terminal identification information, content distribution date and time, and content identification information are stored. Only for the target.

  The process will be described with reference to FIG. 19, which is an example of the flow of the process of step S <b> 400 </ b> B that is executed by the user association unit F <b> 300 and performs the user association process.

  First, in step S410B, like the first example and the second example of the embodiment, for each user terminal, a preference value based on the emotion and the emotion level estimated by analyzing the content is calculated. Hereinafter, the preference value corresponding to this is referred to as a content preference value. The calculation of the content preference value may be exactly the same as that described in step S310 of the first example of the embodiment of the present invention.

Next, in step S420B, a preference value based on user preference information obtained from activity information is calculated for each user terminal. Hereinafter, a preference value corresponding to this is referred to as an activity preference value.

Hereinafter, an example of a method for calculating the activity preference value will be described. For example, for the user terminal 200a, the calculated value of the k-th activity information m ^k when viewing the i-th viewed content is represented as qi (m ^k ), and the number of viewed content is represented as N. Each qi (m ^k ) is preferably normalized to a numerical value between 0 and 1, for example. At this time, for example, the activity preference value Qa (m ^k ) for the k-th activity information m ^k of the user terminal 200a represents a weighted average value.

  It is conceivable to calculate by Here, vi is a weight. For example, it can be considered that the value is larger as the time when viewing the content is more recent. In addition, a forgetting model incorporating psychological knowledge such as an Ebbinghouse forgetting curve may be introduced, and the weight may be determined by associating it with the viewing time of past content.

  By such weighting processing, it is possible to calculate a preference value taking into account a change in user preference, and to perform association with higher timeliness.

Similarly, each user terminal 200b, 200c, for also ..., activity preference value Qb for each activity information ^{m k (m k), Qc} (m k), calculated as ....

  The content preference value and activity preference value obtained by the above calculation are combined into a preference value, and based on this, the similarity between the user terminals of the user terminals 200a, 200b,... Is calculated in step S430B. The similarity indicates, for example, how similar the preference values of the other user terminals 200b, 200c,... Are with respect to the preference value of the user terminal 200a. Hereinafter, an example of a method for calculating the similarity between the user terminal 200a and the user terminal 200b will be described.

  For example, the similarity fs (a, b) between the user terminal 200a and the user terminal 200b is expressed by K for the emotion category index set, the number # (K), the activity information index set L, and the number # (L)

  Can be calculated by:

  For example, the denominator may be replaced by the sum of p-order average norms of content preference values and activity preference values between target users, for example.

  Furthermore, the past preference value may be stored, and the preference value may be calculated by adding correction based on this value. By taking into account past preference values, it is possible to capture invariance and large changes in user preferences. When calculating the similarity, for example, the numerical values may be weighted based on criteria such as the reliability of each preference value. Thereby, for example, when there is a user preference or emotion that is strongly reflected in the activity preference value, the calculation of the similarity with higher accuracy can be executed by processing such as increasing the weight for the activity preference value.

  Similarly, the similarity between the user terminal 200a and the user terminals 200c, 200d... Is calculated as fs (a, c), fs (a, d). For *, similarity fs (b, c), fs (b, d),... With other terminals is calculated. By such processing, the similarity between all target two user terminals can be calculated.

  Hereinafter, step S500B in the third example of the embodiment of the present invention may be exactly the same as step S400 in the first example or the second example of the embodiment of the present invention.

  Heretofore, an example of the method in the embodiment of the user support method according to the present invention has been described in detail. Other than the example shown as the embodiment of the present invention, the embodiment can be appropriately changed within the scope of the embodiment that can be taken based on the principle of the present invention.

  Below, the Example which performs user assistance by this invention is shown. In the embodiment described below, the degree of similarity between each of the user terminals 200a, 200b,... Connected to the information control unit 300 and the server device 500 including the database 400 storing contents is calculated by Internet communication. This is a case where user support is performed by sharing emotions.

  As pre-processing, it is assumed that a speech model for calculating an emotion probability based on at least speech information is stored as a statistical model for calculating an emotion probability in the emotion extraction unit F100.

Each user is given a unique identification information (user ID) by accessing a predetermined content sharing site supplied by the server device 500 provided with the information control unit F300 and logging in to this site. And
[First embodiment]
In this embodiment, the emotion estimation unit F100 extracts the emotion and emotion level of the content from the audio information in the content, obtains the user preference information based on this, and performs the association. FIG. 20 shows the configuration of a specific apparatus of the first embodiment. The implementation procedure is as follows.
[Procedure 1] When the server apparatus 500 receives a content disclosure request from each of the user terminals 200a, 200b,..., An ID number is assigned as unique identification information. The emotion estimation unit F100 estimates the emotion and emotion level of the content from the audio information in the content, and stores this emotion and emotion level information in the HDD 304 in the information control unit 300 together with the content identification information. Furthermore, for the user who published the content, the identification information of the published content and the publication date / time are stored in the user preference information recording unit F200 for each user ID.
[Procedure 2] A distribution request for content that each user wants to view is transmitted to the server device 500 through each user terminal, and the server device 500 transmits the content to each corresponding user terminal 200a, 200b,. To deliver. At this time, the server device 500 stores, for each user ID, the distribution date and the identification information of the distributed content in the user preference information recording unit F200 as user preference information.
[Procedure 3] If 75% of all user terminals have released or viewed content at least once and have been in a state where 15 minutes or more have passed since the previous similarity calculation, user association The unit F300 executes association for a user (user ID) who has received the content distribution at least once.
[Procedure 4] As a result of similarity calculation, with respect to each user (user ID), the user information such as the user name and the similar users immediately before the top three users (user IDs) with high similarity The content information of the content that was viewed at the time is delivered and notified by the notification unit F400.
[Second Embodiment]
In this embodiment, the emotion estimation unit F100 extracts the emotion and emotion level of the content from the audio information and the video information in the content, obtains the user preference information based on this, and performs the association. FIG. 20 shows the configuration of a specific device of the second embodiment. The implementation procedure is as follows.
[Procedure 1] When the server apparatus 500 receives a content disclosure request from each of the user terminals 200a, 200b,..., An ID number is assigned as unique identification information. Further, the emotion estimation unit F100 estimates the emotion and emotion level of the content from the audio information and video information in the content, and sends the emotion and emotion level information together with the content identification information to the HDD 304 in the information control unit 300. Remember. Furthermore, for each user ID who has made the content public, the published content identification information, emotion and emotion level information, and release date are stored in the user preference information recording unit F200.

Subsequent [Procedure 2] to [Procedure 4] are exactly the same as those in the first embodiment.
[Third embodiment]
In this embodiment, the emotion estimation unit F100 extracts the emotion and emotion level of the content from the audio information and the video information in the content, and the activity information recognition unit F500 extracts the user activity information so that the user preference is obtained. This is a case where information is obtained and association is performed based on the information. FIG. 21 shows a specific apparatus configuration of the second embodiment. The implementation procedure is as follows.
[Procedure 1] to [Procedure 2] are exactly the same as in the second embodiment.
[Procedure 4] Immediately after content distribution to each user terminal, the server device 500 transmits an operation information acquisition request to each user terminal. Receiving this, each user terminal acquires the activity information of the user by the body sensor 250, and transmits this to the server device 500 and the activity information recognition unit F500. The activity information recognition unit F500 stores the user activity information in association with the content identification information for each user ID together with the information stored in [Procedure 3].
[Procedure 5] If 75% of all user terminals have watched the content at least once and have passed 15 minutes or more since the previous similarity calculation, the user association unit F300 The association is executed for the user (user ID) who has received the content once.
[Procedure 6] As a result of similarity calculation, for each user (user ID), the user information such as the user name and the similar users immediately before the top three users (user IDs) with high similarity The content information of the content that was viewed at the time is delivered and notified by the notification unit F400.

  Further, a part or all of the functions of each means in the user support apparatus of the present embodiment can be configured by a computer program, and the program can be executed using the computer to realize the present invention. It goes without saying that the procedure in the user support apparatus can be constituted by a computer program and the program can be executed by the computer, and the computer-readable recording medium, for example, FD, can be realized by the computer. (Floppy (registered trademark) Disk), MO (Magneto-Optical disk), ROM (Read Only Memory), memory card, CD (Compact Disk) -ROM, DVD (Digital Versatile) Disk) -ROM, CD-R, CD-RW, HDD, removable disk, etc., and can be stored and distributed. It is also possible to provide the above program through a network such as the Internet or electronic mail.

The flowchart explaining the flow of a process of the method in the 1st example and 2nd example of embodiment of this invention. The block diagram explaining the structure of the apparatus in the 1st example and 2nd example of embodiment of this invention. The block diagram explaining the structure of the apparatus of the user terminal in the 1st example and 2nd example of embodiment of this invention. The figure explaining the information which matched the identification information, the emotion, and the emotion level of the content in the embodiment of the present invention. The flowchart explaining the emotion detection method in embodiment of this invention. The flowchart which shows the processing detail of the step which comprises the statistical model in embodiment of this invention. The conceptual diagram of the method of calculating | requiring the time fluctuation characteristic of a fundamental frequency. The conceptual diagram which shows the time behavior of an audio | voice feature-value. The conceptual diagram of a generalized state space model. The figure which shows an example of the emotional state probability at the time of dealing with the emotional state of joy, sorrow, and calm. The conceptual diagram which shows the relationship between an audio | voice small paragraph and emotion degree. The figure explaining the user preference information in the embodiment of the present invention. The flowchart explaining step S300 in embodiment of this invention. The conceptual diagram which shows the relationship between an audio | voice paragraph and emotion degree. The flowchart explaining the flow of a process of the method in the 3rd example of embodiment of this invention. The block diagram explaining the structure of the apparatus in the 3rd example of embodiment of this invention. The block diagram explaining the structure of the apparatus of the user terminal in the 3rd example of embodiment of this invention. The figure explaining the user preference information including the activity information in the embodiment of the present invention. The flowchart explaining step S400B in embodiment of this invention. The block diagram which shows the specific structure of the user assistance apparatus 100 in the 1st and 2nd Example of this invention. The block diagram which shows the specific structure of the user assistance apparatus 100 in 3rd Example of this invention.

Explanation of symbols

DESCRIPTION OF SYMBOLS 100 ... User assistance apparatus 200, 200a, 200b ... User terminal 210 ... Input part 220 ... Control part 230 ... Storage part 240 ... Display part 300 ... Information control part 400 ... Database 500 ... Server apparatus F100 ... Emotion estimation part F200 ... User information Storage unit F300 ... user association unit F400 ... notification unit F500 ... activity information recognition unit

Claims (7)

A user support method for associating a plurality of users,
The estimation means extracts a feature amount from one or more of the audio information and video information of the content, and based on the extracted feature amount and a pre-configured statistical model, a plurality of emotions of the content An estimation step for estimating each emotion level,
A content attribute storage step in which content attribute storage means stores the identification information of the content in association with the emotion estimated for the content and the emotion level;
User information storage means identifies, for each user terminal, at least one of the content published by the user terminal, the content being viewed, and the content viewed, and the user terminal A user information storage step for associating information to be stored and storing this as user preference information;
Based on the user preference information, the user association means obtains, for each user terminal, a weighted average of emotion level as a preference value of the emotion category for each of a plurality of emotions for content associated with each user terminal. A user association step of associating one or more user terminals other than the first user terminal with the first user terminal based on the similarity of the obtained preference value of the user terminal;
A user support method comprising: The method of claim 1, wherein
The user information storing step includes, for each user terminal, information for identifying the content of at least one of the content published by the user terminal, the content being viewed, and the content viewed, and the user terminal Activity information consisting of at least one of operation information, audio information, and biometric information of a user who uses the user terminal when the content is viewed or viewed is acquired, and this is obtained as a user of the user terminal Remember it as preference information,
The user association step associates the user terminal based on a preference value of the user terminal and a similarity calculated from the activity information.   3. The method according to claim 1, wherein the user information storing step further stores a time change of the user preference information. A user support device that associates a plurality of users,
A feature amount is extracted from one or more of audio information and video information of the content, and each emotion level of the plurality of emotions of the content is extracted based on the extracted feature amount and a statistical model configured in advance. Estimating means for estimating
Content attribute storage means for storing the content identification information, the emotion estimated for the content, and the emotion level in association with each other;
For each user terminal, information for identifying the content and information for identifying the user terminal are associated with at least one of the content published by the user terminal, the content being viewed, and the content viewed. User information storage means for storing this as user preference information;
Based on the user preference information, for each of a plurality of emotions for content associated with each user terminal, a weighted average of emotion levels is obtained for each user terminal as a preference value of an emotion category, and the obtained user terminal User association means for associating one or more user terminals other than the first user terminal with the first user terminal based on the similarity of the preference values of
A user support apparatus comprising: The apparatus according to claim 4.
The user information storage means includes, for each user terminal, information for identifying the content of at least one of the content published by the user terminal, the content being viewed, and the content viewed, and the user terminal Activity information consisting of at least one of operation information, audio information, and biometric information of a user who uses the user terminal when the content is viewed or viewed is acquired, and this is obtained as a user of the user terminal Remember it as preference information,
The user association means associates a user terminal based on a preference value of the user terminal and a similarity calculated from the activity information.   6. The user support apparatus according to claim 4, wherein the user information storage unit further stores a time change of the user preference information.   A user support program, characterized in that it is a program for causing a computer to execute each step according to any one of claims 1 to 3.

Images