JP2008204193A - Content retrieval/recommendation method, content retrieval/recommendation device, and content retrieval/recommendation program - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide a content retrieval/recommendation device, capable of performing retrieval/recommendation of contents according to a user's emotion. <P>SOLUTION: The device comprises an emotion estimation part F100 which estimates emotion and degree of emotion for a multimedia content from voice signal data contained in the multimedia content; a content accumulation part F200 which accumulates contents with the emotion and degree of emotion estimated by the estimation part F100 as metadata; a retrieval request receiving part F300 which receives a retrieval request corresponding to the emotion or the emotion and the degree of emotion; a similarity calculation part F400 which calculates, based on the retrieval request, a similarity of the contents or partial contents; and a result presentation part F500 which presents, based on the similarity, a retrieval/recommendation result of the contents or partial contents. <P>COPYRIGHT: (C)2008,JPO&INPIT

Description

  The present invention estimates a feeling of content or a partial content thereof, and an emotion level indicating the strength of the emotion, and a content search / recommendation method, a content search / recommendation device for searching / recommending content based on this, and Related to content search and recommendation program. In this invention, the content refers to video / audio content, and the audio in the present invention includes not only human speech but also singing audio, music, environmental sound, and the like. In addition, emotions include emotions, emotions, and psychological states such as moods, as well as atmospheres and impressions.

  Currently, viewing of content is increasing not only in broadcasting but also on websites and personal PCs. The types of contents are also very diversified, such as movies, dramas, home videos, news, documentaries, music, and so on.

  As a result, there arises a problem that it becomes difficult for the user to efficiently find content that suits his taste from various contents. In particular, it is necessary to actually view and understand the content in order to confirm whether or not the content is suitable for one's preference, but the time cost to perform this is very large. For this reason, a content search technique and a content recommendation technique for automatically recommending content in accordance with the preference are indispensable. Furthermore, for the same reason, it is desirable that information related to content, such as metadata to be referred to at the time of search execution, is automatically given regardless of the manual operation.

  As a conventional technique related to content search, there is a method described in Patent Document 1 below. In Patent Document 1, physical quantities such as the size and color of a subject are classified into “large / medium / small”, “red / yellow / green”, and the like, and a search based on the words is executed. A method is disclosed.

Note that the fundamental frequency and power extraction methods related to the present invention are described in Non-Patent Document 1 below, the speech speed is described in Non-Patent Document 2 below, and the speech model parameter estimation method is described below. Documents 3 and 4 describe natural language processing in Non-Patent Documents 5 and 6 and Patent Document 2 below. Extraction of video feature values is described in Non-Patent Document 7 and Patent Documents 3, 4 and 5 below. The optical flow calculation method is described in Non-Patent Document 8 below.
Japanese Patent Laid-Open No. 5-282380 Japanese Patent Laid-Open No. 5-73317 Japanese Patent No. 3408117 JP 2005-157911 A Patent No. 3098276 “Digital Speech Processing, Chapter 4, 4.9 Pitch Extraction”, Sadaaki Furui, Tokai University Press, pp. 57-59, September 1985 “Personality information included in dynamic scale of speech”, Shigeki Hiyama, Fumitada Itakura, Proceedings of the Spring Society of Japan 1979 Spring Research Presentation, 3-2-7, pp. 589-590, 1979 “Easy-to-understand pattern recognition”, Kenichiro Ishii, Noriyoshi Ueda, Eisaku Maeda, Hiroshi Murase, Ohmsha, pp 52-54, 1998 “Computational Statistics I Chapter 3 3EM Method 4 Variation Bayes Method”, Nobuyoshi Ueda, Iwanami Shoten, pp. 157-186, June 2003 "Japanese Vocabulary University", supervised by NTT Communication Science Laboratories, Satoru Ikehara, Masahiro Miyazaki, Raku Shirai, Akio Yokoo, Hiroaki Nakaiwa, Kentaro Ogura, Yoshifumi Oyama, Yoshihiko Hayashi, Iwanami Shoten, 1997 "Basic Technology of Natural Language Processing", IEICE, Corona, March 1988 “Research on structured video handling mechanism and video interface based on video feature indexing, Chapter 3 Video indexing based on image processing”, Yoshinobu Tonomura, Ph.D. 15-23, 2006 "Computer image processing", edited by Hideyuki Tamura, Ohmsha, pp. 242-247, December 2002

  The conventional method realizes a similarity search based on the physical characteristics of video. However, since the physical characteristics and the user's preference are not directly connected, search / recommendation reflecting the user's preference cannot be executed.

  In addition, when viewing content, the preference of each user is important, but the emotion of the user at the time of viewing the video is also very important. This is because the content that the user wants to watch is not always the same, and this also changes dynamically according to the user's emotions. However, the conventional method cannot execute search / recommendation according to the user's emotion.

  The present invention has been made in view of the above points, and an object of the present invention is to provide a content search / recommendation method, a content search / recommendation device, and a content search / recommendation program that can search and recommend content according to the user's emotions. There is.

  The present invention has a function of automatically estimating the emotion and emotion level of the content by analyzing the content, and automatically assigning this to the content as metadata. By executing search / recommendation based on this metadata, content search / recommendation suitable for a search request reflecting emotions input from the user is realized.

  That is, in the content search / recommendation method according to claim 1 of the present invention, the emotion estimation means estimates the emotion and the emotion level of the content from the audio signal data included in the multimedia content, and the content accumulation A content storage step in which means stores content including the emotion estimated by the emotion estimation unit and the emotion level as metadata, and a search request reception unit corresponds to the emotion or the emotion and the emotion level. A search request accepting step for accepting a search request, a similarity calculating means for calculating a similarity of the content or partial content based on the search request, and a result presenting means based on the similarity A result presentation step for presenting search / recommendation results of content or partial content; It is characterized in that it comprises.

  In addition, the content search / recommendation device according to claim 7 includes: an emotion estimation unit that estimates an emotion and an emotion level of content from audio signal data included in multimedia content; and the emotion estimated by the emotion estimation unit. Content accumulation means for accumulating content having the emotion level as metadata, search request reception means for receiving a search request corresponding to the emotion or the emotion and the emotion level, based on the search request, the content or It includes a similarity calculation unit that calculates the similarity of partial content, and a result presentation unit that presents a search / recommendation result of the content or partial content based on the similarity.

  With the above configuration, by analyzing the audio signal data included in the content and extracting the emotion, it is possible to automatically generate metadata about the emotion and emotion level of the content and attach it to the content. The content can be searched and recommended according to the user's emotion based on the emotion and the emotion level.

  The content search / recommendation method according to claim 2 is the method according to claim 1, wherein the emotion estimation step includes, for each analysis frame, a fundamental frequency, a temporal variation characteristic of the fundamental frequency, an rms value of amplitude, and an amplitude. A feature amount extraction step for extracting at least one of a time variation characteristic of rms value, power, time variation characteristic of power, voice speed, and time variation characteristic of voice speed as a voice feature quantity, and learning voice signal data. According to one or more statistical models configured in advance, at least one of the appearance probability of the voice feature amount in the emotion and the transition probability in the time direction of one or more states corresponding to the emotion An emotion probability calculation step for calculating an emotion probability based on the emotion probability of the partial content including one or more analysis frames based on the emotion probability And emotions calculation step of calculating, based on said emotion probability is characterized in that it comprises, emotion determination step of determining the emotion of the partial content composed of one or more of the analysis frame.

  The content search / recommendation device according to claim 8 is the device according to claim 7, wherein the emotion estimation means includes, for each analysis frame, a fundamental frequency, a time variation characteristic of the fundamental frequency, an rms value of amplitude, and an amplitude. A feature amount extracting means for extracting at least one of a time variation characteristic of rms value, power, time variation characteristic of power, voice speed, time variation characteristic of voice speed as a voice feature quantity, and learning voice signal data. According to one or more statistical models configured in advance, at least one of the appearance probability of the voice feature amount in the emotion and the transition probability in the time direction of one or more states corresponding to the emotion An emotion probability calculating means for calculating an emotion probability based on the emotion probability, and a feeling for calculating the emotion degree of the partial content including one or more analysis frames based on the emotion probability. A degree calculating unit, based on the serial emotion probability, is characterized by comprising a determining emotion determination means the feeling of the partial content composed of one or more of the analysis frame.

  With the above configuration, voice features that are important for estimating emotions and emotion levels are extracted, and by performing probabilistic estimation, emotions can be stably and highly accurate regardless of the sound source factors of various contents. Emotion level can be estimated.

  The content search / recommendation method according to claim 3 includes an emotion estimation step in which the emotion estimation means estimates the emotion and the emotion level of the content from the audio signal data and the video signal data included in the multimedia content, and the content accumulation. A content storage step in which means stores content including the emotion estimated by the emotion estimation unit and the emotion level as metadata, and a search request reception unit corresponds to the emotion or the emotion and the emotion level. A search request accepting step for accepting a search request, a similarity calculating means for calculating a similarity of the content or partial content based on the search request, and a result presenting means based on the similarity Result presentation step for presenting search / recommendation results for content or partial content It is characterized in that it comprises a.

  The content search / recommendation device according to claim 9 is estimated by the emotion estimation means for estimating the emotion and the emotion level of the content from the audio signal data and the video signal data included in the multimedia content, and the emotion estimation means. Based on the search request, a content storage unit that stores content including the emotion and the emotion level as metadata, a search request reception unit that receives a search request corresponding to the emotion or the emotion and the emotion level, and And a similarity calculation means for calculating the similarity of the content or the partial content, and a result presentation means for presenting a search / recommendation result of the content or the partial content based on the similarity.

  With the configuration described above, it is possible to further improve the estimation accuracy of the emotion and the emotion level of the content by analyzing the video signal data in addition to the audio signal data included in the content.

The content search / recommendation method according to claim 4 is the method according to claim 3, wherein the emotion estimation step includes, for each analysis frame, a fundamental frequency, a temporal variation characteristic of the fundamental frequency, an rms value of amplitude, and an amplitude. Rms value time variation characteristic, power, time variation characteristic of power, audio speed, time variation characteristic of audio speed, and video signal data, shot length, color histogram, time variation characteristic of color histogram, A feature extraction step for extracting at least one of the motion vectors as a video feature, one or more statistical models pre-configured using the learning audio signal data, and pre-configuration using the learning video signal data One or more statistical models, and the appearance probability of the voice feature quantity in the emotion and the time of one or more states corresponding to the emotion An emotion probability calculation step for calculating an emotion probability based on at least one of the transition probabilities toward the direction, and the emotion level of the partial content including one or more analysis frames based on the emotion probability An emotion level calculating step for calculating the emotion level, and an emotion determination step for determining the emotion of the partial content configured by one or more analysis frames based on the emotion probability,
It is characterized by including.

  The content search / recommendation device according to claim 10 is the device according to claim 9, wherein the emotion estimation means includes, for each analysis frame, a fundamental frequency, a time variation characteristic of the fundamental frequency, an rms value of amplitude, and an amplitude. Rms value time variation characteristic, power, time variation characteristic of power, audio speed, time variation characteristic of audio speed, and video signal data, shot length, color histogram, time variation characteristic of color histogram, Feature amount extracting means for extracting at least one motion vector as a video feature amount, one or more statistical models preconfigured using learning audio signal data, and preconfigured using learning video signal data The probability of appearance of the voice feature quantity in the emotion and the time direction of one or more states corresponding to the emotion by one or more statistical models. Emotion probability calculation means for calculating an emotion probability based on at least one of transition probabilities, and calculating the emotion level of the partial content including one or more analysis frames based on the emotion probability It includes an emotion level calculation means and an emotion determination means for determining the emotion of the partial content constituted by one or more analysis frames based on the emotion probability.

  With the above configuration, voice features and video features that are important in estimating emotions and emotion levels are extracted, and further probabilistic estimation is performed, regardless of the sound source factors and imaging conditions of various contents. Emotion and emotion level can be estimated stably and accurately.

  The content search / recommendation method according to claim 5 is the method according to any one of claims 1 to 4, wherein the search request reception step is a content that the user is viewing and / or viewing. Alternatively, the search request determined based on the emotion of the partial content or the emotion and the emotion level is received.

  The content search / recommendation device according to claim 11 is the device according to any one of claims 7 to 10, wherein the search request reception unit is a content that the user is watching and / or watching. Alternatively, the search request determined based on the emotion of the partial content or the emotion and the emotion level is received.

  With the above configuration, it is possible to search / recommend content with emotions and emotional levels that match the user's preference without requesting the user to make a search request based on the content that the user is currently viewing or viewed in the past. It becomes possible.

  The content search / recommendation method according to claim 6 is the method according to any one of claims 1 to 5, wherein the result presentation step ranks content or partial content based on the similarity, Based on the ranking result, at least one of the attribute information of the content or partial content, the emotion, the emotion level, the thumbnail, and the summary content is listed and presented.

  The content search / recommendation device according to claim 12 is the device according to any one of claims 7 to 11, wherein the result presentation unit ranks content or partial content based on the similarity, and the ranking is performed. Based on the result, at least one of the attribute information of the content or partial content, the emotion, the emotion level, the thumbnail, and the summary content is listed and presented.

  In addition to the conventional search / recommendation method, such as presenting in the order of similarity according to the above configuration, results such as presenting the content title, content attribute information, emotion, emotion level, thumbnail, and partial content as summary content By displaying together, it is possible to promote understanding of the content of the user, and it is possible to search and recommend content that matches the user's request.

  The content search / recommendation program according to claim 13 is a program for causing a computer to execute each step of the content search / recommendation method according to any one of claims 1 to 6. .

  With the above configuration, the method according to the present invention can be executed by a computer.

(1) According to the first and seventh aspects of the invention, by analyzing the audio signal data included in the content and extracting the emotion, the metadata about the emotion and the emotion level of the content is automatically generated. The content can be given to the content, and the content can be searched and recommended according to the emotion of the user based on the emotion and the emotion level of the content.
(2) According to the second and eighth aspects of the present invention, the voice feature quantity that is important in estimating the emotion and the emotion level is extracted, and further, the probabilistic estimation is performed, so that sound sources of various contents can be obtained. Regardless of factors, emotions and emotion levels can be estimated stably and accurately.
(3) According to the third and ninth aspects of the invention, the accuracy of estimation of emotion and emotion level of content can be further improved by analyzing video signal data in addition to audio signal data included in the content. Can do.
(4) According to the inventions described in claims 4 and 10, various voice features and video feature values that are important in estimating emotions and emotion levels are extracted, and further probabilistic estimation is performed. It is possible to estimate emotion and emotion level more stably and accurately regardless of the sound source factor of the content and the imaging situation.
(5) According to the inventions described in claims 5 and 11, the user's preference is met without requesting the user for a search request based on the content that the user is currently viewing or viewed in the past. Search / recommend content with emotion and emotion level.
(6) According to the inventions of claims 6 and 12, in addition to the conventional search / recommendation method such as presenting in order of similarity, the content title, content attribute information, emotion, emotion level, thumbnail By displaying the result of displaying partial content as summary content, etc., it is possible to promote understanding of the content of the user, and it is possible to search and recommend content that matches the user's request. .
(7) According to the invention described in claim 13, the content search / recommendation method according to the present invention can be executed by a computer.

Hereinafter, embodiments of the present invention will be described with reference to the drawings, but the present invention is not limited to the following embodiments.
[First Example of Embodiment: Content Search / Recommendation Using Only Audio Signal Data]
The first example of the embodiment of the present invention is a case where the emotion and the emotion level are estimated using only the audio signal data among the information included in the content. This embodiment will be described with reference to FIGS.

  A content search / recommendation method and a content search / recommendation apparatus according to an embodiment of the present invention will be described. FIG. 1 is a flowchart illustrating a processing flow of a content search / recommendation method according to a first example of an embodiment of the present invention, and FIG. 2 is a content search / recommendation apparatus according to a first example of an embodiment of the present invention. FIG.

  In the content search / recommendation apparatus 100 in this embodiment, a search request input from the user terminal 200 is transmitted to the information control unit 300 by a predetermined communication means, and the information control unit 300 is similar to the search request. Alternatively, the partial content is searched from the content stored in the database 400 or the partial content, and the search / recommendation result is presented to the user terminal 200 by a predetermined communication means.

  A block diagram illustrating the configuration of the user terminal 200 is shown in FIG. The user terminal 200 includes, for example, a keyboard 211, an input unit 210 including a pointing device 212 typified by a mouse, a CPU (Central Processing Unit) 221, a ROM (Read Only Memory) 222, and a RAM (Random Access Memory) 223. A control unit 220 including a storage unit 230 including a HDD (Hard Disc Drive) 231 and a monitor screen 241 such as a liquid crystal screen. Information output from the control unit 220 according to an operation of the input unit 210 It is assumed that a display unit 240 for displaying is provided.

  The information control unit 300 is configured by connecting a CPU 301, a ROM 302, a RAM 303, an HDD 304, and the like. The various processes in the present invention are all performed by the information control unit 300, and all programs and data for realizing the various processes are stored in a storage device such as the ROM 302 and the HDD 304, and appropriately read out to the RAM 303. Processing is executed in the CPU 301.

  Hereinafter, the flow of processing will be described for each function unit provided in the information control unit 300 and the database 400.

  FIG. 4 is a block diagram illustrating the configuration of the emotion estimation unit F100 that is the emotion estimation means of the present invention. The emotion estimation unit F100 calculates, from the audio signal data included in the content, the fundamental frequency, the time variation characteristic of the fundamental frequency, and the rms value of the amplitude (root mean square; square root of the root mean square; area average of the oscillating sine wave) Value), time variation characteristic of rms value of amplitude, power, time variation characteristic of power, voice speed, time variation characteristic of voice speed, and a voice feature quantity extraction unit F101 (voice feature quantity) And an emotion probability calculation unit F102 (emotion) that calculates an emotion probability as a probability of occurrence of a voice feature amount by a voice model, which is one or more statistical models preliminarily configured using the voice signal data for learning) A probability calculation means) and an emotion level for calculating the emotion level of a partial content including one or more analysis frames based on the emotion probability A calculation unit F 103 (emotional calculation means), based on the emotion probability, constituted by determining emotion determination unit emotions composed partial contents by one or more analysis frames F 104 (emotion determination means). The emotion level calculation unit F103 and the emotion determination unit F104 further estimate the content emotion and emotion level based on the emotion probability, the partial content emotion level, or the partial content emotion and emotion level, respectively.

Step S100 executed by the emotion estimation unit F100 is a step performed in advance according to the present invention before actually searching or recommending content or partial content, and estimates the emotion and emotion level of the content and partial content. It is a step to do. FIG. 5 shows a flowchart for explaining the flow of processing in step S100. Step S110 is a process for constructing a statistical model for calculating an emotion probability necessary for obtaining the emotion and the emotion level, and Step S120 and Step S130 are for calculating the emotion probability of the content and the partial content. It is processing of. Step S140 is processing for estimating the emotion and the emotion level of the content and the partial content.
First, as in an example of a procedure to be described later, in step S110, a speech model for calculating an emotion probability is acquired in advance based on learning speech signal data.

  In step S120, the audio feature amount extraction unit F101 calculates and extracts a desired audio feature amount for each analysis frame (hereinafter referred to as a frame) from the acquired audio signal data of the content. This voice feature amount is one of fundamental frequency, time variation characteristic of fundamental frequency, rms value of amplitude, time variation characteristic of amplitude rms value, power, time variation characteristic of power, voice speed, and time variation characteristic of voice speed. Consists of two or more elements.

  In step S130, the emotion probability calculation unit F102 has previously acquired, in step S110, the probability that the voice feature amount appears in the emotion of the content or partial content for each frame based on the voice feature amount calculated in step S120. The emotion probability is calculated by calculating with a speech model.

  In step S140, based on the emotion probabilities for each frame of the content and partial content calculated in step S130, the emotion level calculation unit F103 and the emotion determination unit F104 estimate the emotion and emotion level of the content and partial content, respectively.

  Details of each step will be described below.

  First, in step S120, a desired audio feature amount is extracted for each frame from the audio signal data of the captured content.

  Hereinafter, an example of the speech feature extraction method will be described.

  Here, each voice feature amount will be described. In estimating emotions of content and partial content, the audio feature value is obtained stably for speech mixed with various sound source factors, compared to phonological information that requires analysis of high-dimensional speech parameters. It is preferable that the content genre is less dependent on the content attribute.

  For example, a method of converting speech into text information using speech recognition or the like requires such phonological information, and is effective for content of a genre in which a speech of a speaker such as a news video can be heard clearly. It is. However, in movies, dramas, home videos, etc., there are various sound source factors such as background music and environmental sounds in addition to utterances, so the utterances cannot be heard clearly and speech recognition is difficult. . Furthermore, the emotion of the content is not always determined only by utterances. For the purpose of estimating emotions including impressions and atmosphere, audio features that can be treated as important factors such as music, sound effects, and environmental sounds is required.

  In order to solve such a problem, in the first example of the embodiment of the present invention, the prosody information, in particular, the fundamental frequency, the time variation characteristic of the fundamental frequency, the rms value of the amplitude (hereinafter simply referred to as rms), and the time of rms. The fluctuation characteristics, power, time fluctuation characteristics of power, voice speed, voice speed time fluctuation characteristics, and the like are extracted. In particular, by using several types of short-time change amounts as the time variation characteristics, it is possible to detect important behaviors in emotional speech when emotions included in content are extracted.

  Examples of time variation characteristics include inter-frame differences and regression coefficients. The power may be a power spectral density. There are various methods for extracting the fundamental frequency and power, but they are known. For details, see the method described in Non-Patent Document 1, for example.

  The voice speed including speech speed, music rhythm, tempo, and the like can be extracted as a dynamic scale by a method disclosed in Non-Patent Document 2, for example. For example, it is possible to detect the voice speed by detecting the peak of the dynamic scale and counting the number, and calculating the average value and variance value of the peak interval corresponding to the time fluctuation characteristic of the voice speed. Thus, a method of detecting the time variation characteristic of the voice speed may be adopted. Hereinafter, in the first example of the embodiment of the present invention, a peak interval average value of a dynamic scale is used as the voice speed.

  An example of a method for extracting these audio feature amounts for each frame will be described. The length of one frame (hereinafter referred to as the frame length) is, for example, 50 ms, and the next frame is formed by a time shift of, for example, 20 ms with respect to the current frame. As shown in FIG. 6, for each of these frames, the average value of each voice feature amount in each frame, that is, the average fundamental frequency, the average frequency variation characteristic of the fundamental frequency, the average rms, the average time variation characteristic of rms, It is assumed that average power, average time variation characteristic of power, average peak interval average value of dynamic scale, and the like are calculated. Alternatively, not only the average value but also the maximum value, the minimum value, or the fluctuation range of each voice feature amount in the frame may be calculated and used.

  Here, in the sound that appears characteristically in the emotional part of the content, it is often difficult to extract the fundamental frequency itself, which is often lost. For this reason, it is preferable to include a time-variation characteristic of the fundamental frequency that can easily obtain the effect of complementing such a defect.

  Furthermore, it is preferable to further include a time variation characteristic of power in order to improve the determination accuracy while keeping speaker dependency low. As described above, the audio feature amount extraction processing performed for each frame is performed on the frames over the entire content, so that the audio feature amounts can be obtained in all the frames.

  In step S130, using the audio feature value of each frame extracted in step S120 and one or more audio models configured in advance in step S110, the audio feature value in the emotion of each content and partial content is determined. Emotion probability is calculated.

  Here, first, an example of the process of step S110 for configuring a speech model will be described. The speech model is acquired by performing learning from the speech signal data for learning. As in the case of the content audio signal data, the audio feature amount is extracted in units of frames in the learning audio signal data, and further, for example, “fun”, “sad”, “scary”, “furious”, “ Emotions such as “cool”, “cute”, “exciting”, “passionate”, “romantic”, “violent”, “gentle”, “healed”, “warm”, “cold”, “creepy” It is assumed that a label of a type determined as a category is given.

Here, each emotion category is expressed in order as e ¹ , e ² ,..., And the number of emotion categories is expressed as # (K). By associating these emotion categories with the speech model, a speech model for calculating the emotion probability for each emotion category is acquired. As this speech model, for example, a normal distribution, a mixed normal distribution, a hidden Markov model, a generalized state space model, or the like is used. Preferably, a time series model such as a hidden Markov model or a generalized state space model that can model the temporal transition of emotion is employed.

  As a method for estimating the parameters of these speech models, for example, a maximum likelihood estimation method, an EM algorithm, a variational Bayes method, and the like are known and can be used. For details, refer to Non-Patent Document 3, Non-Patent Document 4, and the like.

  In step S130, the voice feature amount for each frame of the content to be analyzed extracted in step S120 is input to the voice model acquired in step S110, thereby calculating the emotion probability for each frame. In step S110, since the speech model is constructed so that the probability can be calculated for each emotion category, the emotion probability that is the probability of each emotion category is calculated by inputting the speech feature amount to each speech model. be able to.

  In step S140, the emotion and the emotion level of the content and the partial content are estimated based on the emotion probability calculated in step S130.

  Hereinafter, an example of the process of step S140 will be described. FIG. 7 is a flowchart for explaining the flow of processing in step S140.

  First, in step S141, partial content composed of one or more frames is generated.

  In the first example of the embodiment of the present invention, a process of collecting a set of speech sections considered to be continuous speech as one section is performed. Hereinafter, the audio section composed of the continuous audio is regarded as partial content. As a method of generating a speech section, for example, there is a method of using a periodicity of continuous speech sections in a speech waveform as a section boundary at a point where the change of the autocorrelation function exceeds a certain value.

  Next, in step S142, the emotion level for each configured partial content is calculated based on the emotion probability calculated for each emotion category in step S130.

Hereinafter, an example of a method for calculating the emotion level will be described. Assume that a set of partial contents S in the content is {S ₁ , S ₂ ,..., S _NS } in order from the earliest time. Here, NS is the total number of partial contents. Also, frames included in a certain partial content S _i are set as {f ₁ , f ₂ ,..., F _NFi }. Here, NFi is the number of frames included in the partial content S _i . Each frame f _t in step S130, the emotion probability pf _t of k-th emotional category e ^k in units of frames (e ^k) is given. Using this, the emotion level pS _i (e ^k ) of the partial content S _i of the ^kth emotion category ek represents, for example, an average value.

  Or the following formula representing the maximum value.

  By performing the above calculation over all partial contents, it is possible to calculate the emotion level for each emotion category for all partial contents (FIG. 8).

  For the emotion of the partial content, for example, the emotion category having the maximum emotion level is set as the emotion of the partial content Si.

  Further, the emotion of the partial content may be estimated based on the transition of the emotion for each frame.

  Hereinafter, an example of a method for calculating the emotion level by this method will be described.

A section including one or more predetermined number L of continuous frames included in the partial content is called a small partial content. At this time, emotions of the respective frames included in this section are expressed as ef ₁ , ef ₂ ,..., Ef _L. For this emotion, using the emotion probability calculated for each frame, for example, the emotion category having the maximum value can be set as the emotion of each frame.

The emotion transition sequence {ef ₁ , ef ₂ ,..., Ef _L } of one or more consecutive frames is modeled by one or more speech models associated with the emotion category. As an example of the speech model, an N-gram, a hidden Markov model, a generalized state space model, or the like can be used. This speech model is acquired by performing learning using the speech signal data for learning. In this case, the learning speech signal data used in step S110 can be used in the same manner as the learning data. Preferably, when this method is employed, the construction of these speech models is also executed in step S110.

  By this method, it is possible to newly calculate the emotion probability in units of small partial contents in the partial contents. The calculation of the emotion level can be performed, for example, by the following procedure.

Small content included in the partial content S _i is set as {S ₁ , S ₂ ,..., S _NSi }. Here, Nsi is a small number of partial contents contained in the partial content S _i. Each small partial content S _t, in step S130, the emotion probability of k-th emotional category e ^k in units of frames ps _t (e ^k) is given. Using this, the emotion level pS _i (e ^k ) of the partial content S _i of the ^kth emotion category ek represents, for example, an average value.

  Or the following formula representing the maximum value.

  In addition to these, there are various methods such as calculating the emotion level after performing filtering within the partial content, but the emotion level may be compared between the partial content, so the emotion level is a certain level It is preferable to be within the range of values, for example, between 0 and 1.

  Next, in step S143, the emotion and emotion level of the content are estimated.

  As an example of this method, for example, by calculating the emotion probability for each frame or the average value or maximum value of the emotion level of all partial contents over the entire content, It is good. As another example, the emotion category having the highest emotion level may be used as the content emotion. More simply, the duration of the partial content determined as each emotion may be calculated, and the emotion category having the longest duration may be determined as the emotion of the content. In this case, only the emotion of the partial content is referred to It is possible to determine the emotion of the content.

  In addition, in the content, how the emotion level changes through the content as shown in FIG. 9 by storing the time at which each partial content exists, for example, the start time, the end time, the central time, etc. It is also possible to obtain time-series information regarding.

  Using this time series information, the emotion category that appears most frequently around a specific time of the content may be used as the emotion of the entire content. For example, in the case of content such as movies and dramas, there are many important scenes that remain in the emotion near the end time of the content, but in such cases, the most frequently appears near the end time. It is possible to use the emotion category as the emotion of the content. At this time, the emotion level near the end time may be referred to, and the emotion category having the highest emotion level may be used as the content emotion.

  The content storage unit F200 (content storage unit) in FIG. 2 stores the content or partial content in the database 400 in association with the emotion and emotion level estimated by the emotion estimation unit F100. The database 400 includes a predetermined storage device and content or partial content data. For example, in the case of a relatively small usage range such as an individual or a home, the predetermined storage device is connected to the user terminal by an HDD in the user terminal, an HDD in the HDD recorder, or a LAN (Local Area Network). It may be an HDD in a predetermined server device. Further, it may be constituted by an external storage device that can be taken out such as a DVD. In the case of a large usage range, a storage device with a predetermined server device connected to the user terminal by a wide area communication network such as the Internet may be used.

  In step S200 of FIG. 1, the content storage unit F200 stores in the database 400 content or partial content including emotion and emotion level, which is content attribute information calculated through step S100 of FIG.

  The search request receiving unit F300 (search request receiving means) in FIG. 2 receives a search request input from the user terminal 200 via a predetermined communication means. Hereinafter, step S300, which is a process in which the user inputs a search request using the user terminal 200 and the search request reception unit F300 receives the search request, will be described. FIG. 10 is a diagram illustrating step S300.

  In step S300A, the user selects the emotion and emotion level of the desired content and inputs this as a search request. This search request may be, for example, an input of emotion and emotion level directly from the keyboard 211 of FIG. 3, or a selection from a pull-down menu displayed on the monitor 241.

  As an example of a method for directly inputting an emotion and an emotion level, an emotion word corresponding to each category and an expression representing its strength are input. For example, the user inputs “emotion: fun, emotion level: 0.9” to make “emotion: fun, emotion level: 0.9” a search request.

  If an emotion word that is directly equal to the emotion category is not input, it can be considered that the emotion word corresponds to the consensus emotion category, and is appropriately converted and processed. For example, even if the user inputs “laughter” and “laughter” is not prepared in the emotion category, for example, an emotion category such as “fun” or “interesting” that is inferred to be consensus is prepared. For example, the processing can be executed assuming that the emotion category belongs.

  At this time, emotion words that are relatively difficult to be attributed to one emotion category may be associated with a plurality of emotion categories. For example, when the user inputs “emotion: creepy, emotion level: 0.6”, even if “eerie” is not prepared in the emotion category, for example, this is expressed as “emotion: scary, emotion level: 0”. .3 AND Emotion: Mysterious, Emotional Level: 0.3 ”can be executed.

  Further, the user input may be a natural sentence. In this case, an emotion word may be extracted from a natural sentence by a method such as morphological analysis and converted into a corresponding emotion and emotion level. For example, when the user inputs “super funny”, it can be converted into “super” → emotion level: 0.9, “interesting” → emotion: fun, and so on.

  Such a conversion can be executed based on a systematic dictionary as found in Non-Patent Document 5 or the like, a method shown in Non-Patent Document 6, or the like. In addition, regarding “super” → emotion level: 0.9 or the like, the method of Patent Document 2 can be used for what can be interpreted as a relative expression. For such a method, a predetermined conversion rule may be designed in advance by a designer, or may be constructed using learning in order to take into account the actual subjective feeling of the user.

  In addition, it may be difficult for the user to express emotional expressions such as emotions and emotion levels accurately as text information. In such a case, it is possible to input a search request through an intuitive and easy-to-understand graphical interface. For example, as shown in FIG. 11, a graph centered on the emotion level of each emotion may be displayed on the monitor 241, and the user may select a preference value using the pointing device 212, and this may be input as a search request. Then, an equalizer-type interface that is a function installed in a music player or the like as shown in FIG. 12 is displayed on the monitor S241, and the user adjusts and inputs the emotion level of each emotion by operating the pointing device 212. You may do it.

  In addition, for these methods, it is not necessary to input the emotion level of all emotion categories, and the emotion level may be selected only for the emotion category desired to be requested. More simply, only the emotion desired by the user may be selected.

  In step 300B, the search request is determined based on the emotion and the emotion level of the content that the user is viewing and viewing. As a determination method, for example, the emotion and the emotion level of the content viewed by the user may be regarded as a direct search request.

  Through this step 300B, even if the user does not directly input a search request, a similarity search can be executed by using the emotion and emotion level of the content being viewed by the user as a search request. There is an advantage.

  At this time, the viewing history of the content may be recorded for each user, and processing for correcting the search request may be performed based on the emotion and emotion level information of the content viewed in the past. By adding this processing, it is possible to search / recommend content that has similar emotions and emotions based on the history of the content that is being viewed or viewed, and adaptively adapt the preferences for each user. It is possible to present the reflected search / recommendation results.

  For example, content that has been viewed in the past may be stored including the viewing time and the like, and for example, a search request may be determined based on the past several hours or the emotion level of several content. For example, a weighted average value or maximum value may be calculated for the emotion level of the viewed content, and this may be regarded as a search request. As a method for determining the weight for the weighted average value, for example, a forgetting model that incorporates psychological knowledge such as increasing the weight of recently watched content or the forgetting curve of Ebbinghouse is introduced. May be determined in association with the viewing time of past contents.

That is, for example, the number of contents viewed by a certain user in the past several hours is set as NK, and the emotion level of the ^kth emotion category ek of each content is pS ¹ (e ^k ), pS ² (e ^k ). ,..., PS ^NK (e ^k ).

At this time, the search request Q (e ^k ) for the ^kth emotion category e ^k is, for example,

  Can be calculated by Here, wj is a weight and may be determined by the above-described weight determination method.

  Further, by the same method, the number of viewings in the past several hours may be recorded for each content, and processing for correcting the search request based on this may be performed. In this case, for example, with respect to the search request, a weighted average value with the emotions and emotional degrees of the top several frequently watched contents, for example, 10 is calculated, and this is calculated as the final search request. It may be. By adding this processing, it is possible to present a search / recommendation result that reflects the emotion and emotion level of content that is often viewed recently.

  As another method, it is also possible to analyze the viewing order pattern of emotion and emotion level of content viewed in the past several hours and correct the search request based on the information. For example, as shown in FIG. 14, a certain user views content in a sequence of “fun” emotional content, “sad” emotional content, and “fun” emotional content along the time axis. If you are a frequent user, this user prefers, or feels like, to watch “fun” and “sad” alternately, rather than continue to watch “fun” content. This pattern may be reflected in the search request by assuming that the user is a certain user.

  That is, for example, if the user is currently watching “fun” content, recommend “sad” content with a high emotional level, and if the user is currently watching “sad” content, recommend “fun” content. So that the search request is corrected.

  Further, as shown in FIG. 9, when the emotion and the emotion level are calculated for each partial content and the temporal change of the emotion and the emotion level can be acquired, the search request input as shown in FIG. 13 is associated with this. The interface may be presented to the user so that the emotional change over time can be freely drawn for each emotion, and this may be used as a search input.

  According to the principle of the present invention, it is possible to realize search / recommendation by executing at least one of steps S300A and S300B. However, a search request is determined and input by combining both. You can also

  In this case, for example, an AND or OR of the search requests obtained in steps S300A and S300B may be used as the final search request. Further, when the emotion level is included in the search request, this weighted average value may be calculated and used as the final search request.

  The search request receiving unit F300 (search request receiving means) in FIG. 2 receives a search request input through the above steps.

  The similarity calculation unit F400 (similarity calculation means) in FIG. 2 calculates the similarity between the search request received by the search request reception unit F300 and the emotions and emotions of the content or partial content stored in the database 400. calculate. Hereinafter, step S400 of FIG. 1 that is a process executed by the similarity calculation unit F400 will be described.

  In step S400, the similarity is calculated by comparing the search request input in step S300 with the emotions and emotions given to the contents in the database stored in advance in step S200.

Here the search request, indicating k th emotions of the emotion category e ^k r (e ^k), the k-th granted to the content or partial content emotional level of emotional category e ^k and p (e ^k). The similarity fs is calculated by comparing the emotion and emotion level input as a search request with the emotion and emotion level given to the content or partial content. For example, the index set of the emotion category selected by the user is K, If the number is # (K),

Can be calculated by:

  Also, if only emotion is selected as the user ’s search request and the emotion level is not entered,

  Can also be calculated. Through the processing as described above, the similarity to the search request can be calculated for each content or partial content.

  The result presentation unit F500 (result presentation unit) in FIG. 2 generates a search result based on the similarity calculated by the similarity calculation unit F400, and transmits the result to the user terminal 200. The user terminal 200 receives this result and presents it on the monitor 241. Hereinafter, step S500 of FIG. 1 which is a processing procedure executed by the result presentation unit F500 will be described.

  In step S500, based on the similarity calculated in step S400, a search result is generated and presented to the user. As a presentation method, at least one of content attribute information, emotion and emotion level, thumbnail, summary, etc. is listed and presented in descending order of similarity of each content.

  As the attribute information, content title, producer, keyword, outline, creation date / time, format, URL or path where the content exists, related content attribute information, and the like can be considered. For example, it can be given in accordance with the description format according to eExtensible Markup Language).

  As for emotion and emotion level, those calculated in step S100 of the present invention are presented on the monitor 241. As a form of this presentation, for example, one that displays emotion and emotion level in text format, one that presents time-series information regarding how the emotion level changes through content, as shown in FIG. 11 and the like, which are presented using a graph with the emotion level of each emotion as an axis.

  As for the summary, if it is a partial content, it may be presented, and if it is a content, one or more partial contents in the descending order of similarity to the search request among the partial contents included in the content may be presented. It may be extracted and presented as a summary. In any case, the summary can be reproduced and viewed by an operation such as the user pointing the area on the monitor 241 where the summary is presented, using the pointing device 212 or the like.

  In addition, since the emotion level is estimated for each partial content, when the user actually views the content, it is possible to edit and reproduce the content so as to suppress the portion that the user does not want to view. For example, when the user does not want to watch the “scary” emotion part, the part that is estimated to be the “scary” emotion in the partial content is notified by, for example, performing countdown in advance. It is possible to edit the video such as mosaicing and darkening, and reducing the volume.

  In addition, with respect to restricted content designated by R or PG, partial content including emotions that are the cause of restriction may be edited and viewed in the same manner, or the content itself may not be reproduced. It becomes possible. By making it possible to change the application / non-application of this editing for each user, for example, it is possible to make settings such as not automatically presenting content that a child does not want to view.

  As for thumbnails, there is a method of extracting and presenting as a still image a predetermined time portion of the above-mentioned summary, for example, the top image, the center image, etc. of the summary video. Alternatively, a predetermined number of emotion probabilities may be extracted in descending order from the frame having the highest emotion probability through content or partial content.

The content search / recommendation method according to the present invention has been described in detail above as an example of the method in the embodiment.
[Second Example of Embodiment: Content Search / Recommendation Using Audio Signal Data and Video Signal Data]
The second example of the embodiment of the present invention is a case where the emotion and emotion level of content are estimated using video signal data in addition to audio signal data. As long as the flow of processing and the specific configuration of the apparatus according to the second example of the embodiment of the present invention are shown in the flowchart of FIG. 1 and the block diagram of FIG. It may be the same as one example.

  The difference from the first example of the embodiment is that the emotion estimation unit F100 calculates the emotion probability using not only the audio signal data but also the video signal data. Hereinafter, step S100 executed by the emotion estimation unit F100 of the second example of the present embodiment will be described. The subsequent processing flow and the specific configuration of the apparatus may all be the same as in the first example of the embodiment.

  The structure of the emotion estimation part F100 is demonstrated using FIG. The emotion estimation unit F100 determines, for each analysis frame, the fundamental frequency, the temporal frequency variation characteristic of the fundamental frequency, the rms value of the amplitude, the temporal variation characteristic of the rms value of the amplitude, power, and the temporal variation of the power from the audio signal data included in the content. A voice feature amount extraction unit F101A that extracts at least one of a characteristic, a voice speed, and a time variation characteristic of the voice speed as a voice feature amount, and a shot length, a color histogram, and a color histogram time for each analysis frame from the video signal data. A statistical model pre-configured using a video feature quantity extraction unit F101B that extracts at least one of a variation characteristic and a motion vector as a video feature quantity, and learning audio signal data and learning video signal data. Emotion probability calculation that calculates emotion probability as the probability of appearance of feature amount by the above audio model and video model F102, an emotion level calculation unit F103 that calculates an emotion level of a partial content including one or more analysis frames based on the emotion probability, and a partial content configured by one or more analysis frames based on the emotion probability It is comprised by the emotion determination part F104 which determines the emotion of. The emotion level calculation unit F103 and the emotion determination unit F104 further estimate the content emotion and emotion level based on the emotion probability, the partial content emotion level, or the partial content emotion and emotion level, respectively.

  Step S100 executed by the emotion estimation unit F100 is performed in advance before the content or partial content is actually searched / recommended according to the present invention.

  FIG. 16 is a flowchart illustrating the process flow of step S100. Step S110 is a process for constructing an audio model and a video model for calculating emotion probabilities necessary for obtaining emotions and emotion levels, and S120 to S140 calculate emotion probabilities for content and partial content. Process. S150 is processing for estimating the emotion and the emotion level of the content and the partial content.

  First, as an example of a procedure to be described later, in step S110, one or more audio models and video models for calculating emotion probabilities in advance based on learning audio signal data and learning video signal data are obtained. Earn it.

  In step S120, the audio feature amount extraction unit F101A calculates and extracts a desired audio feature amount for each analysis frame (hereinafter referred to as an audio frame) from the audio signal data of the captured content. The video feature quantity extraction unit F101B calculates and extracts a desired video feature quantity for each analysis frame (hereinafter referred to as a video frame) from the video signal data of the captured content. The voice feature amount is one of the basic frequency, the time variation characteristic of the fundamental frequency, the rms value of the amplitude, the time variation characteristic of the rms value of the amplitude, the power, the time variation characteristic of the power, the voice speed, and the time fluctuation characteristic of the voice speed. The one or more elements and the video feature amount are composed of one or more elements among a shot length, a color histogram, a time variation characteristic of the color histogram, and a motion vector.

  In step S130, the emotion probability calculation unit F102, based on the audio feature amount and the video feature amount calculated in step S120, in the emotion of the content and the partial content, for each audio frame and video frame, the audio feature amount and the video feature, respectively. The voice emotion probability and the video emotion probability are obtained as the probability that the quantity appears. At this time, the audio model and video model acquired in advance in step S110 are used.

  In step S140, based on the voice emotion probability calculated for each audio frame in step S130 and the video emotion probability calculated for each video frame, an emotion probability for each frame in which the audio frame and the video frame are shared is obtained.

  In step S150, based on the emotion probabilities for each frame of the content and partial content calculated in step S140, the emotion level calculation unit F103 and the emotion determination unit F104 estimate the emotion and emotion level of the content and partial content, respectively.

  Details of each step will be described below.

  First, in step S120 in FIG. 16, desired audio feature amounts and video feature amounts are extracted for each frame from the audio signal data and video signal data of the captured content. Since the extraction of the audio feature amount is the same as that of the first example of the embodiment, an example of the video feature amount extraction method will be described below.

As the video feature amount, a shot length, a color histogram, a temporal variation characteristic of the color histogram, a motion vector, and the like are extracted. An example of the time variation characteristic is an inter-frame difference, for example.
The video feature amount is extracted for each video frame. As the video frame, for example, the length of one video frame may be 33 ms, and the next video frame may be formed by a time shift of, for example, 33 ms with respect to the current video frame.

  Here, there are various basic extraction methods such as a shot length, a motion vector, and a color histogram, but these are known, and for example, a method shown in Non-Patent Document 7 can be used.

  Since it is practically impossible to extract the shot length within a 33 ms video frame, for example, the shot length may be extracted as the length of a shot including the target video frame. Further, an average value, maximum value, minimum value, or the like of the shot length in a certain section including one or more shots may be used.

  The color histogram is extracted as follows, for example.

  A hue (Hue) is extracted for each pixel in the video frame. This hue is quantized to a predetermined number Q such as 11 or 256, for example, so that it can be determined which of Q quanta corresponds to all pixels. By executing this over all the pixels and counting the number of appearances for each quantum, the hue histogram of the video frame can be extracted.

  Alternatively, a histogram for only a specific area may be extracted.

  The motion vector can be extracted as a vector composed of an X component and a Y component, for example, by calculating an optical flow. As a method for calculating the optical flow, for example, Non-Patent Document 8 can be used. In addition, for example, the norm may be calculated for each video frame, or camera operations such as panning, tilting, and zooming are detected using the method disclosed in Patent Document 3, and each unit time is individually detected. It may be quantified as the amount of operation around.

  In step S130, the content using the audio feature quantity and video feature quantity of each audio frame extracted in step S120 and one or more audio models and video models configured in advance in step S110 are used. Then, the voice emotion probability and the video emotion probability in the emotion of the partial content are respectively calculated.

  Here, first, an example of the process of step S110 for configuring a statistical model will be described. The speech model may be obtained by the same method as in the first example of the embodiment. Hereinafter, a video model acquisition method will be described.

  The video model is acquired by performing learning from the video signal data for learning. The learning video signal data, like the content video signal data, has video feature quantities extracted in units of video frames, and has been manually assigned the types of labels defined as emotion categories as described above. And In the second example of this embodiment, it is assumed that the emotion category classified by the video signal data is the same as the emotion category for the speech model to estimate.

  By associating these emotion categories with each video model, a video model for calculating a video emotion probability is acquired for each emotion category. As these models, for example, a normal distribution, a mixed normal distribution, a hidden Markov model, a generalized state space model, or the like may be used. Preferably, a time series model such as a hidden Markov model or a generalized state space model that can model the temporal transition of emotion is employed.

  As methods for estimating the parameters of these video models, for example, a maximum likelihood estimation method, an EM algorithm, a variational Bayes method, and the like are known and can be used. For details, refer to Non-Patent Document 4, Non-Patent Document 5, and the like.

  In step S130, by inputting the audio feature value for each audio frame and the video feature value for each video frame extracted in step S120 to the audio model and video model acquired in step S110, Calculate voice emotion probability and video emotion probability. In step S110, since the audio model and the video model are constructed so that the probability can be calculated for each emotion category, the audio feature quantity is input to each audio model, and the video feature quantity is input to each video model. Emotion probability and video emotion probability can be calculated.

  For the video emotion probability, a region determined to be a face in the video is detected by a method disclosed in Patent Document 4, and the facial expression is further detected by a method disclosed in Patent Document 5. The recognized result may be reflected.

  For example, when the result of recognizing a facial expression corresponds to a certain emotion category, the video emotion probability of the emotion category is increased, and the video emotion probability of other emotion categories is It is possible to take a method of reducing and normalizing so as to satisfy the axiom of.

  Here, an example of a method of sharing one frame and calculating one emotion probability based on the voice emotion probability calculated by the voice model and the video emotion probability calculated by the video model (step S140) will be described. .

For example, as shown in FIG. 17, when the audio frame length is 50 ms and the video frame length is 33 ms, the video emotion probability pV of the video frame that overlaps for the longest time among video frames that overlap one audio frame, for example. and by the voice emotion probability pA of the audio frame, integrating each emotional category, or by introducing a predetermined weight, to calculate the like weighted average of pA and pV, which newly emotion probability pf _t That's fine.

Further, as another method, out of pA and pV, it may adopt a larger value as the pf _t. In the case of video content without audio signal data, for example, scaling adjustment such as squaring pV may be performed, and this may be set as pf _t .

  In these cases, the frame is shared with the audio frame.

  Next, in step S150, the emotion and the emotion level of the content and the partial content are estimated based on the emotion probability calculated for each common frame. This process may be executed in the same manner as step S140 (FIGS. 5 and 7) of the first example of the embodiment.

  In the first example of the embodiment of the present invention, when a partial content is generated, a set of audio sections that are considered to be continuous audio are collected as one section, and this is set as the partial content. This method may also be adopted in the second example of the embodiment, but the shot length is extracted as the video feature amount, but this shot may be used as the partial content.

  Hereinafter, the flow of processing after step S200 in FIG. 1 may be executed in the same manner as in the first example of the embodiment of the present invention.

  Other than the example shown as the embodiment of the present invention, the embodiment can be appropriately changed within the scope of the embodiment that can be taken based on the principle of the present invention.

Hereinafter, a specific embodiment for searching and recommending desired content or partial content according to the present invention will be described.
First Example: Search / Recommendation of Content in User Terminal HDD This example is an example in which content stored / recommended in the HDD 231 in the user terminal is searched. FIG. 18 shows an example of the configuration of a specific apparatus of the present invention in this embodiment.

  In this embodiment, the information control unit 300 is built in the user terminal 200, and the CPU 221, ROM 222, RAM 223, and HDD 231 (FIG. 3) in the user terminal are the same as the CPU 301, ROM 302, RAM 303, and HDD 304 in the information control unit, respectively. Good.

  Therefore, hereinafter, in the description of the present embodiment, the notation relating to the device in the information control unit uses the notation relating to the device in the corresponding user terminal.

As pre-processing, the emotion estimation unit F100 estimates the emotion and emotion level by using the audio signal data for the content stored in the user terminal HDD 231, and the content storage unit F200 transmits the content or partial content together with this information to the HDD 231. Accumulate in. The procedure is as follows.
[Procedure 1] The user launches a search request input screen as shown in FIG. 19, and inputs emotion as a search request using the keyboard 211 and the pointing device 212. For example, enter “fun” or “cool”.
[Procedure 2] The search request receiving unit F300 receives the search request, and the similarity calculation unit F400 determines the similarity between the search request and the content stored in the HDD 231 or the emotion of the partial content according to the equation (7). calculate.
[Procedure 3] The result presentation unit F500 refers to the similarity of each content and ranks in descending order to generate a list. Further, the attribute information, emotion, emotion level, and summary are presented on the monitor 241 in the ranking order.
[Procedure 4] The user uses the keyboard 211 and the pointing device 212 to select content to be viewed.
[Procedure 5] The user terminal 200 reads the content selected by the user from the HDD 231 and presents and reproduces it on the monitor 241.
[Procedure 6] The search request accepting unit F300 receives the weighted average value calculated according to the equation (5) as the search request for the emotion level of the content reproduced in the past and the currently viewed content, and the similarity calculation unit The F400 calculates the similarity between the search request and the emotion of the content or partial content stored in the HDD 231 according to the equation (6).
[Procedure 7] The result presentation unit F500 refers to the similarity of each content, and ranks in descending order of similarity to generate a list. Further, a predetermined number, for example, three of the top rankings, their attribute information, and thumbnails are presented on the monitor 241.
Thereafter, [Procedure 4] to [Procedure 7] may be repeated until the user ends the use, or the user may input a new search request.
Second Embodiment: Search / Recommendation of Content on the Web In this embodiment, contents stored in the database 400 included in the server device 500 including the information control unit 300 are connected to each other by a wide area network. In this example, search / recommendation is performed by inputting a search request from the user terminals 200a, 200b,. In particular, in this embodiment, a description will be given of search and recommendation of content on the Web by Internet communication as an example. FIG. 21 shows an example of the configuration of a specific apparatus according to the present invention in this embodiment. It is assumed that the user accesses a predetermined site supplied by the server device 500 provided with the information control unit 300 and inputs a search request through this site.

As pre-processing, for content stored in the database 400, the emotion estimation unit F100 estimates the emotion and emotion level from the audio signal data and the video signal data, and the content storage unit F200 stores the content or partial content together with this information in the database. Accumulate in 400. The procedure is as follows.
[Procedure 1] A user operates the keyboard 211 and the pointing device 212 to access a predetermined Web site.
[Procedure 2] The server device 500 presents a search request input screen as shown in FIG. 19 on the monitor 241 of the user terminal 200.
[Procedure 3] The user inputs a search request by adjusting the emotion level of each emotion by operating the pointing device 212.
[Procedure 4] When the search request is received by the search request receiving unit F300, the similarity calculation unit F400 determines the similarity between the search request and the emotion or emotion of the content or partial content stored in the database 400. Calculate according to equation (6).
[Procedure 5] The result presentation unit F500 refers to the similarity of each content, generates a list including attribute information, emotion, emotion level, and summary of each content in descending order, and distributes the list to the user terminal. .
[Procedure 6] The user terminal 200 presents the distributed list on the monitor 241. A user uses the keyboard 211 and the pointing device 212 to select content to view.
[Procedure 7] The server device 500 reads the content selected by the user from the database 400 and distributes it to the user terminal 200.
[Procedure 8] The user terminal 200 receives the content distributed from the server device, and presents and reproduces it on the monitor 241.
[Procedure 9] The search request receiving unit F300 receives the emotion and the emotion level of the content being played back as a search request, and the similarity calculation unit F400 receives the search request and the content or partial content stored in the database 400. The similarity with the emotion level is calculated according to the equation (6).
[Procedure 10] List in which result presentation unit F500 refers to the similarity of each content, and includes a predetermined number, for example, three from the top, information such as attribute information, emotion, emotion level, and summary of each content Is generated and distributed to the user terminal 200.
Thereafter, [Procedure 6] to [Procedure 10] may be repeated until the user ends use, or the user may input a new search request.
(Third Embodiment): Search / Recommendation of Content on the Web Combined with Text Search This embodiment is stored in the database 400 included in the server device 500 including the information control unit 300, as in the second embodiment. In this example, search and recommendation are performed by inputting a search request from each of the user terminals 200a, 200b,.

  In particular, in this embodiment, in addition to the content search / recommendation apparatus according to the present invention, further, text information such as a title of a content, a producer, and a genre is input as a search request, which has been conventionally used as a content search method. This is an embodiment in the case where a search apparatus 600 that searches for contents having matching attribute information among contents previously given as attribute information based on this search request is used in combination.

  One example of the configuration of a specific apparatus of the present invention in this embodiment is shown in FIG. It is assumed that the user accesses a predetermined site supplied by the server device 500 provided with the information control unit 300 and inputs a search request through this site.

As pre-processing, the emotion estimation unit F100 estimates the emotion and the emotion level from the audio signal data and the video signal data for the content stored in the database 400, and this information and a title given to each content in advance. The content storage unit F200 stores the attribute information including the text information such as the producer and the genre and the keyword extracted from the surrounding text described in the surroundings of the content in the database 400 together with the information. To accumulate. The procedure is as follows.
[Procedure 1] A user operates the keyboard 211 and the pointing device 212 to access a predetermined Web site.
[Procedure 2] The server device 500 presents a search request input screen as shown in FIG. 19 or 20 on the monitor 241 of the user terminal 200.
[Procedure 3] The user inputs the title or the like of the content to be viewed as text information on the search screen, and further adjusts the emotion level of each emotion by operating the pointing device 212 to input a search request.
[Procedure 4] The search apparatus 600 searches the database 400 for content to which attribute information that matches information such as a title input as text information is included in the search request, and generates a candidate list.
[Procedure 5] Of the search requests, the search request receiving unit F300 receives the emotion and the emotion level, and the similarity calculation unit F400 determines the emotion and the emotion level of the content included in the candidate list by the search request and the procedure 4. Is calculated according to the equation (6).
[Procedure 6] The result presentation unit F500 refers to the similarity of each content, generates a list including information such as attribute information, emotion, emotion level, and summary of each content in descending order, and distributes the list to the user terminal. .
[Procedure 7] The user terminal 200 presents the distributed list on the monitor 241. A user uses the keyboard 211 and the pointing device 212 to select content to view.
[Procedure 8] The server device 500 reads the content selected by the user from the database 400 and distributes it to the user terminal 200.
[Procedure 9] The user terminal 200 receives the content distributed from the server device, and presents and reproduces it on the monitor 241.
[Procedure 10] The search request receiving unit F300 receives the emotion and the emotion level of the content being reproduced as a search request, and the similarity calculation unit F400 receives the search request and the content or partial content stored in the database 400. The degree of similarity with emotion is calculated according to the equation (6).
[Procedure 11] List in which result presentation unit F500 refers to the similarity of each content, and includes a predetermined number, for example, 3 from the top, information such as attribute information, emotion, emotion level, and summary of each content Is generated and distributed to the user terminal 200.
Thereafter, [Procedure 7] to [Procedure 10] may be repeated until the user ends use, or the user may input a new search request. Further, in this embodiment, the search device 600 first performs a search based on information based on text and searches the content of the searched candidate list by narrowing down the emotion and emotion level, but conversely, the emotion and emotion first. A candidate list may be generated depending on the degree, and a narrow search based on text information may be executed.

  Also, a program for causing a computer to execute the content search / recommendation method is constructed.

  It is also possible to supply a recording medium recording the program to a system or apparatus, and the CPU (MPU) of the system or apparatus reads and executes the program stored in the recording medium. In this case, the program itself read from the recording medium realizes the functions of the above-described embodiment, and examples of the recording medium on which the program is recorded include CD-ROM, DVD-ROM, CD-R, CD- There are RW, MO, and HDD.

The flowchart explaining the flow of a process of the method in embodiment of this invention. The block diagram explaining the structure of the apparatus in embodiment of this invention. The block diagram explaining the structure of the apparatus of the user terminal in embodiment of this invention. The block diagram explaining the structure of the apparatus of the emotion estimation part F100 in the 1st example of embodiment of this invention. The flowchart of the process which the emotion estimation part F100 in the 1st example of embodiment of this invention performs. The figure explaining extraction of the audio | voice feature-value in embodiment of this invention. FIG. 6 is a flowchart for explaining the processing flow of step S140 in FIG. 5. The figure explaining the emotion level of the partial content in embodiment of this invention. The figure which shows an example of the time series information of the emotion degree in embodiment of this invention. The flowchart explaining the flow of a process of step S300 of FIG. The figure which shows the graph (radar graph) centering on the emotion category in embodiment of this invention. The figure which shows the equalizer-type interface which adjusts the emotion degree for every emotion category in embodiment of this invention. The figure which shows the search request input interface by the time series information of the emotion degree for every emotion in embodiment of this invention. The figure which shows an example of the user's viewing history in embodiment of this invention. The block diagram explaining the structure of the apparatus of the emotion estimation part F100 in the 2nd example of embodiment of this invention. The flowchart of the process which the emotion estimation part F100 in the 2nd example of embodiment of this invention performs. The figure explaining the method of calculating an emotion probability from the audio | voice emotion probability and image | video emotion probability in embodiment of this invention. The block diagram which shows an example of the specific structure of the apparatus in 1st Example of embodiment of this invention. The figure which shows an example of the search request input screen in embodiment of this invention. The figure which shows an example of the search request input screen in embodiment of this invention. The block diagram which shows an example of the specific structure of the apparatus in 2nd Example of embodiment of this invention. The block diagram which shows an example of the specific structure of the apparatus in 3rd Example of embodiment of this invention.

Explanation of symbols

  F100 ... Emotion estimation unit, F200 ... Content storage unit, F300 ... Search request reception unit, F400 ... Similarity calculation unit, F500 ... Result presentation unit, 200, 200a, 200b ... User terminal, 211 ... Keyboard, 212 ... Pointing device, 221, 301 ... CPU, 222, 302 ... ROM, 223, 303 ... RAM, 231, 304 ... HDD, 241 ... monitor, 300 ... information control unit, 400 ... database, 500 ... server device, 600 ... search device.

Claims (13)

An emotion estimation step in which the emotion estimation means estimates the emotion and the emotion level of the content from the audio signal data included in the multimedia content;
A content accumulation step in which content accumulation means accumulates content including the emotion estimated by the emotion estimation means and the emotion level as metadata;
A search request receiving means for receiving a search request corresponding to the emotion or the emotion and the emotion level,
A similarity calculation means for calculating the similarity of the content or partial content based on the search request;
A result presentation step in which a result presentation means presents a search / recommendation result of content or partial content based on the similarity; and
A content search / recommendation method characterized by including: 2. The method according to claim 1, wherein the emotion estimation step includes, for each analysis frame, a fundamental frequency, a temporal variation characteristic of the fundamental frequency, an rms value of the amplitude, a temporal variation characteristic of the amplitude rms value, a power, and a temporal variation of power. A feature amount extraction step for extracting at least one of a characteristic, a voice speed, and a time variation characteristic of the voice speed as a voice feature quantity;
By using one or more statistical models configured in advance using learning speech signal data, the appearance probability of the speech feature amount in the emotion and the transition probability in the time direction of one or more states corresponding to the emotion An emotion probability calculation step for calculating an emotion probability based on at least one of the two,
An emotion level calculating step of calculating the emotion level of the partial content including one or more of the analysis frames based on the emotion probability;
An emotion determination step of determining the emotion of the partial content configured by one or more of the analysis frames based on the emotion probability;
A content search / recommendation method characterized by including: An emotion estimation step in which the emotion estimation means estimates the emotion and the emotion level of the content from the audio signal data and the video signal data included in the multimedia content;
A content accumulation step in which content accumulation means accumulates content including the emotion estimated by the emotion estimation means and the emotion level as metadata;
A search request receiving means for receiving a search request corresponding to the emotion or the emotion and the emotion level,
A similarity calculation means for calculating the similarity of the content or partial content based on the search request;
A result presentation step in which a result presentation means presents a search / recommendation result of content or partial content based on the similarity; and
A content search / recommendation method characterized by including: 4. The method according to claim 3, wherein the emotion estimation step includes, for each analysis frame, a fundamental frequency, a temporal variation characteristic of the fundamental frequency, an rms value of the amplitude, a temporal variation characteristic of the rms value of the amplitude, a temporal variation of power and power. A feature that extracts at least one of a shot length, a color histogram, a color histogram, a time variation characteristic of a color histogram, and a motion vector as a video feature amount from at least one of a characteristic, an audio speed, an audio speed time variation characteristic, and video signal data A quantity extraction step;
Appearance of the voice feature amount in the emotion by one or more statistical models pre-configured using the learning audio signal data and one or more statistical models pre-configured using the learning video signal data An emotion probability calculation step of calculating an emotion probability based on at least one of the probability and the transition probability in the time direction of one or more states corresponding to the emotion;
An emotion level calculating step of calculating the emotion level of the partial content including one or more of the analysis frames based on the emotion probability;
An emotion determination step of determining the emotion of the partial content configured by one or more of the analysis frames based on the emotion probability;
A content search / recommendation method characterized by including: 5. The method according to claim 1, wherein the search request reception step includes the emotion of the content or partial content that the user is viewing and / or viewing, or the emotion and the emotion level. The content search / recommendation method is characterized in that the search request determined based on the request is received. 6. The method according to claim 1, wherein the result presentation step ranks content or partial content based on the similarity, and attribute information of the content or partial content based on the ranking result. A content search / recommendation method characterized in that at least one of the emotion, the emotion level, a thumbnail, and summary content is listed and presented. Emotion estimation means for estimating the emotion and emotion level of the content from the audio signal data included in the multimedia content;
Content accumulation means for accumulating content comprising the emotion and the emotion level estimated by the emotion estimation means as metadata;
Search request receiving means for receiving a search request corresponding to the emotion or the emotion and the emotion level;
Similarity calculation means for calculating the similarity of the content or partial content based on the search request;
Result presentation means for presenting search / recommendation results of content or partial content based on the similarity,
Content search / recommendation device characterized by including: 8. The apparatus according to claim 7, wherein the emotion estimation means includes, for each analysis frame, a fundamental frequency, a temporal variation characteristic of the fundamental frequency, an rms value of the amplitude, a temporal variation characteristic of the amplitude rms value, a power, and a temporal variation of power. A feature amount extraction means for extracting at least one of a characteristic, a voice speed, and a time variation characteristic of the voice speed as a voice feature quantity;
By using one or more statistical models configured in advance using learning speech signal data, the appearance probability of the speech feature amount in the emotion and the transition probability in the time direction of one or more states corresponding to the emotion An emotion probability calculation means for calculating an emotion probability based on at least one of them,
An emotion level calculating means for calculating the emotion level of the partial content including one or more of the analysis frames based on the emotion probability;
Emotion determination means for determining the emotion of the partial content constituted by one or more of the analysis frames based on the emotion probability;
Content search / recommendation device characterized by including: Emotion estimation means for estimating the emotion and the emotion level of the content from the audio signal data and the video signal data included in the multimedia content;
Content accumulation means for accumulating content comprising the emotion and the emotion level estimated by the emotion estimation means as metadata;
Search request receiving means for receiving a search request corresponding to the emotion or the emotion and the emotion level;
Similarity calculation means for calculating the similarity of the content or partial content based on the search request;
Result presentation means for presenting search / recommendation results of content or partial content based on the similarity,
Content search / recommendation device characterized by including: 10. The apparatus according to claim 9, wherein the emotion estimation means includes, for each analysis frame, a fundamental frequency, a temporal variation characteristic of the fundamental frequency, an rms value of the amplitude, a temporal variation characteristic of the amplitude rms value, a power, and a temporal variation of power. A feature that extracts at least one of a shot length, a color histogram, a color histogram, a time variation characteristic of a color histogram, and a motion vector as a video feature amount from at least one of a characteristic, an audio speed, an audio speed time variation characteristic, and video signal data A quantity extraction means;
Appearance of the voice feature amount in the emotion by one or more statistical models pre-configured using the learning audio signal data and one or more statistical models pre-configured using the learning video signal data An emotion probability calculation means for calculating an emotion probability based on at least one of the probability and the transition probability in the time direction of one or more states corresponding to the emotion;
An emotion level calculating means for calculating the emotion level of the partial content including one or more of the analysis frames based on the emotion probability;
Emotion determination means for determining the emotion of the partial content constituted by one or more of the analysis frames based on the emotion probability;
Content search / recommendation device characterized by including: 11. The apparatus according to claim 7, wherein the search request receiving unit is the content or partial content that the user is watching and / or watching, or the emotion and the emotion level. The content search / recommendation device is characterized in that the search request determined based on the request is received. 12. The apparatus according to claim 7, wherein the result presentation unit ranks content or partial content based on the similarity, and attribute information of the content or partial content, the emotion based on the ranking result. A content search / recommendation device that presents at least one of the emotion level, thumbnail, and summary content in a list. 7. A content search / recommendation program, characterized in that a program for causing a computer to execute each step of the content search / recommendation method according to claim 1 is provided.

Images