JP2007097047A - Contents editing apparatus, contents editing method and contents editing program - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To produce a summary video picture more reflecting individual sensibility. <P>SOLUTION: Expressions of the face of a viewer viewing video contents are imaged, uttered voices are collected and on the basis of them, the emotion of the viewer is inferred. Similarly, the emotion of a player is inferred from the expressions of the face and the voices of the player in the video contents. A frame matching emotions, based on face expressions, of the player and the viewer of the video contents and a frame matching emotions based on voices are defined as a viewer sympathized part, the frames of such a sympathized part are extracted, and a video image constructed from said frames is stored, as a summary video image, in a predetermined storage area in accordance with a title of the video contents as a source of the summary video image. <P>COPYRIGHT: (C)2007,JPO&INPIT

Description

  The present invention relates to a content editing apparatus for creating a content summary, a content editing method, and a content editing program.

  Conventionally, as a method for editing video content, a break for summarizing the content is detected and editing is performed by paying attention to scene switching, and a person in the video is automatically managed to manage the content. Have been proposed that can be recognized and edited based on this. In addition to the above, there are also proposed ones that are used for content search based on keywords (metadata) of video content. With the increase in content, needs for various editing methods such as efficient search and generation of summary video are increasing.

As described above, various editing methods have been proposed. However, in the case of home video shot by an individual, it is rare that the user himself adds a keyword (metadata) expressing the video content, and it is used for editing. The amount of information is limited. Therefore, adding various information has been devised.
For example, Patent Literature 1 proposes a search method in which information content such as video and audio is searched based on emotions analyzed from biological information. In this search method, a sensor or the like is used to analyze a viewer's emotion, psychological state such as mood, and the like, and an attempt is made to search content based on emotion information.
JP 2005-124909 A

However, in the above-described method, biological information is treated as emotion information, and only the viewer's emotion is evaluated in a scene in which the viewer is interested. For this reason, a more effective editing method using subjective information such as emotions has been desired.
Therefore, the present invention has been made paying attention to the above-mentioned conventional unsolved problems, and provides a content editing apparatus, a content editing method, and a content editing program that make more effective use of individual sensibilities. It is an object.

  In order to solve the above-described problems, a content editing apparatus according to the present invention is a content editing apparatus that creates a summary of content in which at least a person appears, and the performer emotion that estimates the emotion of the performer of the content Guessing means, viewer feeling guessing means for guessing viewers' feelings when viewing the content, and performers and viewers guessed by the performer feeling guessing means and the viewer feeling guessing means Among the contents, a high co-sensitivity scene inferring means for inferring a scene with high co-sensitivity to the performer of the viewer, and a scene with high co-sensitivity inferred by the high co-sensitivity scene inferring means And a summary content generation means for generating the summary content of the content.

  According to the above configuration, the content summary is estimated by using the scene with high consensus that the content viewer sympathizes with the content performer by estimating the emotion of the content performer and the emotion of the content viewer. Since the content is generated, it is possible to create summary content composed of scenes in accordance with the viewer's sensibility, and the summary of the original content can be recognized easily and accurately by referring to the summary content. .

In the content editing apparatus described above, the high co-sensitivity scene estimation means includes the performer's emotion estimated by the performer emotion estimation means and the viewer's emotion estimation means estimated by the viewer emotion estimation means. It is characterized by inferring a scene that matches emotion as a highly co-sensitive scene.
According to the above configuration, since the scene where the emotion of the content performer matches the emotion of the viewer of the content is estimated as a scene with high co-sensitivity, the scene with high co-sensitivity of the viewer is easily and accurately estimated. can do.

In the content editing apparatus described above, the content includes one of a video signal and an audio signal of the person, and the performer emotion estimation means is based on either the video signal or the audio signal. It is characterized by guessing the emotions of the performers.
According to the above configuration, since the performer's emotion is estimated based on either the performer's video signal or audio signal included in the content, the performer's emotion can be easily detected.

  In the content editing apparatus described above, the viewer emotion estimation unit includes an imaging unit that captures an image of the viewer's facial expression, a sound collection unit that collects sound generated by the viewer, and the biological information of the viewer. And at least one of biological information detection means for detecting the viewer, and based on any one of the imaging information of the imaging means, the sound collection information of the sound collection means, and the detection information of the biological information detection means It is characterized by guessing emotions.

According to the above configuration, based on at least one of the facial expression of the viewer imaged by the imaging means, the sound emitted by the viewer collected by the sound collection means, and the biological information of the viewer detected by the biological information detection means. Thus, the viewer's emotion can be easily estimated.
In the content editing apparatus, the content includes a video signal and an audio signal of the person, and the performer emotion estimation unit is based on the emotion based on the video signal and the audio signal for the performer. The viewer's emotion estimation means comprises an imaging means for capturing the viewer's facial expression and a sound collection means for collecting the sound emitted by the viewer, and the viewer is Emotions based on imaging information and emotions based on the sound collection information of the sound collection means are estimated, respectively, and the high co-sensitivity scene estimation means is based on emotions based on the video signals of the performers and emotions based on the imaging information of the viewers A scene having the same emotion and a scene having the same emotion based on the audio signal of the performer and an emotion based on the collected sound information of the viewer as a scene with high co-sensitivity It is characterized by guess Te.

  According to the above configuration, a scene in which an emotion that can be seen from the appearance such as an emotion based on the video signal of the content performer and an emotion based on the imaging information of the content viewer match, and the emotion and content based on the audio signal of the content performer By comparing the expressions of the same kind of emotions, it is assumed that the scenes where the emotions based on the voices produced by the viewers and viewers of the content, such as emotions based on the audio signals of the viewers of the viewers, coincide with each other are considered as high-sensitivity scenes. A scene with higher co-sensitivity can be accurately estimated, and a scene with high co-sensitivity can be obtained by considering both emotions even if the emotion is not understood only from the appearance or the audio signal. Can be guessed accurately.

Further, in the content editing apparatus, the performer emotion estimation unit includes a facial expression recognition unit that recognizes the facial expression of the performer based on the video signal, and from the facial expression of the performer recognized by the facial expression recognition unit It is characterized by guessing emotions.
According to the above configuration, since the facial expression of the performer of the content is recognized based on the video signal and the emotion of the performer is estimated based on this, the performer's emotion can be easily estimated.

In the content editing apparatus described above, the performer emotion estimation means includes an analysis means for analyzing a vibration frequency representing vibration of the performer's laryngeal original sound based on the audio signal, and an analysis result of the analysis means It is characterized by guessing the performer's emotion based on the above.
According to the above configuration, since the vibration frequency of the laryngeal original sound emitted by the content performer is analyzed based on the audio signal and the performer's emotion is estimated based on this, the performer's emotion is easily estimated. be able to.

In the content editing apparatus, the viewer emotion estimation means includes a facial expression recognition means for recognizing the viewer's facial expression based on the imaging information, and the viewer's facial expression recognized by the facial expression recognition means It is characterized by guessing emotions.
According to the above configuration, since the viewer's facial expression is recognized based on the imaging information obtained by capturing the viewer and the viewer's emotion is estimated based on the viewer's facial expression, the viewer's emotion can be easily estimated.

Further, in the content editing apparatus, the viewer emotion estimation unit includes an analysis unit that analyzes a vibration frequency representing a vibration of the viewer's laryngeal original sound based on the sound collection information, and the analysis unit analyzes The viewer's emotion is estimated based on the result.
According to the above configuration, the vibration frequency of the laryngeal original sound emitted by the viewer is analyzed based on the sound collection information collected by the sound collecting means, and the viewer's emotion is estimated based on the analysis. It can be estimated easily and accurately.

  Also, the content editing method of the present invention is a content editing method for creating a summary of the content in which at least a person appears, and when the emotion of the performer of the content is detected and the content is viewed Based on the estimated performer and the viewer's emotion, the scene in which the viewer sympathizes with the performer is identified in the content, and the identified scene is used. Then, a summary content of the content is created.

According to the above configuration, since the content that the viewer of the content sympathizes with the performer of the content is identified and the summary content of the content is created using the scene, the summary content in accordance with the viewer's sensitivity By referring to the summary content, it is possible to easily recognize the outline of the original content.
The content editing program of the present invention includes a step of estimating the emotion of a performer of the content, a step of estimating the emotion of the viewer when viewing the content, and the emotion of the performer and the viewer Based on the above, in the content, the step of specifying a scene in which the viewer sympathizes with the performer and the step of creating a summary content of the content using the specified scene are executed by a computer It is characterized by that.

  According to the above configuration, since the content that the viewer of the content sympathizes with the performer of the content is identified and the summary content of the content is created using the scene, the summary content in accordance with the viewer's sensitivity By referring to the summary content, it is possible to easily recognize the outline of the original content.

Embodiments of the present invention will be described below.
FIG. 1 is a functional block diagram showing an example of a content editing apparatus according to the present invention.
As shown in FIG. 1, this content editing apparatus reads content information of designated video content from content files such as home video stored in a predetermined storage area, such as an auxiliary storage device, and stores it. Reproduction means 1 for reproducing, content performer emotion estimation unit 2 for estimating the emotion of the performer of the specified video content, and viewer emotion estimation for estimating the emotion of the viewer who has viewed the specified video content And a consensus calculating means 4 for calculating the consensus based on the emotion of the performer estimated by the content performer emotion estimation unit 2 and the viewer's emotion estimated by the viewer emotion estimation unit 3; Based on the co-sensitivity calculated by the co-sensitivity calculating means 4, a high-co-sensitivity part is extracted, and a summary video creating means 5 for generating a summary video based on the extracted part is provided.

  As shown in FIG. 1, the performer emotion estimation unit 2 includes video signals and audio signals of video content specified by a viewer among files such as home videos stored in a predetermined storage area such as an auxiliary storage device. The video / audio signal input means 11 for capturing the image, the face detection means 12 for extracting the face image of the performer of the video content from the video signal input by the video / audio signal input means 11, and extracted by the face detection means 12 Facial expression recognition means 13 for analyzing a facial image and recognizing the facial expression, and voice emotion estimation means 14 for estimating the emotion of the performer of the video content based on the audio signal input by the video / audio signal input means 11. .

  The face detection means 12 extracts a face image from the video signal by a known procedure. For example, in order to express a face, a face image feature amount generated from luminance or an edge is used. For example, as described in JP-A-9-50528, for a certain input image, first, the presence or absence of a human skin color area is determined, and a mosaic size is automatically determined for the human skin color area. Mosaic the candidate area. Then, by calculating the distance from the human face dictionary, the presence or absence of the human face is determined, and the human face is cut out, thereby reducing erroneous extraction due to the influence of the background, etc. For example, a method for automatically finding a file can be used. The face detection means 12 can be realized by a known statistical method such as a multilayer neural network, a support vector machine, or discriminant analysis. Also, for example, the technical document “Detection of face independent of background and face direction and estimation of face direction” (Science Technical Report, PRMU 2001-217, pp. 87-94 (2002-01), Yuichi Araki, Shin Shimada) As described in Rei, Yoshiaki Shirai), a face part may be obtained with probability, and the face may be detected using a relaxation method. In this embodiment, estimation of the face direction is not necessary.

  The facial expression recognition means 13 classifies the facial expression with respect to the facial image extracted by the face detection means 12 by a known statistical method such as discriminant analysis. For example, Ekman et al.'S “anger”, “fear”, “sadness”, “joy”, “disgust”, “surprise” face images that have been collected in advance are collected in large numbers. Originally, the facial expressions extracted by the face detecting means 12 are classified into facial expressions, and emotion information of content performers is obtained.

This facial expression detection can also be realized by configuring a discriminator that classifies facial expressions by learning using a neural network.
For example, as described in “Facial expression recognition device” in Japanese Patent Application Laid-Open No. 8-249453, a feature vector of a facial image is extracted, quantized according to categories such as emotions, and the facial expression of each category. May be recognized.

  The voice emotion estimation means 14 performs a known general voice emotion analysis based on the vibration frequency representing the vibration of the laryngeal original sound. That is, since the vibration frequency representing the vibration of the laryngeal original sound generally tends to increase when excited, the vibration frequency tends to be low, and the expression of sadness or love tends to be low. Based on standard deviation, classify by statistical method. Further, as described in, for example, “Emotion recognition device” of Japanese Patent Laid-Open No. 5-12023, the normalization deviation amount between the feature signal and the reference data is calculated based on the audio signal, and the duration time or Emotions may be determined based on the direction of signal shift, signal level, rising speed, and the like.

The emotion of the content performer based on the video signal detected by the facial expression recognition unit 13 and the emotion of the content performer based on the audio signal detected by the audio emotion estimation unit 14 are emotions at any point in the video content. It is managed in association with timing information for specifying whether or not there is.
On the other hand, the viewer emotion estimation unit 3 captures the facial expression of the viewer when viewing the designated video content and also inputs video / audio signal input means for inputting audio generated by the viewer. 21. Face detection means 22 for extracting a viewer's face image from the video signal input by the video / audio signal input means 21, and facial expression recognition means 23 for analyzing the face image extracted by the face detection means 22 and recognizing a facial expression. And voice emotion estimation means 24 for recognizing the viewer's emotion based on the voice signal input by the video / audio signal input means 21.

The video / audio signal input means 21 is a collection of imaging means such as an image sensor such as a CCD (Charge Coupled Device) for recognizing the facial expression of the viewer and a microphone for collecting the audio of the viewer. Consists of sound means.
Further, the face detection means 22 is configured in the same manner as the face detection means 12 of the content performer emotion estimation unit 2, and is viewed from a video signal obtained by imaging a viewer obtained by imaging with an imaging means such as a CCD. A person's face image is extracted. Then, the facial expression recognition unit 23 recognizes the facial expression from the viewer's face image detected by the face detection unit 22 in the same procedure as the facial expression recognition unit 13 of the content performer emotion estimation unit 2, and Get emotional information.

Further, the voice emotion estimation means 24 is adapted to perform voice emotion estimation means of the content performer emotion estimation unit 2 with respect to voice signals such as voices, laughter, and cry voices uttered by the viewer detected by the sound collecting means such as the microphone. Analysis is performed in the same procedure as in FIG. 14, and the viewer's emotion based on the audio signal is estimated.
The emotion estimated from the viewer's facial expression recognized by the facial expression recognition means 23 and the viewer's emotion estimated based on the audio signal by the voice emotion estimation means 24 indicate at which point in time the video content occurred. It is managed in association with timing information.

  The co-sensitivity calculation means 4 includes the emotion information of the content performer estimated by the facial expression recognition means 13 and the voice emotion estimation means 14 of the content performer emotion estimation unit 2, and the facial expression recognition unit 23 of the viewer emotion estimation unit 3. Then, the emotion information of the viewer estimated by the voice emotion estimation means 24 is compared, and the degree of coincidence between the emotion of the content performer and the emotion of the viewer, that is, co-sensitivity is calculated. The calculation of this sensitivity is performed when the emotion of the content performer and the emotion of the viewer match the timing of the content performer and the viewer's emotion. Judge that the degree is high. On the contrary, when neither the emotion based on the face image nor the emotion based on the audio signal match, it is determined that the co-sensitivity is low.

The summary video creation means 5 extracts the video signal of the video content at the time when the cosensitivity is determined to be high based on the cosensitivity calculation result in the cosensitivity calculation means 4 and A file composed of video signals is generated and used as a summary video, which is stored in a predetermined storage area such as an auxiliary storage device.
The content editing apparatus can be realized as a computer 100 and a program executed by the computer 100, as shown in FIG.

FIG. 2 is a block diagram illustrating a configuration of the computer 100.
As shown in FIG. 2, the computer 100 includes an arithmetic processing unit (CPU) 101 that controls arithmetic operation and the entire system based on a control program, a ROM 102 that stores a control program executed by the arithmetic processing unit 101, It is composed of a RAM 103 for storing data read from the ROM 102 and the like, a calculation result required in the calculation process of the calculation processing unit 101, and an I / F 104 that mediates input / output of data to an external device. These are connected to each other via a bus 105 which is a signal line for transferring data so that data can be exchanged.

  The I / F 104 stores video content such as a home video file as an external device, an auxiliary storage device 106 such as an HDD for storing a summary video of the content generated by the content editing device, and a human interface. And an input device 107 such as a keyboard or mouse capable of inputting data, and an output device 108 such as a display for displaying video content and displaying a summary video thereof based on the video signal. A viewer sensor 110 is connected which includes an imaging means for capturing the facial expressions of the viewers and a sound collecting means for collecting the sound produced by the viewers.

The arithmetic processing unit 101 includes a microprocessing unit (MPU) or the like, and activates a predetermined processing program stored in a predetermined area of the ROM 102 to generate a summary video of video content.
FIG. 3 is a flowchart showing an example of a processing procedure when a face image is extracted from a content signal or a captured image of a viewer and the emotion is estimated, and is based on the face image in the content performer emotion estimation unit 2 Since the estimation of emotion and the estimation of emotion based on the face image in the viewer emotion estimation unit 3 are performed in the same procedure, processing for the content signal will be described here.

First, the video signal of the video content designated by the input device 107 is read from the auxiliary storage device 106 and taken out frame by frame from successive frames (step S1).
Then, the face detection unit 12 detects a face image of the extracted frame, and detects a face in the frame. At this time, as the detected “face”, the position (X, Y coordinates) in the frame, the size including the contour, and the like are detected (step S2).

Next, the process proceeds to step S3, where it is determined whether a face image exists in the frame based on the detection result of the face image in step S2. If no face image exists, the process returns to step S1. The same processing is performed for the next frame.
On the other hand, if a face image exists in the frame, the process proceeds to step S4, and the face image is located at any position in the frame based on the position of the face image detected in step S2 and the size thereof. The facial expression recognition unit 13 performs facial expression recognition on the image at the estimated position. For example, it is classified into facial expressions such as anger, fear, sadness, joy, disgust, and surprise. Then, the identification information for specifying the frame is used as the timing information, and the timing information and the facial expression recognition result are associated with each other and stored as emotion information of the frame in a predetermined recording area (step S5).

Next, the process proceeds to step S6. If the emotion information is not detected for all the frames of the designated video content, the process returns to step S1 to detect the emotion information for the next frame in the same procedure as described above. Then, when the detection of the emotion information for all the frames of the designated video content is completed, the process is terminated.
With the above processing, facial expression recognition based on the video signal of the performer appearing in the designated video content is completed. For example, as shown in FIG. 4, when there are face images in frames f2 to f4, f6 and f7 in a series of frames (FIG. 4A) (FIG. 4B), The emotion information is detected only for a certain frame. For example, the emotions of the content performers in the frames f2 to f4 are identified as “joy”, and the emotions of the content performers in the frames f6 and f7 are identified as “surprise”. (FIG. 4 (c)).

FIG. 4 schematically shows a change state of emotion in a frame. In FIG. 4B, the hatched frame represents a frame with a face image. In FIG. 4C, the hatched frame represents a frame in which emotion estimation has been performed, and the type of emotion represents the type of hatching.
Emotion information is detected frame by frame in a similar procedure for a video signal that is captured by an imaging means such as a CCD in the same procedure and that captures the facial expression of the viewer when viewing this video content.

FIG. 5 is a flowchart showing an example of a processing procedure for detecting the emotion from the content signal or the audio signal of the viewer, and the emotion detection and viewing based on the audio signal in the content performer emotion estimation unit 2 Since the emotion estimation based on the audio signal in the person emotion estimation unit 3 is performed in the same procedure, the processing for the content signal will be described here.
First, the audio signal for the video content designated by the input device 107 is read from the auxiliary storage device 106 at once for all the frames (step S11). Here, in order to analyze the frequency of the audio signal, all the audio signals are read at once.

Next, the process proceeds to step S12, where the voice signal is handled as a frequency component, and the voice emotion estimation means 14 analyzes emotion information. That is, the emotion is estimated based on the vibration frequency of the laryngeal original sound. At this time, the start time and end time when the emotion occurs are also detected.
Next, the process proceeds to step S13, where the emotion start time and end time based on the audio signal detected in step S12 are used as timing information, and each emotion and its timing information are associated with each other and stored in a predetermined storage area.

  Thus, emotion information based on the audio signal of the content performer is detected. For example, as shown in FIG. 6, when characteristic frequency components representing emotion information appear in the vibration frequency at time points t1 to t2 and time points t3 to t4 (FIG. 6A), the emotion information is specified. For example, it is recognized as “joy” (FIG. 6B). In FIG. 6A, the horizontal axis represents time, and the vertical axis represents amplitude. In FIG. 6B, the type of emotion is represented by the type of hatching.

In the same manner, the emotion of the viewer who viewed this content collected by sound collecting means such as a microphone is estimated based on the audio signal, and is associated with the start time and end time when the emotion occurs. Stored in the storage area.
FIG. 7 is a flowchart illustrating an example of a processing procedure of summary video creation processing for creating a summary video based on emotion information estimated based on a face image and an audio signal.

  In this summary video creation process, first, emotion information of content performers and viewers estimated based on the face image and video signal in step S21 is read from a predetermined storage area (step S21). Next, the process proceeds to step S22, and the emotion information that can be read from the facial expressions of the content performer and the viewer is matched based on the timing information associated with the emotion information based on the face image, that is, the identification information of the frame. Specify the frame to be played. Similarly, the timing information associated with the emotion information based on the audio signal, that is, the time point at which the emotion information that can be read from the audio signal of the content performer and the viewer coincides with each other based on the emotion start time and end time. Identify.

Next, the process proceeds to step S23, where the time point at which the emotional information of the content performer and the viewer based on the face image matches and the time point at which the emotional information of the content performer and the viewer based on the audio signal match are specified. Judge that co-sensitivity is high.
Then, the timing of the emotion specified when the emotion information of the content performer and the viewer based on the audio signal matches the timing specified by the identification information of the frame where the content information of the content performer and the viewer based on the face image matches. When the timing specified by the start time and the end time matches, it is determined that the co-sensitivity at this time is the highest.

Further, when neither the content performers' and viewers' emotion information based on the face image nor the content performers and viewers' emotion information based on the audio signal match, it is determined that the co-sensitivity at this point is low.
Next, the process proceeds to step S24, and the frame at the time when the co-sensitivity is determined to be high in step S23 is extracted from the content signal in the order of the frames. Then, a video signal composed of the extracted frames is generated, stored in the auxiliary storage device 106 as a summary video of the designated content, and the process ends.

Next, the operation of the above embodiment will be described.
First, video content for which a summary is to be created is designated by the input device 107 and is reproduced by the reproducing means 1. At this time, the viewer sensor 110 including the image pickup means and the sound collection means picks up the facial expression of the viewer who is viewing and collects the sound emitted by the viewer and stores it in a predetermined storage area.

Next, emotion information of the content performer is acquired for the content to be summarized. Specifically, the target video content is designated by the input device 107, and the processing shown in FIGS. 3 and 5 is executed to acquire emotion information about the video signal and the audio signal.
The arithmetic processing unit 101 reads one frame at a time from consecutive frames of the video content (step S1 in FIG. 3), and detects a face image (step S2). Then, facial expression recognition is performed on the frame in which the face image is detected (step S4).

  As a result, for example, as shown in the schematic diagram of FIG. 8, the frames in which the face images are detected are classified into six facial expressions such as anger, fear, sadness, joy, disgust, and surprise (FIG. 8A). This is stored in a predetermined storage area in association with the frame identification information. In the case of FIG. 8A, frames f12 to f14, f16 to f17, f21 to f25, and f27 to f29 are detected as frames having face images, and frames f12 to f14 and f21 to f25 are “joy, frames f16 to f17”. And f27 to f29 are recognized as “surprise”.

Further, based on the audio signal, as shown in FIG. 8B, the times t12 to t16 and the times t28 to t30 are recognized as “joy”, and the times t20 to t23 are recognized as “surprise”. In FIG. 8, each frame and each time point are displayed with matching timing, and each timing in FIGS. 8A to 8E is matched.
Next, the emotion information of the viewer who viewed this video content is detected in the same manner. In this case, processing is performed on the video signal and audio signal of the viewer that has been collected and stored in the predetermined storage area, and the emotion information is associated with the timing information in the predetermined storage area. Store.

Thus, for example, as shown in FIG. 8C, for the video signal, the frames f13 to f16 and f20 to f23 are recognized as “joy”, the frames f26 to f30 are recognized as “surprise”, and the audio signal is As shown in FIG. 8D, the times t14 to t15 and the time t30 are recognized as “joy”, and the times t22 to t23 are recognized as “surprise”.
Then, a summary video creation process shown in FIG. 7 is performed on the emotion information detection results based on the video signal and the audio signal, and a location where the emotion of the content performer and the emotion of the viewer are identified is specified. The

  In the case of FIG. 8, the emotion information based on the video signal is “pleasant” in the frames f13 to f14 and the frames f21 to f23, and “surprise” in the frames f27 to f29, as shown in FIG. 8 (a) and FIG. 8 (c). "Matches. In addition, as shown in FIGS. 8 (b) and 8 (d), emotion information based on the audio signal is coincident with time t14 to t15, time t22 to t23 “joy”, and time t30 “surprise”. Therefore, in the video signal, the frames f13 to f14 and f21 to f23 in which the emotion of the content performer and the viewer's emotion match with “joy” and the frames f27 to f29 that match with “surprise” have high cosensitivity. Identified as a frame. Similarly, the time t14 to t15 and the time t30 when the emotion of the content performer and the emotion of the viewer coincide with “joy” and the time t22 to t23 when “surprise” coincide with each other for the audio signal are high in sensitivity. Identified as a location.

  Accordingly, the frame where the emotion based on the video signal and the emotion based on the audio signal match between the content performer and the viewer are extracted as a highly co-sensitive portion, that is, FIG. 8 (e). As shown, frames f13 to f15 having an emotion of “joy” and frames f21 to f23 and f27 to f30 having an emotion of “joy” and “surprise” are extracted, and based on this, a summary video is generated. This summary video is stored in the auxiliary storage device 106 as a summary video of the designated video content. For example, as shown in FIG. 9, the summary video is stored in the auxiliary storage device 106 with the program title specifying the video content for which the summary is created and the recording date / time added.

The user can grasp the outline of the video content by reading each summary video stored in the auxiliary storage device 106 and playing it back by the playback means 1.
Here, as described above, the summary video is composed of frames in which emotions of content performers and viewers match. Therefore, it is possible to create a characteristic summary video in accordance with the sensitivity of the viewer side, that is, the portion where the viewer has a large degree of emotion transfer to the video content. Therefore, by referring to this summary video, it is possible to quickly recognize the content of the original video content and to quickly search for the video content.

At this time, since the emotion information of the content performer and the viewer is estimated based on the facial expression and the audio signal, it is possible to easily extract a portion with high co-sensitivity.
In addition, by extracting high-sensitivity locations, it is possible to create a summary video by extracting characteristic scenes such as locations where the emotions of the content performers have changed significantly, so the content of the video content can be grasped efficiently. The desired video content can be easily searched.

  In addition, human emotions may not appear both in the face and voice, but may appear only in either the face or voice. As described above, not only emotions based on face images but also emotions based on voice signals are detected. However, since the part where the viewer is sympathetic is extracted based on the emotions obtained from both, the part where the viewer is sympathetic even if the emotion does not appear both in the face and the voice Can be accurately extracted.

In particular, in the case of a home video or the like that is personally captured at home, it is a troublesome task to add metadata such as a keyword for video content to be used for content search. However, as described above, since a summary video can be automatically created, a desired home video can be easily searched by referring to this summary video.

In the above-described embodiment, this part is extracted because either the emotion obtained from the face image or the emotion obtained from the audio signal has high co-sensitivity when the emotions of the content performer and the viewer match. However, for example, for emotions obtained from face images and emotions obtained from audio signals, only the points with the highest co-sensitivity where the emotions of the content performers and viewers match are the same. Alternatively, it may be extracted as a summary video.

  In the above embodiment, a case has been described in which the viewer's emotion is estimated based on the facial expression of the viewer and the audio signal emitted by the viewer, but the emotion that can be used to estimate the viewer's emotion As long as it is an estimation means, the viewer's emotion may be estimated based on any information. For example, the viewer's biological information such as heartbeat, pulse, blood pressure, brain wave, breathing, sweating, and blinking may be used. Biometric information detection means for detecting may be provided, whereby the biometric information is acquired, and the viewer's emotion may be estimated based on the biometric information. In this case, for example, as described in “biological information analysis apparatus and biological information analysis method” of Japanese Patent No. 3310498, a fluctuation signal of an electroencephalogram is input and a neural network is used to input information such as comfort level from an electroencephalogram or the like. What is necessary is just to measure emotion information quantitatively. Then, the estimation results of the plurality of emotion estimation means may be used as the viewer's emotions, and any one of the estimation results of the plurality of emotion estimation means or the estimation results of the plurality of emotion estimation means may be used. You may combine.

  Moreover, in the said embodiment, the case where the emotion based on a face image and the emotion based on an audio | voice signal are estimated and the location where the content performer's emotion and viewer's emotion correspond are extracted based on these are demonstrated. However, only one of the emotion based on the face image and the emotion based on the audio signal may be estimated, and the portion where the emotions of the content performer and the viewer are matched may be extracted. Based on the above, a location where the emotions of the content performer and the viewer match may be extracted.

In the above-described embodiment, the case where the summary video is generated only from the frames having the same sensitivities has been described. However, for example, a predetermined amount of frames before and after the frames including the same sensitivities, or the sympathy A predetermined amount of frames may be extracted from frames having the same degree and used as a summary video.
In the above embodiment, the case where the summary video of the video content of the home video is generated has been described. However, the present invention is not limited to this, and the present invention can be applied to other video content such as a drama. Further, not only video content but also audio content that does not include video can be applied. In this case, a portion where the emotion based on the audio signal of the performer of the audio content and the emotion based on the audio uttered by the viewer who has watched the audio content match is extracted as a location with high co-sensitivity, and this is summarized audio And it is sufficient.

Here, in the above embodiment, the content performer emotion estimation unit 2 of FIG. 1 corresponds to the performer emotion estimation unit, the viewer emotion estimation unit 3 corresponds to the viewer emotion estimation unit, and the co-sensitivity calculation unit 4 Corresponds to the high co-sensitivity scene estimation means, and the summary video creation means 5 corresponds to the summary content generation means.
Further, the face detection means 12, the expression recognition means 13, the face detection means 22 and the expression recognition means 23 in FIG. 1 correspond to the expression detection means, respectively, and the voice emotion estimation means 14 and 24 correspond to the vibration frequency analysis means. Yes.

It is a functional block diagram which shows an example of the content editing apparatus to which this invention is applied. It is a figure which shows the hardware constitutions which implement | achieve the content editing apparatus of this invention. It is a flowchart which shows the flow of a process of the detection method of the emotion information based on a face image. It is explanatory drawing for demonstrating the detection method of the emotion information based on a face image. It is a flowchart which shows the flow of a process of the detection method of the emotion information based on an audio | voice signal. It is explanatory drawing for demonstrating the detection method of the emotion information based on an audio | voice signal. It is a flowchart which shows the flow of a process of summary video generation processing. It is explanatory drawing for demonstrating the production | generation method of a summary image | video. It is an example of the summary image | video memorize | stored in the auxiliary storage device.

Explanation of symbols

  2 content performer emotion estimation unit, 3 viewer emotion estimation unit, 4 co-sensitivity calculation unit, 5 summary video creation unit, 11, 21 video / audio signal input unit, 12, 22 face detection unit, 13, 23 facial expression recognition unit , 14, 24 Voice emotion estimation means, 101 arithmetic processing unit, 106 auxiliary storage device, 110 viewer sensor

Claims (11)

A content editing device that creates a summary of content in which at least people appear,
Performer emotion guessing means for guessing the performer's emotion of the content;
Viewer emotion estimation means for estimating the viewer's emotion when viewing the content;
Based on the performer and viewer's emotion information estimated by the performer's emotion estimation means and the viewer's emotion estimation means, a scene with high sensitivity to the performer of the viewer is estimated from the content. A high co-sensitivity scene estimation means;
An apparatus for editing content, comprising: summary content generation means for generating summary content of the content using a scene with high cosensitivity estimated by the high cosensitivity scene estimation means.   The high co-sensitivity scene estimation unit is configured to detect a scene in which the performer's emotion estimated by the performer emotion estimation unit matches the viewer's emotion estimated by the viewer emotion estimation unit. The content editing apparatus according to claim 1, wherein the content is estimated as a high scene. The content includes either one of the person's video signal and audio signal,
The content editing apparatus according to claim 1 or 2, wherein the performer emotion estimation means estimates the performer's emotion based on either the video signal or the audio signal. The viewer emotion estimation means is at least one of an image pickup means for picking up an image of the viewer's facial expression, a sound collection means for collecting a sound uttered by the viewer, and a biological information detection means for detecting the biological information of the viewer. Or one
4. The viewer's emotion is estimated based on any one of imaging information of the imaging means, sound collection information of the sound collection means, and detection information of the biological information detection means. The content editing apparatus according to any one of the above. The content includes a video signal and an audio signal of the person,
The performer emotion estimation means estimates the emotion based on the video signal and the emotion based on the audio signal for the performer,
The viewer emotion estimation unit includes an imaging unit that captures an image of the viewer's facial expression and a sound collection unit that collects a sound emitted by the viewer. Guess each emotion based on the sound collection information of the sound collection means,
The high co-sensitivity scene estimation means includes a scene in which an emotion based on the video signal of the performer and an emotion based on the imaging information of the viewer match, an emotion based on the audio signal of the performer, and a sound collection of the viewer The content editing apparatus according to claim 1, wherein a scene with matching emotions based on information is estimated as a scene with high co-sensitivity. The performer emotion estimation means includes facial expression recognition means for recognizing the performer's facial expression based on the video signal,
6. The content editing apparatus according to claim 3, wherein the emotion is inferred from the facial expression of the performer recognized by the facial expression recognition means. The performer emotion estimation means includes analysis means for analyzing a vibration frequency representing vibration of the laryngeal original sound of the performer based on the audio signal,
The content editing apparatus according to any one of claims 3, 5, and 6, wherein an emotion of the performer is estimated based on an analysis result of the analysis means. The viewer emotion estimation means includes facial expression recognition means for recognizing the viewer's facial expression based on the imaging information,
6. The content editing apparatus according to claim 4, wherein the emotion is estimated from a facial expression of the viewer recognized by the facial expression recognition means. The viewer emotion estimation means includes analysis means for analyzing a vibration frequency representing vibration of the viewer's laryngeal original sound based on the sound collection information,
The content editing apparatus according to any one of claims 4, 5, and 8, wherein the viewer's emotion is estimated based on an analysis result of the analysis means. A content editing method that creates a summary of at least a person ’s content,
Guessing the emotions of the performers of the content, and guessing the emotions of the viewers when viewing the content,
Based on the guess of the performer and the viewer's feelings, in the content, specify the scene that the viewer is sympathizing with the performer,
A content editing method, wherein summary content of the content is created using the identified scene. Guessing the emotions of the content performers,
Inferring viewers' emotions when viewing the content;
Identifying the scene in which the viewer is sympathizing with the performer in the content based on the emotions of the performer and the viewer;
A content editing program causing a computer to execute a summary content of the content using the identified scene.

Images