JP2009111938A - Device, method and program for editing information, and record medium recorded with the program thereon - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To allow execution of automatic editing and editing support by automatically detecting emotional expressions in video images or in sound and capturing each section in an emotional state on the basis of the information obtained by the detection. <P>SOLUTION: An information editing device is provided with a section division means 13 for dividing one or more video images or sound into prescribed sectional units, an emotion detection means 14 for calculating the sureness of one or more types of emotional expression states by using video image information and sound information for each divided section, an image output means 15 for outputting representative images, video images or sound of each divided section, or the representative images and the video images or sound of each divided section for each divided section, an image processing means 16 for executing processing and imparting information to the representative images on the basis of the sureness, a layout means 17 for outputting the processed representative images to a user after spatially laying out the representative images, and a generation means 19 for generating video images or sound by connecting each section, corresponding to at least one representative image selected by the user, with each other. <P>COPYRIGHT: (C)2009,JPO&INPIT

Description

  The present invention relates to an information editing apparatus for editing video content or audio content, an information editing method, an information editing program, and a recording medium on which the program is recorded.

  Due to the widespread use of content distribution / sharing sites as well as broadcasts so far, websites and personal PCs are increasingly viewing content. For example, the types of content include movies, dramas, home videos, News, documentary, music, etc. are very diverse. With the cooperation of communication and broadcasting, it is easily expected that the number of users who enjoy viewing content will increase further in the future.

  Among them, on the content sharing site, content shot or created by individuals can be released and shared among users. In content sharing sites, the amount of content that can be published is often limited, and the number of content is enormous, so in order for more viewers to view the content, It is preferable to publish the edited content so that the section to be viewed is centered.

However, most of the accumulated content is unedited content that has not undergone the editing process. The reason for this is that editing the content requires equipment and technology to perform the editing. Understand the content of the content and confirm which section of the content is the section you want to watch. It is necessary to keep
This is because it is difficult for the user due to these two factors.

To solve this problem,
1) Editing facilities and users who do not have technology can easily perform editing.
2) Efficiently confirm the section in the content that the user wants to watch.
Editing technology that realizes these two points is necessary.

  As conventional techniques, techniques for automating or supporting editing of contents are described in, for example, Patent Documents 1 to 3.

  In the technique described in Patent Document 1, a video is divided into several sections, a representative image of each section is selected, and further, the layout and the representative image are narrowed down to list the video contents in a list. It describes the editing support technology that can grasp this.

  In the technique described in Patent Document 2, the video is divided into several sections, motions such as camera shake and camera work in each section are detected, and the stable motion sections are presented to the user as video materials suitable for editing. It describes the editing support technology to be presented.

According to these prior arts, dividing the video into several sections makes it easy to connect the sections,
1) Editing facilities and users who do not have technology can easily perform editing.
It is an editing support technology that meets the requirements.

  On the other hand, there is a technique described in Patent Document 3 as an editing technique that can reflect the contents of each section of the content. Here, based on the audio information included in the content, the audio is divided into predetermined unit intervals, and the sections in the emphasized state are selected by determining whether or not each section is the audio in the emphasized state. Furthermore, an automatic editing technique for connecting the selected sections is described.

  According to this technique, editing centered on the highlighted section can be automatically executed based on the audio information of the content.

In addition, the reference of the following nonpatent literatures 1-6 exists as a reference of the technique relevant to this invention.
Japanese Patent Laid-Open No. 11-308567 JP 2000-261757 A JP 2005-333205 A “Digital Speech Processing, Chapter 4, 4.9 Pitch Extraction”, Sadaaki Furui, Tokai University Press, pp. 57-59, September 1985 Shigeki Hatakeyama, Fumitada Itakura, “Personality information included in the dynamic scale of speech”, Proc. 589-590 “Easy-to-understand pattern recognition”, Kenichiro Ishii, Noriyoshi Ueda, Eisaku Maeda, Hiroshi Murase, Ohmsha, pp 52-54, 1998 “Computational Statistics I Chapter 3 3EM Method 4 Variation Bayes Method”, Nobuyoshi Ueda, Iwanami Shoten, pp. 157-186, June 2003 “Research on structured video handling mechanism and video interface based on video feature indexing, Chapter 3 Video indexing based on image processing”, Yoshinobu Tonomura, Ph.D. 15-23, 2006 "Computer image processing", edited by Hideyuki Tamura, Ohmsha, pp. 242-247, December 2002

  A typical example of a section that the user wants to watch is a section where emotional expression in video or audio is large. In other words, there are many users who want to watch a fun section in the video and many users who want to watch a sad section. Such sections are automatically classified and automatically edited. Development of technology and technology that can be easily edited is desired.

  However, in these techniques, the information about each section shows only a limited index such as a section in which movement is stable or a section in an emphasized state. No information was presented. For this reason, there is a problem in that it is difficult for such a user to immediately confirm whether or not the section is desired to be viewed, and it is difficult to be an editing technique useful for such a user. It was.

  The present invention has been made on the basis of the above-mentioned problems, and its object is to automatically detect emotional expression in video or audio by analyzing one or more video or audio, and by this detection. An object of the present invention is to provide an information editing apparatus, an information editing method, an information editing program, and a recording medium on which the program is recorded, which can execute automatic editing that captures a section in an emotional state based on information obtained and editing support.

  In the present invention, emotion includes emotion, atmosphere, impression, and the like. The voice includes not only human speech but also singing voice, music, environmental sound, and the like.

  In order to solve the above-mentioned problem, the invention of claim 1 is an apparatus for editing one or more videos or sounds, and section dividing means for dividing one or more videos or sounds into predetermined sections. , For each of the divided sections, using one or more of video information and audio information, emotion detection means for determining the probability of one or more emotion expression states, and based on the probability, Automatic generation means for generating video or audio by connecting one or more of the sections.

  The invention according to claim 2 is an apparatus for editing one or more videos or sounds, section dividing means for dividing one or more videos or sounds into predetermined sections, and for each of the divided sections , Emotion detection means for determining the probability of one or more emotion expression states using at least one of video information and audio information, and a representative image or the divided image for each of the divided sections Output means for outputting video or audio for each section or a representative image and video or audio for each of the divided sections, and processing the representative image to any one of processing and information addition based on the probability. Image processing means for executing at least one image, layout means for spatially laying out the processed representative image and outputting to the user, and the section corresponding to at least one representative image selected by the user. Connect and project Or characterized in that it comprises generation means for generating a sound, a.

  The invention according to claim 3 is a method for editing one or more videos or sounds, wherein the section dividing means divides one or more videos or sounds into predetermined sections, and emotion detection. An emotion detecting step for determining the probability of one or more emotion expression states using one or more of video information and audio information for each of the divided sections, and an automatic generation means, And an automatic generation step of generating video or audio by connecting one or more of the sections based on the certainty.

  The invention according to claim 4 is a method for editing one or more videos or sounds, wherein the section dividing means divides one or more videos or sounds into predetermined sections, and emotion detection. An emotion detecting step for determining the probability of one or more emotion expression states using one or more of video information and audio information for each of the divided sections, and an output means, For each divided section, an output step for outputting a representative image or video or audio for each of the divided sections, or a representative image and video or audio for each of the divided sections; An image processing step that executes one or more of processing and information addition based on the certainty of the image, and a layout unit spatially lays out the processed representative image to the user. To present And out step, the generation means, characterized in that it comprises a generation step of generating a video or audio by connecting the section corresponding to at least one of the representative image selected by the user.

  The invention described in claim 5 is an information editing program for causing a computer to function as each means described in claim 1 or 2.

  A sixth aspect of the present invention is a computer-readable recording medium on which the information editing program according to the fifth aspect is recorded.

  According to the first and third aspects of the present invention, the likelihood of the emotional expression state is automatically calculated from the video or audio, and the edited content can be automatically generated based on the calculated likelihood of automatic expression. .

  According to the second and fourth aspects of the present invention, the likelihood of the emotional expression state is automatically calculated from video or audio. Based on this certainty, the user can determine which section is in what emotional state, can generate edited content including the desired section, and can support editing.

  According to the invention of claim 5, an information editing program capable of automatic editing or editing support is provided.

  According to the sixth aspect of the invention, the information editing program according to the fifth aspect can be incorporated into a computer as a recording medium.

(1) According to the first and third aspects of the invention, since automatic editing of content can be executed based on the likelihood of the emotional expression state, automatic editing that appropriately includes a section that the user wants to watch is performed. Can be executed.
(2) According to the inventions of claims 2 and 4, the user can select a section to be included in the edited content while referring to a representative image to which processing and information are given based on the likelihood of the emotional expression state. Therefore, without confirming the contents of each section, it is possible to grasp the section that is in the emotional expression state efficiently, and efficiently execute the editing including the section that you want to watch. Can be supported.
(3) According to the invention of claim 5, an information editing program capable of automatic editing or editing support can be provided.
(4) According to the invention of claim 6, the invention of claim 1 or 2 can be realized on a computer.

  These can contribute to the information editing field.

Hereinafter, embodiments of the present invention will be described with reference to the drawings, but the present invention is not limited to the following embodiments.
[First example of embodiment]
The first example of the embodiment of the present invention is a case where the present invention is implemented as an apparatus for executing editing support, and corresponds to claims 2 and 4. FIG. 1 shows an example of an information editing apparatus 1 according to this embodiment.

  The information editing apparatus 1 includes a content input unit 11 to be edited, a storage device 12 for storing video / audio data 10 input from the content input unit 11, and one or more of video information and audio information. Using the section dividing means 13 that divides the content data into predetermined section units, and using one or more of video information and audio information, an emotional expression is provided for each divided section. An emotion detection means 14 for calculating the degree of emotion which is the probability of the state, an image output means 15 for determining and outputting a representative image for each section using one or more of video information and audio information; Based on the emotion level, the image processing means 16 for processing the representative image, the layout means 17 for laying out the representative image and presenting it to the user, and the reproducing means for reproducing the section corresponding to the representative image designated by the user 8, by joining a section corresponding to the representative image selected by the user, and a generation means 19 for generating an edited content, each function is performed for example by a computer.

  FIG. 2 illustrates an example of a processing flow of the information editing apparatus 1 according to the present embodiment. Based on this processing flow, the detailed operation of the information editing apparatus 1 shown in FIG. 1 will be described.

  In step S21, when the content input unit 11 accepts video / audio data in response to a request from the user, the content input unit 11 stores it in the storage device 12 in step S22.

  Next, in step S23, the section dividing unit 13 divides the video / audio data into predetermined section units based on one or more of video information and audio information.

  As a method of segmentation, there is a method using a media processing technique.

  For example, when video information is used, a cut point of the video is detected, and a portion surrounded by the cut points is defined as a section. At this time, the section unit is a shot. As a method for detecting the cut point, for example, the method described in Non-Patent Document 5 can be used.

  For example, when voice information is used, voiced / unvoiced determination using voice spectrum information is performed, a voiceless part continuing for a certain amount or more is detected, and a part surrounded by the voiceless part is defined as a section. At this time, the section unit is a continuous voiced section.

  The voiced / unvoiced determination may be executed based on, for example, the presence / absence of a fundamental frequency in addition to the spectrum information.

  When using spectral information, calculate one or more frequencies with peaks in the power spectral density, and if these frequencies are within a predetermined frequency band, they are voiced, otherwise they are silent What is necessary is just to determine as an area. As a frequency band, for example, when a frequency having a peak between 50 and 350 Hz (Hertz) is observed, it may be determined as a voiced section.

  In the case of using the fundamental frequency, for example, the fundamental frequency is obtained by using the method described in Non-Patent Document 1, and when the value is in a range that falls within a predetermined frequency band, the voiced interval is used. May be determined as a silent section. As a frequency band, you may set as 50-350 Hz (hertz) etc., for example.

  Further, for example, a threshold value may be set for the absolute value of the amplitude value of the speech waveform, and when this condition continues for a predetermined time, it may be the end / start of the section.

  When the input content is audio, segment division may be performed using audio information, and when the content is video, segment division may be performed using video information. Further, if possible, both the shot and the continuous voiced section may be adopted as a section unit.

  When there are one or more input video or audio, the section division method may be applied to each video or audio.

  Next, in step S24, the emotion detection means 14 calculates the emotion level for each section based on one or more of video information and audio information.

  In an example of the present embodiment, as the emotion expression state to be detected, for example, “fun”, “sad”, “scary”, “violent”, “cool”, “cute”, “exciting”, “passionate”, There are "romantic", "violent", "calm", "healed", "warm", "cold", "creepy", "laughter". In step S24, for each emotion expression state, an emotion level representing the certainty is calculated.

  Here, the emotion detection means 14 according to the first example of the embodiment of the present invention is configured as the emotion detection means 3 shown in FIG. 3, for example.

  The emotion detection unit 3 includes a feature amount extraction unit 31 that extracts a feature amount for each analysis frame from video and audio signal data included in the content, and a statistical model that calculates a probability of an emotion expression state based on the feature amount. Emotion probability calculation means 32 provided with, emotion level calculation means 33 for calculating the emotion level of the section based on the emotion probability, and emotion determination for determining the emotion expression state of the section based on the emotion level Consists of means 34.

  Here, the statistical model is created by learning before actually executing editing according to the present invention.

  FIG. 4 shows an example of the processing flow of the emotion detection means 3 according to the first example of the embodiment of the present invention. Based on this processing flow, the detailed operation of the emotion detection means 3 shown in FIG. 3 will be described.

  When calculating the emotion level for each section, the necessary video feature amount and audio feature amount are extracted, based on video information, audio information, or both, respectively. Calculate feelings.

  In step S41, the feature extraction unit 31 calculates and extracts a desired audio feature amount for each analysis frame (hereinafter referred to as a frame) from the audio signal data of the captured content.

  An example of a method for extracting each feature amount will be described.

  First, an example of a method for extracting video feature amounts will be described.

  In one example of the present embodiment, as the video feature amount, a shot length, a color histogram, a time variation characteristic of the color histogram, a motion vector, and each time variation characteristic are extracted. An example of the time variation characteristic is an inter-frame difference, for example.

  The video feature amount is extracted for each video frame. At this time, for the purpose of shortening the processing time, it may be devised to extract every several frames, for example, every 5 frames.

  Here, there are various basic extraction methods such as a shot length, a motion vector, and a color histogram, but these are known, and for example, a method shown in Non-Patent Document 5 or the like can be used.

  Since it is practically impossible to extract the shot length within a single video frame, for example, it may be extracted as the length of a shot including the target video frame. Further, an average value, maximum value, minimum value, or the like of the shot length in a certain section including one or more shots may be used.

  The color histogram is extracted as follows, for example.

  A hue (Hue) is extracted for each pixel in the video frame. This hue is quantized to a predetermined number Q such as 11 or 256, for example, so that it can be determined which of Q quanta corresponds to all pixels. By executing this over all the pixels and counting the number of appearances for each quantum, the hue histogram of the video frame can be extracted.

  At this time, the entire video frame may be divided into one or more regions, and a histogram may be extracted for each region.

  The motion vector can be extracted as a vector composed of an X component and a Y component, for example, by calculating an optical flow vector. As a method for calculating the optical flow vector, for example, the method described in Non-Patent Document 6 or the like can be used. In addition, for example, the vector norm may be calculated for each video frame, or camera operations such as panning, tilting, and zooming are detected using the method disclosed in Non-Patent Document 5 and the like. It may be quantified as an operation amount per unit time.

  An example of a method for extracting speech feature amounts will be described.

  The voice feature amount is, for example, a fundamental frequency, a temporal variation characteristic of the fundamental frequency, an rms value of the amplitude, a temporal variation characteristic of the rms value of the amplitude, a power, a temporal variation characteristic of the power, a speech speed, or a temporal variation characteristic of the speech speed It shall consist of one or more elements.

  Details of an example of a method for extracting each voice feature amount will be described.

  The feature quantity used in an example of the embodiment of the present invention for various contents is as follows, even for speech mixed with various sound source factors, compared to phonological information that requires analysis of high-dimensional speech parameters. Those obtained stably are preferred.

  For example, a method of converting speech into text information using speech recognition or the like requires such phonological information, and is effective for, for example, content in a genre where a speaker's speech such as a news video can be clearly heard. .

  However, in movies, dramas, home videos, etc., there are various sound source factors such as background music and environmental sounds in addition to utterances, so the utterances cannot be heard clearly and speech recognition is difficult. . Furthermore, since it is not always possible to determine the emotional expression state of content based on utterances, music, sound effects, environmental sounds, etc. can be handled in order to detect emotional expression states including impressions and atmosphere. An audio feature that can be used is necessary.

  Therefore, in the first example of the present embodiment, the prosody information, in particular, the fundamental frequency, the time variation characteristic of the fundamental frequency, the rms value of amplitude (hereinafter simply referred to as rms), the time variation characteristic of rms, power, and the time of power. Extract fluctuation characteristics, utterance speed, and temporal fluctuation characteristics of utterance speed.

  In particular, by using several types of short-time change amounts as the time variation characteristics, it is possible to detect important behaviors in emotional speech when emotions included in content are extracted.

  Examples of time variation characteristics include inter-frame differences and regression coefficients. The power may be a power spectral density. There are various methods for extracting the fundamental frequency and power, but they are known. For details, see the method described in Non-Patent Document 1, for example.

  Further, the speech speed including the speech speed, music rhythm, tempo, and the like can be extracted as a dynamic scale by the method disclosed in Non-Patent Document 2, for example. For example, it is possible to detect the speech rate by detecting the peak of the dynamic scale and counting the number, and calculating the average value and variance value of the peak interval corresponding to the temporal variation characteristic of the speech rate. Thus, a method of detecting the temporal variation characteristic of the speech rate may be taken. Hereinafter, in the first example of the embodiment of the present invention, the peak interval average value of the dynamic scale is used as the speech rate.

  An example of a method for extracting these audio feature amounts for each frame will be described. The length of one frame (hereinafter referred to as the frame length) is, for example, 50 ms, and the next frame is formed by a time shift of, for example, 20 ms with respect to the current frame.

  As shown in FIG. 5, for each of these frames, the average value of each voice feature amount in each frame, that is, the average fundamental frequency, the average frequency variation characteristic of the fundamental frequency, the average rms, the average time variation characteristic of rms, It is assumed that average power, average time variation characteristic of power, average peak interval average value of dynamic scale, and the like are calculated. Alternatively, not only the average value but also the maximum value, the minimum value, or the fluctuation range of each voice feature amount in the frame may be calculated and used.

  Here, it is often difficult to extract the fundamental frequency itself in the voice that appears characteristically in the emotional expression part of the content, and it is often lost. For this reason, it is preferable to include the time-variation characteristic of the fundamental frequency that can easily obtain the effect of complementing such defects.

  Furthermore, it is preferable to further include a time variation characteristic of power in order to improve the determination accuracy while keeping speaker dependency low. As described above, the audio feature amount extraction processing performed for each frame is performed on the frames over the entire content, so that the audio feature amounts can be obtained in all the frames.

  When the emotion level is calculated using both the video feature value and the audio feature value, it is preferable to unify the frame used for the video feature value extraction and the frame used for the audio feature value extraction.

  As an example of a method for implementing this, for example, there is a method in which a video feature value is converted according to a frame used for audio feature value extraction, and recalculated with a frame width of 50 ms and a time shift of 20 ms.

  Next, in step S42, based on the feature amount extracted for each frame in step S41, the emotion probability calculation means 32 calculates the probability of the emotion expression state to be detected as the emotion probability.

  The emotion probability is calculated for each emotion category using the statistical model by associating the category of the emotion expression state (emotion category) to be detected with the statistical model. As this statistical model, for example, a normal distribution, a mixed normal distribution, a hidden Markov model, a generalized state space model, or the like is used. Preferably, a generalized state space model that can model the temporal transition of emotion is adopted.

  As a method for estimating a parameter of a statistical model, for example, a maximum likelihood estimation method, an EM algorithm, a variational Bayes method, and the like are known and can be used. For details, refer to Non-Patent Document 3, Non-Patent Document 4, and the like.

  Next, in step S43, based on the emotion probability calculated in step S42, the emotion level calculation means 33 calculates the emotion level for each section.

  An example of a method for calculating the emotion level for each section will be described.

Each emotion category is expressed in order as e ¹ , e ² ,..., And the number of emotion categories is expressed as # (K). Assume that a set of sections S in the content is {S ₁ , S ₂ ,..., S _NS } in order from the earliest time. Here, NS is the total number of sections.

Frames included in a certain section S _i are set as {f ₁ , f ₂ ,..., F _NFi }. Here, NFi is the number of frames included in the section S _i . For each frame f _t , the emotion probability pf _t (e ^k ) of the ^kth emotion category ek on a frame basis is obtained in step S42. Using this, the emotion level pS _i (e ^k ) in the section S _i of the ^kth emotion category ek represents, for example, the average emotion probability in the section S _i .

  Or the following formula representing the maximum value.

The averaging performed by the equation (1) may each section S _i further divided to generate a small section, or may be to run on this small section units.

In this small section, for example, a small section width of 1 second and a small section shift are set to 0.5 seconds, and averaging is performed for each small section. Then, the average value of all the small sections included in the section S _i may take the maximum value as the emotion level of the section S _i . In addition to these, for example, there are various methods such as calculating the emotion level after performing filtering within the interval, but the emotion level may be compared between the intervals, so the emotion level may be constant. It is preferable to be within the range of values, for example, between 0 and 1.

  By performing the above calculation over all the sections, the emotion level for each emotion category can be calculated for all the sections.

  Next, in step S44, the emotion determination means 34 determines the emotion expression state for each section based on the emotion level calculated in step S43.

  As a method for implementing this, for example, an emotion category having the maximum emotion level is obtained as the emotion of the section.

  This makes it possible to determine what emotional expression state prevails in each section.

  When a plurality of images or sounds are input, these images or sounds may be continuously connected and regarded as one image or sound, and the emotion level may be calculated.

  Next, in step S25 of FIG. 2, the image output means 15 selects and outputs a representative image for each section.

  First, an example of processing for selecting a representative image for a video will be described.

Selection of the representative image, for example, to select the top frame when arranged all of the video frames included in a certain section S _i in the time direction, the center of the frame, to extract and end of the frame, it as a representative image But you can.

  Further, the representative image may be extracted based on the emotion expression state, the emotion level, and the emotion probability obtained in step S24.

  For example, a processing method based on the emotion expression state and the emotion probability will be described with reference to FIG.

First, to get the emotional expression status of the section S _i for selecting a representative image, emotion probability corresponding to the emotional expression state obtains time T _Frame taking the maximum value.

Next, the video frame closest to the time T _Frame is selected as the representative image 62 from all the video frames 61 included in the section.

  By this method, it is possible to select the video frame that is assumed to have the strongest emotional expression state in that section as the representative image, and confirm the contents of the section in an easy-to-understand manner can do.

  When the input file is audio, an icon 71 indicating audio as shown in FIG. 7 may be used as the representative image as a representative image, or the audio waveform 72 of the section is used as the representative image. It may be.

  Next, in step S26, the image processing means 16 processes the representative image based on the emotion level for each section.

  An example of processing for processing a representative image will be described with reference to FIG.

  First, processing contents are determined in advance for each emotion category according to the level of emotion level. For example, a yellow outer frame may be assigned to the “fun” emotion category, and a blue outer frame may be similarly applied to the “sad” emotion category. Further, the size of the representative image may be increased when the emotion level is high, and the size of the representative image may be decreased when the emotion level is low. In addition to this, image transmissivity, contrast, brightness, hue, number of pixels, and the like may be set as processing items.

For example, when the emotion level pS _i (e ^k ) in the section S _i of the emotion category e ^k takes a value between 0 and 1, the determination is made as follows: 0 ≦ pS _i (e ^k ) < “Low” if 0.3, “Medium” if 0.3 ≦ pS _i (e ^k ) <0.7, “High” if 0.7 ≦ pS _i (e ^k ) ≦ 1.0. Or the like.

  In the stage of actual processing, first, the emotion category that is the most prevalent emotion expression state is obtained. In this method, the emotion category having the highest emotion level may be the most dominant emotion category.

  Next, this dominant emotion category and processing corresponding to the emotion level may be performed on the representative image.

  In addition to the processing as described above, an icon may be assigned.

  Even when this processing is performed, an icon to be assigned is determined in advance for each emotion category according to the level of emotion.

  An example in which an icon is added to the representative image is shown in FIG. For example, when the emotion category of the section corresponding to the representative image is “fun” and the emotion level is “high”, for example, an icon 91 indicating “smile” is added to the representative image 92. For example, when the emotion category is “sad” and the emotion level is “high”, for example, an icon 93 indicating “crying face” is added to the representative image 94.

  By giving such an icon, it is possible to intuitively understand the emotional state of each section, and the confirmation work can be made more efficient.

  Next, in step S27, the layout means 17 lays out and outputs each representative image and presents it to the user.

  An example of this processing method will be described with reference to FIG.

  In the example shown in the figure, the icon and the size of the representative image are processed by the above-described method according to the emotion level. For example, the horizontal (left / right) axis 101 of the presentation plane 100 may be taken as the time axis, and the layout may be sequentially performed from the left in accordance with the time information of the section.

  Furthermore, as shown in FIG. 11, the vertical (vertical) axis 111 may be taken as the emotion level, and a layout reflecting the emotion level may be performed. For example, the section corresponding to the representative image 112 may indicate “high”, the section corresponding to the representative image 113 may be “low”, and the section corresponding to the representative image 114 may be “medium”.

  Further, for example, the processed representative image may be laid out by taking the vertical (up / down) axis of the presentation plane as the time axis and the horizontal (left / right) axis as the axis representing the angle of view of the camera work by panning. In addition, each axis may indicate the angle of view of camerawork such as tilt or the degree of emotion.

  Furthermore, for example, as shown in FIG. 12, the presentation plane 120 is a group of a series of sections, and a presentation area 121 for presenting a representative image of a section belonging to the first group in a unit of the group. The layout position may be divided, such as a presentation area 122 that presents a representative image of a section belonging to the second unit.

  The group used at this time may be a group surrounded by a special cut such as a gradual cut such as a dissolve by using video information, or may be silence for a certain period of time, for example, one second or more. It is good also as a unit surrounded by a section.

  The number of groups is shown in FIG. 12 as two examples. However, the number of groups is not necessarily two. The number of groups is changed according to the input video and audio files, the section division, and the method of configuring the groups. It is.

  With these layouts, the user can grasp the entire section more in a list than in the case where all are presented in a horizontal row. Further, for example, the representative images that have been laid out may be divided into a group when they reach the right end 123 of the presentation plane.

  In addition, for example, similarly to the method of assigning an icon, it may be divided at a point where the emotion level is “high”, and the column is changed after that. That is, the end of the line is a section where the emotion level is “high”, that is, the emotional expression is large. By this method, it is possible to include a series of contexts in which emotional expression greatly appears for each column.

  In addition, when one or more input original video or audio exists, these video or audio are continuously connected and regarded as one video or audio and displayed on the presentation plane 100 of FIG. May be.

  Moreover, it is good also as what switches each presentation area and presents. An example of this layout method will be described with reference to FIG.

  FIG. 13 shows a case where three contents, which are video or audio, are input. At this time, for example, three presentation planes 130 are installed for each content, and by pressing a presentation plane switching tab, the displayed presentation planes are switched and presented.

  Next, when the user requests playback of the section, the playback means 18 executes playback of the designated section in step S28. That is, it is determined whether or not a section playback request has been accepted (step S28a), section playback (step S28b), and whether or not to end representative image selection (step S28c).

  As an example of a method for requesting playback of a section, for example, when a user designates a representative image using a pointing device or the like, the request may be made with that.

  An example of a method for executing reproduction from an arbitrary section in accordance with a user's reproduction request will be described.

  A user requests playback of a section by designating a representative image or the like. At this time, each representative image is associated in advance with a video frame serving as a reproduction start point of the corresponding section or a timeline time. With this method, when a representative image is specified, it is possible to determine which frame position in the video or playback should start from time, so playback can be performed from any section by seeking the playback position. Can do.

  As an example of another method, there is a method in which all sections are individually converted into video and moving image files in advance. For example, the first video frame to the last video frame of each section are cut out and stored as a single video file. The video file corresponding to the section generated in this way may be associated with the representative image corresponding to the section, and the corresponding video file may be reproduced when the representative image is designated.

  Finally, in step S29, when the user selects a representative image and receives a request to finish selecting a section to be used for the edited content, the generation unit 19 generates the edited content by connecting the selected sections. To do.

  As an example of this connection method, for example, connection may be performed in order from the earliest time on the timeline of the original video, or, for example, connection may be performed in order from a section having a higher emotion level or a section having a lower emotion level. Moreover, you may design so that a user can designate the connection order.

The above is an example of the information editing apparatus and the information editing method according to the first example of the embodiment of the present invention.
[Second Example of Embodiment]
The second example of the embodiment of the present invention is a case where it is implemented as an apparatus for executing automatic editing, and corresponds to claims 1 and 3.

  FIG. 14 shows an example of the information editing apparatus 140 according to this embodiment.

  The information editing device 140 includes a content input unit 141 to be edited, a storage device 142 that stores the video / audio data 10 input from the content input unit 141, and one or more of video information and audio information. Using the section dividing unit 143 that divides the content data into predetermined section units, and using one or more of video information and audio information, an emotional expression is provided for each divided section. Emotion detecting means 144 for calculating the emotional degree that is the probability of the state, selecting means 145 for selecting the sections based on the emotional degree, and generating means 146 for connecting the selected sections to generate edited content; The functions are executed by a computer, for example.

  FIG. 15 illustrates an example of a processing flow of the information editing apparatus 140 according to the present embodiment. Based on this processing flow, the detailed operation of the information editing apparatus 140 shown in FIG. 14 will be described.

  Each process of step S151 to step S154 may be exactly the same as step S21 to step S24 of FIG. 2 shown in the description of the first example of the embodiment of the present invention.

  After completing each process up to step S154, in step S155, the selection unit 145 selects a section to be included in the edited content based on the emotion level calculated for each section. In step S154, the emotion level is obtained for the emotion category of each section as shown in FIG.

  An example of the selection process will be described.

  As the section included in the edited content, for example, a section having a high emotion level is adopted. This method will be described. First, reference is made to the emotion category and emotion level of each section shown in FIG. Among these, for example, ranking is performed for the emotion level of the “fun” emotion category, and the ranking is adopted in order from the section having the highest value until the number of sections determined as the upper limit is reached.

  In the case of emotion categories other than “fun”, the same process may be performed.

  As the method of determining the upper limit section number, it may be determined in advance according to the specifications of the system to be used, or a user request may be accepted and a desired number may be set.

  Further, the time of the edited content may be set as the upper limit, not the upper limit of the number of sections. In this case, if the time for each section is calculated separately, the time for the entire adopted section can be calculated.

  Even when the upper limit is set for the edited content time, as with the upper limit of the number of sections, it may be determined in advance according to the specifications of the system to be used, etc. May be set as the upper limit.

  Next, in step S156, the generation unit 146 connects the selected sections and generates edited content.

  As an example of this connection method, for example, connection may be performed in order from the earliest time on the timeline of the original video, or, for example, connection may be performed in order from a section having a higher emotion level or a section having a lower emotion level.

  Heretofore, an example of the method in the embodiment of the user support method according to the present invention has been described in detail.

  Further, in the information editing method described above, it is possible to determine what emotional state is in each section. Based on this, editing content may be further processed.

  For example, if the edited content includes a lot of fun sections when video is input, a pleasant background music is given.

  As an example of this method, the ratio of the number of sections of the edited content in which the “fun” emotion level is “high” with respect to the total number of sections is obtained, and if this ratio is a certain level or more, background music may be added. .

  Other than the example shown as the embodiment of the present invention, the embodiment can be appropriately changed within the scope of the embodiment that can be taken based on the principle of the present invention.

  Hereinafter, specific examples of the present invention will be described.

  In this example, the function described in the first example of the embodiment of the present invention is used as application software that operates on a local PC.

In the first embodiment, a storage medium storing a program for executing an information editing method according to the present invention is mounted on a local PC. The implementation procedure is as follows.
[Procedure 1] The user designates an arbitrary video or audio file and reads it by a program.
[Procedure 2] The information editing program operates, and after segmentation, emotion detection, determination of the representative image, and icon assignment to the representative image in accordance with the processing described in FIG. 2, the layout is performed as shown in FIG. And present it to the user. In the screen of FIG. 17, sections are laid out in order of timelines for the video file.
[Procedure 3] A user previews a section, determines a section to be adopted for editing content by selecting a representative image, and notifies the end of editing by pressing an editing start button.
[Means 4] The information editing program operates to connect the sections adopted by the user and output a video or audio file of the edited content.

  In this example, the function described in the first example of the embodiment of the present invention is used as application software that operates on a server connected via the Internet network.

  A specific configuration of the apparatus 180 according to the second embodiment will be described with reference to FIG.

The server device 181 and a plurality of user terminals 182a, 182b,... Are connected in a form that allows mutual communication via the Internet network. The server device 181 includes a storage unit 183, a control unit 184, and a database 185. The storage unit 183 is a storage medium that stores the information processing method according to the present invention as a computer-readable information editing program. The control unit 184 includes a CPU 186, a RAM 187, a ROM 188, and the like, and executes various arithmetic processes of server processing. The database 185 stores user identification information, video content uploaded by each user, edited content, and the like. The implementation procedure is as follows.
[Procedure 1] The user designates an arbitrary video or audio file through the user terminals 182a, 182b, and uploads it to the server device 181.
[Procedure 2] When the server device 181 confirms uploading of the video or audio file, the information editing program in the storage unit 183 operates, and in accordance with the processing described in FIG. 2, section division, emotion detection, representative image determination, After the icon is assigned to the representative image, it is laid out as shown in FIG. 17 and presented to the user who uploaded the content.
[Procedure 3] A user previews a section, determines a section to be adopted for editing content by selecting a representative image, and notifies the end of editing by pressing an editing start button.
[Procedure 4] The information editing program operates, connects the sections adopted by the user, outputs a video or audio file of the edited content, and stores it in the database 185.

  In this example, the function described in the second example of the embodiment of the present invention is used as application software that operates on a server connected via the Internet network.

The specific configuration of the apparatus according to the third embodiment is the same as the configuration diagram 18 of the apparatus according to the second embodiment. The implementation procedure is as follows.
[Procedure 1] The user designates an arbitrary video or audio file through the user terminals 182a, 182b, and uploads it to the server device 181. At this time, the upper limit time of the edited content and the emotion category of the section adopted for the edited content are designated.
[Procedure 2] When the server device 181 confirms uploading of the video or audio file, the information editing program in the storage unit 183 operates and executes section division and emotion detection according to the processing described in FIG.
[Procedure 3] Among all the sections, the sections are adopted in descending order of the emotion level of the emotion category designated by the user until the length reaches the upper limit time.
[Procedure 4] The adopted sections are connected, and the video or audio file of the edited content is output and stored in the database 185.

  In this example, the function described in the first example of the embodiment of the present invention is used as application software that operates on a server connected via the Internet network. The difference from the second embodiment is that a plurality of videos or sounds are input and finally one edited content is generated.

The configuration of the apparatus in Example 4 is the same as that of the first example of the embodiment, and will be described with reference to FIG. The implementation procedure is as follows.
[Procedure 1] The user designates one or more arbitrary video or audio files and uploads them to the server device 181.
[Procedure 2] When the server device 181 confirms uploading of the video or audio file, the information editing program in the storage unit 183 operates, and in accordance with the processing described in FIG. 2, section division, emotion detection, representative image determination, After the icon assignment to the representative image, each video or audio file is laid out and displayed as shown in FIG. In the example of FIG. 19, an example in which three video or audio files are input is illustrated. In this example, a file to be presented can be switched by a presentation plane switching tab installed at the upper part of the presentation plane.
[Procedure 3] A user previews a section, determines a section to be adopted for editing content by selecting a representative image, and notifies the end of editing by pressing an editing start button.
[Procedure 4] The information editing program operates, connects the sections adopted by the user from each video or audio file, outputs the video or audio file of the edited content, and stores it in the database 185.

  Note that the present invention can be realized by configuring some or all of the functions of each means in the information editing apparatus of the present embodiment by a computer program and executing the program using the computer. It goes without saying that the procedure in the information editing method can be constituted by a computer program, and the program can be executed by the computer, and the computer-readable recording medium, for example, FD, can be realized by the computer. (Floppy (registered trademark) Disk), MO (Magneto-Optical disk), ROM (Read Only Memory), memory card, CD (Compact Disk) -ROM, DVD (Digital Versatile Diary) k) -ROM, CD-R ,, CD-RW, HDD, and recorded in a removable disk, or stored, it is possible to or distribute. It is also possible to provide the above program through a network such as the Internet or electronic mail.

The block diagram explaining the structure of the information editing apparatus which concerns on the 1st example of embodiment of this invention. The flowchart explaining the flow of the process which concerns on the 1st example of embodiment of this invention. The block diagram explaining the emotion detection means in embodiment of this invention. The flowchart explaining the flow of a process of the emotion detection means in embodiment of this invention. The flowchart explaining the audio | voice feature-value extraction method in embodiment of this invention. Explanatory drawing explaining the representative image selection method in embodiment of this invention. Explanatory drawing which shows an example of the representative image when the audio | voice content in embodiment of this invention is input. Explanatory drawing explaining the processing rule of the representative image in embodiment of this invention. Explanatory drawing explaining the method to implement icon provision to the representative image in embodiment of this invention. Explanatory drawing which shows an example of the method of the layout in embodiment of this invention. Explanatory drawing which shows the other example of the method of the layout in embodiment of this invention. Explanatory drawing which shows the other example of the method of the layout in embodiment of this invention. Explanatory drawing which shows the method of layout when the some video or audio | voice content in the embodiment of this invention is input. The block diagram explaining the structure of the information editing apparatus which concerns on the 2nd example of embodiment of this invention. The flowchart explaining the flow of the process which concerns on the 2nd example of embodiment of this invention. Explanatory drawing explaining the matching of the emotion category for every area and emotion level in embodiment of this invention. Explanatory drawing which shows the layout screen which concerns on Example 1 thru | or 3 of this invention. The block diagram explaining the structure of the apparatus which concerns on Example 2 thru | or 4 of this invention. Explanatory drawing which shows the layout screen which concerns on Example 4 of this invention.

Explanation of symbols

  DESCRIPTION OF SYMBOLS 1,140 ... Information editing device, 3, 14, 144 ... Emotion detection means, 11, 141 ... Content input part, 12, 142 ... Memory | storage device, 13,143 ... Section division means, 15 ... Image output means, 16 ... Image Processing means, 17 ... Layout means, 18 ... Reproduction means, 19,146 ... Generation means, 31 ... Feature quantity extraction means, 32 ... Emotion probability calculation means, 33 ... Emotion degree calculation means, 34 ... Emotion determination means, 145 ... Selection Means, 181 ... server device, 182a, 182b ... user terminal, 183 ... storage unit, 184 ... control unit, 185 ... database.

Claims (6)

A device for editing one or more video or audio,
Section dividing means for dividing one or more video or audio into predetermined section units;
Emotion detection means for determining the probability of one or more emotion expression states using one or more of video information and audio information for each of the divided sections;
Automatic generation means for generating video or audio by connecting one or more of the sections based on the probability;
An information editing apparatus comprising: A device for editing one or more video or audio,
Section dividing means for dividing one or more video or audio into predetermined section units;
Emotion detection means for determining the probability of one or more emotion expression states using one or more of video information and audio information for each of the divided sections;
Output means for outputting a representative image or video or audio for each of the divided sections, or a representative image and video or audio for each of the divided sections, for each of the divided sections;
Image processing means for executing one or more of processing and information provision on the representative image based on the probability;
Layout means for spatially laying out the processed representative image and outputting it to a user;
Generating means for connecting the sections corresponding to at least one representative image selected by the user to generate video or audio;
An information editing apparatus comprising: A method of editing one or more video or audio comprising:
A section dividing step in which the section dividing means divides one or more video or audio into predetermined section units;
An emotion detecting step for determining the probability of one or more emotion expression states using one or more of video information and audio information for each of the divided sections;
An automatic generation step, wherein the automatic generation means connects one or more of the sections based on the certainty to generate video or audio;
An information editing method comprising: A method of editing one or more video or audio comprising:
A section dividing step in which the section dividing means divides one or more video or audio into predetermined section units;
An emotion detecting step for determining the probability of one or more emotion expression states using one or more of video information and audio information for each of the divided sections;
An output unit that outputs a representative image or video or audio for each of the divided sections, or a representative image and video or audio for each of the divided sections for each of the divided sections;
An image processing step in which the image processing means executes one or more of processing and information addition on the representative image based on the certainty;
A layout step in which the layout means spatially lays out the processed representative image and presents it to the user;
A generating step of generating video or audio by connecting the sections corresponding to at least one representative image selected by a user;
An information editing method comprising:   An information editing program for causing a computer to function as each means according to claim 1 or 2.   A computer-readable recording medium on which the information editing program according to claim 5 is recorded.