JP2013207530A - Information processing device, information processing method and program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To generate a digest by which a rough content of time series data can be grasped.SOLUTION: From each chapter obtained by segmenting time series data constituted by a plurality of time series data, a chapter segment extraction unit extracts a chapter segment that represents a predetermined chapter portion. Among each chapter obtained by segmenting the time serial data, a feature peak segment extraction unit extracts a feature segment. An effect addition unit combines the chapter segment with the feature segment in the order of time series, to generate a digest that reflects a rough content of the time series data. The present technique is applicable to a recorder for recording contents and the like.

Description

  The present disclosure relates to an information processing device, an information processing method, and a program, and in particular, for example, an information processing device, an information processing method, and an information processing method that can easily generate a digest that understands the rough contents of time-series data such as content. Regarding the program.

  For example, there is a digest generation technique for generating a digest in which a rough content of a sports broadcast program is reflected by detecting a highlight scene from a sports broadcast program that relays sports such as soccer.

  In this digest generation technology, for example, in a sports broadcast program, a scene where a cheer is raised is detected as a highlight scene of a sports program by using a feature value that characterizes the sound excitement (cheer) (see, for example, Patent Document 1). .

  A digest is generated by connecting the detected highlight scenes.

JP 2008-185626 A

  However, in the above-described digest generation technique, when the digest generation target is a content such as a news program or a moving image shot by an individual, the scene where the cheer is raised is not necessarily a highlight scene.

  For this reason, depending on the genre of the content, it may not be possible to generate a digest reflecting the general content.

  The present disclosure has been made in view of such a situation, and makes it possible to easily generate a digest in which rough contents of time-series data such as content can be understood.

  An information processing apparatus according to an aspect of the present disclosure includes a chapter segment representing a predetermined portion representing the chapter from each chapter obtained by dividing time-series data including a plurality of pieces of data arranged in time series. A chapter segment extracting unit for extracting, a feature segment extracting unit for extracting the feature segment from chapters having a feature segment representing a characteristic part of the chapter among the chapters obtained by dividing the time series data, and An information processing apparatus including a generation unit that generates a digest reflecting a rough content of the time-series data by combining the chapter segments and the feature segments in a time-series order.

  The generation unit can generate the digest having a length set by a user's setting operation by combining the chapter segment and the feature segment in chronological order.

  Based on the time series data, a symbol string generation unit that creates a symbol string in which symbols representing the attributes of the plurality of data are arranged in time series, and the time series data based on the distribution of symbols in the symbol string And a partitioning section for partitioning into a plurality of chapters.

  The division unit can divide the time-series data into chapters having the number of divisions based on the length set by the user's setting operation based on the variance of each symbol constituting the symbol string.

  A feature quantity extraction unit that extracts a feature quantity representing the feature of the time series data from the time series data can be further provided, and the feature segment extraction section includes a chapter having the feature segment based on the feature quantity. From the above, the feature segment can be extracted.

  The feature segment extraction unit may extract, from the chapter, the feature segment including a location where the feature amount is one of maximum or maximum in a section from the start to the end of the chapter based on the feature amount. it can.

  In the feature segment extraction unit, based on the feature amount, the feature amount is one of the maximum or maximum in a section from the start to the end of the chapter, and the feature amount is determined in advance. The feature segment including a portion that is equal to or greater than the threshold can be extracted from the chapter.

  The feature segment extraction unit includes a portion where the maximum feature amount is maximized in a section from the start to the end of the chapter among the plurality of different feature amounts based on the plurality of different feature amounts. The feature segment can be extracted from the chapter.

  The generation unit can generate the digest in which a voice prepared in advance with a corresponding weight is added to each of the chapter segment and the feature segment.

  The feature segment extraction unit extracts the feature segment from chapters having the feature segment based on a plurality of different feature amounts, and the generation unit extracts a voice feature from the plurality of different feature amounts. The digest in which the voice is added to the feature segment extracted based on the feature quantity to be expressed with a weight smaller than that of the other feature segment can be generated.

  The generating unit can generate the digest to which the voice is added with a weight that is continuously changed and switched.

  An information processing method according to an aspect of the present disclosure is an information processing method of an information processing device that generates a digest, and classifies time-series data including a plurality of data arranged in time series by the information processing device. A chapter segment extraction step for extracting a chapter segment representing a predetermined part of the chapter from each obtained chapter, and a characteristic portion of the chapter among the chapters obtained by dividing the time-series data. A feature segment extracting step for extracting the feature segment from the chapter having the feature segment, and a digest reflecting a rough content of the time series data by combining the chapter segment and the feature segment in a time series order. An information processing method including a generation step of generating.

  A program according to an aspect of the present disclosure extracts a chapter segment representing a predetermined portion of the chapter from each chapter obtained by dividing time series data including a plurality of pieces of data arranged in time series by a computer. A feature segment extraction unit that extracts a feature segment from a chapter having a feature segment representing a characteristic part of the chapter among the chapters obtained by dividing the time series data, and This is a program for functioning as a generation unit that generates a digest reflecting the rough contents of the time-series data by combining the chapter segments and the feature segments in time-series order.

  According to the present disclosure, a chapter segment representing a predetermined portion of the chapter is extracted from each chapter obtained by dividing time-series data including a plurality of data arranged in time series, and the time-series data Among the chapters obtained by classifying the feature segments, the feature segments are extracted from the chapters having the feature segments representing the characteristic portions of the chapters, and the chapter segments and the feature segments are combined in a time-series order. Thus, a digest reflecting the rough contents of the time series data is generated.

  According to the present disclosure, it is possible to easily generate a digest in which rough contents of time-series data such as content can be understood.

It is a block diagram which shows the structural example of the recorder which is 1st Embodiment. It is a figure which shows an example of the symbol sequence which the symbol sequence production | generation part of FIG. 1 produces | generates. It is a block diagram which shows the structural example of the content model learning part of FIG. FIG. 3 is a diagram illustrating an example of a left-to-right type HMM. It is a figure which shows an example of an Ergodic type HMM. FIG. 3 is a diagram illustrating an example of a two-dimensional neighborhood constrained HMM that is an HMM having a sparse structure. It is a figure which shows an example of HMM of a sparse structure other than a two-dimensional neighborhood constraint HMM. It is a figure which shows the process of extraction of the feature-value by the feature-value extraction part of FIG. It is a flowchart for demonstrating the content model learning process which the content model learning part of FIG. 3 performs. FIG. 2 is a block diagram illustrating a configuration example of a symbol string generation unit in FIG. 1. It is a figure for demonstrating the outline | summary of the symbol sequence production | generation process which the symbol sequence production | generation part of FIG. 1 performs. It is a flowchart for demonstrating the symbol sequence production | generation process which the symbol sequence production | generation part of FIG. 1 performs. It is a figure which shows an example when the division part of FIG. 1 divides | segments a content into a some segment based on a symbol row | line. It is a flowchart for demonstrating the recursive bisection process which the division part of FIG. 1 performs. It is a flowchart for demonstrating the annealing division | segmentation process which the division | segmentation part of FIG. 1 performs. It is a flowchart for demonstrating the content division | segmentation process which the recorder of FIG. 1 performs. It is a block diagram which shows the structural example of the recorder which is 2nd Embodiment. It is a figure which shows an example of the chapter point data produced | generated by the division part of FIG. It is a figure for demonstrating the outline | summary of the digest production | generation process which the digest production | generation part of FIG. 17 performs. It is a block diagram which shows the detailed structural example of the digest production | generation part of FIG. It is a figure for demonstrating a mode that the feature-value extraction part of FIG. 20 produces | generates audio | voice power time series data. It is a figure which shows an example of the motion vector of a flame | frame. It is a figure which shows an example of a zoom-in template. It is a figure for demonstrating the process which the effect addition part of FIG. 20 performs. It is a flowchart for demonstrating the digest production | generation process which the recorder of FIG. 17 performs. It is a block diagram which shows the structural example of the recorder which is 3rd Embodiment. It is a figure which shows an example of a mode that chapter point data change by a user's designation | designated operation. It is a figure which shows an example of the flame | frame made into the chapter point. It is a figure which shows an example when displaying a thumbnail image by the space | interval of 50 frames to the right of the flame | frame used as the chapter point. It is a 1st figure which shows an example of the display screen of a display part. It is a 2nd figure which shows an example of the display screen of a display part. It is a 3rd figure which shows an example of the display screen of a display part. It is a 4th figure which shows an example of the display screen of a display part. It is a block diagram which shows the detailed structural example of the presentation part of FIG. It is a 5th figure which shows an example of the display screen of a display part. It is a 6th figure which shows an example of the display screen of a display part. It is a 7th figure which shows an example of the display screen of a display part. It is an 8th figure which shows an example of the display screen of a display part. It is a 9th figure which shows an example of the display screen of a display part. It is a flowchart for demonstrating the presentation process which the recorder of FIG. 26 performs. It is a flowchart which shows an example of a mode that a display mode transfers. It is a block diagram which shows the structural example of a computer.

Hereinafter, embodiments of the present disclosure (hereinafter referred to as embodiments) will be described. The description will be given in the following order.
1. First embodiment (an example of dividing content into semantically coherent segments)
2. Second embodiment (an example of generating a digest that shows the general content)
3. Third embodiment (an example of displaying thumbnail images of chapters constituting a content)
4). Modified example

<1. First Embodiment>
[Configuration Example of Recorder 1]

  FIG. 1 shows a configuration example of a recorder 1 according to the first embodiment.

  The recorder 1 in FIG. 1 is, for example, an HD (Hard Disk) recorder, and various contents such as a television broadcast program, content provided via a network such as the Internet, content shot with a video camera, and the like. Can be recorded (recorded) (stored).

  1, the recorder 1 includes a content storage unit 11, a content model learning unit 12, a model storage unit 13, a symbol string generation unit 14, a division unit 15, a control unit 16, and an operation unit 17.

  The content storage unit 11 stores (records) content such as a television broadcast program, for example. The storage of the content in the content storage unit 11 results in the recording of the content, and the recorded content (the content stored in the content storage unit 11) corresponds to, for example, a user's playback operation using the operation unit 17 Played.

  For example, the content model learning unit 12 self-organizes the content stored in the content storage unit 11 in a predetermined feature amount space and expresses the content structure (time-space structure) (hereinafter, content model). (Also called statistical learning). The content model learning unit 12 supplies a content model obtained as a result of learning to the model storage unit 13.

  The model storage unit 13 stores the content model supplied from the content model learning unit 12.

  The symbol string generation unit 14 reads content from the content storage unit 11. Then, the symbol sequence generation unit 14 obtains a symbol representing the attribute of each frame (or field) constituting the read content, and creates (generates) a symbol sequence in which a plurality of symbols obtained for each frame are arranged in time series. And supply to the dividing unit 15.

  That is, for example, the symbol sequence generation unit 14 creates a symbol sequence composed of a plurality of symbols using the content stored in the content storage unit 11 and the content model stored in the model storage unit 13, This is supplied to the dividing unit 15.

  Here, as the symbol, for example, a cluster ID representing a cluster including the feature amount of the frame among a plurality of clusters that are the partial spaces constituting the feature amount space can be employed.

  The cluster ID is a value corresponding to the cluster represented by the cluster ID. That is, for example, the cluster IDs are closer to each other as the positions of the clusters are closer. Accordingly, the cluster IDs of frames are closer to each other as the feature amounts of the frames are more similar.

  Further, for example, as a symbol, a state ID representing a frame state among state IDs representing a plurality of different states may be adopted. The state ID is a value corresponding to the state represented by the state ID. That is, for example, the state IDs are closer to each other as the frame states are closer.

  When a cluster ID is employed as a symbol, each frame corresponding to the same symbol has a similar object displayed in the frame.

  Further, when the state ID is adopted as the symbol, the frames corresponding to the same symbol have similar objects displayed in the frames and similar temporal relationships.

  That is, for example, when the cluster ID is adopted as a symbol, the frame in which the train just before the departure is displayed and the frame in which the train just before the stop is displayed are the same symbol.

  This is because when a cluster ID is adopted as a symbol, the symbol is assigned to the frame only depending on whether or not the objects are similar.

  On the other hand, when the state ID is adopted as the symbol, the frame in which the train just before the departure is displayed is different from the frame in which the train just before the stop is displayed.

  This is because when the state ID is adopted as the symbol, the symbol is assigned to the frame in consideration of whether the object is similar or not, and also the temporal context.

  Therefore, when the state ID is employed as the symbol, the symbol represents the frame attributes in more detail than when the cluster ID is employed.

  In the first embodiment, the point is that the content is divided into a plurality of segments based on the variation (dispersion) of each symbol in the symbol string.

  Therefore, in the first embodiment, when the state ID is adopted as the symbol, the content can be divided into a plurality of segments that are semantically grouped with higher accuracy than when the cluster ID is adopted as the symbol. .

  If a learned content model is already stored in the model storage unit 13, the recorder 1 can be configured without providing the content model learning unit 12.

  Here, the content data stored in the content storage unit 11 includes data (stream) of images, sound, and necessary text (caption).

  Here, it is assumed that only the image data of the content data is used for the content model learning process and the process using the content model.

  However, it is possible to use not only image data but also audio and text data for the content model learning process and the process using the content model. In this case, the accuracy of the process can be improved. it can.

  In addition, it is possible to use only audio data, not images, for the content model learning process and the process using the content model.

  The dividing unit 15 reads the same content as the content used when generating the symbol sequence from the symbol sequence generating unit 14 from the content storage unit 11. Then, the dividing unit 15 divides (divides) the read content into a plurality of segments that are semantically grouped based on the variation (dispersion) of each symbol in the symbol sequence from the symbol sequence generating unit 14.

  That is, for example, the dividing unit 15 divides content into a plurality of segments that are semantically grouped for each corner of a program or for each topic of news.

  For example, the control unit 16 controls the content model learning unit 12, the symbol string generation unit 14, and the division unit 15 based on an operation signal from the operation unit 17.

  The operation unit 17 is an operation button or the like operated by the user, and supplies an operation signal corresponding to the user operation to the control unit 16 in response to the operation by the user.

  Next, FIG. 2 shows an example of a symbol string generated by the symbol string generator 14.

  In FIG. 2, the horizontal axis represents time t, and the vertical axis represents the symbol of the frame (frame t) at time t.

  Here, the time t is, for example, a time based on the beginning of the content, and the frame t at the time t means the t-th frame from the beginning of the content. Note that the top frame of the content is frame 0.

  Further, as the symbols (values) are closer to each other, the attributes of the frames corresponding to the symbols are closer.

  In FIG. 2, a thick line segment extending in the vertical direction in the drawing represents a dividing line that divides a symbol string composed of a plurality of symbols into six partial series.

  As shown in FIG. 2, this symbol string includes a first partial sequence (a partial sequence having a “rest” characteristic) in which relatively few types of symbols are frequently observed, and a relatively large number of types. It is composed of a second partial series (a partial series having “large variance” characteristics) in which symbols are observed.

  In FIG. 2, only four first partial series and two second partial series are shown.

  The present inventors conducted an experiment for drawing a dividing line for dividing a symbol string as shown in FIG. 2 into N pieces (N = 6 in the case of FIG. 2) for a plurality of subjects. . And the present inventors obtained the following experimental results.

  That is, the subject draws a dividing line on the boundary between the first partial series and the second partial series, the boundary between the first partial series, and the boundary between the second partial series in the symbol string. The experiment result that there are many.

  In addition, even when the content corresponding to the symbol string shown in FIG. 2 is divided at the position of the dividing line drawn by the subject, the content is generally divided into a plurality of segments that are semantically organized. I understood it.

  Therefore, the dividing unit 15 divides the content into a plurality of segments that are semantically grouped by drawing a dividing line at the same position as the subject based on the symbol sequence from the symbol sequence generating unit 14.

  Specific processing performed by the dividing unit 15 will be described in detail with reference to FIGS. 13 to 15.

[Configuration Example of Content Model Learning Unit 12]
FIG. 3 shows a configuration example of the content model learning unit 12 of FIG.

  The content model learning unit 12 performs learning (model learning) of a state transition probability model defined by a state transition probability that a state transitions and an observation probability that a predetermined observation value is observed from the state. Further, the content model learning unit 12 extracts a feature amount of each frame of an image of learning content that is content used for cluster learning for obtaining cluster information described later. Further, the content model learning unit 12 performs cluster learning using the feature amount of the learning content.

  That is, the content model learning unit 12 includes a learning content selection unit 21, a feature amount extraction unit 22, a feature amount storage unit 26, and a learning unit 27.

  The learning content selection unit 21 selects content used for model learning and cluster learning from the content stored in the content storage unit 11 as learning content, and supplies it to the feature amount extraction unit 22.

  Here, the learning content selection unit 21 selects, for example, one or more contents belonging to a predetermined category from the contents stored in the content storage unit 11 as learning contents.

  The content belonging to a predetermined category is, for example, a content structure hidden in the content such as a program of the same genre, a continuous program, a program broadcasted every week or every other day (a program having the same title), and the like. Means common content.

  As the genre, for example, a rough classification such as a sports program or a news program can be adopted, but it is desirable that the classification is a fine classification such as a soccer game program or a baseball game program. .

  In addition, for example, a soccer game program can be classified into contents belonging to different categories every time the channel (broadcast station) is different.

  It is assumed that what category is adopted as the content category is set in advance in the recorder 1 in FIG. 1, for example.

  The content categories stored in the content storage unit 11 include, for example, metadata such as program titles and genres transmitted together with programs by television broadcasting, information on programs provided by sites on the Internet, and the like. Can be recognized from.

  The feature amount extraction unit 22 demultiplexes (separates) the learning content from the learning content selection unit 21 into image and audio data, extracts the feature amount of each frame of the image, and extracts the feature amount storage unit. 26.

  That is, the feature quantity extraction unit 22 includes a frame division unit 23, a sub-region feature quantity extraction unit 24, and a combination unit 25.

  Each frame of the learning content image from the learning content selection unit 21 is supplied to the frame dividing unit 23 in time series.

  The frame dividing unit 23 sequentially sets the frames of the learning content supplied from the learning content selection unit 21 in time series as the attention frame. Then, the frame division unit 23 divides the frame of interest into a plurality of sub-regions, which are sub-regions, and supplies the sub-region feature amount extraction unit 24 to the sub-region feature amount extraction unit 24.

  The sub-region feature amount extraction unit 24 extracts the feature amount of the sub-region (hereinafter also referred to as a sub-region feature amount) from each sub-region of the frame of interest from the frame division unit 23 and supplies it to the combining unit 25.

  The combination unit 25 combines the sub-region feature amounts of the sub-regions of the target frame from the sub-region feature amount extraction unit 24, and supplies the combination result to the feature amount storage unit 26 as the feature amount of the target frame.

  The feature amount storage unit 26 stores the feature amount of each frame of the learning content supplied from the feature amount extraction unit 22 (the combination unit 25) in time series.

  The learning unit 27 performs cluster learning using the feature amount of each frame of the learning content stored in the feature amount storage unit 26.

  That is, the learning unit 27 divides a feature amount space, which is a space of the feature amount, into a plurality of clusters using the feature amount (vector) of each frame of the learning content stored in the feature amount storage unit 26. Cluster learning is performed to obtain cluster information that is cluster information.

  Here, as the cluster learning, for example, the k-means method can be adopted. When the k-means method is used for cluster learning, the cluster information obtained as a result of cluster learning corresponds to a representative vector that represents a cluster in the feature space and a code that represents the representative vector (the cluster that it represents). It will be a codebook attached.

  In the k-means method, the representative vector of the target cluster of interest is the feature amount (distance from each representative vector of the codebook (Euclidean distance)) among the feature amount (vector) of the learning content. The average value (vector) of the feature amount having the shortest distance from the representative vector of the cluster of interest.

  The learning unit 27 further uses the cluster information obtained from the learning content to convert the feature amount of each frame of the learning content stored in the feature amount storage unit 26 into one of the plurality of clusters. By performing clustering, a code representing a cluster to which the feature amount belongs is obtained, thereby converting a time series of the feature amount of the learning content into a code sequence (determining a code sequence of the learning content).

  Here, when the k-means method is adopted as the cluster learning, the clustering performed using the code book as the cluster information obtained by the cluster learning is vector quantization.

  In vector quantization, for each representative vector of the codebook, the distance from the feature quantity (vector) is calculated, and the code of the representative vector that minimizes the distance is output as the vector quantization result.

  When the learning unit 27 converts the time series of the feature amount of the learning content into a code series by clustering, the learning unit 27 performs model learning that is learning of the state transition model using the code series.

  Then, the learning unit 27 supplies a set of the state transition probability model after model learning and the cluster information obtained by cluster learning as a content model to the model storage unit 13 in association with the learning content category. .

  Therefore, the content model is composed of a state transition probability model and cluster information.

  Here, a state transition probability model (a state transition probability model in which learning is performed using a code sequence) constituting the content model is also referred to as a code model.

[State transition probability model]
A state transition probability model in which the learning unit 27 in FIG. 3 performs model learning will be described with reference to FIGS.

  For example, an HMM (Hidden Marcov Model) can be adopted as the state transition probability model. When the HMM is adopted as the state transition probability model, the HMM learning is performed by, for example, the Baum-Welch re-estimation method.

  FIG. 4 shows an example of a left-to-right type HMM.

  A left-to-right type HMM is an HMM in which the states are aligned in a straight line from left to right. From the state to the self-transition (transition from one state to the state) Can also transition to the state on the right. The left-to-right type HMM is used, for example, for speech recognition.

  The HMM shown in FIG. 4 includes three states s1, s2, and s3. As a state transition, a self transition and a transition from a certain state to a state on the right side thereof are permitted.

The HMM is defined by the initial probability π _i of the state s _i , the state transition probability a _ij , and the observation probability b _i (o) at which a predetermined observation value o is observed from the state s _i .

Here, the initial probability [pi _i, the state s _i is the probability of the initial state (initial state), the left-to-right type HMM, the initial probability [pi ₁ of the leftmost state s ₁ is is 1.0, the initial probability [pi _i of the other state s _i, it is 0.0.

The state transition probability a _ij is a probability of transition from the state s _i to the state s _j .

Observation probability b _i (o), upon state transition to the state s _i, a probability that the observed value o is observed from the state s _i. As the observation probability b _i (o), when the observation value o is a discrete value, a probability value (discrete value) is used, but when the observation value o is a continuous value, the probability distribution function Is used. As the probability distribution function, for example, a Gaussian distribution defined by an average value (average vector) and a variance (covariance matrix) can be employed. In the present embodiment, a discrete value is used as the observed value o.

  FIG. 5 shows an example of an Ergodic type HMM.

An ergodic type HMM is an HMM with no restrictions on state transition, that is, an HMM capable of state transition from an arbitrary state s _i to an arbitrary state s _j .

The HMM in FIG. 5 includes three states s ₁ , s ₂ , and s ₃ , and arbitrary state transitions are permitted.

The ergodic HMM is the HMM having the highest degree of freedom of state transition. However, as the number of states increases, the HMM parameters (initial probability π _i , state transition probability a _ij , and observation probability b _i (o) Depending on the initial value of), it may converge to the local minimum and an appropriate parameter may not be obtained.

  Therefore, the learning unit 27 adopts the hypothesis that “most of the phenomena in the natural world, camera work and program structure that generate video content can be expressed by a sparse connection like a small world network”. Let us adopt an HMM in which state transitions are restricted to a sparse structure.

  Here, a sparse structure is not a dense state transition such as an ergodic HMM that can make a state transition from a certain state to an arbitrary state, but a state that can make a state transition from a certain state is very It is a limited structure (a structure in which state transition is sparse).

  Note that here, even in a sparse structure, at least one state transition to another state exists, and a self-transition exists.

  FIG. 6 shows an example of a two-dimensional neighborhood constrained HMM that is an HMM having a sparse structure.

  In addition to the sparse structure, the HMM in FIG. 6A and FIG. 6B has a constraint that the states constituting the HMM are arranged in a lattice pattern on a two-dimensional plane.

  Here, in the HMM in FIG. 6A, the state transition to another state is limited to a horizontally adjacent state and a vertically adjacent state. In the HMM of FIG. 6B, the state transition to another state is limited to a horizontally adjacent state, a vertically adjacent state, and a diagonally adjacent state.

  FIG. 7 shows an example of an HMM having a sparse structure other than the two-dimensional neighborhood constraint HMM.

  That is, A in FIG. 7 shows an example of an HMM with a three-dimensional grid constraint. FIG. 7B shows an example of an HMM based on a two-dimensional random arrangement constraint. FIG. 7C shows an example of an HMM by a small world network.

  In the learning unit 27 in FIG. 3, the learning of the HMM having the sparse structure shown in FIGS. 6 and 7 having the state of, for example, about 100 to several hundreds is performed on the image stored in the feature amount storage unit 26 ( This is performed by the Baum-Welch re-estimation method using the feature code sequence (extracted from the frame).

  The HMM, which is a code model obtained as a result of learning in the learning unit 27, can be called a Visual HMM because it is obtained by learning using only the feature amount of the content image (Visual).

  Here, the code sequence of the feature amount used for HMM learning (model learning) is a discrete value, and a probability value is used as the observation probability bi (o) of the HMM.

  Regarding HMM, for example, co-authored by Laurence Rabiner and Biing-Hwang Juang, “Basics of Speech Recognition (Up / Down), NTT Advanced Technology Co., Ltd.” and Japanese Patent Application No. 2008-064993 previously proposed by the applicant. It is described in. The use of an ergotic type HMM or a sparse structure HMM is described in, for example, Japanese Unexamined Patent Application Publication No. 2009-223444 previously proposed by the present applicant.

[Feature extraction]
FIG. 8 shows a feature amount extraction process by the feature amount extraction unit 22 of FIG.

  In the feature amount extraction unit 22, each frame of the learning content image from the learning content selection unit 21 is supplied to the frame division unit 23 in time series.

The frame dividing unit 23 sequentially sets the frames of the learning content supplied in time series from the learning content selecting unit 21 as the attention frame, divides the attention frame into a plurality of subregions R _k, and subregion features It supplies to the quantity extraction part 24.

Here, in FIG. 8, the frame of interest is equally divided into ₁₆ sub-regions R ₁ , R ₂ ,.

Note that the number of sub-regions R _k when dividing one frame into sub-regions R _k is not limited to 16 of 4 × 4. That is, one frame can be divided into, for example, 5 × 4 20 sub-regions R _k and 5 × 5 25 sub-regions R _k .

Further, in FIG. 8, one frame have been divided into sub-regions R _k of the same size (equal), the size of the sub regions may not be the same. That is, for example, the central portion of the frame can be divided into small-sized sub-regions, and the peripheral portion of the frame (such as a portion adjacent to the image frame) can be divided into large-sized sub-regions.

The sub-region feature quantity extraction unit 24 (FIG. 3) extracts the sub-region feature quantity f _k = FeatExt (R _k ) of each sub-region R _k of the frame of interest from the frame division unit 23 and supplies it to the combining unit 25. .

That is, the sub-region feature quantity extraction unit 24 uses the pixel values of the sub-area R _k (for example, RGB components, YUV components, etc.) and converts the global feature quantity of the sub-area R _k into the sub-area feature quantity f _k. Asking.

Here, the global feature amount of the sub region R _k, without using the information of the position of the pixels constituting the sub region R _k, using only pixel values, is additively calculated, for example, a histogram This means the feature quantity.

  As the global feature quantity, for example, a feature quantity called GIST can be adopted. Regarding GIST, for example, A. Torralba, K. Murphy, W. Freeman, M. Rubin, "Context-based vision system for place and object recognition", IEEE Int. Conf. Computer Vision, vol. 1, no. 1 , pp. 273-280, 2003.

  The global feature amount is not limited to GIST. That is, the global feature value may be a feature value that is robust (absorbs change) (robust) with respect to changes in appearance such as local position, brightness, and viewpoint. Such feature amounts include, for example, HLCA (Local Higher Order Correlation), LBP (Local Binary Patterns), and a color histogram.

  Details of HLCA are described in, for example, N. Otsu, T. Kurita, "A new scheme for practical flexible and intelligent vision systems", Proc. IAPR Workshop on Computer Vision, pp.431-435, 1988. . For details on LBP, see, for example, Ojala T, Pietikainen M & Maenpaa T, "Multiresolution gray-scale and rotation invariant texture classification with Local Binary Patterns", IEEE Transactions on Pattern Analysis and Machine Intelligence 24 (7): 971-987. (Pietikainen and Maenpaa's "a" is exactly the letter with "..." added to the top of "a").

  Here, global feature quantities such as GIST, LBP, HLCA, and color histogram described above tend to have a large number of dimensions, but also tend to have a high correlation between dimensions.

Therefore, in the sub region feature amount extracting unit 24 (FIG. 3), from the sub-region R _k, after extracting the GIST or the like, it is possible to perform a principal component analysis of the GIST or the like (PCA (principal component analysis)) . Then, the sub-region feature quantity extraction unit 24 compresses (limits) the number of dimensions such as GIST so that the cumulative contribution rate becomes a somewhat high value (for example, a value of 95% or more) based on the PCA result. The compression result can be used as a sub-region feature amount.

  In this case, a projection vector obtained by projecting GIST or the like onto a PCA space in which the number of dimensions is compressed becomes a compression result obtained by compressing the number of dimensions such as GIST.

The combining unit 25 (FIG. 3) combines the sub-region feature amounts f _{1 to} f ₁₆ of the sub-regions R _{1 to} R ₁₆ of the target frame from the sub-region feature amount extraction unit 24, and the result of the combination is displayed as the target frame. The feature value is supplied to the feature value storage unit 26.

That is, the combining unit 25 combines the sub-region feature amounts f _{1 to} f ₁₆ from the sub-region feature amount extraction unit 24 to generate a vector having the sub-region feature amounts f _{1 to} f ₁₆ as components, The vector is supplied to the feature amount storage unit 26 as the feature amount _Ft of the frame of interest.

  Here, in FIG. 8, the frame at time t (frame t) is the frame of interest.

  In the feature amount extraction unit 22 in FIG. 3, each frame of the learning content is sequentially set as a frame of interest from the top, and the feature amount Ft is obtained as described above. Then, the feature amount Ft of each frame of the learning content is supplied and stored from the feature amount extraction unit 22 to the feature amount storage unit 26 in time series (in a state in which the temporal context is maintained).

As described above, the feature amount extraction unit 22, a sub-region feature f _k, global feature amount of the sub region R _k is determined, a vector which its sub region feature amount f _k and components, frames determined as the feature amount F _t.

Therefore, the frame feature value F _t is robust against local changes (changes that occur within a sub-region), but is discriminative (sensitive) to changes in the pattern arrangement of the entire frame. It is a feature quantity that is a property that distinguishes differences.

[Content model learning process]
Next, processing (content model learning processing) performed by the content model learning unit 12 of FIG. 3 will be described with reference to the flowchart of FIG.

  In step S <b> 11, the learning content selection unit 21 selects one or more contents belonging to a predetermined category from the contents stored in the content storage unit 11 as learning contents.

  That is, for example, the learning content selection unit 21 selects any one content that has not yet been set as the learning content from the content stored in the content storage unit 11 as the learning content.

  Furthermore, the learning content selection unit 21 recognizes a category of one content selected as the learning content, and when other content belonging to the category is stored in the content storage unit 11, the content ( Other content) is further selected as learning content.

  The learning content selection unit 21 supplies the learning content to the feature amount extraction unit 22, and the process proceeds from step S11 to step S12.

  In step S12, the frame dividing unit 23 of the feature amount extraction unit 22 has not yet selected the learning content from the learning content selection unit 21 as the attention learning content (hereinafter also referred to as attention content). One of the learning contents is selected as the attention content.

  Then, the process proceeds from step S12 to step S13, and the frame dividing unit 23 selects, as the attention frame, the most temporally preceding frame that has not yet been set as the attention frame among the frames of the attention content. Advances to step S14.

  In step S14, the frame dividing unit 23 divides the frame of interest into a plurality of sub-regions and supplies the sub-region feature amount extracting unit 24, and the process proceeds to step S15.

  In step S15, the sub-region feature value extraction unit 24 extracts the sub-region feature values of each of the plurality of sub-regions from the frame division unit 23, supplies them to the combining unit 25, and the process proceeds to step S16.

  In step S16, the combining unit 25 generates a feature amount of the target frame by combining the sub-region feature amounts of the plurality of sub-regions constituting the target frame from the sub-region feature amount extraction unit 24, and performs processing. Advances to step S17.

  In step S <b> 17, the frame dividing unit 23 determines whether or not all frames of the content of interest have been used as the frame of interest.

  If it is determined in step S17 that there is a frame that has not yet been set as the target frame among the frames of the target content, the process returns to step S13, and the same process is repeated thereafter.

  If it is determined in step S17 that all the frames of the content of interest have been used as the frame of interest, the process proceeds to step S18, and the combining unit 25 determines the feature amount ( Are supplied to the feature amount storage unit 26 and stored therein.

  Then, the process proceeds from step S18 to step S19, and the frame dividing unit 23 determines whether or not all of the learning content from the learning content selection unit 21 is the content of interest.

  If it is determined in step S19 that there is a learning content that has not yet been set as the content of interest in the learning content, the processing returns to step S12, and the same processing is repeated thereafter.

  If it is determined in step S19 that all of the learning content is the content of interest, the process proceeds to step S20, and the learning unit 27 stores the learning content stored in the feature amount storage unit 26. The content model is learned using the feature amount (the time series of the feature amount of each frame).

  That is, the learning unit 27 divides a feature amount space, which is a space of the feature amount, into a plurality of clusters using the feature amount (vector) of each frame of the learning content stored in the feature amount storage unit 26. Cluster learning is performed by the k-means method, and a codebook of, for example, 100 to several hundred clusters (representative vectors) as a predetermined number is obtained as cluster information.

  Further, the learning unit 27 performs vector quantization for clustering the feature amounts of each frame of the learning content stored in the feature amount storage unit 26 using the code book as the cluster information obtained by the cluster learning, The time series of the feature amount of the learning content is converted into a code series.

  The learning unit 27 performs model learning, which is learning of an HMM (discrete HMM), by clustering the time series of the feature amount of the content for learning and converting it into a code series, using the code series.

  Then, the learning unit 27 associates a set of a code model that is an HMM after model learning and a code book as cluster information obtained by cluster learning as a content model with a category of learning content, and stores the model Output (supply) to the unit 13, and the content model learning process is terminated.

  The content model learning process can be started at an arbitrary timing.

  According to the content model learning process described above, in the HMM that is a code model, the content structure (for example, a program structure, a structure created by camera work, etc.) hidden in the learning content is acquired in a self-organizing manner.

  As a result, each state of the HMM as a code model in the content model obtained by the content model learning process corresponds to an element of the content structure acquired by learning, and the state transition is between the elements of the content structure. Of time transitions.

  The state of the code model is close to the spatial distance in the feature amount space (the feature amount space extracted by the feature amount extraction unit 22 (FIG. 3)) and has similar temporal context. Represent a group of frames (ie “similar scenes”) together.

[Configuration Example of Symbol Sequence Generation Unit 14]
FIG. 10 shows a configuration example of the symbol string generation unit 14 of FIG.

  The symbol string generation unit 14 includes a content selection unit 31, a model selection unit 32, a feature amount extraction unit 33, and a maximum likelihood state sequence estimation unit 34.

  The content selection unit 31 selects content for generating a symbol string as content of interest from the content stored in the content storage unit 11 according to control from the control unit 16.

  For example, the control unit 16 controls the content selection unit 31 based on an operation signal corresponding to the user's selection operation from the operation unit 17, and sets the content selected by the user's selection operation as the content of interest. Let them choose.

  In addition, the content selection unit 31 supplies the content of interest to the feature amount extraction unit 33. Further, the content selection unit 31 recognizes the category of the content of interest and supplies it to the model selection unit 32.

  The model selection unit 32 selects a content model in the category that matches the category of the content of interest from the content selection unit 31 from among the content models stored in the model storage unit 13 (a content model associated with the category of the content of interest). ) Is selected as the model of interest.

  Then, the model selection unit 32 supplies the model of interest to the maximum likelihood state sequence estimation unit 34.

  The feature amount extraction unit 33 extracts the feature amount of each frame (image) of the content of interest supplied from the content selection unit 31 in the same manner as the feature amount extraction unit 22 of FIG. The feature amount (time series) is supplied to the maximum likelihood state sequence estimation unit 34.

  The maximum likelihood state sequence estimation unit 34 uses the cluster information of the attention model from the model selection unit 32 to cluster the feature amounts (time series) of the attention content from the feature amount extraction unit 33, and the (features) of the attention content Determine the code sequence.

  Further, the maximum likelihood state sequence estimation unit 34 observes the code sequence of the content of interest (feature amount) from the feature amount extraction unit 33 in the code model of the target model from the model selection unit 32 according to, for example, the Viterbi algorithm. A maximum likelihood state sequence (a sequence of states constituting a so-called Viterbi path), which is a state sequence in which a state transition with the highest likelihood occurs, is estimated.

  The maximum likelihood state sequence estimator 34 then obtains the maximum likelihood state sequence (hereinafter referred to as the attention code for the attention content) when the code sequence of the attention content is observed in the code model of the attention model (hereinafter also referred to as the attention code model). The model maximum likelihood state sequence) is supplied to the dividing unit 15 as a symbol string.

  The maximum likelihood state sequence estimation unit 34 uses the code sequence (cluster ID sequence) of the content of interest obtained by clustering as a symbol string instead of the maximum likelihood state sequence of the code model of interest for the content of interest as a symbol sequence. You may make it supply to.

  Here, the state at time t (t-th state from the top constituting the maximum likelihood state sequence) with respect to the top of the maximum likelihood state sequence of the attention code model for the attention content is expressed as s (t), Let T denote the number of frames of the content of interest.

  In this case, the maximum likelihood state sequence of the attention code model for the attention content is a sequence of T states s (1), S (2),..., S (T), of which the t-th state ( The state (time t) s (t) corresponds to the frame (frame t) of the content of interest at time t.

Also, if it represents the total number of states of the attention code model and N, the state at time t s (t) is the one of N states _{s 1, s 2, ···,} s N .

Furthermore, each of the N states s ₁ , s ₂ ,..., S _N is assigned a state ID (Identification) that is an index for specifying the state.

Assuming that the state s (t) at the time t of the maximum likelihood state sequence of the code model of interest for the content of interest is the i-th state s _i of the _N states s _{1 to} s _N , The frame corresponds to state s _i .

Therefore, each frame of the content of interest corresponds to one of _N states s _{1 to} s _N.

The entity of the maximum likelihood state sequence of the attention code model for the attention content is a state ID sequence of one of the _N states s _{1 to} s _N corresponding to the frame at each time t of the attention content. .

  FIG. 11 shows an outline of the symbol string generation process performed by the symbol string generation unit 14 of FIG.

  11A shows a time series of frames of content selected as content of interest by the content selection unit 31. FIG.

  B of FIG. 11 shows a time series of time-series feature amounts extracted by the feature amount extraction unit 33 in the frame of A of FIG.

  C of FIG. 11 shows a code sequence of a code obtained by clustering the time series of the feature values of B of FIG. 11 in the maximum likelihood state sequence estimation unit 34.

  D in FIG. 11 is a maximum likelihood state sequence in which the code sequence (of the feature amount time series) of the content of interest in FIG. 11C is observed in the attention code model estimated by the maximum likelihood state sequence estimation unit 34. The maximum likelihood state sequence of the attention code model for the attention content) is shown.

  When the code sequence shown in C of FIG. 11 is supplied as a symbol sequence to the dividing unit 15, the symbol sequence generating unit 14 supplies each code (cluster ID) constituting the code sequence to the dividing unit 15 as a symbol. To do.

  In addition, when the symbol sequence generation unit 14 supplies the maximum likelihood state sequence shown in D of FIG. 11 as a symbol sequence to the dividing unit 15, the symbol sequence generation unit 14 uses each state ID constituting the maximum likelihood state sequence as a symbol as a symbol. 15 is supplied.

[Description of operation of symbol string generation unit 14]
Next, a symbol string generation process performed by the symbol string generation unit 14 will be described with reference to the flowchart of FIG.

  This symbol sequence generation processing is performed when, for example, the user performs a selection operation for selecting content for generating a symbol sequence from the content stored in the content storage unit 11 using the operation unit 17. Be started.

  At this time, the operation unit 17 supplies an operation signal corresponding to the user's selection operation to the control unit 16. The control unit 16 controls the content selection unit 31 based on the operation signal from the operation unit 17.

  That is, in step S <b> 41, the content selection unit 31 selects content for generating a symbol string as attention content from the content stored in the content storage unit 11 according to control from the control unit 16.

  Then, the content selection unit 31 supplies the content of interest to the feature amount extraction unit 33. Further, the content selection unit 31 recognizes the category of the content of interest and supplies it to the model selection unit 32.

  In step S42, the model selection unit 32 selects a content model (corresponding to the category of the content of interest) from the content model stored in the model storage unit 13 that matches the category of the content of interest from the content selection unit 31. Selected content model).

  Then, the model selection unit 32 supplies the model of interest to the maximum likelihood state sequence estimation unit 34.

  In step S43, the feature amount extraction unit 33 extracts the feature amount of each frame (image) of the content of interest supplied from the content selection unit 31 in the same manner as the feature amount extraction unit 22 of FIG. Are supplied to the maximum likelihood state sequence estimator 34.

  In step S44, the maximum likelihood state sequence estimation unit 34 uses the cluster information of the model of interest from the model selection unit 32 to cluster the feature amounts (time series) of the content of interest from the feature amount extraction unit 33, and The code sequence of the content (feature value) is obtained.

  Further, the maximum likelihood state sequence estimation unit 34 observes the code sequence of the content of interest (feature amount) from the feature amount extraction unit 33 in the code model of the target model from the model selection unit 32 according to, for example, the Viterbi algorithm. A maximum likelihood state sequence (a sequence of states constituting a so-called Viterbi path), which is a state sequence in which a state transition with the highest likelihood occurs, is estimated.

  The maximum likelihood state sequence estimator 34 then obtains the maximum likelihood state sequence (hereinafter referred to as the attention code for the attention content) when the code sequence of the attention content is observed in the code model of the attention model (hereinafter also referred to as the attention code model). The model maximum likelihood state sequence) is supplied to the dividing unit 15 as a symbol string.

  The maximum likelihood state sequence estimation unit 34 supplies the code sequence of the content of interest obtained by clustering to the dividing unit 15 as a symbol string instead of the maximum likelihood state sequence of the code model of interest for the content of interest. Also good. This completes the symbol string generation process.

  Next, FIG. 13 shows an example when the dividing unit 15 divides the content into a plurality of segments that are semantically grouped based on the symbol sequence from the symbol sequence generating unit 14.

  Note that FIG. 13 is configured in the same manner as FIG. That is, for example, in FIG. 13, the horizontal axis represents time t, and the vertical axis represents the symbol of frame t.

FIG. 13 also shows a dividing line (indicated by a thick line segment) for dividing the content into six segments S ₁ , S ₂ , S ₃ , S ₄ , S ₅ , S ₆ . The dividing line is arranged (drawn) at an arbitrary time t.

  Here, when a code sequence is adopted as the symbol sequence, the symbol is each code (code indicated by C in FIG. 11) constituting the code sequence. Further, when the maximum likelihood state sequence is adopted as the symbol string, the symbol is a code constituting the maximum likelihood state sequence (a code indicated by D in FIG. 11).

  As described with reference to FIG. 2, the dividing unit 15 sets the boundary between the first partial series and the second partial series, the boundary between the first partial series, and the boundary between the second partial series. The content is divided by drawing a dividing line.

  That is, for example, the dividing unit 15 sets the dividing line so that the total sum Q of entropy H (Si) of each segment Si (i = 1, 2,..., 6) shown in FIG. Just pull it. Here, the entropy of the segment Si represents the degree of symbol variation in the segment Si.

  When the dividing line is arranged at an arbitrary time t, the content is divided with the frame t as a boundary. That is, for example, in content that has not yet been divided, if the dividing line is placed at the position of an arbitrary time t, the content includes the segment including the first frame 0 to the frame t-1 and the last from the frame t. Is divided into segments including up to frame T.

  The dividing unit 15 determines the division position (position to draw the dividing line) for dividing the content based on the variation (dispersion) of each symbol in the symbol string as shown in FIG. 13 from the symbol string generating unit 14. calculate.

  Then, the dividing unit 15 reads the content corresponding to the symbol sequence from the symbol sequence generating unit 14 from the content storage unit 11, and divides the content into a plurality of segments at the calculated division position.

  That is, for example, the dividing unit 15 divides the content into D segments Si (i = 1, 2,..., D) with the total number of divisions D specified by the user specifying operation using the operation unit 17. To do.

  Specifically, for example, the dividing unit 15 calculates entropy H (Si) for each segment Si by the following equation (1).

Here, in equation (1), the probability P ^[Si] (k) is the kth symbol (symbol with the kth smallest value), for example, when symbols are arranged in ascending order in the segment Si. Represents the probability of

Further, in the equation (1), P ^[Si] (k) = (number of appearance frequencies of k-th symbol in segment Si) / (total number of symbols in segment Si).

Further, the dividing unit 15 calculates the sum Q of entropies H (S ₁ ) to H (S _D ) of all the segments S _{1 to} S _D using the following equation (2).

Segments S ₁ , S ₂ , S ₃ , S ₄ , S ₅ , S ₆ ,... S _D when the total sum Q is minimum are divided by segment lines S ₁ , _{S 2, S 3, S 4} , S 5, S 6, a ... S _D.

Therefore, the dividing unit 15 divides the content into a plurality of segments S _{1 to} S _D by solving the minimization problem that minimizes the calculated sum Q, and supplies the divided content to the content storage unit 11. To remember.

  In order to solve the minimization problem of the sum Q, for example, recursive bisection processing or annealing division processing can be used. Note that the method for solving the minimization problem of the sum Q is not limited to this, and the minimization problem can also be solved by using, for example, a tabu search or a genetic algorithm.

  Here, the recursive bisection process is a process of dividing the content recursively (repeatedly) at a division position where the total entropy of each segment after division is the minimum, thereby dividing the content into a plurality of segments. The process to divide. The recursive bisection process will be described in detail with reference to FIG.

  The annealing division process is a process of dividing the content into a plurality of segments by performing a process of changing the division position where the content is arbitrarily divided into division positions that minimize the total entropy. . The annealing division process will be described in detail with reference to FIG.

[Description of operation of dividing unit 15]
Next, the recursive bisection process performed by the dividing unit 15 will be described with reference to the flowchart of FIG.

  This recursive bisection process is started, for example, when the user performs a designation operation for designating the total content division number D using the operation unit 17. At this time, the operation unit 17 supplies an operation signal corresponding to the user's designated operation to the control unit 16.

  The control unit 16 controls the dividing unit 15 in accordance with the operation signal from the operation unit 17 and causes the dividing unit 15 to divide the symbol string by the total number of divisions D specified by the user.

  That is, in step S81, the dividing unit 15 sets the number of divisions d held in advance in a built-in memory (not shown) to 1. This division number d represents the division number obtained by dividing the symbol string in the recursive bisection process. When the division number d = 1, it indicates that the symbol string has not been divided yet.

  In step S82, the dividing unit 15 has not yet added a dividing line among the additional points Li representing the time at which the dividing line can be added based on the variance of each symbol in the symbol string from the symbol string generating unit 14. For each additional point Li, the total entropy Q = Q (Li) when the dividing line is added is calculated.

  Here, the additional point Li is the time t corresponding to the frames 1 to T among the frames 0 to T constituting the content.

  In step S83, the dividing unit 15 sets L * when the sum Q (Li) is minimum among the entropy sum Q (Li) calculated in step S82. In step S84, the dividing unit 15 adds a dividing line to the additional point L *, and in step S85, adds (increments) 1 to the number of divisions d.

  Thereby, the dividing unit 15 divides the symbol sequence from the symbol sequence generating unit 14 at the additional point L *.

  In step S86, the dividing unit 15 determines whether or not the division number d is equal to the total division number D specified by the user's specifying operation, and determines that the division number d is not equal to the total division number D. The process returns to step S82, and thereafter the same process is repeated.

In step S86, if the division unit 15 determines that the division number d is equal to the total division number D, that is, determines that the symbol string is divided into _D segments S _{1 to} S _D , The division process ends.

Then, the dividing unit 15 reads the same content as the content converted into the symbol sequence by the symbol sequence generating unit 14 from the content storage unit 11, and the same division position as the division position where the read content is divided Divide by. The dividing unit 15 supplies the content divided into the plurality of segments S _{1 to SD} to the content storage unit 11 for storage.

As described above, according to the recursive bisection processing of FIG. 14, the content is divided into _D segments S _{1 to} S _D having the minimum total Q of entropy H (Si).

  Therefore, according to the recursive bisection process of FIG. 14, it is possible to divide the content into segments that are semantically coherent in the same manner as the subject.

  That is, for example, the content can be divided into a plurality of segments into program corners and news topics.

  Further, according to the recursive bisection process of FIG. 14, the content is divided using a relatively simple algorithm. For this reason, in the recursive two-division process, it is possible to divide content quickly with a relatively small amount of calculation.

[Description of Other Operations of Dividing Unit 15]
Next, the annealing division process performed by the dividing unit 15 will be described with reference to the flowchart of FIG.

  This annealing division process is started, for example, when the user performs a designation operation for designating the total division number D of content using the operation unit 17. At this time, the operation unit 17 supplies an operation signal corresponding to the user's designated operation to the control unit 16. The control unit 16 controls the dividing unit 15 in accordance with the operation signal from the operation unit 17 and causes the dividing unit 15 to divide the symbol string by the total number of divisions D specified by the user.

In step S111, the dividing unit 15 arbitrarily selects D-1 additional points Li from the additional points Li representing the time at which the dividing line can be added, and divides them into the selected D-1 additional points Li. Add (place) a line. Thereby, the dividing unit 15 provisionally divides the symbol sequence from the symbol sequence generating unit 14 into _D segments S _{1 to} S _D.

  In step S112, the dividing unit 15 sets variables t and j held in advance in a built-in memory (not shown) to 1, respectively. Further, the dividing unit 15 sets (initializes) a temperature parameter temp previously held in a built-in memory (not shown) to a predetermined value.

  In step S113, the dividing unit 15 determines whether or not the variable t is a predetermined threshold value NREP. If it is determined that the variable t is not the threshold value NREP, the process proceeds to step S114.

  In step S114, the dividing unit 15 determines whether or not the variable j is a predetermined threshold value NIREP. If it is determined that the variable j is the threshold value NIREP, the process proceeds to step S115. Note that the threshold value NIREP is desirably a value sufficiently larger than the threshold value NREP.

  In step S115, the dividing unit 15 sets the multiplication result temp × 0.9 obtained by multiplying the temperature parameter temp previously held in a built-in memory (not shown) by 0.9 as the changed temp. In step S116, the dividing unit 15 adds (increments) 1 to the variable t, and sets the variable j to 1 in step S117.

  Thereafter, the process returns to step S113, and the dividing unit 15 performs the same process thereafter.

  In step S114, when the dividing unit 15 determines that the variable j is not the threshold value NIREP, the process proceeds to step S118.

  In step S118, the dividing unit 15 determines an arbitrary additional point Li among the D-1 additional points Li to which the dividing line has been added, and calculates the front and rear RNG widths of the determined additional point Li. Here, the front / rear RNG width represents a range from the additional point Li-x to the additional point Li + x. It is assumed that the natural number x is preset by the dividing unit 15.

  In step S119, the dividing unit 15 sets the additional point Li determined in step S118 to the additional point Ln (n is a natural number from ix to i + x) included in the front and rear RNG width calculated in step S118. Q (Ln) when moving is calculated.

  In step S120, the dividing unit 15 determines Ln at which Q (Ln) is minimum among the plurality of Q (Ln) calculated in step S119, and calculates Q (L *). The dividing unit 15 calculates Q (Li) before the dividing line is moved.

  In step S121, the dividing unit 15 obtains a difference ΔQ = Q (L *) − Q (Li) obtained by subtracting Q (Li) before moving the dividing line from Q (L *) after moving the dividing line. Is calculated.

  In step S122, the dividing unit 15 determines whether or not the difference ΔQ calculated in step S121 is less than 0. If it is determined that the difference ΔQ is less than 0, the process proceeds to step S123.

  In step S123, the dividing unit 15 moves the dividing line arranged at the additional point Li determined in step S118 to the additional point L * determined in step S120, and the process proceeds to step S125.

  In step S122, when the dividing unit 15 determines that the difference ΔQ is not less than 0 (0 or more), the process proceeds to step S124.

  In step S124, the dividing unit 15 uses the probability of exp (ΔQ / temp) (= the natural logarithm base e to the power of ΔQ / temp) as the dividing line arranged at the additional point Li determined in step S118. The process moves to the additional point L * determined in S120, and the process proceeds to step S125.

  In step S125, the dividing unit 15 adds 1 to the variable j, returns the process to step S114, and thereafter performs the same process.

  In step S113, when the dividing unit 15 determines that the variable t is a predetermined threshold value NREP, the annealing division process in FIG. 15 is terminated.

Then, the dividing unit 15 reads the same content as the content converted into the symbol sequence by the symbol sequence generating unit 14 from the content storage unit 11, and the same division position as the division position where the read content is divided Divide by. The dividing unit 15 supplies the content divided into the plurality of segments S _{1 to SD} to the content storage unit 11 for storage.

  As described above, according to the annealing division process of FIG. 15, it is possible to divide the content into semantically coherent segments as in the recursive bisection process of FIG.

  By the way, the dividing unit 15 divides the content read from the content storage unit 11 by the total number of divisions D specified by the user specifying operation. However, for example, the dividing unit 15 may divide the content by the total division number D that minimizes the total entropy Q among the total number of divisions into which the content can be divided.

  Also, for example, when the total number of divisions D is specified by the user's specifying operation, the total number of divisions D is specified, and when the total number of divisions D is not specified, the total division that minimizes the total entropy Q You may make it divide | segment by the number D. FIG.

[Description of operation of recorder 1]
Next, referring to the flowchart of FIG. 16, when the total number of divisions D is designated by the user's designation operation, the recorder 1 is not designated with the designated total number of divisions D. In some cases, a content division process for dividing the entropy by the total division number D that minimizes the total entropy Q will be described.

  In step S151, the content model learning unit 12 performs the content model learning process described with reference to FIG.

  In step S152, the symbol string generation unit 14 performs the symbol string generation process described with reference to FIG.

  In step S153, based on the operation signal from the operation unit 17, the control unit 16 determines whether or not the total number of divisions D has been designated within a predetermined period by a user's designation operation.

  Then, when the control unit 16 determines that the total number of divisions D is designated by the user's designation operation based on the operation signal from the operation unit 17, the control unit 16 controls the division unit 15 to let the division unit 15 The content is divided by the total division number D designated by the designation operation.

  That is, for example, the dividing unit 15 divides the content at a dividing position (position where a dividing line is arranged) obtained by the recursive bi-dividing process of FIG. 14 or the annealing dividing process of FIG. Then, the dividing unit 15 supplies the content divided into the total number of divisions D to the content storage unit 11 for storage.

  In step S153, when the control unit 16 determines that the total division number D is not specified by the user's specifying operation based on the operation signal from the operation unit 17, the process proceeds to step S155.

  In the processing after step S155, the control unit 16 controls the dividing unit 15 to calculate the total division number D that minimizes the total entropy Q among the total number of divisions into which the content can be divided. The content to be divided is divided by the division number D.

That is, in step S155, the dividing unit 15 divides the symbol string by a predetermined total number of divisions D (for example, D = 2), for example, using a division process that is one of the recursive two-division process and the annealing division process. calculating the entropy of the total Q _D when.

In step S156, the dividing unit 15 calculates the average entropy mean (Q _D ) = Q _D / D based on the calculated total sum Q _D.

In step S157, the dividing unit 15 calculates the total entropy Q _{D + 1} when the symbol sequence is divided by the total number of divisions D + 1 using the same division process as in step S155.

In step S158, the dividing unit 15 based on the calculated Q _{D + 1,} the average entropy _{mean (Q D + 1) =} Q D + 1 / (D + 1) is calculated.

In step S159, the dividing unit 15, the average entropy mean calculated in step S158 (Q _{D + 1),} it calculates the difference Δmean obtained by subtracting the average entropy mean calculated in step S156 (Q _D).

  In step S160, the dividing unit 15 determines whether or not the difference Δmean calculated in step S159 is less than a predetermined threshold TH, and determines that the difference Δmean is not less than the threshold TH (is greater than or equal to the threshold TH). If so, the process proceeds to step S161.

  In step S161, the dividing unit 15 sets the addition result D + 1 obtained by adding 1 to the predetermined total number of divisions D as a new predetermined total number of divisions D, returns the process to step S157, and so on. Perform the process.

  In step S160, when the dividing unit 15 determines that the difference Δmean calculated in step S159 is less than the threshold value TH, the total entropy Q when the symbol string is divided by the predetermined total division number D is the smallest And the process proceeds to step S162.

  In step S162, the dividing unit 15 divides the content at the same division position as the division position where the symbol string is divided, and the content divided by the predetermined total division number D obtained by the division is stored in the content storage unit. 11 to be stored. Thus, the content dividing process in FIG. 16 is completed.

  As described above, in the content division processing of FIG. 16, when the total division number D is designated by the user's designation operation, the content is divided by the designated total division number D. For this reason, the content can be divided by the desired total division number D designated by the user.

  Further, according to the content division processing of FIG. 16, when the total number of divisions D is not specified by the user's designation operation, the content is divided by the total number of divisions D that minimizes the total entropy Q of the symbol string I made it. Therefore, it is possible to save the user from specifying the total number of divisions D when dividing the content.

  In the first embodiment, the recorder 1 divides the content into a plurality of segments that are semantically grouped. Thereby, the user of the recorder 1 can select and reproduce a desired segment (for example, a predetermined corner that is a part of a program) from a plurality of segments that are semantically grouped.

  In the first embodiment, the recorder 1 divides the content into a plurality of segments. However, the division target is not limited to the content, and may be, for example, audio data, an electroencephalogram waveform, or the like. . In other words, any data can be used as long as the data is time-series data arranged in time series.

  By the way, if a digest (summary) of the segment is generated for each segment, the user can select and reproduce a desired segment more easily by referring to the generated digest.

  For this reason, it is desirable to generate a digest for each of the plurality of segments in addition to dividing the content into a plurality of segments that are semantically grouped.

  Next, with reference to FIGS. 17 to 25, a recorder 51 that divides content into a plurality of segments that are semantically grouped and generates a digest for each of the plurality of segments will be described.

<2. Second Embodiment>
[Configuration Example of Recorder 51]
Next, FIG. 17 shows a configuration example of the recorder 51 according to the second embodiment.

  In the recorder 51 of FIG. 17, the same reference numerals are given to the same components as those of the recorder 1 (FIG. 1) according to the first embodiment. Omitted.

  That is, the recorder 51 is configured in the same manner as the recorder 1 of FIG. 1 except that a dividing unit 71 is provided instead of the dividing unit 15 of FIG. 1 and a digest generating unit 72 is newly provided. .

  The dividing unit 71 performs the same processing as the dividing unit 15 in FIG. Then, the dividing unit 71 supplies the content after being divided into a plurality of segments to the content storage unit 11 via the digest generation unit 72 and stores the content.

  In addition, the dividing unit 71 assigns a chapter ID for uniquely identifying the first frame of each segment (the frame t at the time t at which the dividing line is arranged) when the content is divided into a plurality of segments. Data is generated and supplied to the digest generation unit 72.

  In the following description, a segment obtained by dividing the content by the dividing unit 71 is also referred to as a chapter.

  Next, FIG. 18 shows an example of chapter point data generated by the dividing unit 71.

  FIG. 18 shows an example in which a dividing line is arranged at the time of each frame corresponding to frame numbers 300, 720, 1115, and 1431 among a plurality of frames constituting the content.

  That is, a chapter (segment) whose content is composed of frames corresponding to frame numbers 0 to 299, a chapter composed of frames corresponding to frame numbers 300 to 719, and each frame corresponding to frame numbers 720 to 1114 Are divided into chapters composed of frames, chapters composed of frames corresponding to frame numbers 1115 to 1430, and so on.

  Here, the frame number t is a number for uniquely identifying the t-th frame t from the beginning of the content.

  The chapter ID is associated with the first frame (frame with the smallest frame number) among the frames constituting the chapter. That is, chapter ID “0” is associated with frame 0 with frame number 0, and chapter ID “1” is associated with frame 300 with frame number 300. The chapter ID “2” is associated with the frame 720 with the frame number 720, the chapter ID “3” is associated with the frame 1115 with the frame number 1115, and the chapter ID “4” is the frame with the frame number 1431. 1431.

  The dividing unit 71 supplies a plurality of chapter IDs as shown in FIG. 18 to the digest generating unit 72 of FIG. 17 as chapter point data.

  Returning to FIG. The digest generation unit 72 reads the same content as the content read by the dividing unit 71 from the content storage unit 11.

  Further, the digest generating unit 72 identifies each chapter of the content read from the content storage unit 11 based on the chapter point data from the dividing unit 71. And the digest production | generation part 72 extracts the chapter segment of predetermined length (basic segment length) from each identified chapter.

  That is, the digest generation unit 72 extracts, from each identified chapter, a portion representing the chapter, that is, a predetermined portion from the beginning of the chapter to the basic segment length, as a chapter segment.

  The basic segment length is, for example, in the range of 5 to 10 seconds. The basic segment length can be changed by a user changing operation using the operation unit 17.

  Furthermore, the digest generation unit 72 extracts feature amount time-series data from the read content, and based on the extracted feature amount time-series data, the feature peak that is a characteristic portion of the basic segment length is extracted from each chapter. Extract a segment.

  The feature amount time-series data represents time-series feature amounts used when extracting feature peak segments. Details of the feature amount time-series data will be described later.

  Moreover, the digest production | generation part 72 may extract a characteristic peak segment by the length different from a chapter segment. That is, the basic segment length of the chapter segment and the basic segment length of the feature peak segment can be different.

  Furthermore, the digest generation unit 72 may extract one feature peak segment from one chapter, or may extract a plurality of feature peak segments. Moreover, the digest production | generation part 72 does not necessarily need to extract the characteristic peak segment from each chapter.

  The digest generation unit 72 generates a digest representing the rough contents of the content by arranging the chapter segments and feature peak segments extracted from each chapter in time series, and supplies the digest to the content storage unit 11 for storage.

  In addition, the digest production | generation part 72 can extract until just before the switching of a scene as a chapter segment, when switching of the remarkable scene has generate | occur | produced within the period which should be extracted as a chapter segment.

  Thereby, the digest production | generation part 72 becomes possible [extracting the chapter segment divided | segmented in the good place of a division | segmentation]. The same applies to the feature peak segment.

  Note that the digest generation unit 72, for example, determines whether significant scene switching has occurred based on whether or not the sum of absolute differences of pixels between temporally adjacent frames is equal to or greater than a predetermined threshold. Determine whether or not.

  Further, for example, the digest generation unit 72 may detect an utterance section in which an utterance is being performed in the chapter based on the voice data of the identified chapter.

  And the digest production | generation part 72 can be comprised so that it may extract as a chapter segment until the utterance is complete | finished, even if the period which should be extracted as a chapter segment passes. The same applies to the feature peak segment.

  In addition, when the utterance section is sufficiently longer than the basic segment length, that is, for example, when the utterance section is twice or more the basic segment length, the digest generation unit 72 determines the chapter segment that was cut off during the utterance. May be extracted. The same applies to the feature peak segment.

  In this case, it is desirable to add to the chapter segment an effect that does not make the user feel uncomfortable due to the chapter segment being interrupted in the middle of speech.

  That is, for example, it is desirable that the digest generating unit 72 applies an effect such that the utterance in the extracted chapter segment fades out (the voice of the utterance is gradually reduced) as the chapter segment ends.

  Incidentally, the digest generation unit 72 extracts chapter segments and feature peak segments from the content divided by the division unit 71.

  However, for example, when the user divides content into a plurality of chapters using editing software or the like, chapter segments and feature peak segments can be extracted for the content. Note that chapter point data is generated by editing software or the like when a user divides content into a plurality of chapters using editing software or the like.

  In the following description, it is assumed that the digest generation unit 72 extracts one chapter segment and one feature peak segment from each chapter and adds only BGM to the generated digest.

  Next, FIG. 19 shows an outline of the digest generation process performed by the digest generation unit 72.

  FIG. 19 shows dividing lines for dividing the content to be digest extracted into a plurality of chapters. A corresponding chapter ID is shown on the dividing line.

  In FIG. 19, for example, voice power time series data 91 and face area time series data 92 are shown as feature amount time series data.

  Here, the audio power time-series data 91 refers to time-series data having a larger value as the audio of the frame t is larger. The face area time-series data refers to time-series data having a larger value as the face (ratio) displayed in the frame t is larger.

  In FIG. 19, the horizontal axis represents time t when content is reproduced, and the vertical axis represents feature amount time series data.

  Further, in FIG. 19, a white rectangle represents a chapter segment indicating the head portion of the chapter, and a rectangle indicated by diagonal lines represents a feature peak segment extracted based on the audio power time series data 91. A black rectangle represents a feature peak segment extracted based on the face area time-series data 92.

  The digest generating unit 72 identifies each chapter of the content read from the content storage unit 11 based on the chapter point data (chapter ID) from the dividing unit 71, and extracts the chapter segment of each identified chapter.

  Moreover, the digest production | generation part 72 extracts the audio | voice power time series data 91 as shown, for example in FIG. 19 from the content read from the content memory | storage part 11. FIG.

  Further, the digest generation unit 72 extracts, as a peak feature frame, a frame when the audio power time-series data 91 has the maximum value in each identified chapter.

  Then, the digest generating unit 72 extracts a feature peak segment including the extracted peak feature frame (for example, a feature peak segment starting from the peak feature frame) from the chapter.

  For example, the digest generation unit 72 determines the extraction points of the peak feature frames at regular intervals. And the digest production | generation part 72 may extract the flame | frame when the audio | voice power time series data 91 becomes the maximum value in the range determined based on the determined extraction point as a peak feature frame.

  For example, the digest generation unit 72 may not extract the peak feature frame when the maximum value of the audio power time-series data 91 is equal to or less than a predetermined threshold. In this case, the digest generation unit 72 does not extract the feature peak segment.

  Further, for example, the digest generation unit 72 may extract a frame when the audio power time-series data 91 becomes a maximum value instead of the maximum value of the audio power time-series data 91 as a peak feature frame. .

  For example, the digest generation unit 72 extracts feature peak segments using a plurality of feature amount time-series data in addition to extracting feature peak segments using one audio power time-series data 91. May be.

  That is, for example, the digest generation unit 72 extracts the face area time-series data 92 in addition to the audio power time-series data 91 from the content read from the content storage unit 11.

  Moreover, the digest production | generation part 72 selects the feature-value time series data in which the maximum value in a chapter becomes large among the audio | voice power time series data 91 and the face area time series data 92.

  Then, the digest generation unit 72 extracts a frame when the selected feature amount time-series data has the maximum value in the chapter as a peak feature frame, and extracts a feature peak segment including the extracted peak feature frame from the chapter. To do.

  In this case, the digest generation unit 72 extracts a portion where the voice is loud in a predetermined chapter as a feature peak segment, and extracts a portion where the ratio of the face is large in other chapters as a feature peak segment. It will be.

  For this reason, in the digest production | generation part 72, it can prevent that a monotonous digest is produced | generated by extracting only the part where the audio | voice is loud, for example as a feature peak segment.

  That is, the digest generation unit 72 can generate a digest with randomness as if the feature peak segment was extracted at random.

  Thereby, the digest production | generation part 72 can prevent the situation where the user who views a digest gets tired because the produced | generated digest forms a pattern.

  In addition, for example, the digest generation unit 72 may extract a feature peak segment for each of a plurality of feature time series data.

  That is, for example, the digest generation unit 72 extracts a feature peak segment including, as a peak feature frame, a frame when the audio power time-series data 91 has the maximum value in each identified chapter. Moreover, the digest production | generation part 72 also extracts the feature peak segment which contains the frame when the face area time series data 92 becomes the maximum value as a peak feature frame. In this case, the digest generation unit 72 extracts two feature peak segments from one chapter.

  Note that, as shown in the lower right of FIG. 19, chapter segments (shown in white rectangles) and features from the dividing line corresponding to chapter ID = 4 to the dividing line corresponding to chapter ID = 5 The peak segment (indicated by the hatched rectangle) is extracted in an overlapping state.

  In this case, the digest generation unit 72 treats the chapter segment and the feature peak segment as one segment.

  For example, the digest generating unit 72 generates a digest by connecting the chapter segments and feature peak segments extracted as shown in FIG. 19 in time series.

  The digest generation unit 72 adds BGM (background music) or the like to the generated digest, and supplies the digest with the BGM added to the content storage unit 11 for storage.

[Details of digest generation unit 72]
Next, FIG. 20 illustrates a detailed configuration example of the digest generation unit 72.

  The digest generation unit 72 includes a chapter segment extraction unit 111, a feature amount extraction unit 112, a feature peak segment extraction unit 113, and an effect addition unit 114.

  The content is supplied from the content storage unit 11 to the chapter segment extraction unit 111 and the feature amount extraction unit 112.

  Further, chapter point data is supplied from the dividing unit 71 to the chapter segment extracting unit 111 and the feature peak segment extracting unit 113.

  The chapter segment extraction unit 111 identifies each chapter of the content supplied from the content storage unit 11 based on the chapter point data from the division unit 71. Then, the chapter segment extraction unit 111 extracts chapter segments from the identified chapters and supplies them to the effect addition unit 114.

  The feature amount extraction unit 112 extracts, for example, a plurality of feature amount time-series data from the content supplied from the content storage unit 11 and supplies the extracted feature amount time series data to the feature peak segment extraction unit 113. The feature amount time-series data will be described in detail with reference to FIGS.

  The feature quantity extraction unit 112 smooths the extracted feature quantity time-series data using a smoothing filter (smoothing filter) or the like to remove noise generated in the feature quantity time-series data. The feature peak segment extraction unit 113 may be supplied.

  Further, the feature amount extraction unit 112 supplies the content from the content storage unit 11 to the feature peak segment extraction unit 113 as it is.

  The feature peak segment extraction unit 113 identifies each chapter of the content supplied from the content storage unit 11 via the feature amount extraction unit 112 based on the chapter point data from the division unit 71.

  Further, the feature peak segment extraction unit 113, based on the plurality of feature amount time series data supplied from the feature amount extraction unit 112, from each identified chapter, the feature peak segment as described with reference to FIG. Is extracted and supplied to the effect adding unit 114.

  For example, the effect adding unit 114 generates a digest by connecting the chapter segments and feature peak segments extracted as shown in FIG. 19 in time series.

  Further, the effect adding unit 114 adds BGM or the like to the generated digest, and supplies it to the content storage unit 11 for storage. The process in which the effect adding unit 114 adds BGM or the like to the digest will be described in detail with reference to FIG.

  Further, the effect adding unit 114 may add an effect such as fading out a frame just before the end of each segment (chapter segment or feature peak segment) constituting the generated digest, or fading in a frame immediately after the start. it can.

[Example of feature time-series data]
Next, with reference to FIGS. 21 to 23, a method in which the feature amount extraction unit 112 in FIG. 20 extracts (generates) feature amount time-series data from content will be described.

  Note that the feature quantity extraction unit 112 uses at least one of, for example, face area time series data, audio power time series data, zoom-in intensity time series data, or zoom-out intensity time series data as content quantity time-series data, Extract from

  Here, in the face area time-series data, the segment including the frame when the ratio of the face area (face area) displayed on the frame is increased in the feature peak segment extraction unit 113 as the feature peak segment. Used when extracting from chapters.

The feature amount extraction unit 112 detects a face area (the number of pixels) that is a region where a human face exists from each frame t constituting the content. Then, the feature amount extraction unit 112 calculates the face region feature value f ₁ (t) = R _t -ave (R _{t ′} ) for each frame t based on the detection result, thereby obtaining a time series of the frame t. In addition, face area time-series data obtained by arranging the face area feature values f ₁ (t) is generated.

Note that the ratio R _t = the number of pixels in the face area / the total number of pixels in the frame, and ave (R _{t ′} ) is the ratio R _t obtained from the frame t ′ existing in the section [tW _L , t + W _L ]. Represents the average of _' . The time t represents the time at which the frame t is displayed, and the value W _L (> 0) is a preset value.

  Next, FIG. 21 illustrates an example when the feature quantity extraction unit 112 generates audio power time series data as the feature quantity time series data.

In Figure 21, the audio data x (t) is the entire interval [t _s, t _e] from time t _s to time t _e represents the audio data to be reproduced in. The horizontal axis represents time t, and the vertical axis represents audio data x (t).

  Here, the voice power time-series data is used when the segment including the frame when the voice (volume) is increased is extracted from the chapter as the feature peak segment in the feature peak segment extraction unit 113.

  The feature quantity extraction unit 112 calculates the audio power P (t) of each frame t constituting the content by the following equation (3).

  Here, in Expression (3), the voice power P (t) represents the square root of the square sum of each voice data x (τ) in the section [t−W, t + W]. Also, τ is a value from t−W to t + W, and W is preset.

Then, the feature extraction unit 112, the interval [tW, t + W] from the average value of the audio calculated at power P (t), the entire interval [t _s, t _e] sound calculated in power P of (t) A difference value obtained by subtracting the average value is calculated as an audio power feature value f ₂ (t) in frame t.

The feature amount extraction unit 112 calculates the sound power feature value f ₂ (t) for each frame t, thereby obtaining the sound power feature value f ₂ (t) in time series of the frame t. Generate voice power time-series data.

  Next, a method in which the feature amount extraction unit 112 generates zoom-in intensity time series data as feature amount time series data will be described with reference to FIGS.

  Note that the zoom-in intensity time-series data is used when the segment including the frame when zoomed in (zoomed up) is extracted from the chapter as the feature peak segment by the feature peak segment extraction unit 113.

  FIG. 22 shows an example of the motion vector of frame t.

  FIG. 22 shows a frame t divided into a plurality of blocks. Each block of the frame t shows a motion vector of the block.

  The feature quantity extraction unit 112 divides each frame t constituting the content into a plurality of blocks as shown in FIG. Then, the feature amount extraction unit 112 detects a motion vector of the block by block matching or the like for each of a plurality of blocks using each frame t constituting the content.

  Here, the motion vector of the block in the frame t is, for example, a vector representing the motion of the block from the frame t to the frame t + 1.

  Next, FIG. 23 shows an example of a zoom-in template composed of motion vectors for which the inner product with the motion vector of each block of the frame t is calculated.

  As shown in FIG. 23, this zoom-in template is composed of motion vectors representing the motion of each block when zoomed in.

Feature quantity extracting unit 112, and each block of the motion vector a _t (FIG. 22) in the frame t, corresponding respectively, the inner product a _t · b between the motion vector b of each block (FIG. 23) of the zoom template calculated calculates the calculation result sum sum of (a _t · b).

In addition, the feature quantity extraction unit 112 calculates the average value ave (sum (a _{t ′} · b) of the sum sum (a _{t ′} · b) calculated for each frame t ′ included in the section [tW, t + W]. ) Is calculated.

Then, the feature amount extraction unit 112 calculates the difference value obtained by subtracting the average value ave (sum (a _{t ′} · b)) from the sum total sum (a _t · b) as the zoom-in feature amount value f ₃ in the frame t. Calculate as (t). The zoom-in feature value f ₃ (t) is proportional to the size of the zoom-in at the frame t.

The feature amount extraction unit 112 calculates the zoom-in feature value f ₃ (t) for each frame t, thereby obtaining the zoom-in intensity obtained by arranging the zoom-in feature value f ₃ (t) in time series of the frame t. Generate time series data.

  Here, the zoom-out intensity time-series data is used when the segment including the frame when zoomed out is extracted from the chapter as the feature peak segment in the feature peak segment extraction unit 113.

  When generating the zoom-out intensity time-series data, the feature amount extraction unit 112 replaces the zoom-in template as shown in FIG. 23 with a motion vector in the opposite direction to the motion vector of the template shown in FIG. Used as a zoom-up template.

  That is, the feature amount extraction unit 112 generates zoom-up intensity time-series data using the zoom-out template in the same manner as when generating zoom-in intensity time-series data.

  Next, FIG. 24 illustrates details when the effect adding unit 114 adds BGM to the generated digest.

  On the upper side of FIG. 24, the weight of the volume of each segment (chapter segment and feature peak segment) constituting the digest is shown.

  24 shows a digest obtained by combining the chapter segment and the characteristic peak segment shown in FIG.

  The effect addition unit 114 combines the chapter segments from the chapter segment extraction unit 111 and the feature peak segments from the feature peak segment extraction unit 113 in time series as shown in the lower side of FIG. Generate a digest of L seconds.

  Here, the length L of the digest is determined by the number and length of chapter segments extracted by the chapter segment extraction unit 111 and the number and length of feature peak segments extracted by the feature peak segment extraction unit 113.

  For example, the user can set the digest length L using the operation unit 17. That is, the operation unit 17 supplies an operation signal corresponding to the length L setting operation by the user to the control unit 16. Based on the operation signal from the operation unit 17, the control unit 16 controls the digest generation unit 72 to cause the digest generation unit 72 to generate the digest having the length L set by the setting operation.

  The digest generation unit 72 extracts chapter segments and feature peak segments until the total length (total length) of the extracted segments reaches the length L.

  In this case, the digest generation unit 72 preferentially extracts chapter segments from each chapter, and then extracts feature peak segments so that at least chapter segments are extracted from each chapter. desirable.

  In addition, for example, when the digest generation unit 72 extracts the feature peak segment after extracting the chapter segment preferentially from each chapter, in one or a plurality of feature amount time-series data, Corresponding feature peak segments are extracted.

  Further, for example, the user can perform a desired length by performing a setting operation for setting the sum S of the lengths of segments extracted from one chapter together with the length L of the digest using the operation unit 17. The digest L is generated by the digest generation unit 72.

  In this case, the operation unit 17 supplies an operation signal corresponding to the user's setting operation to the control unit 16. The control unit 16 identifies L and S set by the user based on the operation signal from the operation unit 17, and calculates (reverse calculation) the total number of divisions D based on the identified L and S.

  That is, the total division number D is an integer value closest to L / S (for example, a value obtained by rounding L / S). For example, when the setting operation by the user is set to L = 30 and the chapter segment of 7.5 seconds and the feature peak segment of 7.5 seconds are set to be extracted from the chapter, that is, S = 15 Consider the case where (7.5 + 7.5) is set.

  In this case, the control unit 16 calculates L / S = 30/15 = 2 based on L = 30 and S = 15, and sets the integer value 2 closest to L / S = 2 as the total division number D. calculate.

  The control unit 16 controls the dividing unit 71 to cause the dividing unit 71 to generate chapter point data corresponding to the calculated total division number D. Thereby, the dividing unit 71 generates chapter point data corresponding to the calculated total number of divisions D in accordance with the control from the control unit 16 and supplies the generated chapter point data to the digest generating unit 72.

  The digest generation unit 72 generates a digest of length L set by the user based on the chapter point data from the division unit 71 and the content read from the content storage unit 11 and supplies the digest to the content storage unit 11. To remember.

  Further, the effect adding unit 114 weights the audio data of each segment (chapter segment or feature peak segment) constituting the digest with the weight α as shown in the upper side of FIG. Is weighted 1-α.

  Then, the effect adding unit 114 mixes the weighted audio data and the weighted BGM, and uses the resulting mixed audio data as the audio data of each segment that constitutes the digest. Associate with.

  The effect adding unit 114 is assumed to store BGM (data) in a built-in memory (not shown) in advance, and the BGM to be added is designated in accordance with a user operation.

  That is, for example, when adding BGM to a chapter segment indicated by a white rectangle, the effect adding unit 114 weights the audio data of the chapter segment with a weight smaller than 0.5 in order to set the volume of the BGM higher. (Multiply).

  Specifically, for example, in FIG. 24, the effect adding unit 114 weights the chapter segment audio data by 0.2 and weights the BGM (data) to be added by 0.8.

  In addition, for example, when the effect adding unit 114 adds BGM to a feature peak segment extracted based on feature amount time-series data different from the audio power time-series data among a plurality of feature amount time-series data, the chapter segment Is added with the same weighting as when adding BGM.

  Specifically, for example, in FIG. 24, the effect adding unit 114 weights and adds 0.2 to the audio data of the feature peak segment (indicated by a black rectangle) extracted based on the face area time-series data. Weight BGM 0.8.

  Further, for example, when adding the BGM to the feature peak segment (indicated by the hatched rectangle) extracted based on the audio power time-series data, the effect adding unit 114 sets the BGM volume to a lower level. A weight greater than 0.5 is weighted to the voice data of the feature peak segment.

  Specifically, for example, in FIG. 24, the effect adding unit 114 weights the feature peak segment audio data extracted based on the audio power time-series data by 0.8, and weights the BGM to be added by 0.2.

  As shown in FIG. 19, for example, when the chapter segment and the feature peak segment are extracted in an overlapped (overlapped) state, they are extracted as one segment.

  In this case, the effect adding unit 114 should apply the time of the first frame to the feature peak segment that is later in time as the weight applied to the audio data of one segment composed of the chapter segment and the feature peak segment. Weights are used.

  Further, for example, as shown in the upper side of FIG. 24, the effect adding unit 114 changes the weight switching continuously instead of discontinuously.

  That is, for example, the effect adding unit 114 does not switch the weight of the digest audio data from 0.2 to 0.8 discontinuously, but changes linearly from 0.2 to 0.8 in a predetermined time (for example, 500 milliseconds). Let Note that the effect adding unit 114 may change the weight in a non-linear manner (for example, change the weight in proportion to the square of time).

  As a result, it is possible to prevent a situation in which the volume of the digest or the volume of the BGM suddenly increases when the weighting is switched, so that the user does not have to feel unpleasant due to a sudden change in the volume.

[Description of operation of recorder 51]
Next, digest generation processing performed by the recorder 51 (particularly, the dividing unit 71 and the digest generating unit 72) will be described with reference to the flowchart of FIG.

  In step S191, the dividing unit 71 performs the same process as the dividing unit 15 in FIG. Then, the dividing unit 71 generates, as chapter point data, a chapter ID for uniquely identifying the head frame of each segment when the content is divided into a plurality of segments.

  The dividing unit 71 supplies the generated chapter point data to the chapter segment extracting unit 111 and the feature peak segment extracting unit 113 of the digest generating unit 72.

  In step S 192, the chapter segment extraction unit 111 identifies each chapter of the content supplied from the content storage unit 11 based on the chapter point data from the division unit 71. Then, the chapter segment extraction unit 111 extracts a chapter segment representing the head part of the chapter from each identified chapter, and supplies it to the effect addition unit 114.

  In step S 193, the feature quantity extraction unit 112 extracts, for example, a plurality of feature quantity time-series data from the content supplied from the content storage unit 11, and supplies it to the feature peak segment extraction unit 113.

  Note that the feature amount extraction unit 112 smooths the extracted feature amount time-series data using a smoothing filter (smoothing filter) or the like to remove noise generated in the feature amount time-series data. The feature peak segment extraction unit 113 may be supplied.

  Further, the feature amount extraction unit 112 supplies the content from the content storage unit 11 to the feature peak segment extraction unit 113 as it is.

  In step S194, the feature peak segment extraction unit 113 identifies each chapter of the content supplied from the content storage unit 11 via the feature amount extraction unit 112 based on the chapter point data from the division unit 71.

  The feature peak segment extraction unit 113 extracts feature peak segments from the identified chapters based on the plurality of feature amount time-series data supplied from the feature amount extraction unit 112, and supplies the extracted feature peak segments to the effect addition unit 114. To do.

  In step S195, the effect adding unit 114 generates a digest by, for example, connecting the chapter segments and feature peak segments extracted as shown in FIG. 19 in time series.

  Then, the effect adding unit 114 adds BGM (background music) or the like to the generated digest, and supplies it to the content storage unit 11 for storage. Thus, the digest generation process of FIG. 25 is completed.

  As described above, according to the digest generation process, the chapter segment extraction unit 111 extracts chapter segments from each chapter. The effect adding unit 114 generates a digest having at least the extracted chapter segment.

  For this reason, for example, the user can view the chapter segment that is the head of each chapter of the content by playing the digest, for example, so that the user can easily grasp the rough content (summary) of the content. It becomes.

  Further, according to the digest generation process, the feature peak segment extraction unit 113 extracts, for example, as a feature peak segment based on a plurality of feature amount time-series data.

  For this reason, it is possible to generate a digest that includes, for example, a scene that is a mountainous area as a feature peak segment in the content that is the digest generation target.

  Here, as the feature peak segment, for example, a scene in which the voice is loud, a scene in which zoom-in or zoom-out is performed, a scene in which the ratio of human faces is increased, and the like are extracted.

  For example, the effect adding unit 114 generates a digest to which an effect such as BGM is added. For this reason, according to the digest generation process, a digest that makes it easier to understand the content is generated.

  Furthermore, since the effect adding unit 114 gradually switches the weighting when mixing BGM, the situation where the sound of the BGM or the original sound of the digest suddenly increases when the weighting is switched is prevented. Is possible.

  By the way, it is desirable that the user can easily reproduce the content from a desired reproduction position when reproducing the content stored in the content storage unit 11.

  Next, a recorder 131 that displays a display screen that allows a user to easily search for a desired reproduction position will be described with reference to FIGS. 26 to 41.

<3. Third Embodiment>
[Configuration Example of Recorder 131]
FIG. 26 shows a configuration example of the recorder 131 according to the third embodiment.

  In the recorder 131 of FIG. 26, the same reference numerals are given to the same components as those of the recorder 1 (FIG. 1) according to the first embodiment. Omitted.

  That is, the recorder 131 is configured in the same manner as the recorder 1 in FIG. 1 except that a dividing unit 151 is provided instead of the dividing unit 15 in FIG. 1 and a presentation unit 152 is newly provided.

  The recorder 131 is connected to a display unit 132 that displays an image. Further, the recorder 131 omits the digest generation unit 72 of FIG. 17, but the digest generation unit 72 may be provided as in the case of FIG. 17.

  The dividing unit 151 performs the same processing as the dividing unit 15 in FIG. Further, the dividing unit 151 generates chapter point data (chapter ID) in the same manner as the dividing unit 71 in FIG.

  Furthermore, the dividing unit 151 supplies each symbol constituting the symbol string supplied from the symbol string generating unit 14 to the presentation unit 152 in association with each corresponding frame constituting the content.

  In addition, the dividing unit 151 supplies the content read from the content storage unit 11 to the presentation unit 152.

  Based on the chapter point data from the dividing unit 151, the presentation unit 152 causes the display unit 132 to display the chapters of the content supplied from the dividing unit 151 in a row.

  That is, for example, the presentation unit 152 causes the display unit 132 to display chapters of the total number of divisions D that change according to the user's designation operation using the operation unit 17 so as to be arranged in rows.

  Specifically, for example, the dividing unit 151 generates new chapter point data corresponding to the changed total number of divisions D in response to the change in the total number of divisions D by the user's designation operation. This is supplied to the presentation unit 152.

  The presentation unit 152 causes the display unit 132 to display the chapters having the total number of divisions D specified by the user's specifying operation based on the new chapter point data supplied from the division unit 151.

  In addition, as illustrated in FIG. 39 described later, the presentation unit 152 displays a frame having the same symbol as the frame selected by the user in a tile shape using the symbols from the division unit 151.

  Next, FIG. 27 shows an example of how the corresponding chapter point data changes in accordance with the change of the total number of divisions D by the user's designation operation.

  FIG. 27A shows an example of a combination of the total number of divisions D and chapter point data corresponding to the total number of divisions D.

  FIG. 27B shows an example of chapter points arranged on the content time axis. Here, the chapter point represents the position where the first frame is arranged among the frames constituting the chapter.

  As shown in FIG. 27A, when the total division number D = 2, in addition to the frame with frame number 0, the frame with frame number 720 is set as the chapter point.

  When the total number of divisions D = 2, as shown in the first row of B in FIG. 27, the content is divided into chapters starting with the frame number 0 and chapters starting with the frame number 720. Will be.

  Since the frame with frame number 0 is always a chapter point, the illustration of frame number 0 is omitted in FIGS. 27A and 27B.

  When the total division number D = 2 to the total division number D = 3, the frame with the frame number 300 is newly set as a chapter point.

  When the total number of divisions D = 3, as shown in the second row of B in FIG. 27, the content includes a chapter that starts with a frame with frame number 0, a chapter that starts with a frame with frame number 300, and a frame. That is, the frame is divided into chapters starting with the frame of the number 720.

  When the total number of divisions D = 3 to total number of divisions D = 4, the frame with frame number 1431 is newly set as a chapter point.

  When the total number of divisions D = 4, as shown in the third row of B in FIG. 27, the content is a chapter starting from the frame with frame number 0, a chapter starting from the frame with frame number 300, and the frame number. This is divided into a chapter starting from the 720th frame and a chapter starting from the frame of the frame number 1431.

  Further, when the total number of divisions D = 4 to the total number of divisions D = 5, the frame with frame number 1115 is newly set as a chapter point.

  When the total number of divisions D = 5, as shown in the fourth line of B of FIG. 27, the content is a chapter starting from the frame with frame number 0, a chapter starting from the frame with frame number 300, and the frame number. This is divided into a chapter starting at the frame 720, a chapter starting at the frame number 1115, and a chapter starting at the frame number 1431.

  Next, a process in which the presentation unit 152 generates display data to be displayed on the display unit 132 will be described with reference to FIGS. FIGS. 28 to 30 illustrate how the presentation unit 152 generates display data when the total number of divisions D = 5.

  FIG. 28 shows an example of a frame that is a chapter point.

  In FIG. 28, a rectangle represents a frame, and a number written in the rectangle represents a frame number.

  The presentation unit 152 extracts each frame of frame numbers 0, 300, 720, 1115, and 1431 determined as chapter points from the content supplied from the division unit 151 based on the chapter point data from the division unit 151.

  In this case, it is assumed that the chapter point data corresponds to the total number of divisions D = 5, and each frame of frame numbers 0, 300, 720, 1115, and 1431 is a chapter point.

  The presentation unit 152 reduces each extracted frame to a thumbnail image, and displays the thumbnail images in the order of frame numbers 0, 300, 720, 1115, and 1431 on the display screen of the display unit 132 from top to bottom as shown in FIG. Let

  Then, the presentation unit 152 displays the frames constituting the chapter as thumbnail images on the display screen of the display unit 132 from the left to the right, for example, at intervals of 50 frames.

  Next, FIG. 29 shows an example in which thumbnail images are displayed at intervals of 50 frames in the right direction of the frame set as the chapter point.

  Based on the chapter point data from the dividing unit 151, the presenting unit 152 extracts each frame of frame numbers 50, 100, 150, 200, and 250 from the content supplied from the dividing unit 151, in addition to the frame with frame number 0 set as the chapter point.

  Then, the presentation unit 152 reduces each extracted frame to a thumbnail image, and displays the thumbnail image in the right direction from the frame of frame number 0 in the order of frame numbers 50, 100, 150, 200, and 250.

  In addition, the presentation unit 152 displays the frames of the frame numbers 350, 400, 450, 500, 550, 600, 650, and 700 in the right direction from the frame of the frame number 300 as thumbnail images in ascending order of the frame numbers.

  Further, the presentation unit 152 similarly displays each frame of frame numbers 770, 820, 870, 920, 970, 1020, and 1070 as thumbnail images in order from the smallest frame number in the right direction from the frame of frame number 720. Further, the presentation unit 152 displays the frames with the frame numbers 1165, 1215, 1265, 1315, 1365, and 1415 as thumbnail images in the order from the smallest frame number in the right direction from the frame with the frame number 1115. Further, the presentation unit 152 displays the frames with the frame numbers 1481, 1531, 1581, 1631,... In the right direction from the frame with the frame number 1431 as thumbnail images in ascending order of the frame numbers.

  Thereby, as shown in FIG. 30, the presentation unit 152 can cause the display unit 132 to display a display in which chapter thumbnail images are arranged in rows for each chapter.

  In addition to arranging the thumbnail images of chapters in a row, the presentation unit 152 may arrange other thumbnail images so as to overlap the thumbnail images.

  Specifically, for example, the presentation unit 152 may display the frame with the frame number 300 as a thumbnail image and arrange the thumbnail images of the frames with the frame numbers 301 to 349 so as to be hidden by the frame. .

  Next, FIG. 30 shows an example of the display screen of the display unit 132.

  In this display screen, as shown in FIG. 30, the thumbnail images of each chapter are displayed in chapter display areas (chapter numbers 1, 2, 3, 4, and 5 respectively) provided for each chapter. ) Are displayed in rows.

  That is, in the first row, frames of frame numbers 0, 50, 100, 150, 200,... Are arranged in the order from the left to the right in the figure as thumbnail images of the first chapter 1 from the top of the content.

  That is, the display unit 132 displays the thumbnail image as a representative image that represents each scene of the chapter 1.

  Specifically, for example, the display unit 132 displays a thumbnail image corresponding to the frame with frame number 0 as a representative image representing a scene composed of the frames with frame numbers 0 to 49. The same applies to chapters 2 to 5 shown in FIG.

  In the second row, frames of frame numbers 300, 350, 400, 450, 500,... Are arranged in the order from the left to the right in the figure as thumbnail images of the second chapter 2 from the top of the content.

  Further, in the third row, frames of frame numbers 720, 770, 820, 870, 920,... Are arranged in the order from the left to the right in the figure as thumbnail images of the third chapter 3 from the top of the content. Also, in the fourth line, as thumbnail images of the fourth chapter 4 from the beginning of the content, frames with frame numbers 1115, 1165, 1215, 1265, 1315,... Are arranged in the order from left to right in the figure. Be placed.

  Also, in the fifth line, as thumbnail images of the fifth chapter 5 from the top of the content, the frames with frame numbers 1431, 1481, 1531, 1581, 1631,... Placed in.

  In addition, as shown in FIG. 30, a slider 171 can also be displayed on the display screen of the display unit 132. The slider 171 is moved (slid) in the left-right direction in the figure when setting the total number of divisions D, and the total number of divisions D can be changed according to the position of the slider 171.

  That is, for example, the total division number D decreases as the slider 171 moves in the left direction in the figure, and the total division number D increases as the slider 171 moves in the right direction in the figure.

  Therefore, for example, when the user performs an operation of moving the slider 171 of the display screen shown in FIG. 30 in the left direction in the drawing using the operation unit 17, the display unit 132 corresponds to the operation. A display screen as shown in FIG. 31 is displayed.

  The dividing unit 151 generates chapter point data of the total number of divisions D corresponding to the slide operation in response to the user's slide operation using the slider 171, and supplies the generated chapter point data to the presentation unit 152. To do.

  The presentation unit 152 generates a display screen as shown in FIG. 31 based on the chapter point data from the dividing unit 151 and causes the display unit 132 to display the display screen.

  Further, each time the user performs a slide operation, the dividing unit 151 may generate chapter point data of the total number of divisions D corresponding to the slide operation, or for each of a plurality of different total number of divisions D. The chapter point data may be generated in advance.

  The division unit 151 supplies chapter point data for a plurality of different total division numbers D to the presentation unit 152 when chapter point data for each of a plurality of different total division numbers D is generated in advance.

  In this case, the presentation unit 152 includes chapter points of the total number of divisions D corresponding to the user's slide operation using the slider 171 among the chapter point data for each of the plurality of different total division numbers D supplied from the division unit 151. Select. Then, the presentation unit 152 generates a display screen to be displayed on the display unit 132 based on the selected chapter point data, and supplies the display screen to the display unit 132 for display.

  Next, FIG. 31 shows an example of a display screen displayed on the display unit 132 when the slider 171 is moved in the direction in which the total division number D decreases.

  The display screen shown in FIG. 31 shows that the number of chapters (total number of divisions D) is reduced from 5 to 3 compared to the display screen shown in FIG.

  In addition, for example, the presentation unit 152 may extract feature amount time-series data from the content from the dividing unit 151 in the same manner as the feature amount extraction unit 112 in FIG. Then, the presentation unit 152 may modify the thumbnail image displayed on the display unit 132 according to the intensity (size) of the extracted feature amount time-series data.

  Next, FIG. 32 shows another example of the display screen of the display unit 132 on which thumbnail images modified according to the strength of the feature amount time-series data are displayed.

  Note that a band display is appropriately added to the thumbnail image shown in FIG. 32 according to the characteristics of a scene including a frame corresponding to the thumbnail image (for example, 50 frames starting from the frame corresponding to the thumbnail image). Is done.

  The band displays 191a to 191f are respectively added to thumbnail images representing scenes with a relatively high face area ratio.

  Now, band displays 191a to 191f are added to the thumbnail images of frame numbers 100, 150, 350, 400, 450, and 1581, respectively.

  Further, the band displays 192a to 192d are added to thumbnail images representing scenes having a relatively high face area ratio and relatively high audio power.

  Furthermore, the band displays 193a and 193b are respectively added to thumbnail images representing scenes with relatively high audio power.

  The band indications 191a to 191f are displayed when, for example, the number of frames in which the ratio of the face area is equal to or greater than a predetermined threshold among the frames constituting the scene is equal to or greater than a predetermined number threshold. Is added to the thumbnail image representing the.

  In addition, for example, in the band displays 191a to 191f, for example, the color of the band displays 191a to 191f becomes darker as the number of frames in which the ratio of the face area is equal to or greater than a predetermined threshold among the frames constituting the scene increases. You may make it do.

  The same applies to the band displays 192a to 192d and the band displays 193a and 193b.

  In FIG. 32, the band display is added to the thumbnail image. However, for example, a human face may be added instead of the band displays 191a to 191f. That is, any display method may be used as long as it represents the characteristics of the scene.

  In FIG. 32, a frame number is assigned to identify each thumbnail image, but in reality, the display screen of the display unit 132 is displayed as shown in FIG. 33, for example.

[Details of presentation unit 152]
Next, FIG. 34 shows a detailed configuration example of the presentation unit 152 of FIG.

  The presentation unit 152 includes a feature amount extraction unit 211, a display data generation unit 212, and a display control unit 213.

  Content is supplied from the dividing unit 151 to the feature amount extracting unit 211. The feature quantity extraction unit 211 extracts feature quantity time-series data and supplies it to the display data generation unit 212 in the same manner as the feature quantity extraction unit 112 in FIG.

  That is, for example, the feature amount extraction unit 211 extracts at least one of face area time-series data, audio power time-series data, zoom-in intensity time-series data, or zoom-out intensity time-series data from the content from the dividing unit 151. And supplied to the display data generation unit 212.

  In addition to the feature amount time-series data from the feature amount extraction unit 211, chapter point data is supplied from the division unit 151 to the display data generation unit 212.

  Based on the feature time series data from the feature amount extraction unit 211 and the chapter point data from the division unit 151, the display data generation unit 212 displays the display screen of the display unit 132 as shown in FIGS. 31 to 33. Display data for making a correct display is generated and supplied to the display control unit 213.

  The display control unit 213 causes the display screen of the display unit 132 to display as shown in FIGS. 31 to 33 based on the display data from the display data generation unit 212.

  Note that the display data generation unit 212 generates display data corresponding to a user operation and supplies the display data to the display control unit 213.

  Then, the display control unit 213 changes the display screen of the display unit 132 according to a user operation based on the display data from the display data generation unit 212.

  In other words, there are three display modes when the display control unit 213 controls the display of the chapters of the content: a layer 0 mode, a layer 1 mode, and a layer 2 mode.

  In the layer 0 mode, the display unit 132 displays as shown in FIGS.

  Next, FIG. 35 illustrates an example of a state when the user indicates the position on the display screen of the display unit 132 in the layer 0 mode.

  Here, in order to make the explanation easy to understand, it is assumed that, for example, a mouse is employed as the operation unit 17. The user can perform a single click or a double click, for example, using the operation unit 17 as a mouse. The operation unit 17 is not limited to a mouse.

  In the layer 0 mode, when the user operates the operation unit 17 as a mouse to move the pointer (cursor) 231 onto the fifth thumbnail image from the left in FIG. 213 displays the display screen of the display unit 132 as shown in FIG.

  That is, in the layer 0 mode, the thumbnail image 232 designated by the pointer 231 is displayed with emphasis. In the example of FIG. 35, the thumbnail image 232 designated by the pointer 231 is displayed larger than the other thumbnail images in a state surrounded by a black frame, for example.

  Thereby, the user can easily grasp the thumbnail image 232 designated by the pointer 231.

  Next, FIG. 36 illustrates an example of a state in which double-clicking is performed in the state in which the thumbnail image 232 is designated with the pointer 231 in the layer 0 mode.

  When the user double-clicks while the thumbnail image 232 is designated by the pointer 231, the content is reproduced from the frame corresponding to the thumbnail image 232.

  That is, for example, as shown in FIG. 36, the display control unit 213 arranges the window 233 on the upper left in the figure on the display screen of the display unit 132. In this window 233, content 233a reproduced from a frame corresponding to the thumbnail image 232 is displayed.

  In the window 233, a clock mark 233b, a timeline bar 233c, a playback position display 233d, and a volume button 233e are arranged on the upper side of the content 233a from the left to the right in the drawing.

  The clock mark 233b is an icon for displaying the playback position (playback time) at which the content 233a is played out of the total playback time of the content 233a with a clock hand. In the clock mark 233b, the total playback time of the content 233a is assigned, for example, to the time of one round of the clock hands (1 hour from 0 minutes to 60 minutes).

  The timeline bar 233c is a horizontally long bar and, like the clock mark 233b, displays the playback position of the content 233a. Note that the total playback time of the content 233a is assigned to the timeline bar 233c from the left end to the right end of the timeline bar 233c, and a playback position display 233d is arranged at a position corresponding to the playback position of the content 233a. The

  In FIG. 36, the playback position display 233d can be configured to be movable as a slider. In this case, the user can reproduce the content 233a from the position of the reproduction position display 233d after the movement by performing a moving operation of moving the reproduction position display 233d as a slider using the operation unit 17.

  The volume button 233e is an icon that is operated when the volume of the content 233a being played is muted (muted) or when the volume is changed.

  That is, for example, when the user moves the pointer 231 onto the volume button 233e using the operation unit 17 and performs a single click, the volume of the content 233a being reproduced is muted.

  For example, when the user moves the pointer 231 onto the volume button 233e using the operation unit 17 and double-clicks the window, a window for changing the volume of the content 233a being reproduced is newly displayed. .

  Next, FIG. 37 shows an example of a state when a single click is performed in the state in which the thumbnail image 232 is designated by the pointer 231 in the layer 0 mode.

  In the layer 0 mode, when the user performs a single click with the pointer 231 indicating the thumbnail image 232 (FIG. 35), the display control unit 213 shifts the display mode from the layer 0 mode to the layer 1 mode.

  Then, for example, as shown in FIG. 37, the display control unit 213 arranges a window 251 on the lower side in the figure on the display screen of the display unit 132. In this window 251, a tile image 251a, a clock mark 251b, a timeline bar 251c, and a reproduction position display 251d are arranged.

  The tile image 251 a represents an image of a list of thumbnail images convolved with the thumbnail image 232 (thumbnail image of a scene represented by the thumbnail image 232).

  For example, when the thumbnail image 232 is a thumbnail image corresponding to the frame having the frame number 300, the thumbnail image 232 includes thumbnail images corresponding to the frames having the frame numbers 301 to 349 as shown in FIG. Is folded.

  In addition, when all the images in the list of thumbnail images convolved with the thumbnail image 232 cannot be displayed as the tile image 251a in the window 251, for example, a part of the thumbnail images is thinned and displayed.

  In addition, for example, a scroll bar may be displayed in the window 251, and the scroll bar may be moved so that all the images in the list of thumbnail images convolved with the thumbnail image 232 may be viewed.

  The clock mark 251b is an icon for displaying the playback position at which the frame corresponding to the single-clicked thumbnail image 232 is played out of the total playback time of the content 233a with a clock hand. It is comprised similarly.

  The timeline bar 251c displays the playback position at which the frame corresponding to the single-clicked thumbnail image 232 is played out of the total playback time of the content 233a as the playback position display 251d. The configuration is the same as that of the bar 233c.

  Further, the timeline bar 251c also displays the playback position of each frame corresponding to the thumbnail image (other than the thumbnail image 232) constituting the tile image 251a using the playback position display similar to the playback position display 251d.

  In FIG. 37, only the reproduction position display 251d of the thumbnail image 232 is described, and the other reproduction position displays are not described, in order to avoid making the drawing complicated.

  In addition, when the user performs a mouse-on operation for designating a predetermined thumbnail image with a pointer 231 among a plurality of thumbnail images constituting the tile image 251 a using the operation unit 17, a predetermined instructed by the pointer 231 is performed. The thumbnail image is displayed with emphasis.

  That is, for example, when the user performs a mouse-on operation for pointing the thumbnail image 271 in the tile image 251a with the pointer 231 using the operation unit 17, a thumbnail image 271 ′ highlighting the thumbnail image 271 is displayed.

  At this time, in the timeline bar 251c, the playback position display of the thumbnail image 271 ′ is displayed with emphasis in the same manner as the thumbnail image 271 ′. That is, for example, the playback position display of the thumbnail image 271 ′ is displayed in an emphasized manner such as a color different from other playback position displays.

  Further, the playback position display highlighted in the timeline bar 251c can be configured to be movable as a slider.

  In this case, the user performs a moving operation for moving the highlighted reproduction position display as a slider using the operation unit 17, for example, by a thumbnail image corresponding to the reproduction position display after the movement. Can be displayed as a tile image 251a.

  Note that the thumbnail image 271 may be displayed in an emphasized manner in the same manner as the thumbnail image 232 described with reference to FIG. 35 in addition to displaying the enhanced thumbnail image 271 ′.

  When the user double-clicks using the operation unit 17 in a state where the highlighted thumbnail image 271 ′ is indicated by the pointer 231, as shown in FIG. 38, the user corresponds to the thumbnail image 271 ′ (271). The content 233a is reproduced from the frame.

  Next, FIG. 38 shows an example of a state in which double-clicking is performed in a state in which the thumbnail image 271 ′ is designated with the pointer 231 in the layer 1 mode.

  In the layer 1 mode, when the user double-clicks with the pointer 231 pointing to the thumbnail image 271 ′ (FIG. 37), the display control unit 213 shifts the display mode from the layer 1 mode to the layer 0 mode. .

  And the display control part 213 arrange | positions the window 233 on the upper left in the figure on the display screen of the display part 132, for example, as FIG. 38 shows. In this window 233, content 233a reproduced from the frame corresponding to the thumbnail image 271 ′ (271) is displayed.

  Next, FIG. 39 shows an example of a state when a single click is performed in a state in which the thumbnail image 271 ′ is designated with the pointer 231 in the layer 1 mode.

  In the layer 1 mode, when the user performs a single click with the pointer 231 pointing to the thumbnail image 271 ′ (FIG. 37), the display control unit 213 shifts the display mode from the layer 1 mode to the layer 2 mode. .

  And the display control part 213 arrange | positions the window 291 in the display screen of the display part 132, for example, as FIG. 39 shows. In this window 291, a tile image 291a, a clock mark 291b, and a timeline bar 291c are arranged.

  The tile image 291a represents a list of thumbnail images having display contents similar to the display contents displayed on the thumbnail image 271 ′ (271).

  That is, the tile image 291a is a list of thumbnail images of frames having the same symbol as the symbol of the frame corresponding to the thumbnail image 271 ′ among the frames constituting the content 233a.

  Here, in addition to the chapter point data from the dividing unit 151, the content 233a and the symbol string of the content 233a are supplied to the display data generating unit 212.

  Based on the symbol string from the dividing unit 151, the display data generating unit 212 extracts a frame having the same symbol as the symbol corresponding to the thumbnail image 271 ′ from the content 233a from the dividing unit 151.

  Then, the display data generation unit 212 sets each extracted frame as a thumbnail image, generates a tile image 291a that is a list of the thumbnail images, and displays display data including the generated tile image 291a to the display control unit 213. Supply.

  The display control unit 213 controls the display unit 132 based on the display data from the display data generation unit 212, and displays a window 291 including the tile image 291a on the display screen of the display unit 132.

  Note that if not all the thumbnail images constituting the tile image 291a can be displayed in the window 291, a scroll bar or the like is added to the window 291. In addition, for example, a part of the thumbnail images may be omitted so that the tile image 291a fits in the window 291.

  The clock mark 291b is an icon for displaying the playback position at which the frame corresponding to the single-clicked thumbnail image 271 ′ is played out of the total playback time of the content 233a with a clock hand. The clock mark 233b in FIG. It is configured in the same way.

  The timeline bar 291c displays the playback position where each frame corresponding to a plurality of thumbnail images as the tile image 291a is played out of the total playback time of the content 233a. The timeline bar 291c is the same as the timeline bar 233c shown in FIG. It is comprised similarly.

  Accordingly, the timeline bar 291c displays, for example, as many playback positions as the number of thumbnail images as the tile image 291a.

  In addition, when the user performs a mouse-on operation for designating a predetermined thumbnail image with a pointer 231 among a plurality of thumbnail images constituting the tile image 291 a using the operation unit 17, a predetermined instructed by the pointer 231 is performed. The thumbnail image is displayed with emphasis.

  At this time, on the timeline bar 291c, the reproduction position of the predetermined thumbnail image indicated by the point 231 is displayed in an emphasized manner, for example, by setting it to a color different from other reproduction positions.

  In FIG. 39, for example, when a mouse-on operation for pointing the thumbnail image 271 with the pointer 231 is performed, the predetermined thumbnail image is highlighted in the same manner as when the highlighted thumbnail image 271 ′ is displayed (FIG. 37). Is displayed.

  Then, when the user double-clicks using the operation unit 17 in a state where the highlighted predetermined thumbnail image is indicated by the pointer 231, the predetermined thumbnail image is displayed in the same manner as described with reference to FIG. The content 233a is reproduced from the frame corresponding to the thumbnail image.

[Description of operation of recorder 131]
Next, a presentation process performed by the recorder 131 (particularly the presentation unit 152) in FIG. 26 will be described with reference to the flowchart in FIG.

  In step S221, the dividing unit 151 performs the same process as the dividing unit 15 in FIG. Further, the dividing unit 151 generates chapter point data (chapter ID) in the same manner as the dividing unit 71 in FIG. 17, and supplies the generated chapter point data to the display data generating unit 212 of the presenting unit 152.

  Further, the dividing unit 151 supplies each symbol in the symbol sequence from the symbol sequence generating unit 14 to the display data generating unit 212 of the presentation unit 152 in association with each corresponding frame of content.

  The dividing unit 151 supplies the content read from the content storage unit 11 to the feature amount extraction unit 211 of the presentation unit 152.

  In step S222, the feature quantity extraction unit 211 extracts feature quantity time-series data in the same manner as the feature quantity extraction unit 112 in FIG.

  That is, for example, the feature amount extraction unit 211 extracts at least one of face area time-series data, audio power time-series data, zoom-in intensity time-series data, or zoom-out intensity time-series data from the content from the dividing unit 151. And supplied to the display data generation unit 212.

  In step S223, the display data generation unit 212 displays, for example, as shown in FIGS. 31 to 33 based on the feature amount time-series data from the feature amount extraction unit 211 and the chapter point data from the division unit 151. Display data for generating the data is generated and supplied to the display control unit 213.

  In addition, for example, the display data generation unit 212 generates display data to be displayed on the display screen of the display unit 132 in accordance with a user operation according to the control from the control unit 16 and supplies the display data to the display control unit 213.

  That is, for example, as illustrated in FIG. 39, when a single click is performed in a state where the thumbnail image 271 ′ is designated by the point 231, the display data generation unit 212 uses the symbols from the division unit 151 to Display data for displaying the window 291 including the tile image 291a is generated and supplied to the display control unit 213.

  In step S224, the display control unit 213 causes the display screen of the display unit 132 to display corresponding to the display data based on the display data from the display data generation unit 212. Thus, the presentation process in FIG. 40 is completed.

  As described above, according to the presentation process of FIG. 40, the display control unit 213 displays a thumbnail image for each chapter constituting the content on the display screen of the display unit 132.

  For this reason, the user can reproduce content from a desired reproduction position in a predetermined chapter by referring to the display screen of the display unit 132.

  Further, for example, according to the presentation process of FIG. 40, the display control unit 213 displays the thumbnail image to which the band display is added. For this reason, the feature of the scene corresponding to the thumbnail image can be easily recognized by the band display.

  In particular, since the user cannot obtain information about the sound from the thumbnail image, a band display representing the feature that the sound is loud is added to the thumbnail image without reproducing the scene. Scene features can be easily recognized.

  Further, according to the presentation process of FIG. 40, for example, as shown in FIG. 37, the display unit 132 displays a thumbnail image of a scene represented by a thumbnail image 232 as a tile image 251a together with its reproduction position. I made it.

  In addition, according to the presentation process of FIG. 40, the display control unit 213, as shown in FIG. 39, for example, displays the thumbnail image of each frame that is the same as the symbol of the frame corresponding to the thumbnail image 271 ′. Along with the reproduction position, the tile image 291a is displayed.

  As a result, the user can easily search for the playback position of the frame to start playback from among the plurality of frames constituting the content 233a. Therefore, the user can easily reproduce the content 233a from a desired start position.

  Next, FIG. 41 shows an example of how the display mode of the display control unit 213 is shifted.

  In step ST1, the display mode of the display control unit 213 is the layer 0 mode. For this reason, the display control unit 213 controls the display unit 132 to display the display screen of the display unit 132 as shown in FIG.

  For example, when the control unit 16 determines, based on the operation signal from the operation unit 17, that the user has performed a double click using the operation unit 17 in a state where none of the thumbnail images is instructed by the pointer 231. The process proceeds from step ST1 to step ST2.

  In step ST2, if there is a window 233 that is playing back the content 233a, the control unit 16 controls the display data generation unit 212 to generate display data for causing the window 233 to be displayed on the front. It is made to supply to the control part 213.

  Based on the display data from the display data generation unit 212, the display control unit 213 changes the display screen of the display unit 132 to a display screen with the window 233 displayed on the front, and the process returns from step ST2 to step ST1. .

  Moreover, in step ST1, the control part 16 advances a process to step ST3 suitably.

  In step ST3, the control unit 16 determines based on the operation signal from the operation unit 17 whether or not a slide operation or the like for sliding the slider 171 has been performed by the user. When the control unit 16 determines that the user has performed a slide operation or the like based on the operation signal from the operation unit 17, the control unit 16 generates display data corresponding to the user's slide operation or the like in the display data generation unit 212. And supply it to the display control unit 213.

  Based on the display data from the display data generation unit 212, the display control unit 213 changes the display screen of the display unit 132 to a display screen corresponding to a user's slide operation or the like. Thereby, the display screen of the display unit 132 is changed from, for example, the display screen shown in FIG. 30 to the display screen shown in FIG. Thereafter, the process returns from step ST3 to step ST1.

  Furthermore, in step ST1, the control part 16 advances a process to step ST4 suitably.

  In step ST4, based on the operation signal from the operation unit 17, the control unit 16 determines whether there is a thumbnail image 232 whose distance from the pointer 231 is equal to or less than a predetermined threshold. If the control unit 16 determines that such a thumbnail image 232 does not exist, the control unit 16 returns the process to step ST1.

  Further, in step ST4, when the control unit 16 determines that there is a thumbnail image 232 whose distance from the pointer 231 is equal to or less than a predetermined threshold based on the operation signal from the operation unit 17, the process proceeds to step ST4. Proceed to ST5.

  Here, the distance between the pointer 231 and the thumbnail image 232 represents, for example, the distance between the center of gravity of the pointer 231 (or the tip of the pointer 231 as an arrow) and the center of gravity of the thumbnail image 232.

  In step ST5, the control unit 16 causes the display data generation unit 212 to generate display data for emphasizing and displaying the thumbnail image 232, and supplies the display data to the display control unit 213.

  Based on the display data from the display data generation unit 212, the display control unit 213 changes the display screen of the display unit 132 to a display screen as shown in FIG.

  Further, in step ST5, the control unit 16 performs a double click or single operation by the user using the operation unit 17 in a state where the distance between the pointer 231 and the thumbnail image 232 is equal to or less than the threshold value based on the operation signal from the operation unit 17. It is determined whether one of the clicks has been performed.

  In step ST5, if the control unit 16 determines that neither double-clicking or single-clicking using the operation unit 17 by the user has been performed based on an operation signal from the operation unit 17, an appropriate process is performed. Is returned to step ST4.

  In step ST5, the control unit 16 determines that, based on the operation signal from the operation unit 17, the user double-clicked using the operation unit 17 in a state where the distance between the pointer 231 and the thumbnail image 232 is equal to or less than the threshold value. If so, the process proceeds to step ST6.

  In step ST6, the control unit 16 causes the display data generation unit 212 to generate display data for reproducing the content 233a from the reproduction position of the frame corresponding to the thumbnail image 232, and supplies the display data to the display control unit 213.

  The display control unit 213 changes the display screen of the display unit 132 to a display screen as shown in FIG. 36 based on the display data from the display data generation unit 212, and the process returns to step ST1.

  In step ST5, the control unit 16 performs a single click by the user using the operation unit 17 in a state where the distance between the pointer 231 and the thumbnail image 232 is equal to or smaller than the threshold value based on the operation signal from the operation unit 17. If it is determined that it has been broken, the process proceeds to step ST7.

  In step ST7, the control unit 16 controls the display control unit 213 to shift the display mode of the display control unit 213 from the layer 0 mode to the layer 1 mode. In addition, the display control unit 213 changes the display screen of the display unit 132 to, for example, the display screen illustrated in FIG. 33 and the display screen in which the window 251 in FIG. 37 is added according to the control from the control unit 16.

  In step ST7, the control unit 16 determines whether or not the user has performed a double click using the operation unit 17 based on the operation signal from the operation unit 17, and the user has performed the double click. If so, the process proceeds to step ST8.

  In step ST8, the control unit 16 causes the display data generation unit 212 to generate display data for reproducing the content 233a from the reproduction position of the frame corresponding to the thumbnail image closest to the pointer 231 and causes the display control unit 213 to generate the display data. Supply.

  Based on the display data from the display data generation unit 212, the display control unit 213 displays the display screen of the display unit 132 on the display screen as shown in FIG. 36, and the process returns to step ST1.

  Furthermore, in step ST7, when it is determined that the double click using the operation unit 17 by the user has not been performed based on the operation signal from the operation unit 17, the control unit 16 appropriately proceeds to step ST9. .

  In step ST9, based on the operation signal from the operation unit 17, for example, the control unit 16 determines whether or not there is a thumbnail image 271 whose distance from the pointer 231 is equal to or less than a predetermined threshold in the window 251. Determine. If the control unit 16 determines that such a thumbnail image 271 does not exist, the control unit 16 advances the processing to step ST10.

  In step ST10, the control unit 16 determines whether or not the pointer 231 has moved outside the area of the window 251 displayed in the layer 1 mode based on the operation signal from the operation unit 17, and the area of the window 251 is determined. If it is determined that the pointer 231 has moved outside, the process returns to step ST1.

  In step ST1, the control unit 16 causes the display data generation unit 212 to generate display data for display corresponding to the layer 0 mode, and supplies the display data to the display control unit 213.

  Based on the display data from the display data generation unit 212, the display control unit 213 changes the display screen of the display unit 132 to, for example, a display screen as shown in FIG. In this case, the display control unit 213 shifts the display mode from the layer 1 mode to the layer 0 mode.

  In step ST10, when the control unit 16 determines that the pointer 231 has not moved outside the area of the window 251 based on the operation signal from the operation unit 17, the process returns to step ST7.

  In step ST9, when the control unit 16 determines that there is a thumbnail image 271 whose distance from the pointer 231 is equal to or less than a predetermined threshold in the window 251, for example, based on the operation signal from the operation unit 17, The process proceeds to step ST11.

  In step ST11, the control unit 16 causes the display data generation unit 212 to generate display data for displaying the thumbnail image 271 with emphasis, and supplies the display data to the display control unit 213.

  Based on the display data from the display data generation unit 212, the display control unit 213 displays a display screen of the display unit 132 on which a thumbnail image 271 ′ highlighting the thumbnail image 271 as shown in FIG. Change to the screen.

  In step ST11, the control unit 16 performs double-clicking by the user using the operation unit 17 in a state where the distance between the pointer 231 and the thumbnail image 271 ′ is equal to or less than the threshold value based on the operation signal from the operation unit 17. It is determined whether one of the single clicks has been performed.

  In step ST11, when the control unit 16 determines that neither double-clicking or single-clicking by the user using the operation unit 17 is performed based on an operation signal from the operation unit 17, an appropriate process is performed. To step ST9.

  In step ST11, based on the operation signal from the operation unit 17, the control unit 16 performs a double click by the user using the operation unit 17 in a state where the distance between the pointer 231 and the thumbnail image 271 ′ is equal to or less than the threshold value. If it is determined that the process has been performed, the process proceeds to step ST12.

  In step ST12, the control unit 16 causes the display data generation unit 212 to generate display data for reproducing the content 233a from the reproduction position of the frame corresponding to the thumbnail image 271 ′, and supplies the display data to the display control unit 213.

  The display control unit 213 changes the display screen of the display unit 132 to a display screen as shown in FIG. 38 based on the display data from the display data generation unit 212, and the process returns to step ST7.

  Further, in step ST11, the control unit 16 performs a single click using the operation unit 17 by the user in a state where the distance between the pointer 231 and the thumbnail image 271 ′ is equal to or less than a threshold based on the operation signal from the operation unit 17. If it is determined that the process has been performed, the process proceeds to step ST13.

  In step ST13, the control unit 16 controls the display control unit 213 to shift the display mode of the display control unit 213 from the layer 1 mode to the layer 2 mode. Further, the display control unit 213 changes the display screen of the display unit 132 to a display screen on which a window 291 is displayed as shown in FIG. 39, for example, according to control from the control unit 16.

  In step ST13, the control unit 16 determines whether or not the user has performed a double click using the operation unit 17 based on the operation signal from the operation unit 17, and the user has performed the double click. If so, the process proceeds to step ST14.

  In step ST14, the control unit 16 causes the display data generation unit 212 to generate display data for reproducing the content 233a from the reproduction position of the frame corresponding to the thumbnail image closest to the pointer 231, and causes the display control unit 213 to generate the display data. Supply.

  Based on the display data from the display data generation unit 212, the display control unit 213 displays the display screen of the display unit 132 on the display screen as shown in FIG. 36, and the process returns to step ST1.

  Furthermore, in step ST13, when it is determined that the double click using the operation unit 17 by the user has not been performed based on the operation signal from the operation unit 17, the control unit 16 appropriately proceeds to step ST15. .

  In step ST15, based on the operation signal from the operation unit 17, for example, the control unit 16 includes a predetermined thumbnail image (included in the tile image 291a) whose distance from the pointer 231 is equal to or less than a predetermined threshold in the window 291. Image) is present. If the control unit 16 determines that such a predetermined thumbnail image exists, the control unit 16 advances the processing to step ST16.

  In step ST16, the control unit 16 causes the display data generation unit 212 to generate display data for emphasizing and displaying a predetermined thumbnail image whose distance from the pointer 231 is equal to or less than a threshold in the window 291 and performing display control. To the unit 213.

  Based on the display data from the display data generation unit 212, the display control unit 213 changes the display screen of the display unit 132 to a display screen on which a predetermined thumbnail image is displayed with emphasis.

  In step ST16, the control unit 16 performs a double-click using the operation unit 17 by the user in a state where the distance between the pointer 231 and the thumbnail image is equal to or smaller than the threshold value based on the operation signal from the operation unit 17. It is determined whether or not. If the control unit 16 determines that the double-click has been performed, the control unit 16 advances the process to step ST17.

  In step ST17, the control unit 16 causes the display data generation unit 212 to generate display data for reproducing the content 233a from the reproduction position of the frame corresponding to the thumbnail image, and supplies the display data to the display control unit 213.

  The display control unit 213 changes the display screen of the display unit 132 to a display screen as shown in FIG. 36 based on the display data from the display data generation unit 212, and the process returns to step ST1.

  In step ST15, the control unit 16 determines a predetermined thumbnail image (tile image 291a) whose distance from the pointer 231 is equal to or less than a predetermined threshold in the window 291, for example, based on the operation signal from the operation unit 17. If it is determined that there is no image included in the image, the process proceeds to step ST18.

  In step ST18, based on the operation signal from the operation unit 17, the control unit 16 determines whether or not the pointer 231 has moved outside the area of the window 291 displayed in the layer 2 mode. If it is determined that the pointer 231 has moved outside, the process returns to step ST1.

  In step ST1, the control unit 16 controls the display control unit 213 to shift the display mode from the layer 2 mode to the layer 0 mode, and thereafter the same processing is performed.

  In step ST18, when the control unit 16 determines that the pointer 231 has not moved outside the area of the window 291 displayed in the layer 2 mode based on the operation signal from the operation unit 17, the process is performed. Returning to step ST13, the same processing is performed thereafter.

<4. Modification>
By the way, this technique can take the following structures.
(1) a chapter segment extraction unit that extracts a chapter segment representing a predetermined portion representing the chapter from each chapter obtained by dividing time series data composed of a plurality of data arranged in time series; Of the chapters obtained by classifying the time series data, a feature segment extracting unit that extracts the feature segment from chapters having a feature segment representing a characteristic part of the chapter, and the chapter segment and the feature segment. An information processing apparatus including: a generation unit that generates a digest reflecting a rough content of the time-series data by combining in time-series order.
(2) The information according to (1), wherein the generation unit generates the digest having a length set by a user setting operation by combining the chapter segment and the feature segment in time series order. Processing equipment.
(3) Based on the time series data, a symbol string generation unit that creates a symbol string in which symbols representing the attributes of the plurality of data are arranged in time series, and based on the variance of symbols in the symbol string, The information processing apparatus according to (2), further including a classification unit that classifies the time series data into a plurality of chapters.
(4) The division unit divides the time-series data into chapters having the number of divisions based on a length set by the user's setting operation based on a variance of each symbol constituting the symbol string. The information processing apparatus according to 3).
(5) It further includes a feature amount extraction unit that extracts a feature amount representing the feature of the time series data from the time series data, and the feature segment extraction unit includes a chapter having the feature segment based on the feature amount. The information processing device according to any one of (1) to (4), wherein the feature segment is extracted from the information segment.
(6) The feature segment extraction unit extracts, from the chapter, the feature segment including a portion where the feature amount is one of maximum or maximum in a section from the start to the end of the chapter based on the feature amount. The information processing apparatus according to (5).
(7) The feature segment extraction unit is a location where the feature amount is one of the maximum or maximum in a section from the start to the end of the chapter based on the feature amount, and the feature amount is The information processing apparatus according to (6), wherein the feature segment including a portion that is equal to or greater than a predetermined threshold is extracted from the chapter.
(8) The feature segment extraction unit maximizes the feature amount that is maximized in a section from the start to the end of the chapter among the plurality of different feature amounts based on the plurality of different feature amounts. The information processing apparatus according to (7), wherein the feature segment including a location is extracted from the chapter.
(9) The information processing according to (5) to (8), wherein the generation unit generates the digest in which speech prepared in advance with a corresponding weight is added to each of the chapter segment and the feature segment. apparatus.
(10) The feature segment extraction unit extracts the feature segment from a chapter having the feature segment based on a plurality of different feature amounts, and the generation unit includes a voice of the plurality of different feature amounts. The information processing apparatus according to (9), wherein the digest is generated by adding the voice to the feature segment extracted based on the feature amount representing the feature of the feature with a weight smaller than that of the other feature segment.
(11) The information processing apparatus according to (10), wherein the generation unit generates the digest to which the voice is added with a weight that is continuously changed and switched.
(12) In the information processing method of the information processing apparatus for generating a digest, from each chapter obtained by dividing time series data composed of a plurality of data arranged in time series by the information processing apparatus, the chapters in advance A chapter segment extraction step for extracting a chapter segment representing a determined portion, and a chapter having a feature segment representing a characteristic portion of the chapter among the chapters obtained by dividing the time-series data. A feature segment extracting step of extracting a chapter, and a generating step of generating a digest reflecting a rough content of the time series data by combining the chapter segment and the feature segment in a time series order .
(13) a chapter segment extraction unit that extracts a chapter segment representing a predetermined portion of the chapter from each chapter obtained by dividing the time series data including a plurality of pieces of data arranged in time series by the computer; Among the chapters obtained by classifying the time-series data, a feature segment extracting unit that extracts the feature segment from chapters having a feature segment representing a characteristic portion of the chapter, the chapter segment, and the feature segment A program for functioning as a generation unit that generates a digest reflecting the rough contents of the time-series data by combining them in time-series order.

[Example of computer configuration to which this technology is applied]
Next, the series of processes described above can be performed by hardware or software. When a series of processing is performed by software, a program constituting the software is installed in a general-purpose computer or the like.

  Therefore, FIG. 42 shows a configuration example of an embodiment of a computer in which a program for executing the series of processes described above is installed.

  The program can be recorded in advance on a hard disk 305 or a ROM 303 as a recording medium built in the computer.

  Alternatively, the program can be stored (recorded) in a removable recording medium 311 attached to the drive 309. Such a removable recording medium 311 can be provided as so-called package software. Here, examples of the removable recording medium 311 include a flexible disk, a CD-ROM (Compact Disc Read Only Memory), an MO (Magneto Optical) disk, a DVD (Digital Versatile Disc), a magnetic disk, and a semiconductor memory.

  In addition to installing the program from the removable recording medium 311 as described above, the program can be downloaded to the computer via a communication network or a broadcast network, and can be installed in the built-in hard disk 305. That is, for example, the program is wirelessly transferred from a download site to a computer via a digital satellite broadcasting artificial satellite, or wired to a computer via a network such as a LAN (Local Area Network) or the Internet. be able to.

  The computer includes a CPU (Central Processing Unit) 302, and an input / output interface 310 is connected to the CPU 302 via the bus 301.

  The CPU 302 executes a program stored in a ROM (Read Only Memory) 303 in response to an instruction input by the user operating the input unit 307 or the like via the input / output interface 310. . Alternatively, the CPU 302 loads a program stored in the hard disk 305 to a RAM (Random Access Memory) 304 and executes it.

  Thereby, the CPU 302 performs processing according to the above-described flowchart or processing performed by the configuration of the above-described block diagram. Then, the CPU 302 causes the processing result to be output from the output unit 306 or transmitted from the communication unit 308 via the input / output interface 310, or recorded on the hard disk 305, for example, as necessary.

  Note that the input unit 307 includes a keyboard, a mouse, a microphone, and the like. The output unit 306 includes an LCD (Liquid Crystal Display), a speaker, and the like.

  Here, in the present specification, the processing performed by the computer according to the program does not necessarily have to be performed in time series in the order described as the flowchart. That is, the processing performed by the computer according to the program includes processing executed in parallel or individually (for example, parallel processing or object processing).

  Further, the program may be processed by one computer (processor) or may be distributedly processed by a plurality of computers. Furthermore, the program may be transferred to a remote computer and executed.

  The embodiment of the present disclosure is not limited to the above-described embodiment, and various modifications can be made without departing from the gist of the present disclosure.

  DESCRIPTION OF SYMBOLS 1 recorder, 11 content memory | storage part, 12 content model learning part, 13 model memory | storage part, 14 symbol sequence production | generation part, 15 division | segmentation part, 16 control part, 17 operation part, 21 content selection part for learning, 22 feature-value extraction part, 23 frame division unit, 24 sub-region feature quantity extraction unit, 25 combination unit, 26 feature quantity storage unit, 27 learning unit, 31 content selection unit, 32 model selection unit, 33 feature quantity extraction unit, 34 maximum likelihood state sequence estimation unit , 51 recorder, 71 division unit, 72 digest generation unit, 111 chapter segment extraction unit, 112 feature amount extraction unit, 113 feature peak segment extraction unit, 114 effect addition unit, 131 recorder, 132 display unit, 151 division unit, 152 presentation Part, 211 feature Out unit, 212 display data generating unit, 213 display control unit

Claims (13)

A chapter segment extraction unit that extracts a chapter segment representing a predetermined portion representing the chapter from each chapter obtained by dividing time series data composed of a plurality of data arranged in time series; and
Of each chapter obtained by classifying the time series data, a feature segment extraction unit that extracts the feature segment from a chapter having a feature segment representing a characteristic part of the chapter;
An information processing apparatus comprising: a generating unit that generates a digest reflecting a rough content of the time-series data by combining the chapter segments and the feature segments in a time-series order. The information processing apparatus according to claim 1, wherein the generation unit generates the digest having a length set by a user's setting operation by combining the chapter segment and the feature segment in time series order. A symbol string generation unit that creates a symbol string in which symbols representing the attributes of the plurality of data are arranged in time series based on the time series data;
The information processing apparatus according to claim 2, further comprising: a division unit that divides the time-series data into a plurality of chapters based on symbol dispersion in the symbol string. The division unit divides the time-series data into chapters of a division number based on a length set by the user's setting operation based on a variance of each symbol constituting the symbol string. Information processing device. A feature amount extraction unit for extracting a feature amount representing the feature of the time series data from the time series data;
The information processing apparatus according to claim 4, wherein the feature segment extraction unit extracts the feature segment from a chapter having the feature segment based on the feature amount. The feature segment extraction unit extracts, based on the feature amount, the feature segment including a portion where the feature amount is one of maximum or maximum in a section from the start to the end of the chapter from the chapter. 5. The information processing apparatus according to 5. The feature segment extraction unit is a location where the feature value is one of maximum or maximum in a section from the start to the end of the chapter based on the feature value, and the feature value is determined in advance. The information processing apparatus according to claim 6, wherein the feature segment including a portion that is equal to or greater than a threshold is extracted from the chapter. The feature segment extraction unit includes a location where the feature amount maximized in a section from the start to the end of the chapter among the plurality of different feature amounts is based on the plurality of different feature amounts. The information processing apparatus according to claim 7, wherein the feature segment is extracted from the chapter. The information processing apparatus according to claim 5, wherein the generation unit generates the digest in which a voice prepared in advance with a corresponding weight is added to each of the chapter segment and the feature segment. The feature segment extraction unit extracts the feature segment from chapters having the feature segment based on a plurality of different feature quantities,
The generator generates the digest in which the speech is added to the feature segment extracted based on a feature amount representing a feature of speech among the plurality of different feature amounts with a weight smaller than that of the other feature segment. The information processing apparatus according to claim 9. The information processing apparatus according to claim 10, wherein the generation unit generates the digest to which the voice is added with a weight that continuously changes and switches. In an information processing method of an information processing apparatus that generates a digest,
According to the information processing apparatus,
A chapter segment extraction step for extracting a chapter segment representing a predetermined portion of the chapter from each chapter obtained by dividing time series data composed of a plurality of data arranged in time series, and
Of the chapters obtained by classifying the time series data, a feature segment extracting step of extracting the feature segment from chapters having a feature segment representing a characteristic part of the chapter;
An information processing method comprising: generating a digest reflecting a rough content of the time-series data by combining the chapter segments and the feature segments in a time-series order. Computer
A chapter segment extraction unit for extracting a chapter segment representing a predetermined portion of the chapter from each chapter obtained by dividing time series data composed of a plurality of pieces of data arranged in time series; and
Of each chapter obtained by classifying the time series data, a feature segment extraction unit that extracts the feature segment from a chapter having a feature segment representing a characteristic part of the chapter;
A program for functioning as a generation unit that generates a digest reflecting the rough contents of the time-series data by combining the chapter segments and the feature segments in time-series order.

Images