JP2005109566A - Video image summary instrument, explanation note forming instrument, video image summary method, explanation note forming method and program - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide video image summary equipment forming a summary video image where an explanation is added to a video section interesting for a user by a telop or voice narration. <P>SOLUTION: A significance calculating section 106 calculates the significance of each event included in a video image. A video section selecting section 107 selects a video section used as a summary video image from that video image. An event selecting section 108 for an explanation note selects an event becoming an object for forming the explanation note from an event included in the video image, and a stipulation forming section 109 forms a stipulation. An integrating section 110 creates summary video information based on a selected video section and the explanation note thus formed. <P>COPYRIGHT: (C)2005,JPO&NCIPI

Description

  The present invention relates to a video summarization apparatus and method for creating summary video information from video information, an explanatory text generation apparatus and method for generating explanatory text from video information, and a program.

  In recent years, with the development of information infrastructure such as multi-channel broadcasting by digital satellite broadcasting, a large amount of video content has been distributed. On the other hand, with the improvement of computer processing capacity and the spread of broadband networks such as ADSL and FTTH, video can be recorded and viewed on a storage device on a personal computer or portable terminal, or video distributed over the network can be viewed. It has become common to do. As a result, users can watch only their part of interest in order to view more content in the same amount of time as before, or briefly summarize videos recorded on a personal computer for viewing on the go. The viewing style is expected to diversify.

  Also, from the viewpoint of the producer of video content, even if the content of the viewing environment of users is the same content, depending on the viewing environment for TV broadcasting, broadband broadcasting, and mobile devices, It becomes necessary to create a plurality of variations, and the manufacturing cost increases.

In view of this, there have been proposed devices and methods for describing events in a video as an event sequence and automatically summarizing them according to matching with user preferences. For example, there is a method of obtaining importance for each event and selecting a section of the corresponding event as a summary video (for example, see Patent Document 1) or a method for generating an explanation for the summary video itself (for example, see Patent Document 2). is there.
Japanese Patent No. 3176893 JP 2001-275058 A

  A summary video obtained by collecting scenes of interest of the user from the original video does not necessarily include an important scene in order to know the content. For example, when creating a summary video centering on a video of a teammate in a sport game, the game development may not be known because the scoring scene of the opponent team is not included.

  Also, if you want to create a summary of the entire game, you may want to include some unimportant scenes, taking into account the overall scene structure. However, in the previous proposal, it has not necessarily been a summarizing method capable of generating content in consideration of such a scene configuration. For example, if there is an important scene only in the second half of the game, a summary video that does not include the first half scene may be created.

  The present invention has been made in consideration of the above circumstances, and a video summarization apparatus, a video summarization method, and a video summarization method for creating a summary video in which a description by a telop or voice narration is added to a video section of interest to a user The purpose is to provide a program.

  Moreover, an object of this invention is to provide the explanatory note production | generation apparatus, the explanatory note production | generation method, and program which produce an explanatory note more effectively.

  The present invention calculates the importance of each event based on the first information regarding the contents of the event included in the video, and calculates the second information regarding the video section of the event included in the video and the importance. Based on the video, a video section to be used as a summary video is selected, an event to be created as an explanatory text is selected from events included in the video, and the event related to the selected event is selected. An explanatory text relating to the event is generated based on the first information, and summary video information is generated based on the selected video section and the generated explanatory text.

  Further, the present invention holds a transition state related to an event included in the video, inputs event information regarding the content of the event included in the video, and inputs the event information and a predetermined transition condition The transition state to be retained is updated based on the transition state, and based on the transition related to the transition state update, the explanatory note template information to be used for the creation of the explanatory note is selected from the plurality of explanatory note template information. An explanatory note is generated using the selected explanatory note template information.

  According to the present invention, it is possible to create a summary video to which a description by telop, voice narration, or the like is added to a video section in which the user is interested.

  Hereinafter, embodiments of the present invention will be described with reference to the drawings.

(First embodiment)
FIG. 1 is a diagram showing a basic configuration example of a video summarizing apparatus according to the first embodiment of the present invention.

  As shown in FIG. 1, the video summarization apparatus includes an importance level calculation unit 106, a video segment selection unit 107, an explanatory text event selection unit 108, an explanatory text creation unit 109, and an integration unit 110.

  The original video information 101 corresponds to various video contents such as TV programs and videos taken by users.

  In the present embodiment, the original video information 101 is digital data. The format of the digital data may be any format (for example, a compressed data format such as MPEG-1 or MPEG-2 is generally used). Of course, the source video may be analog data, but in this case, the video is converted into digital data externally in advance, or the video summarization apparatus includes analog-digital conversion.

  The video content can be one or more, and any video content can be played from any position, given the information that uniquely identifies the video content, such as title and ID, and the playback start time or frame number. It is.

  The video attribute information 102 is various attribute information (metadata) related to the video content. FIG. 2 shows an example of attribute information. The attribute information includes an event sequence listing events that occurred in the video content as events. The event includes information such as names and movements of people and objects (such as “XX player's shot” in the case of soccer) and time breaks (such as “match start” and “match end”) along with the time of occurrence. Described. The description of the occurrence time may be a format describing the start point / end point of the section as in the scene information, or may be a format describing the time when the event occurs. In addition, the attribute information may include information not related to a specific time in the video content, such as a sport item, a battle team name, a date, a place, a participating player, and a game result. In addition, although a sport is given here as a specific example, the attribute information can be appropriately described according to the content. For example, drama and information programs can be described in the same manner by using characters and topic titles.

  The summary creation information 103 is information including user preferences when creating a summary, summary video time, summary algorithm parameters, and the like. The user's preference is information such as a keyword for obtaining a portion that the user wants to watch preferentially. In the case of a sporting game, information such as a favorite team and a favorite player corresponds to this. The keyword may be described only for the object that the user wants to see, or it may be possible to describe not only the object that the user wants to see but also the object that the user does not want to see. These pieces of information may be input directly, or may be stored in advance on an internal or external storage device in order to save the trouble of inputting the same condition every time.

The explanatory note creation information 104 is information including a template (template) used when creating an explanatory note from an event included in the video attribute information 102. The template used here is, for example,
"(Time), (Team) chance. (Player) (Play)!"
In this way, the character string is given by a combination of a fixed character string and a variable character string. In this example, the character string indicated by () such as “(time)” represents a variable character string.

  The summary video information 105 is summary video generated by the video summary device. The summary video information includes a part of the original video content and an explanatory text (eg, telop or narration) using visible information such as generated characters and audible information such as audio. Also, content prepared separately from the original video content (for example, a title screen) may be included. In addition, the summary video information may be information that can be played back independently as video content different from the original video, or a text or audio description may be provided while playing back with reference to the original video content. It may be displayed / reproduced. For the latter, a description language such as SMIL that can display a plurality of media synchronously can be used. For speech reproduction, for example, speech synthesis can be used.

The importance calculation unit 106 calculates the importance for each event included in the video attribute information 102 based on the input video attribute information 102 and summary creation information 103. The event importance level is calculated by obtaining an evaluation value of the degree of matching between the character string included in the event and the keyword included in the summary creation information 103. For example, if the importance is w and the total number of keywords is N, the importance w can be obtained by the following example. In the following expression, α _k is a parameter indicating the weight value of the k-th keyword included in the summary creation information 103.
w = Σa _k M (χ)
Here, Σ is a summation target in the range of k = 1 to N.
If the χ th keyword matches, M (χ) = 1,
If the χ th keyword does not match, M (χ) = 0.

In addition, in the configuration that allows the description of the object that you do not want to see as a keyword, for M (χ),
If the χ th keyword matches and the χ th keyword is a keyword related to the video that you want to see, set M (χ) = 1,
If the χ th keyword matches and the χ th keyword is a keyword related to a video that you do not want to see, set M (χ) = − 1,
If the χ th keyword does not match, M (χ) = 0.

  The video segment selection unit 107 selects a segment of the original video content to be included in the summary video according to the calculated importance.

  FIG. 3 shows an example of the section selection procedure (first section) when sections (start time and end time) corresponding to individual events are obtained (in the present embodiment, described in the video attribute information 102). Selection method).

  In step S11, an event having the highest importance level is searched from the event sequence for videos not yet included in the summary video, and this is set as event i.

  In step S12, the total time of the events already included in the summary video and the time of event i are added, and this is compared with the time of the summary video specified by the summary creation information 103. If not, the process proceeds to step S13, and this event i is added as one of the events constituting the summary video. Then, the process returns to step S11 to search for the next event.

  Thus, in step S12, if the sum of the time of the selected event and the time of event i exceeds the summary video time, the process ends.

  This procedure was to generate a summary video using the time as a threshold to match the specified time. However, if the time of the generated summary video is not important, for example, the importance level may be changed. An event having an importance level equal to or higher than the threshold value may be selected as the threshold value.

  Further, a method of performing the procedure of FIG. 3 for only events having an importance level equal to or higher than a threshold value is also possible.

  In addition, various variations are possible for the section selection procedure using the threshold value of the importance level of the event and the threshold value of the time of the summary video.

  FIG. 4 shows another example of the section selection procedure (second section selection method). In this example procedure, each event is expressed as a function of importance with respect to time, and a section of the summary video is selected. Hereinafter, the outline of this procedure will be described (see Japanese Patent Application No. 2002-287861 for details of this procedure).

  In step S21, an event importance curve is calculated from the event importance and the importance function.

  The event importance function is a function that represents a change in importance according to the time of an event section. The function expression can be freely determined for each event. This function formula is changed based on the importance to obtain an event importance curve.

As an example, the event importance function is multiplied by the event importance. If the function formula of the i-th event is f _i (t), the formula of the event importance curve is
E _i = (w _i + e _i ) f _i (t)
It becomes. Here, w _i is the importance of the i-th event. E _i is a correction term and is used when a factor other than w _i is to be reflected.

  In step S22, as illustrated in FIG. 5A, the event importance curve 201 and the time importance curve 202 are added to calculate an importance curve 203 indicating the change in importance of the entire content over time.

  Here, the time importance curve indicates a change in importance with time of the entire content, independently of the occurrence of an event.

If the time importance curve equation is T (t), the importance curve for the entire content is
ER (t) = Max (E _i (t)) + T (t)
It becomes the sum of both the importance of the event and the importance by time. However, Max (E _i (t)) is the maximum value of E _i (t) at time t.

Instead of the above, ER (t) = ΣE _i (t) + T (t) may be used. Here, Σ (E _i (t)) is the sum of E _i (t) for all events.

  In step S23, as illustrated in FIG. 5B, a section where the importance curve exceeds the threshold is selected as a section included in the summary video, and video section information is obtained. As shown in FIG. 5B, the smaller the threshold value, the longer the video section selected. Note that the threshold value used for selecting the section may be directly specified, for example, or may be obtained by obtaining the threshold value so as to meet the specified time.

  In addition to the first and second section selection methods described above, various variations are possible for the section selection procedure.

  The explanatory note event selection unit 108 selects an event to be used for generating an explanatory sentence (telop, narration, etc.) using characters and sounds used for the summary video from the event sequence. Basically, a process for generating an explanatory text only for the entire summary video, a process for generating an explanatory text for only a part of the summary video, A process for generating an explanatory text for a predetermined part and a process for generating an explanatory text only for a predetermined part other than the summary video are conceivable. These event selection processes may be performed independently of the video segment selection unit 107, or data of the video segment selection unit 107 may be used.

  The algorithm used in the explanatory note event selection unit 108 may basically be based on the same idea as the algorithm used in the video section selection unit 107. For example, as in the first section selection method, a method for selecting an event having an importance level equal to or higher than a threshold based on the importance level for each event (first event selection method), or a second section selection Similar to the method, there is a method (second event selection method) of obtaining an importance curve of the entire content and selecting an event included in a section having an importance level equal to or higher than a threshold value. Of course, other methods are possible.

  When the explanatory text event selector 108 uses the data of the video segment selector 107, as an example, the same selection method is used by changing only the threshold between the segment selection method and the event selection method. There is something. At that time, for example, if you want to generate a descriptive text with priority on some particularly important events in the video section, set a higher threshold so that only the most important events are selected. do it. Also, if you want to provide detailed information by generating a descriptive text for events that are not included in the video section, and you want to provide more detailed information, set a lower threshold and select a wider range of events. What should I do? The threshold value used for event selection does not need to be one, and a plurality of threshold values may be used depending on the application, such as an explanatory text expressed in characters or an explanatory text expressed in speech.

  As another example of using the data of the video section selection unit 107, an event used for creating an explanatory note is selected from events included in the previously selected video section. The event selection method at this time may be the same as or different from the video section selection method. As a different example, a video section is selected by the second section selection method, and an event is selected from events included in the video section by using the first event selection method.

  On the other hand, as an example of selecting an event independently of the video section selection unit 107, a video section is selected from the event sequence by the first section selection method, an explanatory text event is selected by the second event selection method, etc. Use a combination of different selection methods. Moreover, even when the same selection method is combined, a case where parameters such as the weight value of the importance calculation unit 106 are different is an example.

  Furthermore, for example, in the case where an explanatory text expressed by visual information such as characters (hereinafter referred to as text explanatory text) and an explanatory text expressed by audible information such as speech (hereinafter referred to as audio explanatory text) are used together, For each event selected by one event selection method, whether to generate a text explanation, a voice explanation, or both is determined according to a predetermined criterion (for example, the importance of the event is high). If the video section is also selected, both criteria are not high importance, but only the audio description when the video section is selected, and only the text description when the video section is not selected ), Or a method in which the user directs and determines each event directly is also possible. At this time, the explanatory note event selecting unit 108 notifies the explanatory note creating unit 109 of information that can specify the above-described determination content for each selected event. This determination process may be performed by the explanatory note creation unit 109.

  In addition, when a text explanation and a voice explanation are used in combination, a method for selecting an event used for the text explanation and a process for selecting an event used for the text explanation are also possible. .

  In addition, for all events selected by one event selection method, both a text explanation and a voice explanation are generated, and the user outputs the explanation with only letters or with only voice. Alternatively, it may be possible to select whether to output both.

  The explanatory note creating unit 109 generates an explanatory sentence (by visual information such as characters or audible information such as voice) from the explanatory note event selected by the explanatory note event selecting unit 108 and the explanatory note creation information 104. Generate.

As described above, the explanatory note creation information is information including a template for generating an explanatory note. A template is, for example,
"(Time), (Team) chance. (Player) (Play)!"
In this way, the character string is given by a combination of a fixed character string and a variable character string. In this example, the character string indicated by () such as “(time)” represents a variable character string.

The variable character string includes a character string that can be obtained directly or indirectly from the video attribute information 102 and the summary creation information 103. A plurality of such templates are prepared depending on the type of event. For example, from the event sequence:
“Time: First 12 minutes, Team: Team A, Player: Player X, Play: Shoot”
If the information is obtained, the template exemplified above, that is,
"(Time), (Team) chance. (Player) (Play)!"
Apply to
“First half 12 minutes, chance of Team A. X player's shot!”
An explanatory note is generated.

  Moreover, you may produce | generate description using the information of two or more event sequences, and one template. For example, if the first event is centering, the second event is a shot, the third event is a goal, and the related events are continuous, this is indicated as “the player who shot (the player who shot the center) ) Is a shoot, a goal! ”And so on.

  In order to determine the relevance of an event, information that enumerates the types of events that are related events may be created and compared with an input event sequence. For example, in the case of the above example, “centering, shoot, goal” and information listing the event types are created. In order to collect only events that are close in time other than the event type, an interval of event occurrence times may be described.

  For example, when there is a possibility of creating both a text explanation and a voice explanation, the explanation creation unit 109 creates a part for creating a text explanation and a voice explanation. It shall be combined with this part. Of course, when only the text description is created, only the part for creating the text description needs to be provided. The same applies to the case where only the audio description is created.

  The integration unit 110 integrates the video section information and the explanatory text by the character telop or voice narration, and creates summary video information including both pieces of information. The display timing of the characters and the voice utterance are performed in accordance with the event occurrence time and the video section. When adjusting to the occurrence time, display and speak at an appropriate interval before the occurrence time, or display and speak so that a specific position in the utterance section (for example, the midpoint) becomes the event occurrence time. The timing may be adjusted. In addition, when adjusting to the video section, display and speaking may be performed with an appropriate interval from the start point and end point of the section. Of course, by combining these methods, for example, the character telop may be used in accordance with the video section, and the voice narration may be used in accordance with the event occurrence time.

  At the time of integration, telop overlay display and voice narration by voice synthesis may be created and included in the summary video, or described as summary video information in the text state, and overlay display voice synthesis is performed on the playback device side. You may go.

  In the above description, the integration unit 11 simply superimposes the selected video section information and the generated explanatory text information and outputs the information as one video summary information. Adjustments may be included.

  FIG. 6 shows a processing procedure example of the integration unit 110 in this case. In this processing procedure, either step S31 or step S32 may be omitted if unnecessary.

  In step S31, the video section information is adjusted according to the timing of display (telop) and reproduction (voice narration) of the explanatory text information.

  For example, when the video section is too short with respect to the speech narration speech time, the video section is extended so that the voice narration can be accommodated. The extended time may be absorbed by shortening other sections (such as sections with short narration), or may be left as it is. Note that the number of characters and the number of words may be used instead of the speech time.

  For example, consider a case where narrations 303a and 303b are to be inserted into a certain video section 301 of the video before adjustment in FIG. In this case, the narration may be inserted before and after the cheering without using the narration in the section 302 near the event occurrence time. In FIG. 7, the time length of the narration is represented by the oblique lines. In the example of FIG. 7, there is no problem with the narration 303a, but since the narration 303b is too long for the insertion destination section, if inserted as it is, the end of the video section 301 is exceeded. Therefore, the video section can be adjusted so that both narrations can be accommodated by shifting backward from 301 to 304.

  In step S32, the volume of the audio included in the video is adjusted. For example, the volume of the video for reproducing the voice narration is reduced so that the voice narration can be easily heard, or the cheer of the part where an event such as a goal occurs is increased.

  In step S33, summary video information in which the adjusted video is combined with the telop and narration is created.

  As described above, according to the present embodiment, from the video information, only the video section in which the user is interested is selected based on the conditions such as the keyword and time input by the user and the user's preference, and a telop or A summary video with voice narration can be created. Also, by supplementing important scenes that have been omitted with explanatory texts, it is possible to create a summary video that makes it easier to understand the contents.

  Therefore, according to the present embodiment, the user can create and reproduce a summary including only a portion of his / her interest without reproducing one whole program. At that time, it is possible to receive explanations based on telops and voice narrations in sentences and tone that match the user's preference, including portions that do not appear in the video.

(Second Embodiment)
FIG. 8 is a diagram illustrating an example of a basic configuration of an explanatory note generating apparatus according to the second embodiment of the present invention.

  As illustrated in FIG. 8, the explanatory note generation apparatus includes a transition control unit 406, a template selection unit 407, and an explanatory note generation unit 408.

  This explanatory note generation device can be used as an explanatory note creation unit of the first embodiment.

  The event information 401 is event information used for description sentence generation. This event information may be all event information attached to the original video or may be only information related to some selected events depending on the explanatory text to be generated.

  The transition state information 402 is information that holds the transition state in the description sentence generation process.

  The transition condition information 403 is information indicating a transition condition and a transition destination node when transitioning from a certain node (state) to another node due to occurrence of an event.

  The description sentence template information 404 is information serving as a template (model) for generating an explanatory sentence by applying event information.

  The description text information 405 is a generated description text (such as character telop or voice narration).

  The transition control unit 406 receives the event information and updates the transition state information according to the transition condition information.

  The template selection unit 407 selects an explanatory note template corresponding to the immediately preceding transition from the explanatory note template information.

  The explanatory note generation unit 408 generates explanatory note information in which event information is applied to the selected template.

  Next, processing of each unit will be described with reference to FIG. FIG. 9 shows an example of state transition when an event occurs.

  In FIG. 9, the current node (501) corresponds to the current state, and transitions to a state represented by node A (502) or node B (503) when an event occurs. As a specific example, in a sporting game, it is assumed that the current node is the same and the current node is the same. Further, a state in which team A has taken the lead is node A, and a state in which team B has taken the lead is node B. Due to the occurrence of a scoring event, a transition is made from the current node to a state led by Team A (Node A) or a state led by Team B (Node B).

  One node can have a plurality of transition destination nodes, and which node is to be transitioned is determined by a transition condition. For example, the transition condition A is defined as “Score for Team A”, and the transition condition B is defined as “Score for Team B”. Of course, the description of the transition condition does not have to be “Score of Team A”, but may be matching for each event item, such as “Team = A Team and Action = Score”. If the meanings are the same, the words may be enumerated or a description language such as XML may be used.

  Transition conditions are obtained by analyzing information obtained directly from events such as team and player names, information obtained indirectly such as score progress and game time, information provided as summary creation conditions, and video data It can be described using various information such as feature quantities such as motion vectors and colors, information obtained from state transitions such as the number of times nodes have passed and state nodes.

  The transition control unit 406 compares the event that has occurred with the transition condition, and updates the current node position included in the transition state information. Using the specific example described above, if the event that occurred is a scoring event for team A, the current node is transitioned to node A, and if it is a scoring event for team B, the current node is transitioned to node B. Let Further, although not applicable in this example, the current node may not be updated as in the transition condition C.

  There is a corresponding explanatory text template for each state transition. However, if it is not necessary to generate an explanatory note, it may be omitted. For example, template A is used when transitioning to node A according to transition condition A, and template B is used when transitioning to node B according to transition condition B. The same template may be used for state transition between different nodes. In the case of the above-mentioned specific example, “(time), (team) chance, (player name) shoot, goal!” However, the values in parentheses are variables determined by the event.

  The explanatory note generation unit 408 applies the information obtained directly or indirectly from the event information and the information on the entire content to the aforementioned explanatory note template to generate the explanatory note information. The information obtained indirectly is, for example, when obtaining time information of a game from time information corresponding to a video. In the case of the above-mentioned specific example, if time: first half 30 minutes, team: Team A, player name: X, “first half 30 minutes, A team chance, X shot. Goal!”

  FIG. 10 shows an example of a state transition rule used for generating narration at the time of scoring in a sports game. In this example, there are five nodes: start, A lead, tie A, B lead, and tie B. The starting node is in the initial state. In addition, A lead: Team A is in a lead state, B lead: Team B is in a lead state, A tie A: A state in which Team A is in a tie and a tie B: Team B The state where the tie is made from the lead state is shown. Among the arrows indicating the state transition, the solid line indicates that “team A score” and the broken line indicates that “team B score” is the transition condition. The text in “” is an explanatory text template corresponding to each transition. Note that two transitions appear from the A lead and the B lead under the same transition condition, but these take into account the score difference other than the team as the transition condition.

  By this method, it can be seen that even if the scoring event is the same, an appropriate explanation can be selected according to the situation. For example, when a scoring event from the same point occurs, if it is a transition from the start node 601, it is a preemptive point and the template 602 is used. After catching up with the same point (node 603), it is the reverse, and the template 604 is used. After catching up with the same point (node 605), an explanatory note indicating that the team has won again can be generated from the template 606.

  FIG. 11 shows another example of the state transition rule used for generating narration at the time of scoring in a sport game. In this example, for a user who likes team A, an explanatory note with team A as the main role is generated. For example, even for the score of team B, a template centered on team A, such as template 701 “reversed” and template 702 “catch up”, is used, with team A as the leading role. Thus, by preparing different sets of explanatory text templates for the same state transition, it is possible to generate explanatory texts according to the user's preference and usage.

  FIG. 12 shows still another example of the state transition rule used for generating a narration at the time of scoring in a sport game. In this example, a plurality of explanatory text templates are associated with one transition like a template 801. These templates are selected according to an explanatory sentence condition different from the transition condition. As with the transition conditions, the explanatory sentence conditions include information obtained directly from events such as teams and player names, information obtained indirectly such as score progress and game time, information given as summary creation conditions, and video data. It is described by various information such as feature quantities such as motion vectors and colors obtained by analysis, information obtained from state transitions such as the number of times a node has passed and the transition source node.

  By using the explanatory sentence conditions, the number of nodes can be reduced and the state transition diagram can be simplified. For example, in the case of FIG. 12, the start, tie point A, and tie point B states are combined into one tie point node. Like, switch templates. Note that the notation of the condition in FIG. 12 uses, as an example, the not-then rule notation of if (condition) then “template”, but other notation methods may of course be used.

  FIG. 13 shows an example of a method using a plurality of state transition rules. In the above description, the state transition rule has been described by taking the state transition of the score progress as an example, but there are a plurality of state transitions other than the score in an actual sports game. For example, taking baseball as an example, there are times and front and back transitions, and there are three out transitions on the front and back of one time. Below that, there is a detailed hierarchical transition for each batter and pitch.

  Although all of these state transitions can be described as one state transition rule, it becomes very complicated. Therefore, a plurality of state transition rules are used. In FIG. 13, state transition rule 1 (901) and state transition rule 2 (902) correspond to state transitions in the same hierarchy and are processed in parallel. On the other hand, the state transition rule 3 (903) is applied when the state transition rule 1 is in the state of the node 4 (904). In the state transition rule 1, if the transition condition from the node 4 is satisfied, the state transitions to the next node regardless of the state of the state transition rule 3. At this time, the state of the state transition rule 3 may be used as a transition condition from the node 4.

  FIG. 14 shows an example of a data structure describing a state transition rule.

  Node information 1001 describes information about one node. The node information includes a node ID 1002 indicating identification information of the node and transition destination information 1003 describing information about the transition destination node. A plurality of pieces of transition destination information may exist depending on the number of transition destination nodes. The transition destination information includes a transition destination node ID 1004 indicating the node ID of the transition destination node, a transition condition 1005 for transitioning to the transition destination node, and explanatory text information 1006 describing the explanatory text template used at this transition. If there is no transition condition, such as when there is only one transition destination and there is a transition, it may be omitted. There may be a plurality of pieces of explanatory note information depending on the number of explanatory note templates. Each piece of explanatory text information includes information on an explanatory text condition 1007 and an explanatory text template 1008. When a plurality of templates are used, switching is performed according to the explanatory text condition. Further, the description sentence condition when there is one piece of description information and there is no need to switch may be omitted.

  FIG. 15 shows an example of a data structure describing a plurality of state transition rules. Here, only differences from FIG. 14 will be described.

  The state transition rule 1101 is information describing the state transition rule, and includes a rule ID 1102 and node information 1103 indicating identification information of the state transition rule. In the node information, in addition to the above-described node ID and transition destination information, another state transition rule 1104 applied under the node is held. The data structure of this state transition rule is the same as that of the state transition rule. The description of the state transition rule may describe an entity as shown in FIG. 15, or may refer to a state transition rule defined at another place by a rule ID. In particular, when the same state transition rule is used a plurality of times, it is efficient to define the state transition rule in one place and refer to it by the rule ID. If such a reference is not performed and the rule ID is unnecessary, the rule ID may be omitted.

  By the way, when a plurality of state transition rules are used, a plurality of explanatory texts corresponding to individual state transition rules may be generated for one event information input. In order to organize and display these explanatory texts (in the case of character telops) or playback (in the case of voice narration), position information indicating how individual explanatory texts are displayed or added is added. Also good. This position information is spatial information indicating the coordinates and size of the display position, or temporal information indicating the reproduction time. These spatial or temporal position information can be described together with the state transition rule. For example, when a common position is used in one state transition rule, it may be described as an element of the state transition rule 1105. Moreover, what is necessary is just to describe as an element of the transition destination information 1003 or the explanatory note information 1006 when changing a position for every transition or every explanatory note.

  FIG. 16 shows an example of a method for describing a relative position centered on an event as temporal position information. Describes whether to play before or after the event time. The hatched portion in FIG. 16 is a section where an explanatory text (voice narration) is inserted. In addition, the order from the event time is described in the case where a plurality of explanatory texts exist before or after. This order need not be sequential. By describing the front and rear information and the order information as temporal position information, even when a plurality of voice narrations are generated for one event, they can be integrated so as to be natural narration. .

  In the above, information indicating the front and back positions (in this example, before or after) and information indicating the order from the event time (in this example, 1 or 2) An example of the description method has been shown, but the present invention is not limited to this.

  For example, temporal position information can be expressed by positive and negative numbers. In this case, for example, starting from the event time, this is represented by 0, the position before the event time is a negative number (the absolute value is the same as the information indicating the order viewed from the event time), and after the event time. Can be represented by a positive number (the absolute value is the same as the information indicating the order viewed from the event time). According to this method, in the specific example of FIG. 16, (front, 2), (front, 1), (back, 1), and (back, 2) according to the previous method are respectively -2, -1 , 1 and 2.

  Hereinafter, an application example of description sentence generation by state transition will be shown.

  FIG. 17 shows an example of a basic writing procedure for obtaining superiority / inferiority information such as a team that is playing a dominant game or an inferior team based on the state transition. The superiority / inferiority information may be expressed by a numerical value for each team, or may be information indicating which team is superior or inferior. This superiority / inferiority information may be used as internal information such as transition conditions and explanatory text conditions, or may be used as information that can be understood by the user.

  In step S41, the superior / inferior value of the current node position in the transition state based on the state transition rule is added to the overall superior / inferior value.

  In step S42, it is determined whether there is another state transition rule to be processed. If there is, the process returns to step S41, and the superior / inferior value of the current node position of the transition state based on the state transition rule is added. This is repeated until there is no state transition rule to be processed, and the process proceeds to step S43.

  In step S43, superiority information is obtained from the overall superiority value.

  Here, the superiority / inferiority value may be expressed, for example, by a numerical value that continuously changes between 0 and 1 or may indicate only which team is superior. In the latter case, it can be considered to take discrete values of 0 or 1.

(Third embodiment)
FIG. 18 is a diagram illustrating a basic configuration example of a video summarizing apparatus according to the third embodiment of the present invention.

  As shown in FIG. 18, the video summarizing apparatus includes an overall description creating unit 1202, an importance level calculating unit 1203, a video section selecting unit 1204, an explanatory text event selecting unit 108, an explanatory text creating unit 109, and an integrating unit 110. I have.

  Since this embodiment is an extension of the first or second embodiment, only differences from the first or second embodiment will be described.

  The summary video configuration information 1201 is information describing the scene configuration of the generated summary video and information attached thereto. The summary video includes various summaries such as a summary of game development and a summary focused on a specific player. In addition, there are cases where it is not desired to know the goal scene or the game result when using the main part as a judgment material for viewing. When paying attention to a specific player, a summary may be created from a plurality of matches instead of a summary of one match. The summary video configuration information describes such a summary scene configuration.

  FIG. 19 shows an example of summary video configuration information. In FIG. 19, the time and contents for each scene are included. The time is the length of the scene and its maximum and minimum values. The contents describe parameters necessary for creating the scene according to the type of the scene (video, summary video, composite video, etc.). If a video is to be extracted, a parameter indicating the location or range is described. If a telop or narration is used, a template or a parameter indicating the position is described.

  The overall description creating unit 1202 creates a description related to the entire content based on the summary video configuration information. The created description is a telop or narration described in the summary video configuration information, or a composite video using the telop or narration. Further, when a specific video is designated as in the title scene of the previous example, the search is also performed. Data obtained from video attribute information or video creation information is used as data to be applied to a telop or narration template.

  The importance level calculation unit 1203 and the video segment selection unit 1204 receive parameters related to the summary video scene included in the summary video configuration information, and create a summary video that satisfies the content. Specifically, the range for extracting the summary video is limited to the range specified by the summary video configuration information, or the threshold is set so that the time of the summary video to be created matches the specified time. . Also, if you do not want to know the results, such as when you want to use the main part as a reference for viewing, you can reduce the importance after the shot so that the scoring scene does not enter, or create a summary from the range excluding that section To do.

  According to the present embodiment, it is possible to create a summary video that not only depends on the user's preference but also satisfies a predetermined scene configuration.

(Fourth embodiment)
FIG. 20 is a diagram illustrating a basic configuration example of a video summarizing apparatus according to the fourth embodiment of the present invention.

  As shown in FIG. 20, the video summarization apparatus includes a summary video configuration information selection unit 1304 and an explanatory note creation information selection unit 1305 in addition to the configuration of FIG. 18.

  Since the present embodiment is an extension of the third embodiment, only differences from the third embodiment will be described.

  The user preference 1301 describes information related to user preferences.

  The summary video configuration information database 1302 is a database that stores a plurality of summary video configuration information. As described in the third embodiment, various scene configurations exist in the summary video configuration information. These various variations are stored in the database 1302.

  The explanatory note creation information database 1303 is a database that stores information such as various explanatory note templates and state transition rules (when an explanatory note is generated by state transition). Even templates that show the same content can have multiple variations due to differences in tone and dialect.

  The summary video configuration information selection unit 1304 selects summary video configuration information that matches the user preferences from the summary video configuration information database.

  The explanatory note creation information selection unit 1305 selects the explanatory note creation information that matches the user preference from the explanatory note creation information database.

  By using the summary video device of the present embodiment, the user describes his / her favorite scene configuration and description format in the user preferences, compared to the case of simply specifying the keyword and time, You can get a summary video that suits your taste.

  Each function described above can be realized as software. The present embodiment can also be implemented as a program for causing a computer to execute predetermined means (or for causing a computer to function as predetermined means, or for causing a computer to realize predetermined functions), The present invention can also be implemented as a computer-readable recording medium on which the program is recorded.

  Note that the present invention is not limited to the above-described embodiment as it is, and can be embodied by modifying the constituent elements without departing from the scope of the invention in the implementation stage. In addition, various inventions can be formed by appropriately combining a plurality of components disclosed in the embodiment. For example, some components may be deleted from all the components shown in the embodiment. Furthermore, constituent elements over different embodiments may be appropriately combined.

1 is a block diagram showing a basic configuration example of a video summarization apparatus according to a first embodiment of the present invention. Diagram showing an example of video attributes The flowchart which shows an example of the process sequence of a 1st selection method. The flowchart which shows an example of the process sequence of a 2nd selection method. The figure for demonstrating the 2nd selection method The flowchart which shows an example of the process sequence of an integration part Diagram for explaining an example of video section adjustment The block diagram which shows the basic structural example of the explanatory note production | generation apparatus which concerns on the 2nd Embodiment of this invention. Diagram for explaining state transition when an event occurs Diagram showing an example of state transition rules The figure which shows the other example of the state transition rule The figure which shows the other example of a state transition rule The figure explaining the case where a plurality of state transition rules are used The figure which shows an example of the data structure of a state transition rule The figure which shows an example of the data structure of the state transition rule in the case of using a several state transition rule Diagram for explaining how to integrate multiple explanations A flowchart showing an example of a basic processing procedure for obtaining superiority information from a transition state The block diagram which shows the basic structural example of the image | video summarization apparatus which concerns on the 3rd Embodiment of this invention. A figure showing an example of summary video composition information The block diagram which shows the basic structural example of the image | video summarization apparatus which concerns on the 4th Embodiment of this invention.

Explanation of symbols

  DESCRIPTION OF SYMBOLS 106 ... Importance calculation part, 107 ... Image | video section selection part, 108 ... Explanation text event selection part, 109 ... Explanation sentence creation part, 110 ... Integration part, 406 ... Transition control part, 407 ... Template selection part, 408 ... Explanation Sentence generation unit 1202 ... Overall description creation unit 1203 ... Importance calculation unit 1204 ... Video section selection unit, 1304 ... Summary video configuration information selection unit, 1305 ... Description sentence creation information selection unit

Claims (24)

Calculation means for calculating the importance of each event based on the first information regarding the content of the event included in the video;
First selection means for selecting a video section to be used as a summary video from the video based on second information on the video section of the event included in the video and the importance;
A second selecting means for selecting an event for which an explanatory note is to be created from events included in the video;
First generation means for generating an explanatory text relating to the event based on the first information relating to the selected event;
A video summarization apparatus comprising: second generation means for generating summary video information based on the selected video section and the created explanatory text.   The second selection unit selects at least one event other than an event included in the video section selected by the first selection unit from among events included in the video. Item 2. The video summarization apparatus according to Item 1.   2. The video summarizing apparatus according to claim 1, wherein the second selection unit selects all events included in the video section selected by the first selection unit.   The method of selecting a video section in the first selection means and the method of selecting an event in the second selection means are used by changing the threshold value of the same selection algorithm, or by combining different selection algorithms. The video summarization apparatus according to claim 1, wherein the video summarization apparatus is used.   The selection algorithm selects an event or a video section related to the event in a range satisfying the threshold in descending order of the importance of the event, or the importance of the event as a function of time 5. The video summarization apparatus according to claim 4, wherein the video summarization apparatus selects a video section of importance or an event included in the video section.   The video summarizing apparatus according to claim 1, wherein the first generation unit generates a single explanatory sentence by collecting a plurality of related events selected by the second selection unit.   The second generation means, when the explanatory text and the video section are played back or displayed at the same time, so that the time required for playback or display of the explanatory text falls within the time related to the video section. The video summarization apparatus according to claim 1, wherein the video section is adjusted.   The second generation means accompanies the video section when the explanatory text and the video section are reproduced or displayed at the same time and both the explanatory text and the video section are accompanied by audio. The video summarization apparatus according to claim 1, wherein the volume of the audio is adjusted.   The calculation means calculates the importance based on a result of comparing a character string indicating the content of the event given as the first information with a character string showing the user's preference given as user information. The video summarizing apparatus according to claim 1, wherein:   The first selection means selects the video section so that a sum of times related to videos included in the summary video information falls within a predetermined time limit range. The video summarizing apparatus according to 1. Further comprising a third generation means for generating an explanatory text and / or video relating to the entire summary video information,
Based on the summary video configuration information including information on the scene configuration to be included in the summary video information and information on the video to be included in each scene and information on the description, the selection of the video section, the selection of the event, and the description of the event 2. The video summarization apparatus according to claim 1, wherein the video summarization apparatus generates a sentence and a description of the entire summary video information and / or video, and generates the summary video information. Means for selecting summary video configuration information that matches user preference information describing information related to user preferences from among the plurality of summary video configuration information;
2. The video summarization apparatus according to claim 1, wherein the summary video information is generated using the summary video configuration information selected by the means. Holding means for holding a transition state related to an event included in the video;
Input means for inputting event information related to the content of the event included in the video;
Updating means for updating the transition state to be held based on the input event information and a predetermined transition condition;
Based on the transition related to the update of the transition state, selection means for selecting the explanatory note template information to be used for the creation of the explanatory note from among a plurality of explanatory note template information,
An explanatory note generating apparatus comprising: generating means for generating an explanatory note using the selected explanatory note template information.   The transition condition is related to information obtained from attribute information about the video including event information, information about conditions for creating summary information of the video, information obtained by analyzing the video, or the transition state 14. The explanatory note generating device according to claim 13, wherein the description generating device is described using information obtained by processing any one of the pieces of information or the information to be processed.   14. The explanatory note generation device according to claim 13, wherein the transition condition and the explanatory note template information are described as a state transition rule.   As the state transition rule, identification information for uniquely identifying a node corresponding to each transition state, a condition for transition from the same node to another node, identification information of a transition destination node when transitioning to another node, and transition to another node The explanatory note generation device according to claim 15, wherein explanatory note information corresponding to the transition in the case of performing is described.   17. The explanatory note generation device according to claim 16, wherein the state transition rule further describes time or spatial position information into which an explanatory note generated from the explanatory note information is to be inserted.   18. The explanatory note generating apparatus according to claim 17, wherein the temporal position information is information indicating a temporal position and order in time with respect to an event in the video.   14. The explanatory note generation device according to claim 13, wherein additional attribute information in the transition state is described.   The explanatory note generation device according to claim 19, wherein information indicating superiority or inferiority of a win or loss is described as the additional attribute information. Calculating the importance of each event based on the first information regarding the content of the event included in the video;
Selecting a video segment to be used as a summary video from the video based on second information regarding the video segment of the event included in the video and the importance; and
Selecting an event to create a description from among the events included in the video;
Generating an explanatory text relating to the event based on the first information relating to the selected event;
A video summarization method comprising: generating summary video information based on the selected video section and the created explanatory text. Maintaining a transition state related to an event included in the video;
Inputting event information relating to the content of the event included in the video;
Updating the transition state to be held based on the input event information and a predetermined transition condition;
Selecting explanatory text template information to be used for creation of explanatory text from a plurality of explanatory text template information based on the transition relating to the update of the transition state;
And a step of generating an explanatory text using the selected explanatory text template information. A program for causing a computer to function as a video summarization device,
A function for calculating the importance of each event based on the first information regarding the content of the event included in the video;
A function for selecting a video segment to be used as a summary video from the video based on the second information regarding the video segment of the event included in the video and the importance;
A function of selecting an event to be a target for creating a description from among the events included in the video;
A function for generating an explanatory text relating to the event based on the first information relating to the selected event;
A program for realizing a function of generating summary video information based on the selected video section and the created explanatory text. A program for causing a computer to function as a video summarization device,
A function to maintain a transition state related to an event included in the video;
A function of inputting event information related to the content of the event included in the video;
A function of updating the transition state to be held based on the input event information and a predetermined transition condition;
Based on the transition relating to the update of the transition state, a function of selecting the explanatory note template information to be used for the creation of the explanatory note from among a plurality of explanatory note template information,
A program for realizing a function of generating an explanatory text using the selected explanatory text template information.

Images