JP2002281449A - Video device - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a video device that displays summary video fitting the user preference. SOLUTION: A summary video scenario table preparing part 104 prepares a summary video scenario that specifies a frame to be included in summary video. A plurality of summary video scenarios reproduction time of which is different in each program are prepared and stored in a summary video scenario table storing part 105. A multiprogram scenario preparing part 110 combines summary video scenarios of appropriate time length of a plurality of programs designated by a user so as to keep the summary video scenarios within prescribed time and outputs the multiprogram to a video display controlling part 107. When a user selection operation is performed, user operation information is stored, and the multiprogram scenario preparing part 110 modifies the multiprogram scenario on the basis of the information.

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a video apparatus for performing display using a summary video created from video data.

[0002]

2. Description of the Related Art There are many sources of video data from professional use to general consumer use. It takes a lot of time and effort to find what you want to watch from these large amounts of video data. For example, in fast-forward playback of a video tape recorder, finding a program to watch requires more time than expected.

To cope with this problem, there have been proposed methods and apparatuses for creating a summary video and grasping the contents of the rough video through reproduction of the summary video. These methods and apparatuses are described in, for example, JP-A-10-32773, JP-A-11-239322, and JP-A-11-1760.
No. 38, for example.

As disclosed in Japanese Patent Application Laid-Open No. 10-32773, a method of roughly grasping the contents of a video can be roughly classified into two methods. One is a method in which video information having a property of being continuous in time series is developed and displayed in space. For example, this corresponds to a case where one screen is divided and a plurality of important frames are displayed as still images on a plurality of divided screens.

[0005] Another method is to display a video in a time series without spatially developing it. The fast forward reproduction described above is the most primitive example.

An example of the latter technique for displaying important frames in time series is disclosed in Japanese Patent Application Laid-Open No. H11-176.
As disclosed in Japanese Patent Publication No. 038, only key frames having high importance are displayed in a fast-forward manner, and when an operation of selecting any key frame is performed during viewing, normal reproduction is started from a position corresponding to the key frame. A way to enter is proposed.

[0007]

The key frame is:
It is obtained by extracting a frame of high importance from video data.

In some cases, a human may manually select from video data to designate a key frame, but in general, when managing a large amount of video data, the key frame is specified by an automatic method. I have to rely on.

When a key frame is automatically designated, there is a problem in that the number of frames determined to have high importance differs for each video source. As a result, in fast-forward playback of a key frame, it is difficult to keep the playback time of the summary video within a fixed time length desired by the user.

Therefore, when displaying a summary video, it is necessary from the user's standpoint to absorb the difference between the video sources and perform the reproduction with a certain fixed time length.

[0011] In addition, after viewing a certain summary video, there may be a demand that the user wants to view a summary video having a longer time to understand even more detailed contents in order to determine whether the user is a desired video source. Therefore, it is necessary to create a plurality of summary videos having a specified time length from one video source.

[0012] Furthermore, the categories of programs considered important by the user are considered to be different for each user depending on the user's preference. In such a case, it is desirable that the summary video of the program in the category that the user considers important be reproduced before the summary video of the program that is not considered to be very important.

It is an object of the present invention to provide an image apparatus which enables a user to reach a desired image in a short time by reproducing a summary image reflecting the user's preference.

[0014]

According to the present invention, there is provided a video apparatus, comprising: video information storage means for storing a plurality of video sequences each including a plurality of frames; and user input means for receiving an instruction from a viewer. A user operation storage unit for storing user operation information as a history of viewer instructions input from the user input unit; and a scenario table storage unit for storing a scenario table corresponding to each video sequence. The scenario table includes a plurality of hierarchical scenario records each corresponding to a plurality of summary videos having different total playback times and including information for specifying a frame used for the summary video from the video sequence. The video device selects and combines scenario records from each of the scenario tables with reference to the user operation information,
Multi-program scenario creating means for creating and outputting a multi-program scenario including information for specifying a frame used for a summary video representing an overview of a plurality of video sequences, and summarizing according to the output of the user input means and the multi-program scenario The image display apparatus further includes: a video display control unit that controls display of a video; and a video reproduction unit that reads data corresponding to a frame from the video information storage unit and reproduces the data as a video signal in response to an instruction from the video display control unit.

[0015] Preferably, the multi-program scenario creator, when receiving an instruction from a viewer during the reproduction of the summary video, summarizes the multi-program scenario to be reproduced after the present time in accordance with a change in user operation information. Change the part corresponding to the video.

Preferably, the user operation information includes a category indicating a classification of the video sequence, and the number of times of reproduction request for each category for which the viewer has requested the reproduction of the video sequence. A multi-program scenario is created so that the summary video corresponding to the video sequence is preferentially arranged in front of the entire summary video according to the number of times the category is requested to be played.

Preferably, the user operation information includes a category indicating a classification of the video sequence, and the number of reproduction requests for each category for which the viewer has requested the reproduction of the video sequence. A multi-program scenario is created in which the ratio of the summary video corresponding to each video sequence to the total summary video is determined according to the number of playback requests of the category to which the category belongs.

Preferably, the user operation information includes a plurality of keywords indicating information contents represented by the video sequence,
The number of times that the viewer has requested the video sequence corresponding to each keyword to be played back. A multi-program scenario is created so that the summary video corresponding to the sequence is preferentially arranged.

Preferably, the user operation information includes a plurality of keywords indicating information content represented by the video sequence,
The number of times the viewer has requested the video sequence corresponding to each keyword to be played back, and the multi-program scenario creating means determines whether each video occupies the entire summary video according to the number of times of playback request corresponding to each keyword. A multi-program scenario in which the ratio of the summary video corresponding to the sequence is determined is created.

[0020]

Embodiments of the present invention will be described below in detail with reference to the drawings. In the drawings, the same reference numerals indicate the same or corresponding parts.

[Explanation of Entire Configuration] First, in the system to which the present invention is applied, it is possible to acquire program-related information such as category information and keywords for each program corresponding to a program such as a TV program to be handled. It shall be. For example, in the case of BS digital broadcasting, video and audio information of a program and information attached to the program can be obtained from a broadcast wave. Further, for example, Japanese Patent Application Laid-Open
As described in 275077, in recent years, a so-called electronic program guide (EPG: Electronic Program G) that displays a guide for selecting a television broadcast program on a display screen.
uide) has been put to practical use. Even if the information is not transmitted by a broadcast wave, the program ancillary information may be supplied by an EPG provider via a communication line like an EPG.

As an example, let us consider a case where the program attachment information is attached to the data of the program body and sent.

FIG. 1 is a block diagram showing a configuration of an image apparatus 10 using a summary image according to an embodiment of the present invention.

Referring to FIG. 1, a video apparatus 10 receives and records a program such as a television program obtained from a communication means such as a broadcast or the Internet, or a video taken by a user with a camera or the like. When a viewer searches for a video desired to be reproduced and gives a reproduction instruction, the recorded original video corresponding to the reproduced video is reproduced. .

The video apparatus 10 includes a video input unit 100 for inputting program data, a video storage unit 101 for storing the input video information, and a video information included in the input data. It includes a video priority determining unit 102 for determining a priority, and a video priority information storage unit 103 for storing the video priority created by the video priority determining unit 102. Note that the video priority is information for specifying a portion that is considered to be important for simply expressing the content from the input video.

The video apparatus 10 further includes a summary video scenario table creation unit 104 that determines which part of the input video is to be a summary video based on the video priority created by the video priority determination unit 102. A summary video scenario table storage unit 105 that stores a summary video scenario table output by the summary video scenario table creation unit 104
And

The video device 10 further includes a video input unit 10
0, a program information storage unit 113 for receiving and storing program ancillary information, a program information management unit 114 for managing the program information, and a reproduction request time indicating a relationship between a digest level indicating the detail level of the summary video and the reproduction request time. A playback request time table storage unit 111 for storing a table and a user operation storage unit 112 for storing a history of user operations are included.

The video apparatus 10 further converts a summary video scenario table corresponding to a plurality of programs into a summary video scenario table based on information from the program information management unit 114, the requested reproduction time table storage unit 111, and the user operation storage unit 112. It includes a multi-program scenario creating unit 110 that reads out from the storage unit 105, selects necessary information, and creates a multi-program scenario in which a plurality of programs are linked.

The video device 10 further includes a user input unit 106 for inputting a video to be viewed by the user while viewing the summary video, and a video input based on a multi-program scenario in accordance with the user's input. Video display control unit 107 for controlling reading and stopping, and video display control unit 107
A video reproducing unit 108 that reads a video from the video storage unit 101 based on an instruction from the
And a video display unit 109 for displaying video data processed by the video reproduction unit 108. For example, if the data is MP
When compressed by the EG method or the like, the processing performed by the video reproduction unit 108 includes decoding processing and the like.

When a user requests digest viewing of a plurality of programs, the request is sent to the video display control unit 1.
07 to the multi-program scenario creation unit 110. Multi-program scenario creation unit 110
Obtains the total playback time of the summary video to be created using the playback request time table, which will be described later, in the playback request time table storage unit 111. The multi-program scenario creating unit 110 is adjusted according to the total playback time.
Creates a multi-program scenario. The multi-program scenario creating unit 110 reads the scenario table of each program from the summary video scenario table storage unit 105 and creates a multi-program scenario.

In response to an operation performed by the user while viewing the summary video, program ancillary information of the currently reproduced program is read from the program information management unit 114 to the multi-program scenario creation unit 110. This program attachment information is stored in the user operation storage unit 112 as a user history.

[Explanation of Video Data] In this specification, FIG.
A unit of an image obtained from a communication means such as a television broadcast or the Internet or an image captured by a user with a camera or the like, which is input to the image device, is referred to as a program. For example, if the input data is a TV program,
One television program corresponds to one program. Further, in the case of a video taken by a user with a camera or the like, for example, a graduation ceremony video, an athletic meet video, and the like correspond to the respective programs. In many cases, audio information is added to video, and in this case, the program includes not only a video sequence as video data but also an audio sequence as audio data.

FIG. 2 is a diagram showing how the video data included in the program in this specification is structured.

Referring to FIG. 2, video sequence 200
Is an image group included in the one program described above. Video sequence 200 includes a plurality of scenes, for example, scenes 211 and 212.

Here, a scene indicates a group of images having the same semantic content. For example, a video sequence corresponding to a program called one news program includes one scene called news of earthquake information and another scene called weather forecast. And are included. One scene includes a plurality of shots.

Here, a shot refers to a group of physically continuous images. For example, an image in which a human moves in the background with respect to the same background is one shot. A boundary between shots is called a cut. For example, scene 211 includes shots 221-225. In other words, the scene 211 is cut 226-2
29 divides shots 221 to 225.

Each shot is composed of a corresponding frame. For example, the shot 221 has a plurality of frames 23
Contains 0. Here, a frame refers to one piece of image data displayed on a screen.

[Explanation of Video Priority] FIG. 3 is a block diagram showing the configuration of the video priority determination unit 102 in FIG.

Referring to FIG. 3, video priority determining section 102
Is composed of an input unit 300 that receives data stored from the video storage unit 101 in FIG. 1, and a video analysis unit 301 that receives data from the input unit 300, derives a feature amount included in video information, and performs analysis. Including. When the program includes audio information and additional data, such as a television program, the video analysis unit 301 also analyzes them.

The purpose of the analysis performed by the video analyzing unit 301 is as follows.
The purpose is to find the priority of each frame when creating a summary video, and to perform cut detection and scene detection as necessary. Various proposals have been made for these analysis methods, for example, "Structure and Retrieval of Information, by Nishio et al., Iwanami Shoten, March 27, 2000, 157-159.
Page ".

As a method of cut detection necessary for specifying a shot, for example, there is a method of calculating a difference in gray value of a corresponding pixel between two temporally adjacent frames.

The scene is extracted by taking into account the continuity of these information such as video information such as color, audio information such as BGM, audio information spoken by humans, and important words included in text information such as telops. Can be considered.

The video priority determining unit 102 further receives a result of the analysis by the video analyzing unit 301 and determines a final frame priority, and outputs an output of the frame priority determining unit 302. And an output unit 303 for outputting to the video priority information accumulation unit 103 in FIG.

FIG. 4 is a schematic diagram showing the priorities analyzed for each frame by the video priority determining unit 102.

Referring to FIG. 4, the priority of each frame is calculated based on histogram 4 corresponding to video sequence 400.
It is represented by 01. For example, a rule that determines that a frame is important if a face is included, a rule that is determined if a moving object is included, or a rule that if short shots are consecutive, those frames and the frame immediately after them are important The priority is determined by a plurality of defined rules. The video sequence 400 is analyzed according to such rules, and the priority is counted for each frame to obtain the histogram 401. Of the plurality of rules, a frame determined to conform to more rules can be considered to have a higher priority.

Here, the priority is derived for each frame. If a shot is specified, for example, the first frame in one shot is set as a representative frame, and the priority of only this representative frame is calculated. By setting the priority for the entire shot, the amount of calculation can be reduced.

When the shots, cuts, and scenes are specified by the video analysis unit 301 in FIG. 3, for example, it is possible to view the summary video in scene units. It is also possible to obtain the priority of a scene based on the priority of each frame, and this can be used when viewing a summary video.

[Explanation of Summary Video Creation] FIGS. 5 and 6 are flowcharts showing the operation of the summary video scenario table creation unit 104 in FIG.

Referring to FIG. 5, before the processing is started in step S500, summary video scenario table creating section 104 previously holds the total playback time of the summary video and the continuous playback time as parameters. It is assumed that

Here, the total reproduction time is a parameter indicating how long the user wants to put the summary video as a whole. In addition, the continuous playback time is a parameter indicating how many frames are to be played continuously starting from the key frame in order to recognize an extracted key frame having a high priority as a moving image.

In this description, it is assumed that the video priority information is accompanied by an index number for specifying a shot.

First, the video priority information storage unit 103 shown in FIG.
Is input to the summary video scenario table creation unit 104, and the process is started in step S500.

In step S501, the summary video scenario table creation unit 104 initializes an index number I indicating a shot to be processed from now on to zero. Further, the summary video scenario table creation unit 104 sets the variable W indicating the priority of the shot to be processed from now on to the highest priority.

Next, in step S502, the summary video scenario table creation unit 104 determines whether the priority of the shot number I is equal to the priority W currently being processed. The priority of the shot number I is determined by the priority of the frame included in the shot. The priority of the frame having the highest priority may be set as the priority of the shot, or the priority of the frame located at the beginning of the shot may be set as the priority of the shot for simplicity of calculation.

If the priority of the shot number I is equal to the priority W currently being processed, the flow advances to step S503 to mark the representative frame of the shot I as a key frame, and via the connector A to step S504 in FIG. Proceed to.

At this time, the summary video scenario table creation unit 104 sets the variable J storing the index number of the frame uniquely allocated from the head of the video to the number of the representative frame of the shot I.

If the priorities are not equal in step S502, the flow advances to step S512. Step S512
Then, the shot number I is incremented. Then, the process advances to step S509 to determine whether or not the last shot in the program currently being processed has been processed. If the processing has been performed up to the last shot, step S510
Then, the shot number I is initialized to 0, and the priority W is reduced by one. Priority W in step S510
Is decreased, the process proceeds to step S502 again.

If the last shot has not been reached in step S509, step S510 is skipped and the process proceeds to step S502 again.

That is, steps S512, S509, S
In 510, when the search is performed up to the last shot of the program, a process of reducing the priority by 1 is performed.

Referring to FIG. 6, in step S504,
The index J of the frame number is incremented. Subsequently, in step S505, it is determined whether the frame currently being processed is the head of the shot or whether the index J has been incremented to the last frame.

The case where the frame corresponds to the head of the shot is the case where the index J is incremented and the position of the frame currently being processed enters the shot next to the shot to which the key frame belongs. . If the shots are different, for example, a frame of a sports program and a frame of a commercial may be included during continuous playback time. In this case, since the video is not smooth even during the continuous reproduction time, the mark of the frame is interrupted.

When the index J is incremented to the last frame of the program, it is impossible to mark the subsequent frames, so that it is necessary to make a judgment.

Therefore, when these conditions are satisfied, the flow advances to step S508. On the other hand, if the condition is not satisfied, the process proceeds to step S506, and it is determined whether or not the total playback time of the frame marked continuously with a certain key frame at the beginning exceeds the continuous playback time. If the total playback time of the marked frames exceeds the continuous playback time, the process proceeds to step S508.

On the other hand, if not exceeded, step S
Proceeding to 507, frame J is marked as an additional frame to the already marked key frame without consideration of priority.

That is, by the processing of steps S505 to S507, after an important key frame, several frames specified as a continuous reproduction time are continuously reproduced to maintain smoothness as a video. If a cut exists between these several frames, continuous reproduction is terminated by making a determination in step S505.

In step S508, step S
It is evaluated whether or not the sum of the reproduction times of the key frames and the additional frames marked in step 503 and step S507 exceeds a predetermined total reproduction time of the summary video.

If the predetermined total reproduction time has elapsed, the flow advances to step S511 to complete the creation of the summary video scenario table. If not, step S50
Proceed again to 2 and repeat the loop.

FIG. 7 is a diagram for explaining that the summary video scenario table is created according to the flowcharts shown in FIGS.

Prior to this processing, it is assumed that the video priority determining unit 102 has previously performed the processing for setting the priority of the frame having the highest priority among the shots as the priority of the shot.

Referring to FIG. 7, a case where shots 1 to 10 are displayed in time series in one video sequence will be described. Priority is given to each shot,
It is assumed that the priority has 10 levels, with 10 being the highest priority. For each shot, shot 1 has priority 2,
Shot 2 has priority 3, shot 3 has priority 4, and shot 4 has priority 10.

Furthermore, shot 5 has priority 9, shot 6 has priority 8, shot 7 has priority 3, and shot 8 has priority 10. Also shot 9
Has priority 9 and shot 10 has priority 7.

In FIG. 7, one circle represents a plurality of frames corresponding to one second of reproduction time. The plurality of frames are called a frame set.

Hereinafter, the process of selecting frames for the summary video will be described. A case where a frame set for a total reproduction time of 10 seconds is selected as a summary video from the frame sets included in shots 1 to 10 will be described.
5 and 6 when such data is given.
An example of how the process is performed will be described with reference to the flowchart of FIG.

First, in step S501, the priority W is set to 10, which is the highest priority, in the subsequent processing. Shots 1-3 have a priority lower than 10, so
It is skipped in the loop of steps S502, S512, and S509.

Subsequently, when the shot number I becomes 4, since the shot 4 has the priority 10, the steps S502 and S502 are executed.
At 503, the representative frame, which is the first frame of shot 4, is marked as a frame included in the summary video. Here, as can be seen from FIG. 7, the shot 4 also includes a frame set 552 following the frame set 551.

FIG. 8 is a diagram for explaining the structure of the frame sets 551 and 552. Referring to FIG. 8, frame 571 which is the first frame of frame set 551
Is the representative frame at the beginning of the shot, and
This is the key frame marked in step S503.
Note that a key frame is a frame that is marked first in a shot. As shown in FIG. 7, the priority of the entire shot is determined by the priority of the highest priority frame included in the shot, or the priority of the representative frame which is the first frame is determined. If so, the priorities of the frames included in the shot are all equal, so the key frame is a marked representative frame.

In order to recognize the key frame as a moving image, several frames following the key frame are automatically included in the digest. For example, a continuous reproduction time of 2 seconds is continuously included in the digest. Frames that are marked continuously with the key frame at the beginning in this manner are referred to as a continuous playback frame set.

Assuming that the frame rate is 30 frames / second, the number of frames for 2 seconds is 60 frames. Therefore, 59 frames following the key frame are selected as additional frames used for the summary video. In this case, the frame from the frame 571 to the frame 572 corresponding to the 60th frame is marked as being included in the digest in step S507.

Referring again to FIGS. 6 and 7, when a desired continuous reproduction frame set is marked per shot,
The process proceeds from step S506 to step S508.

Subsequently, in step S508, it is determined whether or not the total reproduction time of the frames marked so far exceeds the total time of the desired summary video. At this time, the frame sets 551 and 552, that is, two seconds from the frames 571 to 572 in FIG. 8 are selected, and the desired total reproduction time is less than 10 seconds.
08 to step S502.

By the above processing, the frame set at the time when the first loop in FIG. 7 has been rotated is selected. At this point, the frame sets 551 and 552 have been selected.

Subsequently, the first frame of the next shot of priority 10 is selected as a key frame. This frame is included in frame set 555.

As in the first loop, the key frames included in the two frame sets 555 and 556 and their additional frames are marked as frames included in the summary video. With these processes, the second loop ends.

The frame sets marked at the end of the second loop are frame sets 551 and 5
52, 555 and 556. Even at this point, since the total playback time is less than 10 seconds, the process proceeds to the third loop.

Since there are no more shots with a priority of 10, the process proceeds to the last shot 10, and the shot number I is initialized to 0 in step S510. Then, the priority is changed from 10 to 9.

In the third loop, the frame set 55
3 will be marked. Here, step S50
Although the frame number J is incremented at 4, the shot number 6, which is the next shot, is reached before the continuous reproduction time of 2 seconds is satisfied. Therefore, the process proceeds from step S505 to step S508. Even at this point, the total playback time is less than 10 seconds, so the loop is repeated. At the end of the third loop,
The selected framesets are 551, 552, 553,
555 and 556.

The loop is repeated in this manner. Finally, the frame set 558 is selected, and the total reproduction time is 10
For more than a second. At this time, from step S508 to step S
Proceeding to 511, the creation of the summary video scenario record ends.

The above processing is repeatedly executed to create a plurality of scenario records corresponding to a plurality of total reproduction times, for example, a scenario record for 30 seconds, for one minute, etc.

FIG. 9 is a schematic diagram showing an example of the summary video scenario table created by the summary video scenario table creation section 104 of FIG.

Referring to FIG. 9, a summary video scenario table is created corresponding to original video 600. This summary video scenario table contains the scenario record 60
1,602,603. Scenario record 601,
602 and 603 indicate a total playback time of 30 seconds and 1
These are scenarios with different lengths corresponding to minutes and minutes.

The scenario record 601 includes continuous playback frame sets 601.1 and 601.1 whose continuous playback time is 3 seconds.
1.2,.

The scenario record 602 includes a continuous playback frame set 602.
1, 602.2, 602.3, and so on.

The scenario record 603 has a continuous playback frame set 603.
1, 603.3, 603.4 and a continuous playback frame set 603.2 with a continuous playback time of 3 seconds.

The contents reproduced by the scenario record 601 include shot number 1, ie, 0 of the original video.
It contains information such as reproduction from minute 0 second to 3 seconds, and reproduction from shot number 29, that is, normal elapsed time 7 minutes 23 seconds for 3 seconds. This information may be the time information itself as shown in FIG. 9 or may be designated by a pointer such as a frame number.

Similarly, scenario records 602 and 603
However, a set of a normal elapsed time of a video and a continuous reproduction time from that position is described, and is expressed as a reproduction scenario of a summary video.

Such a scenario record can be sequentially obtained by giving the total reproduction time and the continuous reproduction time as parameters in the flowcharts shown in FIGS. Further, in FIG. 9, the total playback times are all different in time length, but a plurality of scenario records having different continuous playback times and the same time length may be included.

[Explanation of Video Playback Operation] FIG. 10 is a display example for explaining the video display unit 109 in FIG.

Referring to FIG. 10, in this example, a PDA (Personal Digital Assembler) is used as one mode of video display unit 109.
(i.e., istant: personal portable information terminal) 700 shows an image being displayed.

The PDA 700 is provided with a touch panel 701 also serving as a display area and operation buttons 702 to 706. An original video and a summary video are displayed on the touch panel 701 according to a user's instruction. By operating the touch panel 701, the user can give an instruction to start reproduction and view a detailed summary video. In addition, the user can control the reproduction status using the operation buttons 702 to 706. For example, it is possible to switch from a summary video that requires a short playback time to coarse content, which has a long playback time but contains more detailed content, or to switch from a detailed summary video to a coarse summary video.

Touch panel 70 also serving as display area
1 corresponds to the image display unit 109 in FIG. Also, the touch panel 701 and the operation buttons 702 to 706
Corresponds to the user input unit 106.

Specifically, when the operation button 702 is pressed,
The PDA 700 returns from the video being played back to the top of the summary video, and restarts playback of the summary video with the shortest playback time.
When the operation button 703 is pressed, the PDA 700 returns to the previous shot of the currently reproduced summary video and resumes the reproduction. When the operation button 704 is pressed, the PDA 700 stops playing the video. When the operation button 705 is pressed, the PDA 7
00 reproduces the original video of the currently viewed summary video. When the operation button 706 is pressed, the PDA 700 shifts from the currently reproduced summary video to a coarser summary video.
The flow of these operations will be described later in more detail using a flowchart.

[Explanation of Change of Multi-Program Scenario] The video apparatus of the present invention grasps the user's preference from the history of the operations performed by the user when viewing and reflects it in the reproduction of the summary video. . The number of viewing requests of the user is registered for each program category. For example, a user who likes a drama and does not often watch the news is provided with a summary video that reproduces the drama program with priority over the news. Then, in response to a user request, the summary video can be switched to a summary video containing more detailed contents during reproduction.

FIG. 11 is a diagram showing an example of the required reproduction time table stored in the required reproduction time table storage unit 111 of FIG.

Referring to FIG. 11, the required reproduction time table includes digest levels and corresponding required reproduction times. In this example, there are three levels of digest levels. Level 1 has the shortest reproduction time with a reproduction request time of 1 minute, level 2 has 3 minutes, and level 3 has a maximum of 10 minutes.
At the start of reproduction, the digest level is normally set to level 1, and can be shifted to level 2 or level 3 according to the user's request. The multi-program scenario creating unit 110 of FIG. 1 obtains the total playback time by referring to the playback request time table according to the digest level.

Initially, the digest level is level 1, and the reproduction of the summary video is 1 if the user does not perform a selection operation.
Although the process is completed in minutes, when the user performs a selection operation during the playback of the summary video to display detailed contents, the digest level shifts to 2, and the total playback time becomes longer.

FIG. 12 shows the user operation storage unit 112 of FIG.
FIG. 8 is a diagram for explaining the number of requests for each category obtained from the operation history of the user recorded in the user.

Referring to FIG. 12, news, sports,
Category information such as documentaries, dramas, varieties, and movies is attached as program-attached information. This is stored in the program information storage unit 113 of FIG.

When the user operates touch panel 701, the number of viewing requests for the category of the program currently being viewed is counted. Then, a histogram as shown in FIG. 12 is formed.

That is, since the user operates the touch panel 701 when he / she wants to view the summary video in detail, the user is interested in the program currently being viewed, and the category of the program is a category that the user prefers. Conceivable. Therefore, in the histogram, it is desired to provide a summary video in which a category having a higher number of viewing requests by the user is preferentially reproduced. Note that the number of viewing requests may be counted when the user presses the operation button 705 in addition to the above operation and selects to view the original of the video. Alternatively, the number of viewing requests may be counted only when the user presses the operation button 705 and selects to view the original of the video.

FIG. 13 is a diagram conceptually illustrating an example in which a multi-program scenario is changed by changing a scenario record according to a user operation.

With reference to FIG. 13, the user
After watching the summary video of, it is assumed that the user has viewed the summary video of drama 1.

It is assumed that the user operates touch panel 701 while watching drama 1 for more details.
Then, the summary video scenario of drama 1 is changed to a scenario record including more detailed contents.

The priority order of the programs displayed following drama 1 is the same as that of FIG.
Based on the histogram shown in FIG. 2, the order is switched so that the drama, documentary, and news are played in the order of the drama, the documentary, and the news by the operation based on the flowchart described later.

A keyword indicating information content of a program may be used instead of or in addition to the category. For example, the keywords include a cast name, a title name, a production country name, and the like of the program.
This information is also transmitted as information accompanying the program,
It is input to the video device and stored in the program information storage unit 113.

For example, when the keyword is a performer name, the histogram of FIG. 12 shows the number of viewing requests for each performer, such as performer A, performer B, performer C, performer D,. Is counted. Then, when the number of viewing requests corresponding to the performer D is the largest, the multi-program scenario creating section 110 creates a multi-program scenario giving priority to the program in which the performer D appears.

As described above, the multi-program scenario is changed so as to reflect the user's preference while considering the entire reproduction time. That is, if no operation is performed, the summary image can be viewed so as to fit within the predetermined reproduction request time, and detailed content can be viewed if the user requests. Therefore, the user can view the summary video according to his / her preference while considering the scheduled time to some extent, and can quickly reach the desired video.

FIGS. 14 and 15 are flowcharts for explaining the operation of the video display control unit 107 in FIG.
For the sake of simplicity, in these flowcharts, the category of the program is described as an example of the user operation information. However, as user operation information,
Similar processing may be performed using the above-described keyword.

Referring to FIGS. 14 and 15, first, a processing routine is started in step S800.

FIG. 16 shows a state where the image storage unit 10 shown in FIG.
FIG. 2 is a diagram for explaining the contents of a video stored in the image data 1;

Referring to FIG. 16, image storage unit 10 shown in FIG.
It is assumed that a program 1, a program 2, a program 3,... For example, when the program is a television program, the program 1, the program 2, and the program 3 are recorded in a non-viewed state by scheduled recording or the like. Each program contains multiple shots. For example, program 1 has shots 1-3
Including multiple shots.

As described above, when the video is stored in the video storage unit 101 in order from the program 1, the user designates the program that he / she wants to view. For example, by specifying a recording time, a program recorded near that time may be preferentially reproduced to reproduce a one-day summary video.

Referring again to FIGS. 14 and 15, the processing routine is started in step S800, and step S801 is performed.
Thus, the first digest playback scenario is created.
This scenario requires the shortest playback time as the initial state.

In the following step S802, the video display control unit 107 waits until there is an instruction from the user input unit 106 to start reproduction. This instruction is performed by touching touch panel 701 in FIG. Subsequently, in step S803, the reproduction of the coarsest summary video, that is, the summary video requiring a short reproduction time is started.

Next, in step S804, the video display control unit 107 checks whether or not the user has given an instruction to select a summary video that is more detailed than the currently reproduced summary video. It should be noted that the user can instruct reproduction of a more detailed summary video than the current layer by touching the touch panel 701 in FIG. If there is a selection instruction, it is considered that the user is interested in the program currently being reproduced, and the process proceeds to step S814 to record the category of the selected program. Then, the histogram as described in FIG. 12 is updated. Then, the process proceeds to step S805.

On the other hand, if no selection has been made in step S804, steps S814 and S805 are skipped, and the flow advances to step S806.

In step S805, it is determined whether or not the video currently being reproduced corresponds to the most detailed summary video. If not, the flow advances to steps S815 and S809. On the other hand, if it is the most detailed summary video, step S8
15, skip step S809 and proceed to step S806.

In step S815, a new digest reproduction multi-program scenario described later is created. Then, in step S809, step S809
Based on the new multi-program scenario created in 815, the reproduction is restarted from the same shot that is currently being reproduced, and the process proceeds to step S806.

Next, in step S806, the video display control unit 107 confirms whether or not the user has given an instruction to select a summary video that is coarser than the currently reproduced summary video. If so, step S80
Go to 7. On the other hand, if the selection instruction has not been issued, step S807 is skipped and the process proceeds to step S812. In addition,
By touching the operation button 706 in FIG. 10, the user can give an instruction to reproduce a summary video whose level is one level higher than the currently reproduced summary video.

In step S807, it is determined whether or not the video currently being reproduced is the coarsest summary video.
If the video is not the coarsest summary video, the process proceeds to step S810. If the video being reproduced is the coarsest summary video, the process skips step S810 and proceeds to step S812.

In step S810, the reproduction is restarted from the next shot close to the same shot currently reproduced, based on the coarser summary video scenario corresponding to the next hierarchy.

When the processing in step S810 ends, the flow advances to step S812. In step S812, it is determined whether or not the reproduction of the currently reproduced program has been completed. If the processing has been completed, the process proceeds to step S813. On the other hand, if not completed, step S813 is skipped and step S8
Proceed to 08.

In step S813, the process moves to the next program, and the shortest digest having the highest level is reproduced again.

When the processing in step S813 ends, the flow advances to step S808. In step S808, it is confirmed whether or not the reproduction processing of the summary video of all programs to be reproduced has been completed. If there is no program to be reproduced, the flow advances to step S811 to end the reproduction.

In step S808, it is also confirmed whether or not a stop instruction has been given by the user. If there is a stop instruction, the process similarly proceeds to step S811 and ends.

FIG. 17 is a flowchart showing the operation of determining the program reproduction order of the multi-program scenario for reproducing the summary video in step S815 in FIG.

Referring to FIG. 17, a user instructs reproduction of a summary video from user input section 106, and the process starts (step S901).

When the user inputs a time through the user input section 106, the time is set to the time Rreq, which is the user's desired recording time (step S902). That is, Rreq indicates “when the user wants to view the recorded program”.

[0138] In the program stored in the video storage unit 101, the recorded time is also recorded. The recording start time Ri and the time Rreq of the i-th program registered in the multiprogram scenario for reproduction are compared, and the absolute value of the difference is set as a variable Bi. That is, the variable Bi is the difference between the time Rreq and the recording start time Ri of the program. Therefore, it indicates that the larger the variable Bi, the more distant the program content is from the recording time desired by the user (step S903).

Then, the index i is incremented as necessary to obtain Bi corresponding to all the programs included in the scenario (step S904).

When the processing for all programs is completed, the cumulative selection number Dj for each category stored in the user operation storage unit 112 of FIG. 1 is obtained. For example, the viewing request count in FIG. 12 corresponds to the cumulative selection count Dj. Then, normalization processing is performed on each of Bi and Dj (step S905). Here, i is the index of the program, and j is the index of the category.

Here, the normalization process is a process for obtaining a normalized feature value by the following equation. Normalized feature value = (measured value−mean value) / standard deviation (1) The normalized feature value is used when comparing feature values of different units, and is a distribution having an average of 0 and a variance of 1. The conversion of the measured values is performed to have This processing is described in "Organization of Information, Nagao et al., Iwanami Shoten, published on July 25, 2000, page 105".

Specifically, the normalized feature value BB of the variable Bi
i is obtained by substituting the variable Bi for the measured value in the equation (1),
Variable B of all programs to be summarized and played back to the average value
It is determined by substituting the average value of i and substituting the standard deviation of each variable Bi in the program to be summarized and reproduced into the standard deviation.

The normalized feature value DD of the cumulative selection number Dj
Similarly, for j, in equation (1), the cumulative selection number Dj is substituted for the measured value, the average value of the cumulative selection numbers Dj of all registered categories is substituted for the average value, and each cumulative value is substituted for the standard deviation. It is determined by substituting the standard deviation of the selected number Dj in the entire registered category.

When the normalization processing is completed in step S905, Bi and Dj are normalized BBi and Dj, respectively.
Dj is obtained. Using these, an evaluation function Ti for each program is obtained by the following equation (S906, S907).

Ti = w1 · BBi−w2 · DDj (2) where w1 and w2 are weighting coefficients. Also, DDj
Is the category cumulative selection number D to which the i-th program belongs
j is the normalized feature value of j.

Then, when the evaluation functions Ti for all the programs are obtained, the evaluation functions Ti are rearranged in ascending order (step S908). As a result, the reproduction order of the programs is determined (step S909).

By the above processing, a program that is included in an important category and has new recorded contents is reproduced with priority.

As a specific example, for example, consider a case where a user likes a drama and retrieves a digest in order to retrieve “a television program recorded around 7 pm on February 20”. In the input history so far, the fact that the user likes the drama is recorded on the video device 10. Therefore, the category "drama" is set to be large for DDj in equation (2). In addition, the user receives “February 20, 7:00 pm”
, The value of BBi in equation (2) becomes smaller as the program is recorded at 7:00 pm on February 20 or at a time closer thereto.

Therefore, in the program recorded at 7:00 pm or close to February 7:00 on February 20, the program having a category of drama has a smaller evaluation function Ti, so that the video device preferentially summarizes this program. Play the video.

FIG. 18 is a flowchart showing the operation of determining the program reproduction time length of the digest reproduction scenario in step S815 of FIG.

Referring to FIG. 18, first, a reproduction request time is acquired from the reproduction request table shown in FIG. 11 in accordance with the current digest level (steps S951 and S951).
952).

The original playback time of the program is Li,
The number of shots included in the program is Ui, and the number of selected categories to which the program belongs is Di. Note that, for example, the number of viewing requests in FIG. 12 corresponds to the number of selections Di. Then, Li, Ui, and Di are normalized, and LLi, U
Ui and DDi are obtained (step S953).

Then, an evaluation function Ci is calculated for all programs by the following equation (steps S954 and S954).
955).

Ci = w3 · LLi + w4 · UUi + w5 · DDi (3) where w3 to w5 are weighting coefficients.

When the evaluation functions Ci of all the programs have been obtained, the required reproduction time is allocated to each program according to the value of the evaluation function Ci (step S956).

That is, the playback time of the summary video of each program is determined by the importance of the category, the time length of the original program, and the number of shots or cuts included in the program.

For example, specifically, a TV program having a longer recording time has a larger evaluation function Ci. Further, the content of a television program with a large number of scene changes cannot be grasped unless the time of the summary video is increased, so that the evaluation function Ci increases. Furthermore, since it is considered that a television program belonging to a category having a history selected by a larger number of viewers includes more contents preferred by the viewer, the evaluation function Ci increases. These are weighted by the equation (3) to determine the distribution of the reproduction time.

Next, an example of the reproducing operation will be described in detail with reference to a flow chart using a simple schematic diagram.

FIG. 19 is a schematic diagram for explaining reproduction when digest levels are prepared for three layers.

Referring to FIG. 19, the first scenario is as follows.
This is a multi-program scenario created by the multi-program scenario creating unit 110 in FIG. 1 in step S801 in FIG. A to E represent each program.

The multi-program scenario creation section 110 acquires the total playback time from the digest level 1 portion of the playback request time table in FIG. The acquired total reproduction time is one minute. Then, an appropriate scenario record is selected from the summary video scenario table as shown in FIG. 9 corresponding to the programs A to E so that the total playback time becomes 1 minute, and these are combined to form the first scenario record. Multi-program scenario.

It is assumed that the reproduction of the digest of the program A is completed, and the user operates the touch panel 701 at the point P1 to issue a more detailed digest viewing request during the reproduction of the subsequent program B. Then, the multi-program scenario creating unit 110 proceeds to step S81 in FIG.
At 5, a new multi-program scenario is created.

First, the total reproduction time corresponding to digest level 2 is obtained from the reproduction time request table of FIG. Then, the total playback time is changed to 3 minutes. Currently, the programs that make up the digest are AE
Therefore, the reproduction time per program is calculated as 36 seconds by 3 (minutes) ÷ 5 (pieces).
Then, a scenario record close to 36 seconds is selected and read from the summary video scenario table of the program B.

Multi-program scenario creating section 110
Replaces the scenario record portion of the program B of the first scenario in FIG. 15 with the newly read scenario record. The reproduction of the summary video is performed continuously from the same shot of the program B.

For the remaining programs C, D and E,
The redistribution of the reproduction time is performed according to the flowchart described with reference to FIG.

The total of the total playback time assigned to the three programs C, D, and E is 40 seconds originally assigned in the first multi-program scenario. In step S952, the total playback time of the remaining programs is obtained. This is obtained from the reproduction request time table at the first time, but the time determined at the first time is inherited from the second time. Note that, for example, the remaining time can also be obtained by obtaining the remaining playback time counted by the video device 10 during the playback.

The total reproduction time of the remaining programs of 40 seconds is distributed again according to the predetermined parameters normalized in step S953. In the operation shown in FIG. 19, the selection operation by the user is performed, so that the number Di of selections of the category to which the program belongs among the parameters has changed.

[0168] The change of the reproduction order is also performed according to the flowchart explained in FIG. In the second scenario, selecting program B results in a change in the number of categories to which program B belongs,
The reproduction order of the program changes in the order of D, E, and C.

Subsequently, suppose that the user has requested viewing of a detailed digest at point P2. Again, the reproduction request time table is referred to, and digest level 3
Is obtained. As a result, the total playback time of the entire summary video is 10 minutes. At digest level 3, the total playback time of one program is 2 minutes of 10 (minutes) ÷ 5 (pieces), so the total playback time of program B is set to 2 minutes. Then, a scenario record close to two minutes is selected and read from the summary video scenario table of the program B. The multi-program scenario creating unit 110 replaces the scenario record part of the program B of the second scenario in FIG. 19 with the newly read scenario record. The reproduction of the summary video is performed continuously from the same shot of the program B.

Regarding the reproduction times and the reproduction order of the programs C, D, and E, the same processing as when the first scenario is changed to the second scenario is performed. As a result, in the third scenario, the playback order is E, D, C
, And the distribution ratio is also changed for the remaining time of 40 seconds.

Next, during reproduction of the third scenario, FIG.
The case where the operation button 0 706 is operated will be described.

FIG. 20 is a schematic diagram for explaining the operation when the operation button 706 is pressed.

Referring to FIG. 20, when a summary video is reproduced based on the third scenario, if the user presses operation button 706 at point P3 to request viewing of a coarse digest (hereinafter referred to as fast-forward operation). ),
The scenario record of the program B is replaced with that of the total playback time corresponding to level 2. That is, FIG.
The same scenario as that of the part of the program B in the second scenario of No. 9 is also used for the fourth scenario of FIG.

As for the programs C, D, and E, the portions of the programs E, D, and C of the third scenario are used as they are. Thus, the fourth scenario can be created by splicing previously created scenarios.

If the fast-forward operation is further performed at point P4, what is used for the next fifth scenario is the program B portion of the first scenario and the third and fourth scenarios in FIG. This is a part related to the programs E, D, and C included. These two parts are joined together to create a fifth scenario.

That is, in the case of the fast-forward operation, a new multi-program scenario can be created by using a previously created multi-program scenario.

As described above, the video apparatus of the present invention can reproduce a summary video in a certain fixed time regardless of the number of programs requested to be viewed. Further, if the user desires, it is possible to reproduce a detailed summary video that takes more time. Therefore, the user can quickly search for a desired video with a simple operation.

The embodiments disclosed this time are to be considered in all respects as illustrative and not restrictive. The scope of the present invention is defined by the terms of the claims, rather than the description above, and is intended to include any modifications within the scope and meaning equivalent to the terms of the claims.

[0179]

As described above, according to the present invention,
Even if there are many unviewed programs, the user can grasp the outline in a short time by reproducing the summary image that fits in a certain time. In addition, when the user wants to know the contents of the summary image a little in detail during the reproduction of the summary image, the digest level can be changed at that point, so that the desired image can be easily found in the summary image. Furthermore, a summary video according to the user's preference can be provided.

[Brief description of the drawings]

FIG. 1 is a block diagram showing a configuration of a video apparatus 10 using a summary video, which is one embodiment of the present invention.

FIG. 2 is a diagram showing how video data included in a program in the present specification is structured.

FIG. 3 is a block diagram showing a configuration of a video priority determining unit 102 in FIG.

FIG. 4 is a schematic diagram showing priorities analyzed for each frame by a video priority determining unit 102;

FIG. 5 is a first half of a flowchart showing the operation of the summary video scenario table creation unit 104 in FIG. 1;

FIG. 6 is the latter half of the flowchart showing the operation of the summary video scenario table creation unit 104 in FIG. 1;

FIG. 7 is a diagram for explaining that a summary video scenario table is created according to the flowcharts shown in FIGS. 5 and 6;

FIG. 8 is a diagram illustrating the structure of frame sets 551 and 552.

9 is a summary video scenario table creation unit 10 of FIG.
FIG. 9 is a schematic diagram showing an example of a summary video scenario table created in 4.

FIG. 10 is a display example for explaining the image display unit 109 in FIG.

11 is a reproduction request time table storage unit 111 shown in FIG. 1;
FIG. 5 is a diagram showing an example of a requested reproduction time table stored in the storage device.

12 is a diagram for describing the number of requests for each category obtained from a user operation history recorded in the user operation storage unit 112 of FIG. 1;

FIG. 13 is a diagram conceptually illustrating an example in which a multi-program scenario changes due to a change in a scenario record in response to a user operation.

FIG. 14 is the first half of a flowchart for explaining the operation of the video display control unit 107 in FIG. 1;

FIG. 15 is a latter half of a flowchart for explaining the operation of the video display control unit 107 in FIG. 1;

FIG. 16 is a diagram for explaining the contents of video stored in the video storage unit 101 of FIG. 1 at the start.

17 is a flowchart showing an operation of determining a program reproduction order of a multi-program scenario for reproducing a digest video in step S815 of FIG.

FIG. 18 is a flowchart showing the operation of determining the program playback time length of the digest playback scenario in step S815 of FIG.

FIG. 19 is a schematic diagram for explaining reproduction when digest levels are prepared for three layers.

FIG. 20 is a schematic diagram for explaining an operation when an operation button 706 is pressed.

[Explanation of symbols]

Reference Signs List 10 video device, 100 video input unit, 101 video storage unit, 102 video priority determination unit, 103 video priority information storage unit, 104 summary video scenario table creation unit, 105 summary video scenario table storage unit, 106
User input unit, 107 video display control unit, 108 video playback unit, 109 video display unit, 110 multi-program scenario creation unit, 111 playback required time table storage unit, 112 user operation storage unit, 113 program information storage unit, 114 program information Management section, 200 video sequences, 211 scenes, 221 shots,
226 cuts, 230, 571, 572 frames,
300 input unit, 301 video analysis unit, 302 frame priority determination unit, 303 output unit, 551-558 frame set, 600 original video, 601-603
Scenario records, 601.1 to 601.2, 602.
1 to 602.3, 603.1 to 603.4, continuous playback frame set, 701 touch panel, 702 to 706
Manual operation button.

Claims (6)

[Claims] 1. A video information storage means for storing a plurality of video sequences each including a plurality of frames, a user input means for receiving an instruction from a viewer, and a viewer instruction input from the user input means. A user operation storage means for storing user operation information as a history; and a scenario table storage means for storing a scenario table corresponding to each video sequence, wherein the scenario table includes a plurality of summaries having different total playback times. A scenario record of a plurality of layers including information for identifying a frame used for a summary video from the video sequence is selected, and a scenario record is selected from each of the scenario tables with reference to the user operation information. Combine and use for a summary video that outlines multiple video sequences Multi-program scenario creation means for creating and outputting a multi-program scenario including information for specifying a frame, and video display control means for controlling display of a summary video according to the output of the user input means and the multi-program scenario And a video playback unit that reads data corresponding to a frame from the video information storage unit in accordance with an instruction of the video display control unit and reproduces the data as a video signal. 2. The multi-program scenario creator, according to a change in the user operation information, in response to a change in the user operation information during reproduction of the summary video, The video device according to claim 1, wherein a portion corresponding to the summary video to be reproduced is changed. 3. The user operation information includes a category indicating a classification of a video sequence, and the number of reproduction requests for each category for which the viewer has requested the reproduction of the video sequence. 2. The multi-program scenario according to claim 1, wherein the multi-program scenario is created such that the summary video corresponding to the video sequence is preferentially arranged in front of the entire summary video according to the number of playback requests of the category to which the video sequence belongs. The imaging device according to the above. 4. The user operation information includes a category indicating a classification of a video sequence, and the number of reproduction requests for each category in which the viewer has requested the reproduction of the video sequence. 2. The multi-program scenario according to claim 1, wherein a ratio of a summary video corresponding to each of the video sequences to a total of the summary video is determined according to the number of reproduction requests of a category to which the video sequence belongs. 3. Video equipment. 5. The user operation information includes a plurality of keywords indicating information content represented by a video sequence, and the number of times the viewer has requested reproduction of the video sequence corresponding to each of the keywords, The multi-program scenario creating means, according to the number of reproduction requests corresponding to each of the keywords, stores the multi-program scenario such that the summary video corresponding to the video sequence is preferentially arranged in front of the entire summary video. The video device according to claim 1, wherein the video device is created. 6. The user operation information includes a plurality of keywords indicating information contents represented by a video sequence, and the number of times the viewer has requested reproduction of the video sequence corresponding to each of the keywords, The multi-program scenario creating means creates the multi-program scenario in which the ratio of the summary video corresponding to each video sequence to the total summary video is determined according to the number of reproduction requests corresponding to each of the keywords. The video device according to claim 1.