JP2008103802A - Image compositing device - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide an image compositing device capable of presenting information on other video images without interfering with viewing desire and understanding to a video image that a user is viewing mainly by selecting the video image related to a video scene for compositing and displaying when the user is viewing the video scene unimportant to the user. <P>SOLUTION: A video image feature detection section 102 acquires feature information from each scene in the image being viewed by the user currently, and an importance calculation section 103 uses the feature information for calculating the index values of the importance to each scene successively, thus detecting an unimportant fixed section (scene) by an unimportant scene identification section 104. At a recorded video image selection section 106, a subvideo image related to the video image under viewing is selected by a storage section 105. A video composite section 109 composites the subvideo image related to the video image under viewing mainly corresponding to an unimportant scene. <P>COPYRIGHT: (C)2008,JPO&INPIT

Description

  The present invention relates to a video composition apparatus that synthesizes and displays a plurality of videos.

  In recent years, with the speeding up of networks and the diversification of channels, the amount of video and video content that can be viewed by individuals has increased dramatically. Also, with the increase in capacity of recording media such as DVDs (Digital Versatile Discs) and hard disk drives (HDDs), a large amount of video is recorded on these recording media and the program can be viewed without being restricted by the broadcast time. Also became common.

If the amount of video handled by an individual becomes enormous in this way, it will not be possible to easily know whether or not there is a video that can be viewed by the user. Therefore, as a technique for obtaining the existence of a video from this enormous amount of video, Patent Document 1 mainly relates to the additional information attached to the video while viewing an arbitrary video to be viewed. A method has been proposed in which a video to be searched is automatically searched and the information or itself is multiplexed and output.
JP 2003-134428 A

  However, Patent Document 1 does not take into account the timing of video insertion, and multiplexes scenes that are considered to be important for the user, resulting in a significant reduction in the user's willingness to view the video. It was.

  In Patent Document 1, another information related to the program is searched at the start of the program using information attached to the video. In general, a video has a plurality of signals and semantically different properties. Since this method is composed of a combination of scenes, this method is not sufficient to select an appropriate different image related to each moment-changing scene.

  Furthermore, when the video is inserted and the section for multiple display becomes longer than the total playback time of the original video, simply playing the inserted video from the beginning will cause the original video to be played back before playback to the end. Thus, there has been a problem that the playback of the inserted video is finished and the inserted video is played halfway, making it difficult for the user to understand the outline of the inserted video.

  Therefore, an object of the present invention is to select a video that is related to the video scene when the user is watching a video scene that is not important, and display the synthesized video to display the video that the user mainly views. It is an object of the present invention to provide a video synthesizing apparatus capable of presenting other video information without disturbing the willingness to watch and understanding.

In view of such circumstances, the video composition device according to the first aspect of the present invention is a video composition device that synthesizes and displays at least two images that are broadcast or recorded.
A feature detection unit that detects feature information corresponding to each scene in the video, a recording unit that records the video and / or the feature information, and the importance of each scene of the first video based on the feature information A calculation unit that calculates an index value for the image, a video selection unit that selects a second video, and a scene in the first video that combines the second video based on the value of the index A scene specifying unit; and a video synthesizing unit that combines the second video with the scene in the first video specified by the scene specifying unit.

  In the video composition device according to the second invention, the video selection unit detects and selects a video similar to or related to the first video as the second video based on the feature information. It is characterized by.

  In the video composition device according to the third invention, the video selection unit selects the second video based on the feature information for each scene of the first video or each continuous scene group. And

  In the video composition device according to a fourth aspect of the invention, the feature information for the video selection unit to select the second video is the latest of the scene in the first video specified by the scene specification unit. The feature information is obtained from a scene or a scene group having a high value of the index.

  The video composition apparatus according to the fifth invention is characterized in that the first video is a video being broadcast and the second video is a video recorded in the device.

  The video composition device according to the sixth invention is characterized in that both the first video and the second video are video recorded in the device.

  According to a seventh aspect of the present invention, there is provided the video composition device, wherein the feature information includes video information, audio information, and text information that change every moment in the video.

  According to an eighth aspect of the present invention, the scene specifying unit specifies a scene in the first video to be combined with the second video by performing threshold processing on the index value. It is characterized by doing.

  In the video composition device according to a ninth aspect of the invention, the calculation unit sets the threshold value and outputs the threshold value together with the index value to the scene specifying unit.

  In the video composition device according to a tenth aspect of the invention, the calculation unit inverts the information on the threshold selection area according to the type of the first video scene.

  According to an eleventh aspect of the present invention, there is provided the video composition device, wherein the video composition unit matches the second video selected by the continuous time length of the scene in the first video specified by the scene specifying unit. It is characterized by being edited as a digest video.

  In the present invention, video information is obtained from each scene in the video currently being viewed by the user, and the importance index value for each scene is sequentially calculated by using the video information. A section (scene) is detected. By multiplexing other video mainly related to the video currently being viewed in accordance with this unimportant scene, information on other video can be obtained without hindering the user's willingness to understand or understanding the video that is mainly viewed. Can be provided.

  In addition, the user can select the selected video in consideration of the characteristics obtained from the scene immediately before the scene that is mainly regarded as unimportant in the currently viewed video, that is, the feature obtained from the recently important scene. Appropriate contents can be selected based on their interests.

  Further, the selected video can be digested in accordance with the length of an unimportant scene and played back, so that an outline of the selected video can be known in a short time.

Hereinafter, embodiments of the present invention will be described with reference to the drawings.
FIG. 1 is a functional block diagram showing a schematic configuration of a video composition device according to an embodiment of the present invention.

  The video composition apparatus 100 in FIG. 1 includes a video feature detection unit 102 that detects a feature of the video from the video obtained from the video input unit 101 and attached information accompanying the video, and each scene from the obtained video feature. An importance calculation unit 103 for calculating the importance of the image, a non-important scene specifying unit 104 for specifying a scene section having a low importance from the calculated importance, and a video already recorded and recorded in the recording unit 105 and its video feature The feature correlation between the input video and the recorded video is calculated, and the recorded video selection unit 106 for selecting the video correlated with the input video from the recorded video, the selected recorded video, the video feature and the unimportant scene specification The digest creation unit 107 that receives the non-important scene section information obtained from the unit 104 and creates a digest, a moving image from the digest of the broadcast content and the recorded content Configured to include an image combining unit 109 to switch the screen display on the radical 19 108.

  In general, when viewing a video, the user is not always focused on the video. In scenes that are generally not very important such as commercials and monotonous scenes, the user temporarily loses interest in the video being viewed.

  In the present invention, the importance level of each scene of the video mainly viewed by the user (hereinafter referred to as “main video”) is calculated and stored according to the scene determined to be low in importance using a threshold value. An appropriate video (hereinafter referred to as “sub-video”) from which the current user's interest has been estimated is selected from the video, and the sub-video itself or a digest re-edited from the sub-video is reproduced.

Here, the importance in the present invention is an index indicating the viewing value of each scene in the video. Here, since the main video and the sub-video are played back at the same time, the content of the main video is never lost while the sub-video is being viewed.
Furthermore, by selecting a video that is estimated to be of high user interest as a sub-video, the user can watch the video without getting bored.

Details of each part of FIG. 1 are shown below.
First, the operation of the video feature detection unit 102 will be described.

  The video feature detection unit 102 is a block that acquires various information from the video and detects the features of the entire video or each scene. The obtained video features are transmitted to the recorded video selection unit 106 and the importance level calculation unit 103, and are used to search for videos similar to or related to the viewed video and to calculate the importance level of the video. . Details of the recorded video selection unit 106 and the importance level calculation unit 103 will be described later.

In order to detect the video feature by the video feature detection unit 102, information obtained from the video itself or information attached to the video can be used.
Examples of information obtained from the video itself include EPG (Electronic Program Guide) and tag information attached to the video as information attached to the video, such as image information, audio information, or caption information.

  Examples of video features obtained from these video information include scene switching position and color histogram information obtained from image information, edge information, telop information, motion vector information, voice level and speech position obtained from voice information, background music (BGM) ) Important keywords such as information, titles obtained from EPG and tag information, genre information, performers, recording date and time.

  The video feature is obtained for each scene, and when the input video obtained from the video input unit 101 is a video being broadcast, it must be processed in real time, but the input video from the video input unit 101 is If the recorded video, that is, the entire video is known before the user starts viewing, it can be processed in advance. In this case, the video feature may be detected in advance and recorded in the recording unit 105.

  The importance level calculation unit 103 uses the video features obtained from the video feature detection unit 102 to calculate the importance level for each scene.

As described above, in general, even when a user selects and views a video of interest by himself / herself, the user does not concentrate on the video all the time.
For example, a monotonous pass scene in a soccer game, a scene that generally lacks excitement, such as when changing offense and defense in a baseball game, or a scene where a favorite talent or singer does not appear in a variety or song program The user temporarily loses interest in the video and does not concentrate on the video.

  In the present invention, the importance of each scene is determined by using a plurality of video features obtained from each scene of a video that changes every moment in a single or complex manner. A scene in which the user loses interest in the video is estimated using this importance.

  For example, a scene in which there are many objects from which a user can obtain useful information, such as a scene in which a person or a telop appears, has a feature that it includes many regions with high edge strength. From this, the importance using the edge strength can be considered.

First, the edge strength e is obtained by the following equation.
e = f (p) (1)

Here, p is the luminance at each pixel of an arbitrary image, and f (p) is a filter for edge acquisition represented by Sobel and Laplacian.
If the pixel in which the edge information e exceeds a preset threshold value t is detected as a region having a high edge strength, and the appearance frequency N of the region having a high edge strength is the importance w, the following equation is obtained.
w = N (e) (2)

  Further, an artificial geometric regularity such as a circle or a straight line may be further extracted by using the Hough transform for the set of pixels having a high edge strength e, and the appearance frequency may be used as the importance.

In addition to this, importance calculation using voice information can be considered.
In order to detect only the rising scene, the voice level obtained from the voice information may be set as the importance level, or in order to detect the scene where the person talks, the voice section is detected from the voice information, and further the voice of the person The utterance interval may be identified from the frequency characteristics of the, and the appearance frequency may be set as the importance. Also, in the case of a program in which the monaural audio is the main program and the stereo audio is commercial, the importance is set so that the commercial scene section is detected using the type of the audio format, and the importance of the commercial scene section is reduced. May be set.

  In the above-described example of creating importance, importance is calculated using information after video is decoded, but of course, importance may be calculated using information before decoding. For example, when video is compressed by an inter-frame predictive encoding method such as MPEG (Moving Picture Experts Group), motion vector information obtained from video data before decoding is used, so that the motion in the video is intense. The interval, that is, whether or not the user is visually stimulated can be set as the importance.

Also, a method of calculating importance using a combination of several video features is conceivable.
FIG. 2 is a diagram illustrating an example of importance calculation.
An example of importance calculation when a sports video is being viewed will be described with reference to FIG.
When watching sports videos such as soccer, the user may be interested in scenes where decisive events such as shoots, decisive passes, and fouls affect game results. Many people are less interested in other scenes.
Therefore, the calculation is performed so that the importance is high for the video information characterizing the decisive scene.

In general, the following video features can be obtained in a scene where the important event occurs.
I. Since the cheering of the audience increases, the sound level increases.
II. The camera switching frequency is high because it is often possible to switch to cameras from various angles one after another in order to capture important scenes from the optimal angle.
III. In a scene with a score, character information is displayed on the screen.

From these features, as shown in FIG. 2A, the audio level, camera switching frequency, and character information appearance frequency are acquired from the video feature detection unit 102 as video features, and the audio level at an arbitrary time t is set to x, time If the camera switching frequency during the unit time before and after t is y, and the character information appearance frequency during the unit time before and after time t is z, the video importance w is calculated as follows using the coefficients a, b, and c. can do.
w = ax + by + cz (3)

FIG. 2B shows an example calculated by the equation (3).
Here, the coefficients a, b, and c are arbitrary values set in advance in the present apparatus. Hereinafter, unless otherwise specified, it is assumed that the preset value is an arbitrary value preset in the present apparatus.

On the other hand, in a video such as a drama, there are many events that make the user interested in the scene where the performer is projected widely or the utterance scene of the performer. The length and number of utterances of performers can be detected as video features and used as an index of the importance of the video.
In addition, it is possible to register images and sounds of your favorite actors in advance, and determine whether the information included in the current scene matches the images and sounds registered in advance using a method such as pattern matching. And it is good also as an index of importance whether it is a scene where these favorite actors appear.

  In addition, in videos such as news and information programs, an announcer is generally arranged in the center of the main screen, and there is a tendency that useful information for the user is included in a scene where a utterance or a telop is included. Therefore, by acquiring the appearance frequency and utterance position of the telop as video features from the video information and using them in combination, the video importance of each scene can be calculated.

  In the description using FIG. 2, the coefficients a, b, and c are values set in advance. However, the coefficients may be arbitrarily determined by the user according to the viewing purpose.

  For example, when the user puts weight on scenes with scores, the scenes are loudly cheered, and character information indicating that scores have been entered is inserted. Increasing c increases the importance of the scored scene.

  In addition, when the user places importance on the decisive scenes that influence the game regardless of the score, the cheers are large in those scenes, and many replays are taken in. By increasing the coefficient b of the camera switching frequency, the user can obtain a degree of importance in which a decisive scene as a whole is important.

FIG. 3 is a diagram illustrating an example of an interface that receives input from the user.
In this case, the input from the user is obtained from the operation input unit 110, and as shown in FIG. 3A, a plurality of important elements prepared in advance and a coefficient for inputting the coefficients are provided in a combo box format. It is also good.

  Alternatively, the importance calculation unit 103 may include a table in which the values of the coefficients a, b, and c are set for each video genre, and calculate the importance by referring to the table.

  Of course, any number of video features used for calculating the importance may be used, and the video importance is not limited to being calculated only from the video features. From the past operation history such as normal playback, fast forward, rewind, pause, etc., set a high importance level for scenes that are paused or rewinded and watched again, and a low importance level for scenes skipped by fast forward Thus, it is possible to estimate a scene that the user likes.

  Further, the importance of the video may be calculated depending on how much the user is focused on the video using information obtained from a linked external device such as hardware for measuring the user's line of sight. Further, means other than the above example may be used, and the above example does not limit the calculation method of importance.

FIG. 4 is a diagram showing the degree of importance when a recorded video is used as an input video.
The non-important scene specifying unit 104 will be described with reference to FIG.
The non-important scene specifying unit 104 specifies a scene for reproducing the sub-picture using the importance and the threshold obtained from the importance calculating unit 103 for the video 401 currently being reproduced. A graph 403 is a graph showing the importance for each scene of the video obtained by the importance calculation unit 103 for an arbitrary scene section 402, with time on the horizontal axis and importance on the vertical axis.

  If the currently viewed video is already recorded, the entire video 401 is known, and the feature detection performed by the video feature detection unit 102 and the importance calculation processing performed by the importance calculation unit 103 are performed in advance. And can be recorded in the recording unit 105. Using this pre-calculated importance level and the threshold value 404 preset in the importance level calculation unit 103, a scene section that is less than the threshold value 404 is identified as an unimportant scene section 406.

FIG. 5 is a diagram showing a method for determining an unimportant scene section.
Here, when the importance level is obtained, if a section less than the threshold 404 is simply set as an unimportant scene section, an unimportant scene section as shown in FIG. 5A is obtained.

  Further, the non-important scene section information obtained here is transmitted to the video composition unit 109 and the digest creation unit 107, and is used to determine whether or not to synthesize and display sub-pictures and to create a digest that matches the section length. .

  For this reason, when a certain length of section length is required for an unimportant scene, a section with an extremely short section length as shown in FIG. 5A is reproduced as a non-important scene section. In addition, the following processing is added.

In order to exclude an extremely short section from an unimportant scene section, the present invention uses a preset scene section minimum continuous time m as shown in FIG. 5B, and has a continuous time of m or more. Will be identified as an unimportant scene.
In addition, by using various filters and smoothing the obtained importance, etc., it is possible to soften a sudden change in importance and remove a scene section that is cut more than necessary.

On the other hand, there is a case where it is necessary to exclude a section having an extremely short section length as described with reference to FIG.
FIG. 6 is a diagram showing the degree of importance when an image being broadcast is used as an input image.
In this case, as shown in FIG. 6, the entire video is not received at the current time 603, and naturally the importance that can be calculated is only for the video portion broadcast before the current time 603. is there. For this reason, as in the case of the recorded video described above with reference to FIG. 4, it is impossible to know in advance how long the non-important scenes are continuous.

  When the input video is a video being broadcast, as shown in FIG. 5 (c), the preset scene segment minimum continuous time n is used, and there are n or more important scene segments or non-important scene segments from the time when the threshold is crossed. The screen display is switched only when it continues.

It is also possible to exclude a section having a short section length of the scene specified by importance by performing time-shifted reproduction in which a video being broadcast is once recorded in a recording unit for a predetermined time and then reproduced.
FIG. 7 is a functional block diagram showing a schematic configuration of the video composition apparatus when performing time shift reproduction. Parts denoted by the same reference numerals as those in FIG. 1 represent the same items. The recording unit 701 stores, in addition to the video and video features recorded by the recording unit 105 in FIG.

As shown in FIG. 7, the recording unit 701 records the video being broadcast for a predetermined time t1, and outputs it to the video composition unit 109.
In addition, the recording unit 701 outputs to the video feature detection unit 102 when a broadcast video is input without waiting for a predetermined time t1. The subsequent processing, that is, the processing for detecting the feature of the input video by the video feature detection unit 102 and calculating the importance by the importance calculation unit 103 is the same as that described with reference to FIG.

  By the way, the fixed time t1 of the time shift is set to be equal to or longer than the scene section minimum continuous time m shown in FIG. Therefore, the non-important scene specifying unit 104 obtains the threshold value and the importance level of the input video from the importance calculating unit 103, and determines whether or not the important scene or the unimportant scene continues at least for a time m. It is possible to confirm the recording unit 701 before outputting the moving image to be the main video to the video composition unit 109.

FIG. 8 is a diagram showing a method for determining an important scene section using time shift.
FIG. 8 is used to specifically show an example in which the received broadcast video is temporarily stored in the recording unit 701 and is reproduced while being shifted by a time t1 that satisfies m ≦ t1. The broadcast video 801 is temporarily stored in the recording unit 701 and is output as the video 802 to the video synthesis unit 109 with a delay of time t1. At this time, it is assumed that the time t1 is equal to or longer than the scene section minimum continuous time m in the non-important scene specifying unit 104.

  On the other hand, the non-important scene specifying unit 104 performs processing by obtaining the output from the importance calculating unit 103. For example, a section where the continuous time of an unimportant scene whose importance is lower than a threshold is m or more is set as an unimportant scene. I will ask. Here, at the current time 803, the broadcast video 801 is accumulated in the storage unit 701 for t1, and is guaranteed to be output to the video synthesis unit 109 later than t1, so at this point in time the non-important scene The specifying unit 104 can check whether or not the important scene or the non-important scene continues at least m or more, and can output the non-important scene section information to the video composition unit 109. That is, even if the video is being broadcast, an important scene section 804 that is cut more than necessary, or an unimportant scene section 805 that is cut more than necessary, which is generated when an unimportant scene is extracted from the video, 806 is excluded, and an important scene section and an unimportant scene section as shown in 802 are finally determined.

  As shown in FIG. 8, the important scene section or the non-important scene section excluding the section having an extremely short section length can be obtained even for the video currently being viewed. .

  In the above description, as an example of the process performed by the unimportant scene specifying unit 104, an image is divided into important and unimportant scenes according to the importance obtained from the importance calculating unit 103.

  By the way, it is considered that the importance derived from the video feature obtained by the importance calculation unit 103 (hereinafter referred to as “video importance”) and the importance considered by the user (hereinafter referred to as “user importance”) do not match. It is done.

For example, let us consider a case in which importance is calculated based on camera work, in which a scene with a large (strong) movement is important.
As described above, when the video is a soccer broadcast program, the scene that the user pays attention to is a scene where the camera work is large (severe) such as a shoot scene, which matches the video importance level and the user importance level. To do.

  On the other hand, when the video is a cooking program, the scenes that the user pays attention to are scenes where the movement of camera work is mainly small, such as scenes that show the hands of cooking, scenes that introduce materials by flip, finished pictures, etc. Video importance and user importance do not match.

FIG. 9 is a diagram illustrating an example of a method for determining an unimportant scene section.
Here, as shown in FIG. 9A, if a scene that is equal to or greater than the threshold value is set as an important scene, an important scene such as a shoot scene that is noticed by a user can be acquired in a soccer broadcast program. Only scenes that are not so important to the user are determined to be important scenes.

  In order to solve this problem, it may be possible to calculate the importance of each video using two different calculation methods, but instead, one video importance, for example, a scene with a large motion in the above example is important. If the video importance level is calculated and a scene with small movements is important, only the importance / non-importance evaluation of the user importance level is reversed as shown in FIG. It is also conceivable that a section that is equal to or higher than the video importance level threshold is set as a non-important scene for the user. That is, a scene with extremely large camera work movement, that is, a section having a certain level of video importance is determined as a non-user important scene, and this non-user important scene is determined as a final non-important scene. And

As described above, with respect to a certain video importance level calculated by the importance level calculation unit 103, any of a certain threshold value and a certain threshold value or less may be regarded as important in the user importance level.
This depends on the combination of the video importance calculation method calculated by the importance calculation unit 103 and the scene property that the user considers important. Whether the non-important scene section is set to be greater than or less than the threshold value is determined at the time of importance calculation in the importance calculation unit 103, and is transmitted to the non-important scene specifying unit 104 together with the threshold value.

  Of course, the user may determine the range of the threshold, or may be determined by an input from the operation input unit 110.

  Although the description has been made assuming that there is one threshold value here, an unimportant scene may be specified using a plurality of threshold values. For example, when there are thresholds s and t satisfying the relationship of s <t, non-important scene sections are obtained by threshold processing using a plurality of thresholds such as threshold s or more and less than threshold t, or less than threshold s or threshold t. May be determined.

  These threshold values may be received from the operation input unit 110 from the user. As shown in FIG. 3 (b), the user's input may be in the form of inputting a threshold value, or as shown in FIG. 3 (c), a graph of importance calculated earlier is displayed. It is good also as a form which makes the threshold value on a graph select visually.

Next, the operation of the recorded video selection unit 106 will be described.
The video feature relating to the currently viewed video is transmitted from the video feature detection unit 102 to the recorded video selection unit 106, and an appropriate sub-video is selected from the storage unit 105 from the recorded video.

  Since the main video currently being viewed by the user is the one that the user has selected and viewed, the user is currently interested in video related to or correlated with the main video. it is conceivable that. Therefore, by selecting an image that is highly correlated or related to the main image as a sub image, it is possible to select an image that attracts the user's interest.

  Furthermore, as a condition for detecting this sub-picture, it is possible to detect a scene more appropriately estimating the current interest of the user by adding video features obtained from an important scene that the user has been watching until immediately before. I can do it.

In this case, the important scene section information is output from the non-important scene specifying unit 104 to the video feature detecting unit 102, and the feature information of the important scene section information is transmitted from the video feature detecting unit 102 to the recorded video selecting unit 106. Is done.
Further, the video feature detection unit 102 needs to store feature information together with time information.

  The correlation between the main video currently being viewed and the sub-video is obtained from the correlation between the video features previously obtained by the video feature detection unit 102 or information associated therewith. Here, it is assumed that the video feature of the sub-video is acquired by using the video feature detection unit 102 at the time of or after the sub-video is recorded and is recorded in the recording unit 105. Examples of these correlations include audio level, music used, video histogram, motion vector, genre information obtained from EPG, performer information, title information, sponsor information, recording date and time information, and the like.

  An example of acquiring a highly correlated video when the user is viewing the main video will be described below with reference to FIG.

FIG. 4 is a diagram showing the state of video recorded in the past as described above.
Focusing on a section 402 in the video 401 recorded in the past, the importance of the section is graphed 403, but the sub-picture to be reproduced in the non-important scene section 406 determined from the set threshold 404 I want to choose.

  The recorded video selection unit 106 calculates the correlation by using the video information included in the important scene section 405 immediately before the non-important scene section 406 in addition to the information accompanying the video such as EPG information. To select a sub-picture.

FIG. 10 is a diagram illustrating an example of information accompanying a video.
For example, it is assumed that the user has selected and viewed video B, which is a variety program, from the video group shown in FIG. At this time, when a related sub-video is selected using only information associated with these videos, the video D classified into the same genre or the video A in which the same performer appears is selected.

  However, in general, a video is composed of a plurality of different scenes, and the correlation calculated for each video unit selects a video that is appropriate for each scene, that is, a user's current interest is well estimated. I can't.

  On the other hand, in the present invention, it is possible to select a video that has well estimated the current user's interest by using the feature of the immediately preceding important scene.

FIG. 11 is a diagram illustrating a state after the input video is classified into an important scene and an unimportant scene.
Assume that non-important scenes 1104 and 1105 are included in the video B 1101 as shown in FIG. In addition, it is assumed that the important scenes immediately before each of the unimportant scenes 1104 and 1105 are the important scene 1102 where the comics are performed and the important scene 1103 where the performer performs a soccer confrontation. At this time, in the unimportant scene 1105, the immediately preceding video features are as follows.
I. Since the area that occupies the screen as a whole is often grass, the color distribution of the image is biased to green.
II. There is a field composed of white lines on the field.
III. There are multiple objects (players) on the field. These are mainly classified into two types (each team) by color.
IV. There is a circular object (ball) that is smaller than the above object, and this object is often at the center of the screen.
Since it has the above image features, it is considered that the user is currently interested in videos with similar features, and from the video group shown in FIG. Select image F.

  In this way, by detecting features from the important scene immediately before the unimportant scene and calculating the correlation using this, it is possible to select an appropriate sub-picture for each scene.

  Other methods for obtaining correlation include extracting important keywords from subtitle information included in the immediately preceding important scene and selecting matching videos, or scenes where people in the video information appear large From the above, a video including the same performer may be selected by pattern matching. Moreover, you may select the image | video etc. which use the same background music (BGM) from audio | voice information.

Next, an example of the operation of the digest creation unit 107 will be described below with reference to FIG.
In this block, the non-important scene section obtained by the non-important scene specifying unit 104 and the video feature recorded in the recording unit 105 are transmitted to the digest creation unit 107, and each video selected by the recorded video selection unit 106 is Create a digest that matches the scene.

  The digest creation unit 107 creates a digest having a length corresponding to this time from the non-important scene continuous time information received from the non-important scene specifying unit 104.

When viewing the recorded video as the main video, the entire video 401 is known, and the feature detection performed by the video feature detection unit 102 and the importance calculation processing performed by the importance calculation unit 103 are performed in advance, and the recording unit 105 can be recorded.
As described above, using the importance calculated in advance, the non-important scene specifying unit 104 can know in advance how long the non-important scene section 406 continues, and the digest editing unit 107 can detect this. Create a digest that matches the continuous time of non-important scene sections.

  In order to create a digest adapted to the time, for example, a technique such as that shown in Patent No. 3640615 that makes it possible to create a digest according to a set digest playback time may be used. The technique may be used.

  On the other hand, since the entire video cannot be known in advance when viewing the currently broadcast video, as described with reference to FIG. 4 in the description of the non-important scene specifying unit, the start point of the non-important scene It is impossible to know in advance when the end point of the non-important scene (digest playback end point) occurs from (digest playback start point). For this reason, it is not possible to create sub-pictures in real time in accordance with the time during which continuous playback is possible.

Therefore, a digest is created by the following method.
First, some highlight scenes cut in about several seconds are selected from sub-pictures in advance. Here, the highlight scene is a scene having a particularly high importance calculated by the importance calculation unit 103, and is mainly used by a user such as a sound excitement, a scene change, or a scene in which a person is reflected in the center. It refers to a scene where an event that interests you occurs. By playing these highlight scenes in the order of importance calculated by the importance calculation unit 103 or along the time axis of the original video until the end point of the non-important scene is reached, a digest is obtained. Is realized. When the end point of the non-important scene is reached, the digest video that was in the middle of playback may end in the middle, or the non-important scene end point may be extended until the digest video finishes playback. Good.

  In response to an instruction from the unimportant scene specifying unit 104, the video composition unit 109 switches the display of the moving image output unit 108. In the scene determined to be important by the non-important scene specifying unit 104, only the main video is displayed. In the scene determined to be unimportant, the sub-video created by the digest editing unit 107 is reproduced together with the main video.

FIG. 12 is a diagram showing a specific example when video is multiplexed.
As a display method, as shown in FIG. 12, the main screen (FIG. 12 (a)) may be reduced, and one or more digests may be displayed in an empty space (FIG. 12 (b)). The main screen and digest may be displayed simultaneously in a P (Picture in Picture) format (FIG. 12C).
In addition to the digest, basic data such as a title and text data such as related keyword information may be displayed (FIG. 12D).

  As for the output audio, when the main video and the sub video are being played back simultaneously, either the main video or the sub video may be muted and only the other audio may be played back. . In addition, while it is determined that the importance of the main video is low, the audio of the main video may be weak and the audio of the sub audio may be multiplexed and reproduced.

The following describes the user's special operations during video viewing.
During the digest reproduction of the sub-video, it is possible to perform a specific operation from an input device such as a remote controller according to the user's interest in the video.
If you are interested in the video played as a digest, you can send a signal such as “I'm interested” from the input device to register the bookmark in the storage unit 105 as a video candidate to be viewed later. good.
On the contrary, if the video reproduced as a digest is not interested, the video may be added to the deletion list in the storage unit 105 as it is.

  Moreover, it is good also as a format which interchanges and views the image | video currently watched and the image | video which flowed by digest.

  It should be noted that the video composition apparatus of the present invention is not limited to the above-described illustrated examples, and it is needless to say that various modifications can be made without departing from the gist of the present invention.

1 is a functional block diagram showing a schematic configuration of a video composition device according to an embodiment of the present invention. It is the figure which showed an example of importance calculation in the video synthesizing | combining apparatus concerning one Embodiment of this invention. It is the figure which showed the example of the interface which receives the input from a user in the video synthesizing | combining apparatus concerning one Embodiment of this invention. It is the figure which showed the mode of the importance at the time of making into the input image | video the recorded image | video in the video composition apparatus concerning one Embodiment of this invention. It is the figure which showed the method of the non-important scene area determination in the video synthesizing | combining apparatus concerning one Embodiment of this invention. It is the figure which showed the mode of the importance when the image | video in broadcast is made into the input image | video in the image | video synthetic | combination apparatus concerning one Embodiment of this invention. 1 is a functional block diagram showing a schematic configuration of a video composition device according to an embodiment of the present invention. It is the figure which showed the technique of the important scene area determination using a time shift in the video synthesizing | combining apparatus concerning one Embodiment of this invention. It is the figure which showed an example of the determination method of the non-important scene area in the video composition device concerning one embodiment of the present invention. It is the figure which showed the example of the information accompanying a video in the video synthesizing | combining apparatus concerning one Embodiment of this invention. It is the figure which showed the mode after classifying an input image | video into an important scene and an unimportant scene in the image | video synthetic | combination apparatus concerning one Embodiment of this invention. It is the figure which showed the specific example at the time of multiplexing an image | video in the video synthesizing | combining apparatus concerning one Embodiment of this invention.

Explanation of symbols

100, 700 Video composition device 101 Video input unit 102 Video feature detection unit 103 Importance calculation unit 104 Non-important scene specification unit 105, 701 Storage unit 106 Recorded video selection unit 107 Digest creation unit 108 Moving image output unit 109 Video synthesis unit 110 Operation input units 401, 601 All video sections 402 Arbitrary sections 404, 606 Threshold values 405, 407, 607 Important scene sections 406 Non-important scene sections 602 Broadcasted sections 603, 803 Current broadcast position 604 Non-broadcast sections 608 Digest playback start point 609 Digest playback end point 801 Broadcast video 802 Shift output video 804, 1102, 1103 Important scene 1101 Video B
805, 806, 1104, 1105 Non-important scene

Claims (11)

A video synthesizing device for synthesizing and displaying at least two videos that are broadcast or recorded,
A feature detector for detecting feature information corresponding to each scene in the video;
A recording unit for recording the video and / or the feature information;
A calculation unit that calculates a value of an index for importance of each scene of the first video based on the feature information;
A video selection unit for selecting a second video;
A scene specifying unit for specifying a scene in the first video for synthesizing the second video based on the value of the index;
An image composition device comprising: an image composition unit that composes the second image with the scene in the first image identified by the scene identification unit.   The video according to claim 1, wherein the video selection unit detects and selects a video similar to or related to the first video as the second video based on the feature information. Synthesizer.   3. The video composition apparatus according to claim 2, wherein the video selection unit selects the second video based on the feature information for each scene of the first video or each continuous scene group. .   The feature information for the video selection unit to select the second video is from a scene or scene group having a high index value closest to the scene in the first video specified by the scene specification unit. The video composition apparatus according to claim 3, wherein the obtained characteristic information is used.   5. The apparatus according to claim 1, wherein the first video is a video being broadcast, and the second video is a video recorded in the apparatus. 6. Video synthesizer.   5. The video composition apparatus according to claim 1, wherein both the first video and the second video are videos recorded in the device. 6.   The video synthesizing apparatus according to any one of claims 1 to 6, wherein the feature information includes video information, audio information, and text information that change every moment in the video.   The said scene specific | specification part specifies the scene in the said 1st image | video which synthesize | combines a said 2nd image | video by performing a threshold value process with respect to the value of the said index, The Claim 1 characterized by the above-mentioned. Item 8. The video synthesizing device according to any one of Items 7.   9. The video composition apparatus according to claim 8, wherein the calculation unit sets the threshold value and outputs the threshold value together with the index value to the scene specifying unit.   The video composition device according to claim 9, wherein the calculation unit inverts the information of the threshold selection range according to a type of the first video scene.   The video composition unit edits the selected second video as a digest video that matches the continuous time length of the scene in the first video specified by the scene specification unit, The video composition device according to any one of claims 1 to 10.

Images