JP2009296344A - Apparatus and method of processing video, program, and computer-readable recoding medium - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To extract an image in which motion of an object or background is emphasized or an image in which the object or background is clearly imaged from a video in a section where the object (a moving body) exists. <P>SOLUTION: When extracting a thumbnail using only motion of an object, a video is input, the object in a video frame is detected, frame correspondence is carried out, and a thumbnail image is extracted around the local maximum point and the local minimum point of the object motion. Meanwhile, when the thumbnail is extracted using motion of the object and background, a video is input, the object in a video frame is detected, frame correspondence is carried out, a motion vector between frames for each image is calculated, and a thumbnail image is extracted around the local maximum point and the local minimum point of the motion of both or any one of the object and the background. <P>COPYRIGHT: (C)2010,JPO&INPIT

Description

  The present invention relates to a video processing apparatus, method, program, and computer-readable recording medium, and more particularly to a video processing apparatus, method, program, and computer-readable recording medium for generating a representative image representing video motion. About.

  There are many techniques for extracting representative images representing video content from video. Here are some examples:

  (1) As a first conventional technique, a thumbnail representing a changed portion of a video scene is detected by detecting a place where the content change between adjacent frame images in a video is reduced from a large one (for example, Patent Documents). 1).

  (2) The second conventional technique is a method of graphing the total amount of motion vectors of a video in time series and employing the local minimum as a representative image. By doing this, we aim to extract impressive scenes and thumbnails that will be milestones for camera and actor movements. Further, a relatively clear image can be extracted as an accompanying effect (see, for example, Non-Patent Document 1).

(3) As a third conventional technique, a motion vector is used to separate an image into a moving object and a background, and a frame image (moving object up frame) in which the moving object is shown is detected, and those adjacent to each other are integrated By doing so, the video section (animal upshot) is calculated. In addition, the photographer pays attention to the moving object and detects a follow shot that is being followed (see, for example, Non-Patent Document 2).
Japanese Patent No. 319437 Wayne Wolf, "Key Frame Selection by Motion Analysis." IEEE ICASSP 96, pp. 1228, 1996 Torii et al., “Animal upshot / followshot detection using video motion”, MIRU2005 Proc. Pp. 24-31 Jul. 2005

  However, the above conventional technique has the following problems.

  The first prior art mainly uses color feature change, but because it is sensitive to brightness and the like, a very large number of thumbnails are detected for one video, making it difficult to see. In general, in thumbnail extraction based on color feature changes, it is difficult to separate a subject / background and extract a thumbnail that takes into consideration the subject / background of the video and its operation. For example, it is impossible to extract a thumbnail corresponding to the movement of a subject photographed with a fixed camera (hereinafter referred to as problem a).

  In the second conventional technique, since it is impossible to determine whether the movement is the movement of the subject or the movement of the camera from the total movement amount of the entire frame, there are cases where the effective thumbnail extraction cannot be performed. For example, in a snowboard video, a jumping spot is a highlight, but it cannot be detected by the local minimum of the total amount of movement (hereinafter referred to as problem b).

  The third conventional technique can detect a scene in which a moving object is large, but cannot select an effective thumbnail from the scene. Further, in detecting the follow shot, there is a problem that excessive detection occurs because the local motion amount detected in units of frames is used.

  The present invention has been made in view of the above points, and in the section where the subject (animal body) exists from the video, an image in which the motion of the subject or the background is emphasized and an image in which the subject or the background is clearly visible are respectively shown. It is an object of the present invention to provide a video processing apparatus and method, a program, and a computer-readable recording medium that can be extracted.

  A further object is to provide a video processing apparatus and method, a program, and a computer-readable recording capable of extracting a shot of a subject of interest relatively stably, and capable of extracting a shot of the subject of interest. To provide a medium.

  FIG. 1 is a diagram for explaining the principle of the present invention.

The present invention (Claim 1) is a video processing method for extracting a representative image representing video content from video,
A video input step (step 1) for the video input means to input the video and store it in the video frame storage means;
A subject detection unit acquires a video frame from the video frame storage unit, detects a subject in the video frame, performs frame association (step 2), and stores a subject in a time-series subject list storage unit (step 2). Step 3) and
The thumbnail image extraction means performs a thumbnail image extraction step (step 4) for extracting a thumbnail image around the maximum and minimum points of the movement of the subject stored in the time-series subject list storage means.

The present invention (Claim 2) is a video processing method for extracting a representative image representing video content from video,
A video input means for inputting video and storing the video in the video frame storage means;
A subject detection step in which the subject detection means acquires a video frame from the video frame storage means, detects a subject in the video frame, performs frame association, and stores it in the time-series subject list storage means;
An image unit correspondence unit calculates an inter-frame motion vector for each image unit and stores it in a time-series subject list storage unit;
A thumbnail image extracting step in which the thumbnail image extracting means extracts thumbnail images around the maximum and minimum points of the movement of the subject stored in either the subject and the background, or one of the time-series subject list storage means, and Do.

The present invention (Claim 3), in the thumbnail image extraction step,
In addition to the maximum / minimum movement of the subject, a thumbnail image is extracted based on a thumbnail extraction condition that combines the position, size, and background movement conditions of the subject.

Further, the present invention (Claim 4), in the thumbnail image extraction step,
One of the maximum and minimum values of the subject motion and the background motion is extracted, and the other motion amount is used as a thumbnail extraction condition for each, and a thumbnail image is extracted based on the extraction condition.

  FIG. 2 is a principle configuration diagram of the present invention.

The present invention (Claim 5) is a video processing apparatus for extracting a representative image representing video content from video,
Video input means 110 for inputting video and storing it in the video frame storage means 102;
A subject detection unit 140 that acquires a video frame from the video frame storage unit 102, detects a subject in the video frame, performs frame association, and stores the frame in the time-series subject list storage unit 104;
The thumbnail image extraction unit 150 extracts thumbnail images around the local maximum and minimum points of the movement of the subject stored in the time-series subject list storage unit 104 and stores the thumbnail image in the storage unit, or the time of the thumbnail image is the thumbnail time. Thumbnail image extraction means 150 stored in the storage means 105.

The present invention (Claim 6) is a video processing apparatus for extracting a representative image representing video content from video,
Video input means for inputting video and storing it in video frame storage means;
Subject detection means for acquiring a video frame from the video frame storage means, detecting a subject in the video frame, performing frame association, and storing it in a time-series subject list storage means;
An image unit correspondence unit that calculates a motion vector between frames for each image unit and stores the motion vector in a time-series subject list storage unit;
A thumbnail image extracting unit extracts a thumbnail image around the maximum and minimum points of the movement of the subject stored in the time series subject list storage unit of both the subject and the background, or one of them, and stores the thumbnail image in the storage unit; Alternatively, the thumbnail image extracting unit stores the time of the thumbnail image in the thumbnail time storage unit.

Further, the present invention (Claim 7) provides a thumbnail image extraction means,
In addition to the maximum / minimum movement of the subject, a means for extracting a thumbnail image based on a thumbnail extraction condition combining the position, size, and background movement condition of the subject is included.

The present invention (Claim 8) provides a thumbnail image extraction means,
Means for taking out the maximum and minimum values of one of the movement of the subject and the movement of the background, setting the other movement amount as a thumbnail extraction condition for each, and extracting a thumbnail image based on the extraction condition;

  The present invention (Claim 9) is a video processing program for causing a computer to function as each means constituting the video processing apparatus according to Claims 5 to 8.

  The present invention (Claim 10) is a computer-readable recording medium storing the video processing program according to Claim 9.

  As described above, according to the present invention, instead of presenting thumbnails of all shots, subject detection processing is performed. For example, attention is paid to a place where a subject is detected, and an image is extracted and presented from there. It is possible to avoid presentation of a meaningful frame image, and to present only a frame image effective for representing an operation, and to solve the above-mentioned problem a.

  In addition, the present invention detects not the entire frame movement, but the target object, and extracts the thumbnail that can express the movement to be noticed by using it separately from the movement of the subject area and the other movement (background movement). To solve it. Also, you can know the state of the operation by showing not only the minimum of movement but also the thumbnail of the maximum. Thereby, the above-mentioned problem b can be solved.

  Further, by considering not only the size of the moving object but also the movement of the moving object and the movement of the background, a thumbnail image that can express the movement of the subject and the flow of the scene is extracted. Moreover, the above-mentioned problem c can be solved by detecting a follow shot in consideration of temporal continuity.

  As described above, in the present invention, the subject and the background are separated, and the motion feature and the image feature are separately used, so that the motion of the subject or the background is emphasized in the section where the subject (animal body) exists from the video. It is possible to distinguish and extract the captured image and the image clearly showing the subject or the background. For example, by alternately displaying the maximum and minimum images of the movement, it is possible to create a slide show that represents the movement of the subject or the movement of the viewpoint of the camera and the process thereof. In addition, it is possible to relatively stably extract a shot in which the camera follows the subject and photograph it, and to extract a thumbnail of the subject of interest.

  Hereinafter, embodiments of the present invention will be described with reference to the drawings.

  FIG. 3 shows the configuration of the video processing apparatus in one embodiment of the present invention.

  The video processing apparatus shown in the figure includes a video input unit 110, a cut point detection unit 120, an image unit correspondence unit 130, a subject detection unit 140, a thumbnail image extraction unit 150, an output unit 160, a video storage unit 101, a frame buffer 102, A subject video section list storage unit 103, a time-series subject list storage unit 104, and a thumbnail time list storage unit 105 are configured.

  The video storage unit 101, the frame buffer 102, the subject video section list storage unit 103, the time-series subject list storage unit 104, and the thumbnail time list unit 105 are storage media such as a hard disk device and a memory.

  The frame buffer 102 stores a frame number (video time) and image data. The subject video section list storage unit 103 stores a start time and an end time. The time-series subject list storage unit 104 stores a subject ID, image unit coordinates (set), classification, and weight (evaluation value such as area). The thumbnail time list storage unit 105 stores the video frame number.

  The video input unit 110 inputs video data from the video storage unit 101 and sequentially outputs a designated frame number (video time) or a read frame image. At this time, video time information is acquired, and a frame image corresponding to the video time is acquired. In addition, it is desirable to store the input video frame in the frame buffer 102 and to efficiently acquire the video data at the specified time. Further, deinterlace (deinterlace) processing of the input video may be performed. In the following, it is assumed that the frame image and its video time are always input / output in pairs.

  The cut point detection unit 120 is a device that determines whether or not a cut point that is a discontinuous point exists between the frame time that is the current processing target and the frame time that is the previous processing target. For this method, for example, the method described in Japanese Patent No. 2839132 can be used. The reliability of motion detection is low at video discontinuous points such as cut points. Therefore, each video processing is performed in units of continuous video sections (called shots) between cut points, and processing between cut points is avoided. This can be omitted if the image to be processed does not include a cut point or if it is known that there is no processing problem at the cut point.

  The subject detection unit 140 processes the frame image input from the video input unit 110, determines whether or not a subject (animal body) exists in the image, and stores the information in the time-series subject list storage unit 104. To do. This is because object detection is performed using object detection (recognition) technology such as a face with respect to an image, an object detection method such as a background subtraction method, or a feature point-based moving object detection method described in Non-Patent Document 2 above. It is possible to realize this by using, for example. The detected subject (animal body) stores the coordinates (collection) where the subject exists in the time-series subject list storage unit 104 with a subject ID for each subject. The coordinates stored in the subject detection unit 140 are, for example, coordinates representing an image unit for detecting a subject and its movement such as an image block, a pixel set, a feature point, and an image template.

  When the motion vector for each image unit and the motion of the subject (association between frames) are internally calculated by the subject detection method described above and the background motion is not used, the image described below is used. The unit correspondence unit 130 can be omitted. Examples of the subject detection method in which the inter-frame correspondence (motion) of the subject is calculated internally include template matching, various object recognition techniques, and subject detection using motion as described in Non-Patent Document 2 above. An example of a subject detection method in which the motion of the subject cannot be detected is the background difference method. In the following, a case where a method capable of detecting the motion of the subject is described will be described. However, if the subject is associated with each other between frames, the same processing can be performed even when a method in which the motion of the subject cannot be detected is used.

  As a method of the above-described inter-frame association, for example, a temporary numerical value is recorded as the subject ID, and the image unit correspondence unit 130 becomes an object to be compared with the previous frame in the overlapping area with each subject area of the previous frame. When the ratio of the area to the area is equal to or greater than a certain value, the method uses the same ID as all processing frames, or calculates the motion vector between the previous frame and the current frame, Various methods are conceivable, such as estimating as an average vector of motion vectors calculated in the region, and associating each subject detected in the current frame with the ID of the region where the overlapping ratio is a certain value or more.

  The time-series subject list storage unit 104 stores a list stored for each frame image to be processed. The storage record includes a subject ID, image unit coordinates (set), classification, and weight (evaluation value such as area). “Subject ID” is a numerical value assigned to each subject or image unit and associated in time series. The time-series subject list storage unit 104 stores a position representing the subject in units of images or a set thereof. The content of “image unit coordinates (collection)” is different depending on the detection unit of subject detection. For example, in the case of subject detection / tracking using a rectangular image unit, the coordinate pair of the upper left and lower right of the rectangle, the point set of pixels that make up that one object when the object is extracted / tracked in pixel units, and the feature points In the case of subject detection / tracking in units, feature point coordinate sets are stored respectively. The “classification” stores the type of the detected subject. Basically, “subject” or “background” is stored as “classification”, but subject classification, for example, semantic classification of characters (telop), animals, people, faces, etc., depending on the subject detection means used. It may be given. The “weight” stores the area (number of pixels) of the detected subject and the weight (Non-Patent Document 2) representing the dominant region for each feature point. Moreover, the evaluation value of the certainty at the time of detection may be stored together as a weight.

  The image unit correspondence unit 130 is an apparatus necessary for generating thumbnails using a motion vector of an area (background) other than the subject. When the background motion vector is not estimated, the motion vector for each image is estimated and stored in the time-series subject list storage unit 104. In this case, a classification other than the subject, such as a numerical value representing “background”, is set in the “classification” column of the time-series subject list storage unit 104.

  The background motion vector calculation method can be calculated, for example, by inputting a frame image at a plurality of times and matching the two frame images for each image unit. As a motion vector calculation method, various techniques such as block matching of existing techniques and a feature point tracking technique described in Non-Patent Document 2 can be used. A unit area (image unit) for motion vector calculation indicates, for example, an image block in block matching, and a feature point in feature point tracking.

  The thumbnail image extraction unit 150 operates when the subject detection unit 140 determines that the subject has disappeared, or when the subject has been detected and reaches the end of the shot, and is stored in the time-series subject list storage unit 104. Using the extracted information, a thumbnail is extracted from the subject video section, and the video frame number or the time of the extracted thumbnail is stored in the thumbnail time list storage unit 105.

Details of the operation in the above configuration will be described below. Below,
(1) Processing for extracting thumbnails using only the movement of a subject, in which the subject detection unit 140 can associate the frames;
(2) Processing for extracting thumbnails using the movement of the subject and background;
This will be explained separately.

  FIG. 4 shows the process (1), and FIG. 5 shows the process (2).

  First, the process (1) will be described. This process shows a case where the image unit correspondence unit 130 is not included in the apparatus configuration.

  FIG. 4 is a flowchart of the operation (using image-based subject detection) in one embodiment of the present invention.

Step 101) Video input update procedure:
The video input unit 110 reads the next processing frame of the video to the frame buffer 102 and the like, and can be used from each device (cut point detection unit 120, image unit correspondence unit 130, subject detection unit 140, thumbnail image extraction unit 150). It is a procedure to do. When processing is started for the first time, the number of frames necessary for processing in the cut point detection unit 120 and the subject detection procedure is read out to the frame buffer 102, for example. For example, in the case of subject detection using feature point tracking, the number is two.

Step 102) Cut point detection procedure:
The cut point detection unit 120 determines whether there is a cut point between the read processing target frames. If there is a cut point, the cut point detection unit 120 returns to the video input update procedure (step 101) to skip the process. If there is no cut point, the process proceeds to the subject detection procedure (step 103).

Step 103) Subject detection procedure:
The subject detection unit 140 determines the presence / absence of detection of the subject, for example, by movement using the feature point as an image unit, stores it in the time-series subject list storage unit 104, and returns to step 102.

Step 104) Thumbnail detection procedure:
The thumbnail image extraction unit 150 selects a frame image to be extracted as a thumbnail based on the information in the time-series subject list storage unit 104 stored in step 103, and displays the video time and the like of the selected image as the thumbnail time list storage unit 105. To remember. A specific procedure will be described later with reference to FIG.

  Step 105) It is determined whether the video is the end. As the determination method, there are various methods such as determining whether to read out the next frame. If the video is at the end, the process proceeds to step 106, and if not, the process proceeds to step 101.

  Step 106) In the output unit 160, the thumbnail time list storage unit 105 or the like performs a termination procedure such as releasing necessary file output or memory.

  Next, the process (2) will be described. Hereinafter, processing when the image unit correspondence unit 130 is included in the apparatus configuration will be described.

  FIG. 5 is a flowchart of the operation (using image-based subject detection) in one embodiment of the present invention. In the figure, the same steps as those in FIG. 4 are denoted by the same step numbers, and the description thereof is omitted.

  The operation shown in FIG. 5 is obtained by adding an image unit correspondence procedure (step 201) to the operation of FIG. Specifically, in the image unit correspondence unit 130, the correspondence between frames for each image unit is taken and stored in the time-series subject list storage unit 104. A method of taking correspondence between frames for each image unit can be realized by using block matching and a feature point tracking method as described above. This is necessary when the subject detection procedure (step 103) does not include subject-to-subject association or motion vector (image unit association) processing.

<Thumbnail extraction method: Using subject movement>
Next, a thumbnail extraction method using subject movement in step 104 will be described.

  FIG. 6 is a flowchart of a thumbnail image extraction procedure according to an embodiment of the present invention.

Step 1041) Subject / Background Motion Amount Calculation Procedure The thumbnail image extraction unit 150 uses the information in the time-series subject list storage unit 104 to calculate the subject / background motion amount. For example, in the method of Non-Patent Document 2 described above, the image unit coordinates (set) are stored as a feature point set. As a vector difference between frames of corresponding feature points corresponding to motion vectors, for example, a median of motion vector absolute values of all subjects is calculated as a subject motion amount. When the background motion amount is used as a thumbnail extraction condition, the median value of the motion vector absolute value is calculated as the background motion amount with respect to the background feature points as in the subject. This calculation is performed between all frames in the shot. Thereafter, pre-processing such as smoothing processing is performed on the motion amount graph as necessary. Thus, a graph of the amount of movement of the subject and the background is created. FIG. 7 shows an example of a graph created for an image of snowboarding. Note that when another method is used for the subject detection unit 140, for example, in the case of an object recognition method in which a rectangle is an image unit, the same is obtained by using the vector difference between the rectangular barycentric coordinates of the subject region as the subject motion amount. Can be processed. Further, although the subject / background motion amount is calculated as the median value, other calculation methods may be used.

Step 1042) Thumbnail extraction condition detection procedure:
A thumbnail is detected in consideration of the maximum and minimum values of the graph of each motion amount obtained in the subject / background motion amount calculation procedure (step 1041) and the position and size of the subject. The detection example is shown below. First, the video time that takes the maximum and minimum values for each of the motion amounts calculated in the subject / background motion amount calculation procedure is selected. Further, the video time that best satisfies the conditions for the position and size of the subject within a certain range before and after each video time is selected. The evaluation criteria for the position and size of the subject are, for example, the product of the reciprocal of the distance between the vertical line at the center of the screen and the subject center-of-gravity coordinates (a constant if the distance is 0) and the ratio of the subject to the screen. Use things as evaluation criteria (larger is better). Then, by using the weighted sum of the evaluation position of the subject position and size as the evaluation value as the evaluation value, the time satisfying the position and size of the subject can be calculated.

  It should be noted that it is also possible to calculate the amount of movement without distinguishing between the subject and the background, and detect thumbnails having maximum and minimum values. In this case, it is possible to extract a thumbnail representing the state of change of the entire screen content. The evaluation value may be calculated by any other arithmetic calculation.

  A plurality of images before and after selected near the maximum value of the movement may be extracted by an arbitrary method (for example, a certain number of images are extracted at equal intervals). For sports images such as snowboards, the movement in the image, that is, the change is so severe that it is necessary to extract a thumbnail that allows detailed observation of the change in the movement.

  When selecting a video time that takes a maximum or minimum, the selection may be narrowed down by putting conditions into the heights of the peaks and valleys. For example, if the difference between the maximum and minimum values before and after is less than a certain value, the extreme value is not selected. It is possible to select only the thumbnail when the subject, the background, or the entire screen has moved greatly. By changing the conditions of the height of the peaks and valleys depending on the processing target video, it is possible to extract thumbnails expressing the video content.

  If the subject motion and the background motion suddenly show an impulse response, and there is not much difference between the subject and the background motion, it may not be detected because it is often not a significant image. The reason for this will be described later in the section “Thumbnail extraction using background motion”.

  Further, when a classification such as a person or a car is stored in the classification of the subject in the time-series subject list storage unit 104, the thumbnail selection may be performed using only the amount of movement of a specific subject. In this case, it is possible to extract a thumbnail that focuses on changes in only the subject of interest.

Step 1043) Detection time storage procedure:
The thumbnail time detected in the thumbnail extraction condition detection procedure (step 1042) is stored in the thumbnail time list storage unit 104. Alternatively, an image at the corresponding time may be acquired by the video input unit 110 and stored in a file, a memory, or the like.

<Thumbnail image extraction procedure: Thumbnail extraction using background motion>
As an example, FIG. 8 shows a graph of the subject / background created in the subject / background motion calculation procedure (step 1041) for the drama video.

  By setting the thumbnail extraction condition to the maximum / minimum of background motion, it is possible to extract thumbnails according to camera movement. It is possible to extract thumbnails that represent scene transitions when the camera is moved to switch the viewpoint or overlook the landscape.

  The method of distinguishing and detecting viewpoint switching and landscape shooting is “viewpoint switching” when the maximum value of both background motion and subject motion is suddenly both above the threshold and the ratio of both is below a certain value. This can be realized by regarding the background movement that occurs in addition to the "landscape photography". That is, in the case of “viewpoint switching”, it is assumed that the camera is moved more quickly than “landscape shooting” because there is no interest in the scene in the middle of moving the camera. Note that the evaluation of the suddenness is performed by, for example, considering that the case where the width of the 50% point (FIG. 8B) of the maximum value that is greater than or equal to the central value of the maximum value is equal to or less than a certain value is “suddenly”. be able to.

  Note that a section where the subject and the background motion are almost equal as shown in FIG. 8C is a section in which the subject is photographed up (in Non-Patent Document 2, abrupt changes that occur in camera work using this are shown. To detect up-shots). In the extraction of the background thumbnail, for example, it is possible to prevent excessive detection by removing this section in advance. The detection method detects the difference between the detected background / subject movement maximum / minimum difference values continuously as a certain number or less as the threshold value, or instead of the maximum / minimum values, the difference within the interval is detected. It can be detected by using an average of absolute values.

<Follow shot detection>
In the present invention, in addition to the background motion described above, the subject motion is also taken into consideration, so that it is possible to detect a follow shot that is tracked and photographed while paying attention to the subject. FIG. 9 shows an example of the graph.

  When the subject of interest walks and the camera follows it, the apparent movement of the subject image becomes relatively small while the movement of the camera occurs. When the thumbnail image extraction unit 150 detects this, a follow shot can be detected. As a specific example, the camera work section is set between the minimum values of the camera motion (FIG. 9A), the subject motion average value in the section is calculated, and the subject motion minimum value exists in the vicinity of the background maximum value. When the local minimum value is smaller than the subject motion average value, it is detected as a follow shot.

  The subject of interest is extracted by extracting thumbnails before and after the detected follow shot in the same manner as the procedures of “Thumbnail Extraction Method: Using Subject Movement” and “Thumbnail Image Extraction Procedure: Thumbnail Extraction Using Background Motion” described above. It is possible to extract the thumbnails that are included.

<Output unit: slideshow presentation method>
The output unit 160 extracts thumbnails from the frame buffer 102 based on the video frame numbers in the thumbnail time list storage unit 105, and presents the thumbnails to the user by a presentation method such as a slide show in which images are sequentially displayed, for example. It is effective to do.

  When presenting a slide show, for example, a thumbnail corresponding to the maximum value of movement is displayed shorter than a thumbnail corresponding to the minimum value of movement. Also, by adding an effect such as dissolve when the images are exchanged, a slide show with more enhanced motion is obtained.

  A thumbnail extraction method suitable as a thumbnail detection method for displaying as a slide show differs depending on the target video. When the movement of the subject is important as in sports, thumbnail extraction using the subject movement is suitable. In addition, when the change of the scene is important as in a drama, thumbnails extracted by both the method using the subject motion and the method using the background motion are used. If there is a tendency to keep track of the subject that the photographer has paid attention to, such as a video taken by an individual, it is also effective to use the thumbnail of the follow shot described above.

  Note that the operation of each component of the video processing apparatus shown in FIG. 3 can be constructed as a program and installed in a computer used as the video processing apparatus, or can be distributed via a network.

  Further, the constructed program can be stored in a portable storage medium such as a hard disk, a flexible disk, or a CD-ROM, and can be installed or distributed in a computer.

  The present invention is not limited to the above-described embodiment, and various modifications and applications can be made within the scope of the claims.

  The present invention can be applied to a technique for extracting a representative image representing video content from a video.

It is a figure for demonstrating the principle of this invention. It is a principle block diagram of this invention. It is a block diagram of the video processing apparatus in one embodiment of this invention. It is a flowchart (the 1) of the operation | movement in one embodiment of this invention. It is a flowchart (the 2) of the operation | movement in one embodiment of this invention. It is a flowchart of the thumbnail image extraction procedure in one embodiment of the present invention. It is an example of extraction thumbnails (use of subject movement) in an embodiment of the present invention. It is a to-be-photographed object / background motion amount graph and extracted thumbnail example (using background motion) in an embodiment of the present invention. 5 is a subject / background motion amount graph and extracted thumbnail example (using background / subject motion) in an embodiment of the present invention.

Explanation of symbols

101 Video storage unit 102 Video frame storage unit, frame buffer 103 Subject video section list storage unit 104 Time series subject list storage unit, time series subject list storage unit 105 Thumbnail time storage unit, thumbnail time list storage unit 110 Video input unit, video Input unit 120 Cut point detection unit 130 Image unit correspondence unit 140 Subject detection unit, subject detection unit 150 Thumbnail image extraction unit, thumbnail image extraction unit 160 Output unit

Claims (10)

A video processing method for extracting a representative image representing video content from a video,
A video input means for inputting video and storing the video in the video frame storage means;
A subject detection step in which subject detection means acquires a video frame from the video frame storage means, detects a subject in the video frame, performs frame association, and stores it in the time-series subject list storage means;
A thumbnail image extracting unit for extracting a thumbnail image around the maximum and minimum points of the movement of the subject stored in the time-series subject list storage unit;
A video processing method characterized by: A video processing method for extracting a representative image representing video content from a video,
A video input means for inputting video and storing the video in the video frame storage means;
A subject detection step in which subject detection means acquires a video frame from the video frame storage means, detects a subject in the video frame, performs frame association, and stores it in the time-series subject list storage means;
An image unit correspondence unit calculates a motion vector between frames for each image unit, and stores it in the time series subject list storage unit;
Thumbnail image extraction means for extracting a thumbnail image around the maximum and minimum points of the movement of the subject stored in the time-series subject list storage means of either the subject or the background, or any one of the subject and background. When,
A video processing method characterized by: In the thumbnail image extraction step,
The video processing method according to claim 1, wherein the thumbnail image is extracted based on a thumbnail extraction condition that combines a position, a size, and a background movement condition of the subject in addition to the maximum and minimum of the movement of the subject. In the thumbnail image extraction step,
4. The maximum or minimum value of one of the movement of the subject and the movement of the background is taken out, the movement amount of the other is used as a thumbnail extraction condition, and the thumbnail image is extracted based on the extraction condition. The video processing method according to claim 1. A video processing apparatus for extracting a representative image representing video content from video,
Video input means for inputting video and storing it in video frame storage means;
Subject detection means for acquiring a video frame from the video frame storage means, detecting a subject in the video frame, performing frame association, and storing it in a time-series subject list storage means;
A thumbnail image extraction unit extracts a thumbnail image around the maximum and minimum points of the movement of the subject stored in the time-series subject list storage unit, and stores the thumbnail image in the storage unit, or the time of the thumbnail image is set to the thumbnail time Thumbnail image extraction means stored in the storage means;
A video processing apparatus comprising: A video processing apparatus for extracting a representative image representing video content from video,
Video input means for inputting video and storing it in video frame storage means;
Subject detection means for acquiring a video frame from the video frame storage means, detecting a subject in the video frame, performing frame association, and storing it in a time-series subject list storage means;
An image unit correspondence unit that calculates a motion vector between frames for each image unit and stores the motion vector in the time-series subject list storage unit;
Thumbnail image extraction means extracts thumbnail images around the maximum and minimum points of the movement of the subject stored in the time series subject list storage means of either the subject and the background, or one of them, and stores it in the storage means Thumbnail image extraction means for storing or storing the time of the thumbnail image in the thumbnail time storage means;
A video processing apparatus comprising: The thumbnail image extraction means includes
7. The video according to claim 5, further comprising means for extracting the thumbnail image based on a thumbnail extraction condition in which the subject's position, size, and background motion conditions are combined in addition to the maximum / minimum movement of the subject. Processing equipment. The thumbnail image extraction means includes
A means for extracting one of the maximum and minimum values of the movement of the subject and the movement of the background, using the other movement amount as a thumbnail extraction condition for each, and extracting the thumbnail image based on the extraction condition. The video processing device according to any one of 5 to 7.   A video processing program for causing a computer to function as each means constituting the video processing device according to claim 5.   A computer-readable recording medium storing the video processing program according to claim 9.

Images