JP2010252013A - Video retrieving apparatus and video retrieving program - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To reliably and efficiently retrieve a scene of a main video part corresponding to a predetermined scene in an edited video. <P>SOLUTION: A main video part preprocessing section 21 calculates image characteristic vectors of a pitching shot and a second shot of a main video part for a game. An edited video vector calculating section 22 calculates image characteristic vectors of a pitching shot and second to fourth shots of a sports news edited video. A candidate shot set selecting section 24 obtains a similarity of the image characteristic vectors of the pitching shots in both videos to select a candidate shot set (d(n), k(n)). A final identifying section 25 obtains a similarity of the image characteristic vector of the second shot of the main video part in the game and the image characteristic vectors of the second to fourth shots of the sports news edited video and identifies a shot set (d<SB>MAX</SB>, k<SB>MAX</SB>). In this way, the pitching shot and batted ball shot of the shot set (d<SB>MAX</SB>, k<SB>MAX</SB>) correspond to a predetermined scene in the sports news edited video. <P>COPYRIGHT: (C)2011,JPO&INPIT

Description

  The present invention relates to a video search device, and more particularly to a technique for searching a main video scene corresponding to a predetermined scene in an edited video such as sports news.

  For example, for sports news that introduces a baseball scene, multiple “highlight scenes” that are key to the game are selected from the video of the baseball game (main game video), and these are edited and broadcast shortly. There are many things to do. That is, the sports news is generally composed of “highlight scenes” that are the key to the game.

  For this reason, viewers who watch sports news may not be satisfied only with a sports news scene, and may desire to watch the main game video corresponding to the scene. In order to realize the viewer's demand, a video search device is required that identifies a sports news scene that the viewer wants to watch carefully from the main game video.

  As such a video search device, in order to search for a desired video, the video is divided into scenes, and the nodes near the nodes (reference points) of the first frame constituting the scene are tracked for each frame, so that the nodes are searched. A technique for classifying and generating feature amounts is known (see Patent Document 1). Search processing is performed using this feature amount.

  Also, a technique is known in which a frame constituting a video is divided into blocks of a predetermined area, a representative image is selected from the block images, and a search reference image is drawn from the representative image (Patent Document 2). See). Search processing is performed using this search reference image. A technique is also known in which color features and texture features are extracted for each image of a divided block, and the similarity between images is calculated by weighting these features (see Patent Document 3).

  Also, for example, as a device for discriminating events such as a scoring scene of baseball and a turning point of inning, a video is divided into scenes, feature quantities of a plurality of frames included in the scene are calculated, and the feature quantities are set in advance. A technique for identifying the type of event in a scene based on numerical data for each type of event is known (see Patent Document 4). In addition, in order to detect the starting point of the pitching scene, the turf and soil regions are extracted from the frame of the video and the feature amount is calculated, and between this feature amount and the preset pitch scene feature amount, A technique is known in which the similarity is calculated and the frame of the starting point of the pitching scene is specified based on the similarity (see Patent Document 5).

Japanese Patent Laid-Open No. 2004-280669 JP 2002-140331 A JP 2003-216649 A JP-A-2005-332206 JP 2009-20621 A

  Such a conventional video search device is configured to generate a scene between sports news and a game main video based on a feature calculated from a predetermined scene in sports news and a feature calculated from each scene in the main game video. Are matched, and the scene of the main game video corresponding to the sports news scene is specified. Thereby, the viewer can watch the desired scene in the sports news carefully with the main game video.

  However, each scene of sports news is not produced using the game main video as it is, but is edited. For this reason, it is difficult for a video search apparatus that simply compares feature quantities between a sports news scene and a game main video scene to specify a game main video scene corresponding to the sports news scene. This is because sports news has a video cut of about 1 or 2 between two consecutive shots selected from the main game video, in addition to the case where the continuous shots selected from the main game video are used as they are. This is because there is a case where it is configured by being inserted, and processing for the inserted video cut is required in addition to comparison between shots. Details of the sports news composition pattern will be described later.

  Accordingly, the present invention has been made to solve the above-mentioned problems, and the purpose of the present invention is to provide a video search capable of reliably and efficiently searching a scene of a main video corresponding to a predetermined scene in an edited video. An apparatus and a video search program are provided.

  In order to achieve the above object, the video search apparatus according to the present invention corresponds to a predetermined shot in the edited video based on a feature quantity of the main video and a feature quantity of the edited video including consecutive shots of the main video. In the video search apparatus for searching for a shot of the main video, a shot switching point is detected based on a similarity between frames in the main video, and a first main shot and a second main shot immediately after the shot switching point are detected. A main set video pre-processing unit that extracts a feature set of the first main shot and a feature quantity of the second main shot, and a first edit shot that is a predetermined shot in the edited video. Edited video calculation unit for calculating feature values and feature values of a plurality of continuous edit shots following the first edit shot Based on the feature amount of the first main shot and the feature amount of the first edit shot, a first similarity between the first main shot and the first edit shot is calculated, and the second A second similarity between the second main shot and the plurality of edit shots is calculated based on the main shot feature and the features of a plurality of consecutive edit shots following the first edit shot, respectively. And a search processing unit that identifies a shot of the main video corresponding to a predetermined shot in the edited video based on the first similarity and the second similarity.

  In the video search device according to the present invention, the search processing unit calculates the similarity between both shots based on the feature amount of the first main shot and the feature amount of the first edit shot, and the similarity is predetermined. A candidate shot set selection unit that selects a shot set including a first main shot higher than the threshold value as a candidate shot set, a feature amount of a second main shot included in the candidate shot set, and the first edited shot. The similarity between the second main shot and the plurality of editing shots is calculated based on the feature values of the subsequent consecutive editing shots, and the calculated similarities of all the candidate shot sets are calculated. Among them, the candidate shot set having the highest similarity is identified, and the first main shot included in the identified candidate shot set is selected as the edited video. And a final identifying section that identifies a definitive shot of the main video corresponding to the predetermined shot, and further comprising a.

  Further, in the video search device according to the present invention, when the search processing unit sets a region in which a frame is divided as a block, the first search unit is based on the feature amount for each block over a predetermined number of consecutive frames. The similarity and the second similarity are calculated.

  In the video search device according to the present invention, the edited video is configured based on two consecutive shots of the main video, and a predetermined number of shots are configured between the shots configured based on the two shots. Other shots are inserted, and the edited video calculation unit calculates the feature quantities of the respective shots configured based on two consecutive shots of the main video and the feature quantities of the predetermined number of other shots. It is characterized by calculating.

  The video search device according to the present invention is characterized in that the main video is a baseball game video and the first main shot and the first edit shot are pitch shots.

  In addition, the video search program according to the present invention is based on the feature quantity of the main video and the feature quantity of the edited video including consecutive shots of the main video, and the main video shot corresponding to the predetermined shot in the edited video. And a step of detecting a shot switching point based on a similarity between frames in the main video, and a first main shot and a second main shot immediately after the shot switching point. A step of extracting a shot set comprising: a step of calculating a feature amount of the first main shot and a feature amount of the second main shot; and a feature amount of a first edit shot that is a predetermined shot in the edited video, And feature amounts of a plurality of continuous edit shots following the first edit shot, respectively. Calculating a first similarity between the first main shot and the first edit shot based on the step, the feature amount of the first main shot, and the feature amount of the first edit shot; The second similarity between the second main shot and the plurality of edit shots based on the feature amount of the second main shot and the feature amounts of a plurality of consecutive edit shots following the first edit shot Performing a step of calculating each of the degrees, and a step of identifying a shot of the main video corresponding to a predetermined shot in the edited video based on the first similarity and the second similarity. Features.

  As described above, according to the present invention, a shot of the main video corresponding to a predetermined shot in the edited video is between the first main shot extracted from the main video and the first edited shot (predetermined shot). The degree of similarity is specified from the degree of similarity between the second main part shot extracted from the main part video and the second edit shot and the like. In this case, even if the edited video is configured based on two consecutive shots of the main video and includes a predetermined number of other shots inserted between these two shots, that is, the edited video Is composed of the first shot of two consecutive shots of the main video, the other shot (s) inserted, and the later shot of the two consecutive shots of the main video. Even if it is a case (for example, FIG. 1 (2) pattern B mentioned later), editing of the edit video corresponding to the second main shot of the main video and the later shot of two consecutive main video shots The similarity between shots is also calculated. The scene of the main video having the highest calculated similarity is the search target scene. This makes it possible to reliably and efficiently search for a scene of the main video corresponding to a predetermined scene in the edited video. Therefore, the viewer can watch the desired scene in the edited video carefully with the main game video.

It is a figure explaining the composition pattern A and B of the sports news edit image | video using the pitching shot and hitting shot of a game main part image | video. It is a block diagram which shows the structure of the video search device by embodiment of this invention. It is a conceptual diagram of the image feature vector in a predetermined video section. It is a figure explaining the image feature vector of the pitch shot and the 2nd-4th shot in a sports news edit image | video. It is a flowchart explaining the process of a candidate shot set selection part. It is a figure explaining the similarity between the pitch shot of a game main part video, and the pitch shot of a sports news edit video. It is a flowchart explaining the process of the last specific part. It is a figure explaining the similarity between the 2nd shot of a game main part picture, and the 2nd-4th shot of a sports news edit picture.

  Hereinafter, embodiments for carrying out the present invention will be described in detail with reference to the drawings. In the following embodiments, a baseball game main video and a sports news editing video obtained by editing the game main video will be described as examples. The video search device searches for a scene in a baseball game main video corresponding to a scene including a predetermined throw shot and a hit shot in a baseball video in sports news. It should be noted that the present invention is not limited to the baseball video as a search target video, but can be applied to other videos. In addition, the present invention does not limit the scene to be searched to a scene including a pitching shot and a hitting shot of a baseball video. For example, a scene of a pitching shot and a catching shot, a scene of a hitting shot and a catching shot This also applies to scoring scenes.

  FIG. 1 is a diagram for explaining configuration patterns A and B of a sports news edited video using a pitch shot and a hit ball shot of the main game video. In the baseball game main video, for the scene where the pitcher throws the ball and the batter hits the ball, the pitch shot is a series of actions that the pitcher throws, and the batter hits the ball from the pitcher The sequence with the video of the shot shot that is the motion is used. A sports news edited video is produced by editing a game main video having a plurality of such shot shots and shot shots (shot set).

  FIG. 1A is an example (pattern A) in which a sports news edited video is configured by using the pitch shot and the hit shot of the game main video as they are. The pitch shot of the sports news edited video is configured by cutting out a frame of a predetermined video section retroactively from the break between the pitch shot shot frames in the main game video. The hit shot of the sports news edited video is configured by cutting out a frame of a predetermined video section counted from a break between the shot shot and the shot shot frame in the main game video. The sports news edited video of this pattern A is a series of pitch shots and hit shots that are continuously arranged as they are with a break.

  FIG. 1B is an example (pattern B) in which a sports news edited video is formed by using other shots in addition to the pitch shot and the hit ball shot of the main game video. The pitch shot of the sports news edited video is configured by cutting out a frame of a predetermined video section from the pitch shot frames in the main game video. The hit shot of the sports news edited video is configured by cutting out a frame of a predetermined video section from the shot shot frame in the main game video. In addition, another video having a shot number of about 1 or 2 is inserted between the pitch shot and the hit shot in the sports news edited video. The sports news edited video of this pattern B is a shot shot, a shot of another video, and a shot shot arranged in order.

  Here, a case is assumed where a scene of a game main video corresponding to a predetermined scene in a sports news edited video is searched. When the sports news edited video is composed of the pattern A in FIG. 1 (1), the pitching shot and the hitting shot of the sports news editing video are arranged as the next shot following the pitching shot is the hitting shot. This is the same as the pitch shot and hit shot sequence in the video. Therefore, the similarity of the video is compared between the pitch shot of the sports news edited video and the pitch shot of the main game video, and the next shot (hit ball shot) of the pitch shot in the sports news edited video and the main video of the game The scene of the main game video can be specified by comparing the similarity of the video with the shot after the pitching shot (hit ball shot).

  However, when the sports news edited video is configured by the pattern B of FIG. 1 (2), when the number of inserted shots is 1, the next shot following the pitch shot is not a shot shot but another shot. Yes, the next shot is a shot. When the number of shots inserted is 2, the next shot following the pitch shot and the next shot are not shots but other shots, and the next shot is a shot shot. As described above, when the sports news edited video is composed of the pattern B, in the same process as the pattern A, the pitch shot and the shot of the sports news edited video are associated with the pitch shot and the shot of the main game video. It is not possible to specify the scene of the game main video.

  Therefore, in the present invention, even in the case of not only the pattern A but also the pattern B, in order to identify the scene of the game main video corresponding to a predetermined scene in the sports news edited video, a pitch shot ( The first and second shots as well as the third and fourth shots are used to compare the similarity of the video between the shot of the sports news edited video and the main game shot. To do. In the following embodiment, the pattern B has a configuration in which two other shots are inserted, and the number of shots to be searched in the sports news edited video is 4. However, the present invention is limited to this number. It is not something. The number of shots to be searched in the sports news edited video is determined according to the number of other shots inserted between the pitch shot and the hit shot in the sports news edited video. For example, if the number of other shots to be inserted is 1, the number of shots to be searched in the sports news edited video may be set to 3, and if the number of other shots to be inserted is 3, sports news editing The number of search target shots in the video may be five. Furthermore, when the number of other shots to be inserted is 0, that is, in the case of pattern A, the number of shots to be searched in the sports news edited video may be set to 2.

[Video Search Device]
FIG. 2 is a block diagram showing a configuration of the video search apparatus according to the embodiment of the present invention. The video search apparatus 1 includes a storage unit 11 in which a game main video d is stored, a storage unit 12 in which sports news edited video is stored, a main video pre-processing unit 21, an edited video vector calculation unit 22, and a search processing unit 23. I have. The search processing unit 23 includes a candidate shot set selection unit 24 and a final identification unit 25. Here, d = 1 to D, d is a number (ID) indicating a main game video file, and D is the number of stored main game video files. When a viewer who views a sports news edited video designates a scene including a pitch shot and a shot to be desired to watch carefully in the main game video, the video search device 1 displays the game main video corresponding to the scene. Search for a scene. Note that the video search device 1 may input the game main video d and the sports news edited video from outside.

(Main video pre-processing section)
First, the main video pre-processing unit 21 will be described. The main video pre-processing unit 21 reads the game main video d (d = 1 to D) from the storage unit 11 and detects a shot switching point (shot break) for each video of the game main video d. The video is divided into shots at the switching point. Specifically, the main video pre-processing unit 21 divides a frame constituting the game main video d into a predetermined number of blocks, calculates an image feature vector for each of the divided blocks, and images of each block in the preceding and following frames. Calculate the similarity of feature vectors. Then, a boundary between frames in which the calculated similarity is lower than a predetermined threshold is detected as a switching point for dividing the video into shots. In addition, a group of frames in which the calculated similarity is equal to or greater than a predetermined threshold is set as one shot. This switching point detection method is known, and for details, see, for example, Japanese Patent Laid-Open No. 2006-270301.

Then, the main video pre-processing unit 21 determines whether or not the shot immediately before the switching point is a pitch shot for two shots straddling the switching point of the shot-divided video. Specifically, the main video pre-processing unit 21 generates a pitch shot reference in advance using the video of the pitch shot, the image feature vector of the shot immediately before the switching point, and the pitch shot reference generated in advance. The similarity is calculated by comparing with the image feature vector. When the similarity is equal to or greater than a predetermined threshold, it is determined that the two shots straddling the switching point are the pitch shot and the second shot, and the combination (shot set) of the shots is assigned the number k. Here, k = 1,..., K ^(d) , and K ^(d) is a shot set of the pitch shot and the second shot detected in the game main video d (hereinafter referred to as 1-2 shot set). Number). As described above, the main video pre-processing unit 21 extracts the 1-2 shot set k that is a combination of the pitch shot and the second shot across the switching point from the game main video d. Here, the second shot is not necessarily a hit shot.

  Note that the main video pre-processing unit 21 may determine whether or not the shot immediately after the switching point is a hit ball shot based on the similarity of images. Specifically, the main video pre-processing unit 21 generates a hit ball shot reference in advance using a hit ball shot video, and an image feature vector of a shot immediately after the switching point and a hit ball shot reference generated in advance. The similarity is calculated by comparing with the image feature vector. Then, when the similarity of the pitch shot immediately before the switching point is equal to or greater than a predetermined threshold and the similarity of the shot shot immediately after the switching point is equal to or greater than the predetermined threshold, the switching point is crossed. It is determined that the two shots are a pitch shot and a second shot (hit ball shot), and a number k is assigned to the combination of the shots to extract a 1-2 shot set k. Thereby, since the 1-2 shot set k can be narrowed down, more efficient search processing can be realized.

The main video pre-processing unit 21 decomposes the pitch shot and the second shot into frames for the 1-2 shot set k of the game main video d. At this time, the first frame number of the pitch shot is t1 ₀ ^{(d, k)} , the length (number of frames ^{) of} the pitch shot is T1 ^{(d, k)} , and the first frame number of the second shot is t2 ₀ ^{(d, k).} (= T1 ₀ ^{(d, k)} + T1 ^{(d, k)} ), and the length (number of frames) of the second shot is T2 ^{(d, k)} .

The main video pre-processing unit 21 receives each frame t (t = t1 ₀ ^{(d, k)} ,..., T1 ₀ ^{(d, k)} + T1 ^{(d, k)} − in the 1-2 shot set k. 1) is divided into M × N blocks, and the image feature vector v1 ^(d) (t, i, j) of each block (i, j) is calculated. Here, i = 1,..., M, j = 1,..., N, and M and N are preset values. That is, the image feature vector v1 ^(d) (t, i, j) is data indicating the image feature of the block (i, j) in the frame t of the main game video d. For example, the image feature vector v1 ^(d) (t, i, j) is an average value of RGB pixel values in the block (i, j) or an average value of luminance in the block (i, j). Is used.

The main video pre-processing unit 21 receives each frame image t (t = t2 ₀ ^{(d, k)} ,..., T2 ₀ ^{(d, k)} + T2 ^{(d, k )} in the 1-2 shot set k. ⁾ -1) is also divided into M × N blocks, and the image feature vector v2 ^(d) (t, i, j) of each block (i, j) is calculated.

Thus, the image feature vector v1 ^(d) (t, i, j) for each block (i, j) in the pitch shot of the 1-2 shot set k calculated by the main video pre-processing unit 21 and the first The image feature vector v2 ^(d) (t, i, j) for each block (i, j) in two shots is output to the search processing unit 23.

(Edited video vector calculator)
Next, the edited video vector calculation unit 22 will be described. The edited video vector calculation unit 22 reads the sports news edited video from the storage unit 12, and of the sports news edited video, the w-th shot (w = 1 indicates the shot shot and the shot shot of the scene designated by the viewer). pitching shot, w = 2, 3, 4, the second shot, respectively, the third shows a shot and the fourth shot, a fourth shot showing a hitting shots.), frame tw _q predated T frame from the last frame Is the first frame, and the image feature vector V (tw _q ) in the video section of the number T of frames from the frame tw _q to the last frame is calculated. Note that the video section of the number T of frames may be anywhere as long as it is within each section of the w-th shot.

FIG. 3 is a conceptual diagram of image feature vectors in a predetermined video section. With reference to FIG. 3, the calculation process of the edit video vector calculation part 22 is demonstrated concretely. The edited video vector calculation unit 22 uses the frame t ₀ (= w _q ) in the video of the w-th shot (w = 1 to 4) including the pitch shot and the shot of the scene designated by the viewer for the sports news edited video. ) As the first frame, T consecutive frames from the shot to t ₀ + T−1 (final frame) are divided into M × N blocks. Then, the edited video vector calculation unit 22 calculates the image feature vector v (t, i, j) of each block (i, j) in each of the T frames t, and in each block (i, j), the image The feature vectors v (t, i, j) are added by T, and the value is divided by T to calculate V ′ (i, j).

Then, the edited video vector calculation unit 22 arranges V ′ (i, j) for all the blocks V (t ₀ ) = (V ′ (1,1), V ′ (2,1),. , V ′ (i, j),..., V ′ (M, N)) are calculated as image feature vectors of this video section (frame t ₀ to frame t ₀ + T−1). That is, the edited video vector calculation unit 22 performs the image feature vector V in the video section of the predetermined number of frames T for the wth shot (w = 1 to 4) including the shot shot and the shot shot of the scene specified by the viewer. (Tw _q ) is calculated.

FIG. 4 is a diagram for explaining the pitch shot and the image feature vectors of the second to fourth shots in the sports news edited video. As shown in FIG. 4, the edited video vector calculation unit 22 calculates the image feature vector V (t1 _q ) in the video section of the number T of frames from the frame t1 _q to the final frame for the pitch shot (w = 1). . Similarly, edited video vector calculation unit 22, the second to fourth shots (w = 2 to 4), the frame t2 _q t4 image features in the video section of the frame number T from _q to the last frame vector V (t2 _q) ~V the (t4 _q) is calculated.

From the frame tw _q to the final frame of the w-th shot including the pitching shot and the hitting shot of the scene specified by the viewer from the sports news edited video calculated by the edited video vector calculation unit 22 in this way. The image feature vector V (tw _q ) in the video section with the number of frames T is output to the search processing unit 23.

(Candidate Shot Set Selection Department)
Next, the candidate shot set selection unit 24 of the search processing unit 23 will be described. The candidate shot set selection unit 24 inputs the image feature vector v1 ^(d) (t, i, j) of the pitch shot in the 1-2 shot set of the main game video from the main video pre-processing unit 21 and the edited video. An image feature vector V (t1 _q ) in a pitched shot (w = 1) of a scene designated by the viewer in the sports news edited video is input from the vector calculation unit 22. Then, the candidate shot set selection unit 24 calculates an image feature vector V (t ₀ ) in a video section of a predetermined number of frames T for the pitch shot of the main game video, and the image feature in the pitch shot of the sports news edited video. A similarity S1 ^(d) (t ₀ ) with the vector V (t1 _q ) is calculated, and a candidate shot set (d (n), k (n)) is selected. This will be specifically described below.

FIG. 5 is a flowchart for explaining the processing of the candidate shot set selection unit 24. FIG. 6 is a diagram for explaining the similarity between the pitch shot of the main game video and the pitch shot of the sports news edited video. The candidate shot set selection unit 24 sets the game main video number d = 1 (step S501), sets the 1-2 shot set number k = 1 in the game main video d (step S502), and sets the game main video. The number t ₀ = t _{1 0} ^{(d, k)} of the first frame at ^d is set (step S503). Here, the frame t1 ₀ ^{(d, k)} indicates the first frame of the pitch shot in the 1-2 shot set k of the main game video d.

Candidate Shots selecting unit 24, the game for pitching shots main video d, of the block (i, j) the image feature vector v1 of ^{(d) (t, i,} j) in the frame t, frame t ₀ ~t ₀ Image feature vector of this video section using v1 ^(d) (t, i, j) (t = t ₀ ,..., T ₀ + T−1) of block (i, j) in the video section of + T V (t ₀ ) is calculated (step S504). The calculation method is the same as the calculation by the edited video vector calculation unit 22.

As shown in FIG. 6, the candidate shot set selection unit 24 calculates the image feature vector V (t1 _q ) in a predetermined video section for the pitch shot (w = 1) of the sports news edited video, and the calculation in step S504. The similarity S1 ^(d) (t ₀ ) is calculated between the image feature vector V (t ₀ ) in a predetermined video section for the pitched shot of the main game video (step S505).

The candidate shot set selection unit 24 compares the similarity S1 ^(d) (t ₀ ) calculated in step S505 with a preset threshold value S1 _th (step S506). When it is determined that the similarity S1 ^(d) (t ₀ ) is larger than the threshold value S1 _th (step S506: Y), the “main shot video d 1-2 shot set k” currently being processed is selected as the “candidate shot set”. ”(D (n), k (n)) (step S507), and the process proceeds to step S508. On the other hand, when it is determined that the similarity S1 ^(d) (t ₀ ) is not greater than the threshold value S1 _th (step S506: N), the process proceeds to step S508. Thereby, the 1-2 shot set information (d which is the ID of the game main video and k which is the number of the 1-2 shot set) including the pitch shot of the main game video similar to the pitch shot of the sports news edited video is obtained. The candidate shot set (d (n), k (n)) is set. n is a number indicating the set order.

The candidate shot set selection unit 24 increments the first frame number t ₀ in the main game video d (step S508), and the first frame number t ₀ and t1 ₀ ^{(d, k)} + T1 ^{(d, k)} −T Are compared (step S509). t1 ₀ ^{(d, k)} indicates the number of the first frame of the pitch shot, T1 ^{(d, k)} indicates the length (number of frames ⁾ of the pitch shot, and T indicates the number of frames in the processing unit. That is, it is determined whether or not the top frame number t ₀ is larger than the frame number of T frames before the last frame. In other words, with respect to the pitch shot in the 1-2 shot set k of the main game video d, the processing from step S504 to step S508 for each T frame number in the video section is the first frame t1 ₀ ^{(d, k).} It is determined whether or not everything has been completed over the last frame.

If the candidate shot set selection unit 24 determines in step S509 that the first frame number t ₀ is larger than t1 ₀ ^{(d, k)} + T1 ^{(d, k)} −T (step S509: Y), that is, If it is determined that all the processes for each T frame in the video section of the pitch shot have been completed, the process proceeds to step S510. On the other hand, when it is determined that the first frame number t ₀ is not larger than t _{1 0} ^{(d, k)} + T 1 ^{(d, k)} −T (step S 509: N), that is, the T frame in the video section of the pitch shot when the processing for each is determined not to be completed, for the processing of T frame from the frame number t ₀ of the top set at step S508, the process proceeds to step S504. As a result, the process with the next frame number t ₀ as the head is performed.

The candidate shot set selection unit 24 increments the 1-2 shot set number k in the main game video d (step S510), and compares the 1-2 shot set number k with K ^(d) (step S511). ). K ^(d) indicates the number of 1-2 shot sets extracted from the game main video d. That is, it is determined whether or not the processing from step S503 to step S510 has been completed for all the shots of the 1-2 shot set in the main game video d.

If the candidate shot set selection unit 24 determines in step S511 that the number k of the 1-2 shot set is larger than K ^(d) (step S511: Y), that is, all 1s in the game main video d. When it is determined that the process has been completed for the -2 shot set, the process proceeds to step S512. On the other hand, when it is determined that the number 1-2 of the 1-2 shot set is not larger than K ^(d) (step S511: N), that is, the processing is completed for all 1-2 shot sets in the main game video d. If it is determined that it is not, the process proceeds to step S503 in order to process the 1-2 shot set k set in step S510. As a result, processing is performed for the next 1-2 shot set k.

  The candidate shot set selection unit 24 increments d, which is the ID of the game main video (step S512), and compares the game main video d with D (step S513). D indicates the number of game main video files. That is, it is determined whether or not the processing from step S502 to step S512 has been completed for all the main game videos d.

  If the candidate shot set selection unit 24 determines in step S513 that d, which is the ID of the main game video, is greater than D (step S513: Y), that is, the processing has been completed for all main game videos d. If determined, the process is terminated. On the other hand, if it is determined that d, which is the ID of the game main video, is not greater than D (step S513: N), that is, if it is determined that processing has not been completed for all the game main video d, the process proceeds to step S512. The process proceeds to step S502 in order to process the main game video d set in step S502. Thereby, a process is performed about the next game main video d.

In this manner, the candidate shot set selection unit 24 determines whether the main game video d, every 1-2 shot set extracted from the main game video d, and the first frame to the last frame in the video section of the pitch shot. Processing is performed for each T frame specified in, and the similarity between the pitch shot of the sports news edited video and the pitch shot of the main game video is calculated, and the candidate shot set (d (n), k (n)) A set {(d (1), k (1)),..., (D (N _ss ), k (N _ss ))} is selected and output to the final specifying unit 25. Thereby, the candidate shot set selection unit 24 selects a 1-2 shot set including a pitch shot similar to the pitch shot of the sports news edited video from the 1-2 shot set k extracted from the main game video as a candidate shot. It can be selected as a set (d (n), k (n)). Here, (d (n), k (n)) is the k (n) -th 1-2 shot set of the video d (n), and N _ss is the number of candidate shot sets.

(Final specific part)
Next, the final specifying unit 25 of the search processing unit 23 will be described. The final specifying unit 25 inputs the image feature vector v2 ^(d) (t, i, j) of the second shot in the 1-2 shot set of the main game video from the main video preprocessing unit 21 and the edited video vector. The image feature vectors V (t2 _q ) to V (t4 _q ) in the second to fourth shots (w = 2 to 4) of the scene designated by the viewer in the sports news edited video are input from the calculation unit 22, respectively. . The candidate shot set selection unit 24 inputs candidate shot sets (d (n), k (n)) determined to have high similarity in pitch shots between the main game video and the sports news edited video. . Then, the final specifying unit 25 calculates the image feature vector V (t ₀ ) in the video section of the number T of frames for the second shot of the main game video, and the images in the second to fourth shots of the sports news edited video. Similarities S2 ^(d) (t ₀ ) to S4 ^(d) (t ₀ ) between the feature vectors V (t2 _q ) to V (t4 _q ) are calculated, and candidate shot sets (d (n), k The shot set having the highest similarity is identified from (n)), and the main game video ID, the start point (start frame number) and end point (end frame number) of the video section are output. This will be specifically described below.

FIG. 7 is a flowchart for explaining the processing of the final specifying unit 25. FIG. 8 is a diagram for explaining the similarity between the second shot of the game main video d (n) and the second to fourth shots of the sports news edited video. The final identification unit 25 sets the maximum similarity S _MAX = 0.0 (step S701), sets the candidate shot set number n = 1 (step S702), and sets the candidate shot set (d (n), The first frame number t ₀ = t 2 ₀ ^{(d (n),} k (n)) in the second shot of ^{k (n))} is set (step S703).

Final identification unit 25, for the second shot of the game main video d (n), the image feature vector v2 of the block (i, j) in the frame ^{t (d (n)) (} t, i, j) of the frame t ₀ ~t ₀ + T blocks in the image section (i, j) of ^{v2 (d (n)) (} t, i, j) by using the _{(t = t 0, ···,} t 0 + T-1) Then, the image feature vector V (t ₀ ) of this video section is calculated (step S704). The calculation method is the same as the calculation by the edited video vector calculation unit 22.

As shown in FIG. 8, the final specifying unit 25 includes image feature vectors V (t2 _q ) to V (t4) in a predetermined video section for the second to fourth shots (w = 2 to 4) of the sports news edited video. _q ) and the similarity S2 ^{(d (n))} between the image feature vector V (t ₀ ) in the predetermined video section for the second shot of the main game video d (n) calculated in step S704. t ₀₎ to S4 of ^{(d (n)) (t} 0) is calculated respectively (step S705).

The final specifying unit 25 selects the maximum value S ^{(d (n))} (from the similarity S2 ^{(d (n))} (t ₀ ) to S4 ^{(d (n))} (t ₀ ) calculated in step S705. t ₀ ) is set (step S706).

The final identifying unit 25 compares the maximum value S _MAX of the similarity with the similarity S ^{(d (n))} (t ₀ ) set in step S706 (step S707). When it is determined that the maximum value S _MAX of the similarity is smaller than the similarity S ^{(d (n))} (t ₀ ) (step S707: Y), S _MAX = S ^{(d (n))} (t ₀ ), d _MAX = d (n) and t _MAX = t ₀ are set. That is, the similarity S ^{(d (n))} (t ₀ ) set in step S706 is set to the maximum value S _MAX , the candidate shot set d (n) is set to the optimal shot set d _MAX , and the head frame t ₀ is set to the optimal head. Each frame t _MAX is set (step S708). Then, the process proceeds to step S709. On the other hand, when it is determined that the maximum value S _MAX of the similarity is not smaller than the similarity S ^{(d (n))} (t ₀ ) (step S707: N), the process proceeds to step S709. As a result, the most similar degree between the second shot of the candidate shot set (d (n), k (n)) in the game main video d (n) and the second to fourth shots in the sports news edited video. It is possible to set d _MAX which is the ID of the main video of the game with high and the number t _MAX of the first frame.

The final identifying unit 25 increments the first frame number t ₀ in the main game video d (n) (step S709), and the first frame number t ₀ and t 2 ₀ ^{(d (n), k (n))} + T2 ^{(d (n), k (n)) −} T is compared (step S710). t2 ₀ ^{(d (n), k (n))} indicates the number of the first frame of the second shot, and T2 ^{(d (n), k (n))} indicates the length (number of frames ⁾ of the second shot. , T indicates the number of frames in a processing unit. That is, it is determined whether or not the top frame number t ₀ is larger than the frame number of T frames before the last frame. In other words, for the second shot in the 1-2 shot set k (n) of the main video d (n) of the game, the processing from step S704 to step S709 for each T frame in the video section is performed as the first frame. It is determined whether or not all of t2 ₀ ^{(d (n), k (n)) is} completed over the last frame.

In step S710, the final specifying unit 25 determines that the leading frame number t ₀ is greater than t 2 ₀ ^{(d (n), k (n))} + T 2 ^{(d (n), k (n)) −} T. In the case (step S710: Y), that is, when it is determined that the processing for every T frame in the video section of the second shot is completed, the process proceeds to step S711. On the other hand, when it is determined that the leading frame number t ₀ is not larger than t 2 ₀ ^{(d (n), k (n))} + T 2 ^{(d (n), k (n)) −} T (step S 710: N) That is, when it is determined that the processing for every T frame in the video section of the second shot has not been completed, the processing for T frames is performed from the first frame number t ₀ set in step S709. The process proceeds to S704. As a result, the process with the next frame number t ₀ as the head is performed.

The final identifying unit 25 increments the candidate shot set number n (step S711), and compares the candidate shot set number n with N _ss (step S712). N _ss indicates the number of candidate shot sets. That is, it is determined whether or not the processing from step S703 to step S711 has been completed for the second shots of all candidate shot sets (d (n), k (n)).

If the final specifying unit 25 determines in step S712 that the candidate shot set number n is larger than N _ss (step S712: Y), that is, all candidate shot sets (d (n), k (n)). ), The process proceeds to step S713. On the other hand, if it is determined that the candidate shot set number n is not larger than N _ss (step S712: N), that is, the processing has been completed for all candidate shot sets (d (n), k (n)). If it is determined that there is not, the process proceeds to step S703 in order to process the candidate shot set (d (n), k (n)) set in step S711. As a result, processing is performed for the next candidate shot set (d (n), k (n)).

The final specifying unit 25 specifies a shot set (d _MAX , k _MAX ) including the video section t _{MAX to} t _MAX + T of the main game video d _MAX (step S713). Thereby, a shot set of any one of the shot shots of the sports news edited video and the second to fourth shots, and a candidate shot set (d (n), k (n)) of the main game video Among them, the shot set (d _MAX , k _MAX ) having the highest similarity is specified.

The final identification unit 25 uses the game main video ID = d _MAX and the video section start point = t1 ₀ ^(dMAX) as a scene of the game main video d corresponding to a scene including a predetermined throw shot and a shot in the sports news edited video. ^{, kMAX)} (match main video d shots (d _MAX of _MAX, k _MAX) start frame number of the pitching shots in) and the end point = t1 ₀ image segment ^{(dMAX, kMAX + 1) -1} ( shots (d _MAX, k _MAX) for the next shot set (d _MAX, k _MAX +1) outputs the immediately preceding frame number) pitching shots in (step S714).

As described above, the final specifying unit 25 performs processing for each candidate shot set and for each T frame from the first frame to the last frame in the video section of the second to fourth shots in the sports news edited video. Sports news edited video from the set {(d (1), k (1)),..., (D (Nss), k (Nss))} of (d (n), k (n)) A shot set (d _MAX , k _MAX ) that includes the second shot having the highest similarity to any one of the second to fourth shots is specified. Thereby, the last specification part 25 can specify the scene of the game main part video corresponding to the scene containing the predetermined pitch shot and hitting shot in a sports news edit video.

  As described above, according to the video search device 1 according to the embodiment of the present invention, the main video pre-processing unit 21 detects the shot switching point of the game main video, and the shot immediately before the shot switching point is a pitch shot. Whether or not it exists is determined based on the similarity of the image with a preset pitch shot reference, and a 1-2 shot set k that is a combination of the pitch shot and the second shot across the switching point is extracted. I made it. As a result, since the search is performed on the 1-2 shot set k narrowed down to the pitch shot, the search process is more efficient than the case where the search is performed on the shots before and after all the shot switching points. Can be realized.

In addition, according to the video search device 1 according to the embodiment of the present invention, the edited video vector calculation unit 22 is the w-th shot including a pitch shot and a shot shot of a scene specified by the viewer in the sports news edited video. The image feature vector V (tw _q ) is calculated for (w = 1 to 4). That is, in addition to the pitch shot that is the first shot (w = 1) of the sports news edited video, the second to fourth shots (w = 2 to 4) including the hit ball shot were also set as search targets. Thereby, even if it is the structure of the pattern B shown in FIG. 1 (2) by which other 1-2 shots are inserted between a pitch shot and a hit ball shot, the predetermined pitch shot and hit ball in a sports news edit image | video It is possible to reliably identify the scene of the game main video corresponding to the scene including the shot.

In addition, according to the video search device 1 according to the embodiment of the present invention, the candidate shot set selection unit 24 uses the similarity S1 ^(d) between the pitch shot of the main game video and the pitch shot of the sports news edited video. t ₀ ) is calculated, and a shot set having a high similarity S1 ^(d) (t ₀ ) is selected as a candidate shot set (d (n), k (n)). Thereby, since the final specifying unit 25 searches for candidate shot sets (d (n), k (n)) with similar pitch shots, the search is performed for all shot sets k. Compared to the above, an efficient search process can be realized.

In addition, according to the video search device 1 according to the embodiment of the present invention, the final specifying unit 25 includes the second shot of the main game video and the sports news edited video in the candidate shot set (d (n), k (n)). Similarities S2 ^(d) (t ₀ ) to S4 ^(d) (t ₀ ) between the second to fourth shots are calculated, and the similarities S2 ^(d) (t ₀ ) to S4 ^(d) are calculated. The shot set having the highest (t ₀ ) is specified, and the game main video ID, the start point (start frame number) and end point (end frame number) of the video section are output. Thereby, it is possible to reliably and efficiently search for a scene of the game main video corresponding to a predetermined scene in the sports news edited video. Therefore, the viewer can watch the desired scene in the edited video carefully with the main game video.

  Note that the video search apparatus 1 is configured by a computer including a volatile storage medium such as a CPU and a RAM, a non-volatile storage medium such as a ROM, an interface, and the like. The functions of the main video preprocessing unit 21, the edited video vector calculation unit 22 and the search processing unit 23 provided in the video search device 1 are realized by causing the CPU to execute programs describing these functions. These programs can also be stored and distributed in a storage medium such as a magnetic disk (floppy (registered trademark) disk, hard disk, etc.), optical disk (CD-ROM, DVD, etc.), semiconductor memory, or the like.

DESCRIPTION OF SYMBOLS 1 Image | video search device 11,12 Storage part 21 Main part image | video pre-processing part 22 Edit image | video vector calculation part 23 Search process part 24 Candidate shot set selection part 25 Final specification part

Claims (6)

In a video search apparatus for searching for a shot of the main video corresponding to a predetermined shot in the edited video based on a feature of the main video and a feature of an edited video including consecutive shots of the main video,
A shot switching point is detected based on the similarity between frames in the main video, a shot set including a first main shot immediately before the shot switching point and a second main shot immediately after the shot switching point is extracted, and the first main shot is extracted. Main feature video pre-processing unit for calculating the feature amount of the second feature shot and the feature amount of the second feature shot,
An edited video calculation unit that calculates a feature amount of a first edit shot that is a predetermined shot in the edited video and a feature amount of a plurality of continuous edit shots following the first edit shot;
Based on the feature amount of the first main shot and the feature amount of the first edit shot, a first similarity between the first main shot and the first edit shot is calculated, and the second main portion shot is calculated. Second similarity between the second main shot and the plurality of edit shots is calculated based on the shot feature amount and the feature amounts of a plurality of continuous edit shots following the first edit shot, respectively. And a search processing unit for identifying a shot of the main video corresponding to a predetermined shot in the edited video based on the first similarity and the second similarity. apparatus. The video search device according to claim 1,
The search processing unit
A similarity between both shots is calculated based on the feature amount of the first main shot and the feature amount of the first edit shot, and a shot set including the first main shot with the similarity higher than a predetermined threshold is selected as a candidate. A candidate shot set selection unit to select as a shot set;
Based on the feature amount of the second main shot included in the candidate shot set and the feature amount of a plurality of continuous edit shots following the first edit shot, the second main shot and the plurality of edit shots And calculating the first similarity shot included in the identified candidate shot set, specifying a candidate shot set having the highest similarity among the calculated similarities in all the candidate shot sets. A final specifying unit that specifies a shot of the main video corresponding to a predetermined shot in the edited video. The video search device according to claim 1 or 2,
The search processing unit determines the first similarity and the second similarity based on a feature amount for each block over a predetermined number of consecutive frames when a region obtained by dividing a frame is a block. A video search device characterized by calculating. In the video search device according to any one of claims 1 to 3,
The edited video is configured based on two consecutive shots of the main video, and a predetermined number of other shots are inserted between the shots configured based on the two shots,
The edited video calculation unit calculates a feature amount of each shot configured based on two consecutive shots of the main video, and a feature amount of the predetermined number of other shots. Search device. In the video search device according to any one of claims 1 to 4,
A video search apparatus, wherein the main video is a baseball game video, and the first main shot and the first edit shot are pitch shots. A program for searching for a shot of the main video corresponding to a predetermined shot in the edited video based on a feature of the main video and a feature of an edited video including consecutive shots of the main video, ,
Detecting a shot switching point based on the similarity between frames in the main video;
Extracting a shot set consisting of a first main shot immediately before the shot switching point and a second main shot immediately after the shot switching point;
Calculating the feature amount of the first main shot and the feature amount of the second main shot;
Calculating a feature amount of a first edit shot that is a predetermined shot in the edited video and a feature amount of a plurality of continuous edit shots following the first edit shot;
Calculating a first similarity between the first main shot and the first edit shot based on the feature amount of the first main shot and the feature amount of the first edit shot;
A second similarity between the second main shot and the plurality of edit shots based on the feature amount of the second main shot and the feature quantities of a plurality of consecutive edit shots following the first edit shot Calculating each of
And a step of specifying a shot of the main video corresponding to a predetermined shot in the edited video based on the first similarity and the second similarity.

Images