JP5522790B2 - Template image generation apparatus and template image generation program - Google Patents

Description

  The present invention relates to a template image generation apparatus and a template image generation program for generating a template image used when template matching is performed from a plurality of program videos.

  In recent years, a large amount of content such as program images has been accumulated and an environment in which these can be easily accessed has been established. At the same time, there is a need for a technique for efficiently searching for a desired video from a large amount of content. Currently, in connection with such content search technology, research is being conducted to analyze content such as program video and automatically add metadata to the content. Here, when automatically assigning metadata to a program video, there is one that detects and uses a rough break of the program, and a specific video or image in the detection of this break may be used. .

  For example, a number of news items are conveyed in one news program, and these news items can be roughly divided by a studio shot of an announcer that conveys the news. Therefore, if it is possible to extract typical production scenes that are used repeatedly, such as the announcer's studio shots in news programs as described above, from a large amount of content, it is effective for automatically adding metadata to the program video. Can be used.

  Here, Non-Patent Documents 1 and 2 propose a technique for dividing content such as a program video by performing template matching using a template image. Further, Patent Documents 1 and 2 propose a technique for generating a template image from images input continuously.

JP-A-11-284997 JP 2006-276948 A

HongJiang Zhang, Shuang Yeo Tan, Stephen W. Smoliar, Gong Yihong: “Automatic parsing and indexing of news video”, Multimedia Systems, Vol.2, pp.256-266, (1995) Deborah Swanberg, Chiao-Fe Shu, Ramesh Jain: “Knowledge Guided Parsing in Video Databases”, SPIE, Vol.1908, pp.13-24, (1993)

  However, in the techniques proposed in Non-Patent Documents 1 and 2, a specific method for generating a template image has not been described in detail. Further, in the technologies proposed in Non-Patent Documents 1 and 2, most of the template images are generated manually, which is inefficient for dividing a large amount of content.

  In addition, the techniques proposed in Patent Documents 1 and 2 simply generate a template image from a video, and cannot generate a template image from a typical effect scene that is repeatedly used in the video. It was.

  Also, for example, in an announcer's studio shot in a news program, the background studio set portion basically does not change, but the appearing announcer may change from day to day. However, in the technique proposed in each of the above-mentioned documents, the template image is generated without considering the fixed portion that does not change in the program video and the variable portion that changes, so the variable portion in the template image There is a problem in that the accuracy of template matching is reduced when the ratio is increased. In template matching, a threshold value is used to determine whether the target image is similar to the template image. However, this threshold value varies when the ratio of the variable portion changes. Therefore, there has been a problem that the techniques proposed in the above-mentioned documents are not practical.

  The present invention has been made in view of the above points, and can automatically generate a template image from a typical effect scene repeatedly used in a plurality of program videos, and a fixed portion in the program video. It is an object of the present invention to provide a template image generation apparatus and a template image generation program capable of generating a template image considering a variable portion.

  In order to solve the above-mentioned problem, a template image generation device according to claim 1 is a template image generation device that generates a template image used when performing template matching from a plurality of program videos, and includes an image feature amount extraction unit, The template candidate cluster extracting unit and the template image generating unit are provided.

  According to such a configuration, the template image generation device divides a plurality of program videos into shots by the image feature amount extraction unit, extracts the representative still images from the shots, and extracts the representative still images from the predetermined still images. The image feature amount is extracted for each block. In addition, the template candidate cluster extracting unit hierarchically clusters the representative still images according to the similarity of the image feature amount extracted by the image feature amount extracting unit, and the tree diagram indicating the result of the hierarchical clustering is cut at a predetermined hierarchy. From each intersection with the cutting line in the case, the cluster is traced down one by one, and a cluster having a branch at a predetermined distance or more is extracted as a template candidate cluster. Further, the template image generation unit generates a template image from the template candidate cluster extracted by the template candidate cluster extraction unit.

  Further, the template image generation apparatus according to claim 1 is configured such that the template image generation means includes a variance value calculation unit, a mask information generation unit, and a template image selection unit.

  According to such a configuration, the template image generation unit of the template image generation apparatus includes a plurality of templates that are candidates for template images included in the template candidate clusters extracted by the template candidate cluster extraction unit by the variance value calculation unit. A variance value of the image feature amount for each block of the candidate image is calculated. Further, when the variance value calculated by the variance value calculation unit exceeds a preset threshold value, the mask information generation unit generates mask information that is information relating to a mask formation position for each block with respect to the template image. Further, the template image selection unit selects, as a template image, a template candidate image whose image feature amount is closest to the average among a plurality of template candidate images included in the template candidate cluster extracted by the template candidate cluster extraction unit. .

  According to a second aspect of the present invention, the template image generating apparatus according to the first aspect is configured such that the template candidate cluster extracting unit includes a hierarchical clustering unit and a candidate cluster extracting unit.

  According to such a configuration, the template image generation device is a representative still image obtained by arranging the image feature amounts for each block of the representative still image extracted by the image feature amount extraction unit by the hierarchical clustering unit in a predetermined order. The representative still images are hierarchically clustered according to the similarity of the feature vectors. In addition, the candidate cluster extraction unit traces a tree diagram indicating the result of the hierarchical clustering by the hierarchical clustering unit at a predetermined level, and then follows each level from the intersection with the cutting line to determine a predetermined distance. Clusters having branches apart from each other are extracted as template candidate clusters.

  According to a third aspect of the present invention, in the template image generating apparatus according to the second aspect, the template candidate cluster extracting unit includes a candidate cluster narrowing-down unit.

  According to such a configuration, in the template image generation device, the candidate cluster narrowing unit extracts the template candidate cluster extracted by the candidate cluster extracting unit as a cluster including a representative still image extracted from the adjacent shots. The first condition for deleting the corresponding representative still image from the template candidate clusters and the number of program videos from which the representative still images included in the template candidate cluster that have passed through the first condition are preset. If the number is not more than the number, the template candidate clusters are narrowed down according to the second condition for deleting the corresponding template candidate clusters.

  According to a fourth aspect of the present invention, there is provided a template image generation program for generating a template image to be used when template matching is performed from a plurality of program videos by using an image feature amount extraction unit, a template candidate cluster extraction unit, a distribution It is configured to function as a value calculation unit, a mask information generation unit, and a template image selection unit.

  According to such a configuration, the template image generation program divides a plurality of program videos into shots by the image feature amount extraction unit, extracts the representative still images from the shots, and extracts the representative still images from the predetermined still images. The image feature amount is extracted for each block. In addition, the template candidate cluster extracting unit hierarchically clusters the representative still images according to the similarity of the image feature amount extracted by the image feature amount extracting unit, and the tree diagram indicating the result of the hierarchical clustering is cut at a predetermined hierarchy. From each intersection with the cutting line in the case, the cluster is traced down one by one, and a cluster having a branch at a predetermined distance or more is extracted as a template candidate cluster. Further, the variance value calculating means calculates the variance value of the image feature amount for each block of the plurality of template candidate images that are candidates for the template image included in the template candidate cluster extracted by the template candidate cluster extracting means. Further, when the variance value calculated by the variance value calculation unit exceeds a preset threshold value, the mask information generation unit generates mask information that is information regarding the mask formation position for each block with respect to the template image. Further, the template image selection means selects, as a template image, a template candidate image whose image feature amount is closest to the average among a plurality of template candidate images included in the template candidate cluster extracted by the template candidate cluster extraction means. .

  According to the first and fourth aspects of the invention, hierarchical clustering of representative still images is performed according to the similarity of image feature amounts, and clusters including a plurality of representative still images that are partially similar are extracted from the result. Thus, a template image can be automatically generated from a typical effect scene repeatedly used in a plurality of program videos. In addition, in order to determine a variable portion in the image by calculating a variance value of image feature amounts in a plurality of template candidate images and generate mask information covering the variable portion, a mask specified by the mask information is used as a template. By synthesizing with an image, the accuracy of template matching can be improved, and variations in threshold values during template matching can be prevented.

  According to the invention according to claim 2, in the tree diagram showing the result of the hierarchical clustering, the cluster having a branch at a predetermined distance or more is searched by following the hierarchy one by one from each intersection with the cutting line. Thus, template candidate clusters can be easily extracted.

  According to the invention according to claim 3, it is possible to accurately extract only typical production scenes that are repeatedly used in a plurality of program videos by narrowing down the template candidate clusters in stages using two conditions. it can.

It is a block diagram which shows the whole structure of the template image generation apparatus which concerns on embodiment of this invention. It is the schematic which shows the process by the shot division part of the template image generation apparatus which concerns on embodiment of this invention, (a) is a figure which shows the process with respect to the program A, (b) is a figure which shows the process with respect to the program B . It is the schematic which shows the process by the representative still image extraction part of the template image generation apparatus which concerns on embodiment of this invention, (a) is a figure which shows the process with respect to program A, (b) is a figure which shows the process with respect to program B, It is. It is the schematic which shows the process by the block division part of the template image generation apparatus which concerns on embodiment of this invention. It is the schematic which shows the outline | summary of hierarchical clustering. It is the schematic which shows the process by the hierarchical clustering part of the template image generation apparatus which concerns on embodiment of this invention, (a) is a dendrogram which shows the result of hierarchical clustering, (b) was enclosed with the broken line in (a). It is an enlarged view of the A section. FIG. 7 is an enlarged view of a portion B surrounded by a broken line in FIG. It is a flowchart which shows the process by the candidate cluster extraction part of the template image generation apparatus which concerns on embodiment of this invention. It is the schematic which shows the process based on the 1st condition by the candidate cluster narrowing part of the template image generation device which concerns on embodiment of this invention, (a) is what extracted several template candidate images from the adjacent shot. FIG. 5B is a diagram illustrating a case where a plurality of template candidate images are not extracted from adjacent shots. It is the schematic which shows the process by the dispersion value calculation part of the template image generation apparatus which concerns on embodiment of this invention. It is the schematic which shows an example of the mask information output by the template image generation apparatus which concerns on embodiment of this invention, and a template image, (a) is the mask information produced | generated by the mask information production | generation part, and the template image selection part The figure which shows the selected template image, (b) is a figure which shows the synthesized template image with mask information.

  A template image generation apparatus and a template image generation program according to an embodiment of the present invention will be described with reference to the drawings.

[Template image generator]
The template image generation device 1 is a device that generates a template image used when performing template matching from a plurality of program videos. As shown in FIG. 1, the template image generating apparatus 1 includes an image feature amount extracting unit 10, a template candidate cluster extracting unit 20, and a template image generating unit 30 as main components. The plurality of programs used in the present invention are two or more videos of the same series of programs, such as news programs with the same program name broadcast at the same time every week.

  The image feature amount extraction means 10 divides a plurality of program videos into shots, extracts representative still images from the shots, and extracts image feature amounts for each block of the representative still images. As shown in FIG. 1, the image feature quantity extraction unit 10 includes a shot division unit 11, a representative still image extraction unit 12, a block division unit 13, and a feature quantity extraction unit 14.

  The shot division unit 11 divides the input program video for each continuous shot. For example, as shown in FIGS. 2 (a) and 2 (b), the shot dividing unit 11 receives a program video of two programs A and B, and cuts off video points such as edit points from the respective videos. Detect and divide each shot according to the break. In addition, since the breaks in the video are different for each program, as shown in FIGS. 2A and 2B, the length of each shot differs for each program.

  As shown in FIG. 1, a plurality of program videos are input to the shot division unit 11 from program video storage means (not shown). Then, the shot dividing unit 11 divides the program video for each shot by the above-described method, and outputs this to the representative still image extracting unit 12.

  The representative still image extraction unit 12 extracts a representative still image from shots constituting a program video. Here, the representative still image refers to a still image that represents the content of each shot. Note that the place where the representative still image is extracted from the shots constituting the program video differs depending on the content of the program video. For example, when the program video is a news program video, the representative still image extraction unit 12 extracts the first frame image of each shot as a representative still image because there is often little change in the video within the shot.

  For example, as shown in FIGS. 3A and 3B, the representative still image extraction unit 12 receives the first frame of each shot when the shots of the two news programs A and B are input from the shot division unit 11. An image is extracted as a representative still image. When the representative still image extraction unit 12 extracts a representative still image from a plurality of shots, the representative still image is extracted from the representative still image from the extracted program name and the number of shots constituting the program video. The number which shows whether it is a thing is provided.

  As shown in FIG. 1, the representative still image extraction unit 12 receives a plurality of shots constituting a plurality of program videos from the shot division unit 11. Then, the representative still image extracting unit 12 extracts a representative still image from a plurality of shots by the above-described method, and outputs this to the block dividing unit 13.

  The block dividing unit 13 divides the representative still image into blocks by a predetermined number. For example, as shown in FIG. 4, when a plurality of representative still images are input from the representative still image extracting unit 12, the block dividing unit 13 converts each representative still image into a preset 18 horizontal × 11 vertical ( 198 blocks).

  As shown in FIG. 1, the block dividing unit 13 receives a plurality of representative still images extracted from a plurality of shots of a plurality of programs from the representative still image extracting unit 12. Then, the block dividing unit 13 divides each representative still image into blocks by the above-described method, and outputs this to the feature amount extracting unit 14.

The feature amount extraction unit 14 extracts an image feature amount for each block of the representative still image. Here, examples of the image feature amount extracted by the feature amount extraction unit 14 include color information such as an average value of RGB components of each block, an average value of L ^* a ^* b ^* components, and a spatial frequency such as a DCT coefficient. Can be used.

  As shown in FIG. 1, a plurality of representative still images after block division are input to the feature amount extraction unit 14 from the block division unit 13. Then, the feature amount extraction unit 14 extracts the image feature amount for each block of the representative still image by the above-described method, and outputs this to the hierarchical clustering unit 21 of the template candidate cluster extraction unit 20.

  The template candidate cluster extraction unit 20 performs hierarchical clustering on the representative still images according to the similarity of the image feature amounts extracted by the image feature amount extraction unit 10, and cuts a tree diagram indicating the result of the hierarchical clustering at a predetermined level. In this case, a cluster having a branch at a predetermined distance or more from each intersection with the cutting line is extracted as a template candidate cluster including a plurality of images that are candidates for the template image. It is. Here, as shown in FIG. 1, the template candidate cluster extracting means 20 includes a hierarchical clustering unit 21, a candidate cluster extracting unit 22, a distance condition storage unit 23, a candidate cluster narrowing unit 24, and a ratio condition storage unit 25. It is equipped with.

  The hierarchical clustering unit 21 performs hierarchical clustering using image feature amounts for each block of the representative still image. Here, hierarchical clustering is a technique in which each of a plurality of data groups is regarded as a single cluster, and the clusters are classified hierarchically based on the respective similarities. Hereinafter, an outline of hierarchical clustering will be briefly described with reference to FIG.

As shown in FIG. 5, for example, consider a case where hierarchical clustering is performed by regarding each of six pieces of data 1 to 6 as a single cluster. In this case, first, a combination having the most similar feature amount is selected from the clusters of data 1 to 6. Then, for example, if the feature amount of data 1 and data 4 are the most similar among all the data, as shown in FIG. 5, coupled by connecting the data 1 and data 4 in a line, the first coupling cluster C ₁ Generate. Here, the first combined cluster C ₁ is a cluster including two clusters of data 1 and data 4.

Next, a combination having the most similar feature amount is selected from the first combined cluster C ₁ and the remaining clusters of data 2, 3, 5, and 6. Note that the feature amount of the first combined cluster C ₁ is represented by an average value of the feature amounts of the data 1 and the data 4. For example, if the first combined cluster C ₁ and the data 5 are most similar among all remaining data, as shown in FIG. 5, the first combined cluster C ₁ and the data 5 are connected by a line. bound, generating a second binding cluster _{C 2.} Here, the second combined cluster C ₂ is a cluster including three clusters of data 1, data 4, and data 5.

Hierarchical clustering, thus feature quantity bonded one after another cluster together similar, as shown in FIG. 5, performs the binding processing until finally all the clusters to form a fourth connection cluster C ₄ one . Here, FIG. 5 is a dendrogram that visually represents the result of hierarchical clustering. The vertical axis in the dendrogram in FIG. 5 indicates the coupling distance (dissimilarity) when the clusters of each data are coupled, and the cluster has a similar feature amount in the hierarchy below the dendrogram, This means that the feature quantity is not similar to the cluster connected in the hierarchy above the tree diagram.

  Specifically, the hierarchical clustering unit 21 calculates the image feature amount for each block of the representative still image extracted by the feature amount extraction unit 14 of the image feature amount extraction unit 10 in a predetermined order, for example, the block of the representative still image block. The features arranged from the upper left to the lower right are used as feature vectors of the representative still image. Then, hierarchical clustering is performed based on the similarity of each feature vector with one representative still image as one cluster.

  A tree diagram shown in FIG. 6A is a visual representation of the result of hierarchical clustering performed by the hierarchical clustering unit 21. In the tree diagram in FIG. 6 (a), the vertical axis indicates the connection distance (dissimilarity) of the feature vectors when the clusters are combined, and the horizontal axis (FIG. 6 (b)) is the horizontal axis. , Each representative still image extracted is shown.

  As shown in FIG. 1, the hierarchical clustering unit 21 receives the image feature amount for each block of the representative still image from the feature amount extraction unit 14. The hierarchical clustering unit 21 performs hierarchical clustering by the above-described method, and outputs a hierarchical clustering result as shown in the tree diagram of FIG. 6A to the candidate cluster extracting unit 22.

  The candidate cluster extraction unit 22 extracts template candidate clusters from the result of hierarchical clustering by the hierarchical clustering unit 21. Here, the template candidate cluster means a cluster including a representative still image that is a template image candidate. When the result of the hierarchical clustering is input from the hierarchical clustering unit 21, the candidate cluster extracting unit 22 extracts a template candidate cluster using a predetermined algorithm shown below. FIG. 7 shows an enlarged view of FIG. 6A for explanation.

Referring to FIG. 7, there is a stepped region in the upper hierarchy. This stepped region, the cluster C _x various clusters are formed by bonding the other cluster is a region formed by combining one after another. Among the clusters coupled to the cluster C _x , as shown in FIG. 7, there are a cluster C ₀ including only one representative still image and a cluster C ₂ including a plurality of representative still images. Exists.

Here, in the cluster C ₀ shown in FIG. 7, an image including r and s is not connected to any other cluster in the lower hierarchy in the tree diagram, and is higher in the dissimilarity higher hierarchy. it is a cluster bound to the cluster C _x only. Therefore, it is estimated that these representative still images are images that are not very similar to other representative still images in the entire image or a part of the image. On the other hand, although the cluster C ₂ shown in FIG. 7 is coupled to the cluster C _{x in the} upper layer having a high dissimilarity, a plurality of clusters are separated in a lower layer separated from the coupling with the cluster C _{x by} a certain distance. It is formed by. Therefore, although the representative still image included in the cluster C ₂ is similar to the other representative still images in the entire image, it is estimated that a part of the image is different from the other representative still images.

Based on the above estimation, the candidate cluster extraction unit 22 extracts a template candidate cluster on the basis of the stepped region shown in FIG. The template candidate cluster extraction algorithm by the candidate cluster extraction unit 22 will be described below with reference to FIG. 7 and along the flowchart of FIG. In FIG. 7, for convenience of explanation, each representative still image is denoted by a to s. An example of a distance condition d _c (threshold) described later is shown in the upper left of the drawing.

  When the candidate cluster extraction unit 22 receives the hierarchical clustering result from the hierarchical clustering unit 21 and starts the candidate cluster extraction process, first, as shown in FIG. 7, the tree diagram which is the classification result of the hierarchical clustering is displayed in a predetermined hierarchy. Disconnect (step S1). Here, cutting at a predetermined hierarchy means, for example, dividing the combined distance of clusters for all clusters, dividing the total number of clusters to obtain an average combined distance, and obtaining a tree diagram at the position of the average combined distance. It means cutting. In addition, there is a method in which a stair-like tree diagram is traced upward from the position of the average coupling distance, and cutting is performed when the distance to the next higher layer exceeds a predetermined threshold.

  Next, the candidate cluster extraction unit 22 adds the intersections B1 to B7 that intersect the cutting line in FIG. 7 to the starting point list (step S2). Next, the candidate cluster extraction unit 22 determines whether the starting point list is empty (step S3). If the departure point list is not empty (No in step S3), the candidate cluster extraction unit 22 selects one departure point (intersection) from the departure point list, and performs processing in the downward direction of the departure point. Start (step S4). On the other hand, when the starting point list is empty (Yes in step S3), the candidate cluster extraction unit 22 ends the process.

  Next, the candidate cluster extraction unit 22 determines whether there is a branch point in the downward direction of the starting point, that is, in the lower hierarchy (step S5). If there is a branch point, the candidate cluster extraction unit 22 proceeds to step S6. On the other hand, when there is no branch point, that is, when the cluster includes only one representative still image as in the representative still images k, l, m, n, r, and s shown in FIG. The current starting point is deleted from the starting point list (step S7), and the process returns to step S4.

Next, the candidate cluster extraction unit 22 determines whether the distance between the branch point and the starting point is a distance above defined to a preset distance condition d _c (step S6). If the distance is greater than or equal to the distance specified in the distance condition (Yes in step S6), the candidate cluster extraction unit 22 extracts clusters that are below the starting point (clusters below the starting point) as template candidate clusters. The starting point is deleted from the starting point list (step S8), and the process returns to step S4.

For example, when the starting point was B5 as shown in FIG. 7, the distance d ₂ between the starting point B5 and the branching point G2 is at least the distance indicated on the distance condition d _c. Therefore, the candidate cluster extraction unit 22 extracts clusters that are equal to or lower than the branch point G2 as template candidate clusters. The representative still images included in the template candidate cluster are three representative still images o, p, q as shown in FIG.

  On the other hand, if the distance is less than the distance specified in the distance condition (No in step S6), the candidate cluster extraction unit 22 deletes the current departure point from the departure point list and instead puts the end points of the branch points into the departure point list. A point is added (step S9), and the process returns to step S3.

For example, if the starting point B1 as shown in FIG. 7, the distance d ₁ between the starting point B1 and the branch point G1 is less than the distance indicated on the distance condition d _c. Accordingly, the candidate cluster extraction unit 22 deletes the branch point G1 from the start point list, adds the points R1 and R2 at both ends of the branch point G1 to the start point list as new start points, and repeats the processing from step S3 onward. . Here, since the point R2 includes only one representative still image k, the candidate cluster extraction unit 22 deletes the point R2 from the starting point list. On the other hand, the point R1 is the distance between the branch point P1 is less distance condition d _c, the candidate cluster extraction unit 22 deletes the point R1 from the starting point list, points at both ends of the point P1 to starting point list F1 , F2 are added, and the processes in and after step S3 are repeated.

  In this way, the candidate cluster extraction unit 22 includes, for example, the cluster including the representative still images b and c, the cluster including the representative still images i and j, and the representative still images o, p, and q in FIG. Three candidate template clusters are extracted. The distance indicated in the distance condition described above is a value obtained experimentally in advance, and is stored in advance in the distance condition storage unit 23 shown in FIG.

  As shown in FIG. 1, the candidate cluster extraction unit 22 receives the hierarchical clustering result from the hierarchical clustering unit 21 and the distance condition from the distance condition storage unit 23. Then, the candidate cluster extraction unit 22 extracts template candidate clusters by the above-described method, and outputs this to the candidate cluster narrowing unit 24.

As described above, the distance condition storage unit 23 stores in advance the distance condition d _c (threshold value) indicating the distance between the departure point and the branch point. Specifically, the distance condition storage unit 23 is implemented by a memory, a hard disk, or the like that can store data. Distance condition storage unit 23, as shown in FIG. 1, and outputs the distance condition d _c in the candidate cluster extraction unit 22. The distance condition storage unit 23 may be provided outside the template image generation device 1 as long as the distance condition can be output to the candidate cluster extraction unit 22.

  The candidate cluster narrowing unit 24 narrows down the number of template candidate clusters based on a predetermined condition. Specifically, the candidate cluster narrowing unit 24 narrows down the number of template candidate clusters extracted by the candidate cluster extracting unit 22 in stages based on the following two conditions. The template image generation apparatus 1 can perform a mask information generation process and a template image selection process, which will be described later, without performing the narrowing down by the candidate cluster narrowing unit 24. By doing so, a more appropriate template image can be generated, and the accuracy of template matching can be improved.

  The first condition is that if the template candidate cluster extracted by the candidate cluster extraction unit 22 is a cluster including a representative still image extracted from adjacent shots, the corresponding representative still image is deleted from the template candidate cluster. It is to do. This is because when a shot constituting a program video includes a video such as a camera flash, the shot division unit 11 erroneously detects that the flash is a video break, and is originally one shot. Is a condition for suppressing such excessive detection of shots.

  Accordingly, as shown in FIG. 9A, the candidate cluster narrowing unit 24, when the representative still image extracted from the adjacent shot is included in the template candidate cluster 1, according to the first condition described above. The corresponding representative still image is deleted from the template candidate cluster, and collation with a second condition described later is performed. On the other hand, as shown in FIG. 9B, the candidate cluster narrowing unit 24 deletes the representative still image when the template candidate cluster 2 does not include the representative still image extracted from the adjacent shot. Without narrowing down, the second condition described later is used for narrowing down.

  The second condition is to delete the corresponding template candidate cluster when the number of program videos from which the representative still images included in the template candidate clusters narrowed down by the first condition are not more than a preset number. That's it. This is because, for example, when 100 program videos are input to the template image generation apparatus 1 and the template candidate cluster includes only representative still images extracted from one program video, the representative This is because there is a possibility that a still image does not indicate a typical effect scene common to a plurality of program videos.

  Therefore, the candidate cluster narrowing unit 24 first detects in which program the representative still image included in the template candidate cluster narrowed down according to the first condition described above is input to the template image generation apparatus 1. If the ratio of the number of programs from which the representative still image is extracted to the number of program videos is not equal to or greater than the ratio indicated in the preset ratio condition, the corresponding template candidate cluster is deleted. As described above, the representative still image included in the template candidate cluster is extracted from the program name extracted by the representative still image extraction unit 12 and the number of shots constituting the program video. And a number indicating.

  Here, the above-described ratio condition is a condition that can be changed as appropriate depending on the accuracy of template matching. That is, for example, when 1000 program videos are input to the template image generation apparatus 1 and it is desired to generate a template image that performs the same effect on all the program videos, the ratio condition may be set to 100%. In this case, the accuracy at the time of template matching is improved, but the number of generated template images is reduced. On the other hand, if, for example, 1000 program videos are input to the template image generation apparatus 1 and it is desired to generate a template image that performs the same effect with 100 program videos, the ratio condition may be set to 10%. In this case, the accuracy at the time of template matching is lowered, but the number of generated template images is increased.

  As shown in FIG. 1, the candidate cluster narrowing unit 24 receives a template candidate cluster from the candidate cluster extraction unit 22 and a ratio condition from the ratio condition storage unit 25. The candidate cluster narrowing unit 24 narrows down the template candidate clusters by the above-described method, and outputs this to the variance value calculating unit 31 and the template image selecting unit 34 of the template image generating unit 30.

  As described above, the ratio condition storage unit 25 stores in advance a ratio condition indicating the ratio of the number of programs from which the representative still image is extracted with respect to the number of program videos input to the template image generating apparatus 1. Specifically, the ratio condition storage unit 25 is implemented by a memory, a hard disk, or the like that can store data. The ratio condition storage unit 25 outputs the ratio condition to the candidate cluster narrowing unit 24 as shown in FIG. The ratio condition storage unit 25 may be provided outside the template image generation apparatus 1 as long as the ratio condition can be output to the candidate cluster narrowing unit 24.

  The template image generation means 30 generates mask information from the template candidate clusters after narrowing down extracted by the template candidate cluster extraction means 20 and selects a template image from the template candidate images. As shown in FIG. 1, the template image generation unit 30 includes a variance value calculation unit 31, a mask information generation unit 32, a threshold storage unit 33, and a template image selection unit 34.

  The variance value calculation unit 31 calculates a variance value for each block of the template candidate images (representative still images) included in the template candidate cluster. Specifically, the variance value calculation unit 31 compares the image feature amounts of the template candidate images included in the template candidate clusters narrowed down by the candidate cluster narrowing unit 24 for each block, and the variance value of the image feature amount for each block. Is calculated. For example, as illustrated in FIG. 10, the variance value calculation unit 31 includes template candidate images 1 to 3 in a template candidate cluster, and these images are processed by the above-described block division unit 13 by 18 squares × vertical. If the block is divided into 11 squares, a variance value for 198 blocks is calculated.

  Here, the fact that the variance value of the image feature value is large indicates that the change in the image feature value of the corresponding block is large. Therefore, a block having a large image feature amount variance value in the template candidate image can be considered as a variable portion in a plurality of program videos. On the other hand, a small dispersion value of the image feature amount indicates that a change in the image feature amount of the corresponding block is small. Therefore, it can be considered that a block having a small image feature amount variance value in a template candidate image is a fixed portion in a plurality of program videos.

  As shown in FIG. 1, the template candidate cluster after narrowing down is input to the variance value calculating unit 31 from the candidate cluster narrowing unit 24. Then, the variance value calculation unit 31 calculates the variance value for each block of the template candidate image included in the template candidate cluster by the above-described method, and outputs these to the mask information generation unit 32. When a plurality of template candidate clusters are input from the candidate cluster narrowing unit 24, the variance value calculation unit 31 calculates the above-described variance value for each template candidate cluster.

  The mask information generation unit 32 generates mask information to be combined with the template image. Here, the mask information means information related to the mask formation position with respect to the template image. Specifically, the mask information generation unit 32 compares the variance value for each block of the template candidate image input from the variance value calculation unit 31 with a preset threshold value, and the variance value exceeds the threshold value. In this case, mask information related to the formation position of the mask covering the corresponding block is generated. Then, the mask information generation unit 32 performs the above-described processing for all the blocks of the template image, and generates mask information for the entire template image, for example, as shown in the left diagram of FIG.

  Here, the threshold value described above is a condition that can be changed as appropriate depending on the accuracy of template matching. That is, if it is desired to mask even a small variable portion in the template candidate image, the threshold value may be lowered. In this case, the accuracy in template matching is improved, but the number of template images to be generated is reduced. On the other hand, if it is desired to mask only a large variable part in the template candidate image, the threshold value may be increased. In this case, the accuracy at the time of template matching is lowered, but the number of generated template images is increased.

  As shown in FIG. 1, the mask information generation unit 32 receives a variance value for each block of template candidate images included in the template candidate cluster from the variance value calculation unit 31, and receives a threshold value from the threshold storage unit 33. Is entered. Then, the mask information generation unit 32 generates mask information of the entire template by the above-described method, and outputs this.

  As described above, the threshold value storage unit 33 stores a threshold value for determining whether or not to generate mask information that covers each block of the template image. Specifically, the threshold storage unit 33 is implemented by a memory, a hard disk, or the like that can store data. As shown in FIG. 1, the threshold storage unit 33 outputs the threshold to the mask information generation unit 32. Note that the threshold storage unit 33 may be provided outside the template image generation device 1 as long as the threshold value can be output to the mask information generation unit 32.

  The template image selection unit 34 selects one template image from the template candidate images included in the template candidate cluster. More specifically, the template image selection unit 34 has the feature vector of the image feature amount closest to the center among the plurality of template candidate images included in the template candidate cluster narrowed down by the candidate cluster narrowing unit 24 (the feature vector of the feature vector). The template candidate image closest to the average is selected as the template image. That is, the template image selection unit 34 extracts the image group closest to the average from the image group included in the cluster.

  As shown in FIG. 1, the template candidate cluster after narrowing down is input to the template image selection unit 34 from the candidate cluster narrowing unit 24. And the template image selection part 34 selects a template image with an above-described method, and outputs this. When a plurality of template candidate clusters are input from the candidate cluster narrowing unit 24, the template image selection unit 34 selects the template image described above for each template candidate cluster.

  The mask information generated by the mask information generation unit 32 and the template image selected by the template image selection unit 34 are combined as shown in FIG. 11A, for example, and a mask as shown in FIG. A template image with information is generated.

  Referring to FIG. 11 (b), it can be seen that the inset image portion that conveys the news item changes for each news, and is therefore masked. In addition, the lower left part of the announcer's face moves and changes when the manuscript is read, so that the mask is put on. Further, the announcer's left arm moves and changes when turning the document, so that it can be seen that the announcer is masked. The announcer's tie changes from day to day, so you can see that the mask is on.

  The template image generating apparatus 1 having the above configuration performs hierarchical clustering of representative still images according to the similarity of image feature amounts, and extracts clusters including a plurality of representative still images that are partially similar from the result. Thus, a template image can be automatically generated from a typical effect scene repeatedly used in a plurality of program videos. In addition, in order to determine a variable portion in the image by calculating a variance value of image feature amounts in a plurality of template candidate images and generate mask information covering the variable portion, a mask specified by the mask information is used as a template. By synthesizing with an image, the accuracy of template matching can be improved, and variations in threshold values during template matching can be prevented.

  In addition, the template image generation apparatus 1 searches a cluster having a branch at a predetermined distance or more by following the hierarchy one by one from each intersection with the cutting line in the tree diagram showing the result of the hierarchical clustering. Thus, template candidate clusters can be easily extracted. Further, by narrowing down the template candidate clusters in stages using two conditions, it is possible to accurately extract only typical effect scenes that are repeatedly used in a plurality of program videos.

[Template image generation program]
Here, the template image generating apparatus 1 can be realized by operating a general computer with a program that functions as each of the above-described units and units. This program can be distributed via a communication line, or can be written on a recording medium such as a CD-ROM for distribution.

[Operation of Template Image Generation Device]
Hereinafter, an example of the operation of the template image generating apparatus 1 will be briefly described with reference to FIG. First, when a plurality of program videos are input to the template image generating apparatus 1, the shot dividing unit 11 detects video breaks such as edit points from the respective videos, and divides each shot according to the cuts.

Next, the representative still image extraction unit 12 extracts, for example, the first frame image of each shot as a representative still image. Next, the block dividing unit 13 divides the representative still image into blocks by, for example, horizontal 18 squares × vertical 11 squares (198 blocks). Next, the feature amount extraction unit 14 extracts an image feature amount including color information such as an average value of RGB components and an average value of L ^* a ^* b ^* components for each block of the representative still image.

  Next, the hierarchical clustering unit 21 arranges the image feature amounts for each block of the representative still image in order from the upper left to the lower right of the image as feature vectors, and performs hierarchical clustering according to the similarity of each feature vector. Next, the candidate cluster extraction unit 22 extracts template candidate clusters from the result of hierarchical clustering. The template candidate cluster extraction algorithm by the candidate cluster extraction unit 22 is as described above. Next, when the candidate cluster narrowing-down unit 24 includes a representative still image extracted from an adjacent shot in the template candidate cluster based on the first condition, the candidate cluster narrowing unit 24 corresponds to the first condition described above. The representative still image to be deleted is deleted from the template candidate cluster. Further, the candidate cluster narrowing unit 24 detects, based on the second condition, which program contained the representative still image included in the template candidate cluster narrowed down according to the first condition, and the template image generating device 1 If the ratio of the number of programs from which the representative still image is extracted to the number of program videos input to is not equal to or greater than the ratio of the preset ratio condition, the corresponding template candidate cluster is deleted.

  Next, the variance value calculation unit 31 calculates the variance value of the image feature amount for each block of the template candidate images included in the template candidate cluster. Next, when the mask information generation unit 32 compares the variance value for each block of the template candidate image input from the variance value calculation unit 31 with a preset threshold value, and the variance value is equal to or greater than the threshold value The mask information for covering the corresponding block is generated and output. The template image selection unit 34 selects and outputs an image closest to the average as a template image among a plurality of template candidate images included in the template candidate cluster.

DESCRIPTION OF SYMBOLS 1 Template image generation apparatus 10 Image feature-value extraction means 11 Shot division part 12 Representative still image extraction part 13 Block division part 14 Feature-value extraction part 20 Template candidate cluster extraction means 21 Hierarchical clustering part 22 Candidate cluster extraction part 23 Distance condition storage part 24 candidate cluster narrowing down section 25 ratio condition storage section 30 template image generation means 31 variance value calculation section 32 mask information generation section 33 threshold storage section 34 template image selection section C _x cluster C ₀ independent cluster C ₁ first combined cluster C ₂ first 2 bond cluster C ₃ 3rd bond cluster C ₄ 4th bond cluster

Claims (4)

A template image generation device that generates a template image used when performing template matching from a plurality of program videos,
An image feature amount that divides the plurality of program videos into shots, extracts a representative still image from the shot, divides the representative still image into a predetermined number of blocks, and extracts an image feature amount for each block Extraction means;
The representative still image is hierarchically clustered according to the similarity of the image feature amount extracted by the image feature amount extraction means, and a tree line showing a result of the hierarchical clustering is a cutting line when cut at a predetermined hierarchy. Template candidate cluster extraction means for following the hierarchy one by one from each intersection and extracting a cluster having branches separated by a predetermined distance or more as a template candidate cluster;
Template image generation means for generating mask information and a template image from the template candidate cluster extracted by the template candidate cluster extraction means,
The template image generation means includes
A variance value calculation unit that calculates a variance value of the image feature amount for each block of a plurality of template candidate images that are candidates for the template image included in the template candidate cluster extracted by the template candidate cluster extraction unit;
A mask information generating unit that generates mask information, which is information relating to a mask formation position for each block with respect to the template image, when the variance value calculated by the variance value calculating unit exceeds a preset threshold;
A template image for selecting, as the template image, the template candidate image having the closest image feature amount among the plurality of template candidate images included in the template candidate cluster extracted by the template candidate cluster extracting unit. A selection section;
A template image generation apparatus comprising: The template candidate cluster extraction means includes
A feature vector of the representative still image obtained by arranging the image feature amounts for each block of the representative still image extracted by the image feature amount extracting unit in a predetermined order, and the representative still image according to the similarity of the feature vector A hierarchical clustering unit for hierarchical clustering;
Clusters having branches separated by a predetermined distance or more from each intersection with the cutting line when the tree diagram showing the result of hierarchical clustering by the hierarchical clustering unit is cut at a predetermined hierarchy. A candidate cluster extraction unit that extracts a template candidate cluster,
The template image generation apparatus according to claim 1, further comprising:   The template candidate cluster extraction means, when the template candidate cluster extracted by the candidate cluster extraction unit is a cluster including the representative still image extracted from the adjacent shot, the corresponding representative still image is The first condition to be deleted from the template candidate clusters, and the number of the program videos from which the representative still images are extracted from the template candidate clusters that have passed the first condition are equal to or greater than a preset number If not, the template image generation device according to claim 2, further comprising a candidate cluster narrowing-down unit that narrows down the template candidate clusters according to a second condition for deleting the corresponding template candidate clusters. In order to generate a template image used when performing template matching from a plurality of program videos,
An image feature amount that divides the plurality of program videos into shots, extracts a representative still image from the shot, divides the representative still image into a predetermined number of blocks, and extracts an image feature amount for each block Extraction means,
The representative still image is hierarchically clustered according to the similarity of the image feature amount extracted by the image feature amount extraction means, and a tree line showing a result of the hierarchical clustering is a cutting line when cut at a predetermined hierarchy. A template candidate cluster extraction means for tracing a hierarchy one by one from each intersection and extracting a cluster having a branch at a predetermined distance or more as a template candidate cluster;
A variance value calculating unit that calculates a variance value of the image feature amount for each block of a plurality of template candidate images that are candidates for the template image included in the template candidate cluster extracted by the template candidate cluster extracting unit;
A mask information generating unit configured to generate mask information that is information relating to a mask forming position for each block with respect to the template image when the variance value calculated by the variance value calculating unit exceeds a preset threshold;
A template image for selecting, as the template image, the template candidate image having the closest image feature amount among the plurality of template candidate images included in the template candidate cluster extracted by the template candidate cluster extracting unit. Selection means,
A template image generation program characterized by being made to function as.

Images