JP2012222450A - Similar video output method, similar video output apparatus and similar video output program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To output pieces of video data which have similar impressions.SOLUTION: A similar video output apparatus 1 includes: a video data storage part 12 in which a plurality of pieces of video data are stored; structural feature similarity calculation means 22 for calculating time in which a structural feature for each section obtained by dividing the video data appears as a structural feature amount for each of the plurality of pieces of video data and calculating structural feature similarity between the video data on the basis of the structural feature amount and outputting structural feature similarity data; and similar video determination means 24 for extracting video data similar to reference video data from the video data storage part 12 on the basis of the structural feature similarity data.

Description

  The present invention relates to a similar video output method, a similar video output device, and a similar video output program for outputting similar video data.

  With the recent development of information processing apparatuses, electronic contents such as video data have increased enormously. For example, in an IPTV (Internet Protocol TeleVision) or video sharing site, a user can select video data of interest from a large amount of video data and browse the selected video data at an arbitrary timing. is there. For example, by specifying a keyword or genre of desired video data, the user can browse video data that matches the specified conditions. Thus, when the image of the video data desired by the user is clear, it is easy to specify the search condition and search for the desired video data from the searched video data.

  However, if the image of the video desired by the user is not clear, the user cannot specify an appropriate search condition. Therefore, it is difficult for the user to find a desired video by the method of specifying the search condition. In addition, every time the desired video data is browsed, an operation for specifying the search condition is required, and the user may feel annoyed.

  One approach for solving such problems is video recommendation technology. This video recommendation is a technique for facilitating the user's video discovery by presenting another related video when a user browses or evaluates a video.

  The video recommendation methods are mainly divided into collaborative filtering and content-based filtering.

  Collaborative filtering is a technique of analyzing and recommending whether users or videos are similar by using a user's browsing history or evaluation history for the video. For example, there is a method of recommending items that are similar in how evaluation values are given by users (see Non-Patent Document 1, for example).

  In this collaborative filtering recommendation method, the viewing history is used to analyze the relationship between videos, as in the technique described in Non-Patent Document 1. The technology described in Non-Patent Document 1 regards users who share the same viewing and evaluation videos as having similar preferences, and recommends unviewed videos among videos viewed by users with similar preferences.

  However, in collaborative filtering, the recommendation is to use the user's history, so when the amount of history is small, the amount of viewing history is small, so it accurately analyzes whether the users who viewed the video are similar. There was a problem that it was impossible to make an effective recommendation. For example, if you do not use the service very much or want to recommend it to users who have just registered in the service, or if you want to recommend videos that are buried or newly added to the service I was unable to make a recommendation.

  Collaborative filtering requires a lot of viewing history, whereas content-based filtering recommends based on metadata attached to video and video feature information extracted from video even when the amount of history is small It can be performed. According to content-based filtering, a video can be recommended even when the amount of available history is small, such as a recommendation for a new user who has just registered in the service or a video that is buried. As a result, the service can be used by more users and more videos can be viewed.

  As a research on content-based filtering, there is a method (Patent Document 1) that measures the frequency of appearance of attributes assigned to content highly evaluated by the user and recommends content assigned attributes with high appearance frequency. In addition, there is a method (for example, Non-Patent Document 2) that recommends based on analysis of video features without using information on metadata provided to users or videos.

  The technique described in Non-Patent Document 2 calculates the similarity between videos based on the appearance and sound information of the videos. As the visual information, a color histogram, the intensity of motion, and the average number of shots per second are used, and as the sound information, the average value and variance of the sound tempo are used.

  In a content-based filtering recommendation method, it is common to recommend a video having the same content and genre as the video.

JP 2009-205418 A

B. Sarwar, G. Karypis, J. Konstan, and J. Riedl, “Item-Based Collaborative Filtering Recommendation Algorithms,” In Proceedings of the 10th international conference on World Wide Web, 2001. B. Yang, T. Mei, X.-S. Hua, L. Yang, S.-Q. Yang and M. Li.Online Video Recommendation Based on Multimodal Fusion and Relevance Feedback.In Proc.ACM International Conference on Image and Video Retrieval (CIVR), pp. 73-80, 2007.

  However, when recommending videos, it is important not only to recommend videos that share the same content and genre, but also to recommend videos that have similar impressions to viewers. For example, for a user who has viewed a bright atmosphere video, it is more likely that the user will view a video with a bright atmosphere rather than a dark atmosphere.

  In content-based filtering, as a method for analyzing the relationship between videos, there are a method of using metadata attached to videos and a method of analyzing video features.

  In the method of Patent Document 1, a video to be recommended is determined based on attributes and keywords in metadata. Therefore, when there is no attribute or keyword related to the impression of the video in the metadata, or when no metadata is given, it is not possible to recommend a video with a similar impression.

  Further, in the technique for analyzing video features for analyzing the relationship between videos disclosed in Non-Patent Document 2, the entire video is analyzed. Therefore, a less important scene that does not remain in the viewer's impression is also subject to similarity calculation, and recommends a video that does not have a similar impression on the viewer's video even though the characteristics of the unimportant scene are similar. There was a problem. In addition, since this technique uses the average value or variance of the entire video for calculating the degree of similarity for each feature, it is difficult to compare information about the time axis of the video, such as many features appearing in the first half of the video. was there. For this reason, videos with similar impressions could not be recommended appropriately.

  Accordingly, an object of the present invention is to provide a similar video output method, a similar video output device, and a similar video output program for outputting video data having similar impressions.

  In order to solve the above problems, a first feature of the present invention relates to a similar video output method for outputting similar video data. That is, the similar video output method according to the first feature of the present invention has a structure in which the appearance time of a structural feature for each section obtained by dividing the video data for each of a plurality of video data stored in the video data storage unit is structured. A structural feature similarity calculating step for calculating a structural feature similarity between video data based on the structural feature amount and outputting structural feature similarity data, and a structural feature A similar video determination step of extracting video data similar to the reference video data from the video data storage unit based on the similarity data.

  Here, the structural feature is any one or more of color, motion, acoustic feature, cut division, music segment, speech segment, and telop segment in the segment.

  For each of a plurality of video data, the feature amount of the important scene included in the video data is calculated as the important scene feature amount, and the important scene feature similarity between the video data is calculated based on the important scene feature amount. An important scene feature similarity calculating step for outputting important scene feature similarity data may be further included. In this case, the similar video determination step extracts video data similar to the reference video data from the video data storage unit based on the structural feature similarity data and the important scene feature similarity data.

  Here, the important scene feature is at least one of the color, motion, and acoustic feature of the important scene.

  The second feature of the present invention relates to a similar video output method for outputting similar video data. The similar video output method according to the second feature of the present invention calculates, for each of a plurality of video data stored in the video data storage unit, an important scene feature amount included in the video data as an important scene feature amount. Calculating an important scene feature similarity between video data based on the important scene feature amount, and outputting the important scene feature similarity data; and based on the important scene feature similarity data A similar video determination step of extracting video data similar to the reference video data from the video data storage unit.

  A third feature of the present invention relates to a similar video output device that outputs similar video data. The similar video output device according to the third aspect of the present invention includes a video data storage unit storing a plurality of video data, and a structural feature for each section obtained by dividing the video data for each of the plurality of video data. Structural feature similarity calculating means for calculating the appearance time as a structural feature amount, calculating the structural feature similarity between video data based on the structural feature amount, and outputting the structural feature similarity data And similar video determination means for extracting video data similar to the reference video data from the video data storage unit based on the structural feature similarity data.

  Here, the structural feature is any one or more of color, motion, acoustic feature, cut division, music segment, speech segment, and telop segment in the segment.

  For each of a plurality of video data, the feature amount of the important scene included in the video data is calculated as the important scene feature amount, and the important scene feature similarity between the video data is calculated based on the important scene feature amount. An important scene feature similarity calculating unit that outputs important scene feature similarity data may be further included. In this case, the similar video determining means extracts video data similar to the reference video data from the video data storage unit based on the structural feature similarity data and the important scene feature similarity data.

  Here, the important scene feature is at least one of the color, motion, and acoustic feature of the important scene.

  A fourth feature of the present invention relates to a similar video output device that outputs similar video data. The similar video output device according to the fourth feature of the present invention provides a video data storage unit storing a plurality of video data and a feature amount of an important scene included in the video data for each of the plurality of video data. An important scene feature similarity calculating means for calculating an important scene feature similarity between video data based on the important scene feature quantity and outputting important scene feature similarity data; And a similar video determining means for extracting video data similar to the reference video data from the video data storage unit based on the similarity data.

  A fifth feature of the present invention is a similar video output program for causing a computer to execute the steps of the similar video output method according to the first feature or the second feature of the present invention.

  According to the present invention, it is possible to provide a similar video output method, a similar video output device, and a similar video output program for outputting video data with similar impressions.

FIG. 1 is a flowchart for explaining a similar video output method according to an embodiment of the present invention. FIG. 2 is a functional block diagram of the similar video output device according to the embodiment of the present invention. FIG. 3 is a diagram for explaining an example of the data structure and data of the structural feature amount data according to the embodiment of the present invention. FIG. 4 is a diagram for explaining an example of the data structure and data of the structural feature similarity data according to the embodiment of the present invention. FIG. 5 is a diagram for explaining an example of the data structure and data of the important scene feature amount data according to the embodiment of the present invention. FIG. 6 is a diagram for explaining an example of the data structure and data of the important scene feature similarity data according to the embodiment of the present invention. FIG. 7 is a diagram for explaining an example of the data structure and data of similarity data according to the embodiment of the present invention. FIG. 8 is a diagram for explaining an example of the data structure and data of the video management data according to the embodiment of the present invention. FIG. 9 is a flowchart for explaining the structural feature similarity calculation processing according to the embodiment of the present invention. FIG. 10 is a diagram for explaining the structural feature similarity calculation processing according to the embodiment of the present invention. FIG. 11 is a flowchart for explaining important scene feature similarity calculation processing according to the embodiment of the present invention. FIG. 11 is a diagram for explaining important scene feature similarity calculation processing according to the embodiment of the present invention. FIG. 13 is a flowchart for explaining similar video determination processing according to the embodiment of the present invention.

  Next, embodiments of the present invention will be described with reference to the drawings. In the following description of the drawings, the same or similar parts are denoted by the same or similar reference numerals.

(Embodiment)
The similar video output method according to the embodiment of the present invention outputs video data with similar impressions.

The similar video output method according to the embodiment analyzes a video feature amount, and based on whether the structural feature of the video is similar or whether the feature of the important scene of the video is similar, From the recommended video data, video data with a similar impression is output as video data recommended to the user. In the embodiment of the present invention, a case in which similar video data is calculated based on both the structural feature similarity and the important scene feature similarity will be described. When the user inputs information on the reference video data, the similar video output method outputs video data similar to the reference video data as video data recommended to the user. There may be one or more recommended video data to be output to the user. In the embodiment, a target to be recommended is described as a user, but this user may be a general user or a system using the similar video output method according to the embodiment.

  With reference to FIG. 1, the outline of the process of the similar video output method according to the embodiment will be described. First, the similar video output method proceeds to step S2 when reference video data is input in step S1.

  In step S2, a structural similarity calculation process is executed. In the structural similarity calculation processing, the appearance time of the structural feature for each section obtained by dividing the video data is calculated as the structural feature amount, and the structural feature similarity between the video data is calculated based on the structural feature amount. Is calculated. Here, the structural features include color, movement, acoustic features, cut division, music section, speech section, and telop section in the section of video data. When viewing the video in chronological order, the point at which the shots are switched is called the cut position, and a shot is one scene shot continuously.

  In step S3, a video similarity calculation process is executed. In the video similarity calculation process, the feature amount of the important scene included in the video data is calculated as the important scene feature amount, and the important scene feature similarity between the video data is calculated based on the important scene feature amount. Here, the important scene feature is at least one of the color, motion, and acoustic feature of the important scene.

  In step S4, similar video data is determined. Here, based on the structural feature similarity calculated in step S2 and the similarity between the video data calculated from at least one of the important scene feature similarity calculated in step S3, the reference video Video data similar to the data is specified.

  In step S5, information on the similar video data specified in step S4 is output. For example, the similar video output device 1 outputs a list of titles, descriptions, recommended scores, and the like of the specified similar video data.

(Similar video output device)
The similar video output method according to the embodiment described with reference to FIG. 1 is realized by the similar video output device 1 shown in FIG. The similar video output device 1 is a general computer including a storage device 10, a central processing control device 20, and a display device 30. The similar video output device 1 is implemented by installing a similar video output program for executing a predetermined process in a general computer. Each component of the similar video output device 1 is an API (Application Program Interface) of a server device, and may be realized by a program for a client terminal to provide video information through this API.

  The storage device 10 includes a reference video data storage unit 11, a video data storage unit 12, a structural feature amount data storage unit 13, a structural feature similarity data storage unit 14, an important scene feature amount data storage unit 15, and an important scene feature similarity. A degree data storage unit 16, a similarity degree data storage unit 17, a similar video list data storage unit 18, and a video management data storage unit 19 are stored. The storage device 10 also stores a similar video output program.

  The reference video data storage unit 11 is a storage area in the storage device 10 in which the reference video data 11a is stored. The similar video output device 1 outputs video data whose impression is similar to the reference video data 11a. The reference video data 11a may be video data itself, or may be an identifier of video data stored in the video data storage unit 12 described later, for example.

  The video data storage unit 12 is a storage area in which a plurality of video data 12a, 12b,.

  The structural feature value data storage unit 13 is a storage area in the storage device 10 in which the structural feature value data 13a is stored. The structural feature quantity data 13a is generated and referred to by the structural feature similarity calculation means 22.

  As shown in FIG. 3, the structural feature amount data 13a is data in which a video identifier and a segment identifier in the video are associated with a structural feature identifier and a structural feature amount in the video and the segment. Here, the structural feature identifier schematically shows the structural features of the present invention such as color, motion, acoustic feature, cut division, music section, speech section, and telop section in the section of video data. . In the example shown in FIG. 3, when the structural feature identifier “FEA001” indicates the structural feature of the color, the color structure in the segment of the video data identified by the segment identifier “DIV001” of the video identifier “MOV001”. This indicates that the target feature is a vector of “(0.0001, 0.0002,...)”. The similar video output device 1 may store data in which the structural feature identifiers and the names of the structural features are associated with each other in the storage device 10.

  The structural feature similarity data storage unit 14 is a storage area in the storage device 10 in which the structural feature similarity data 14a is stored. The structural feature similarity data 14a stores the structural feature similarity between arbitrary video data. The structural feature similarity data 14 a is generated by the structural feature similarity calculation unit 22 and is referred to by the similar video determination unit 24.

  As shown in FIG. 4, the structural feature similarity data 14a includes a first video identifier, a second video identifier, and a structural feature between the video data specified by the first and second video identifiers. It is data in which the similarity is associated with. The structural feature similarity is calculated from the structural feature data 13a described with reference to FIG.

  The important scene feature value data storage unit 15 is a storage area in the storage device 10 in which important scene feature value data 15a is stored. The important scene feature data 15a is generated and referred to by the important scene feature similarity calculating unit 23.

  As shown in FIG. 5, the important scene feature data 15a is data in which a video identifier and an important scene identifier in the video are associated with an important scene feature identifier and an important scene feature in the video and the important scene. Here, the important scene feature identifier is an identifier of an important scene feature such as the color, motion, and acoustic feature of the important scene. For example, an L * a * b * histogram can be considered as the color feature of the important scene, and an optical flow can be considered as the feature of the important scene movement. The similar video output device 1 may store data in which the important scene feature identifier and the name of the important scene feature are associated with each other in the storage device 10.

  The important scene feature similarity data storage unit 16 is a storage area in the storage device 10 in which important scene feature similarity data 16a is stored. The important scene feature similarity data 16a stores the important scene feature similarity between arbitrary video data. The important scene feature similarity data 16 a is generated by the important scene feature similarity calculating unit 23 and is referred to by the similar video determining unit 24.

  The important scene feature similarity data 16a includes, as shown in FIG. 6, important scene features between the first video identifier, the second video identifier, and the video data specified by the first and second video identifiers. It is data in which the similarity is associated with. The important scene feature similarity is calculated from the important scene feature data 15a described with reference to FIG.

  The similarity data storage unit 17 is a storage area in the storage device 10 in which similarity data 17a is stored. The similarity data storage unit 17 is generated and referred to by the similar video determination unit 24.

  The similarity data 17a corresponds to the first video identifier, the second video identifier, the similarity between the video data specified by the first and second video identifiers, and the recommendation score corresponding to the similarity. It is attached data. The similarity between the video data is the structural feature similarity of the structural feature similarity data 14a described with reference to FIG. 4 and the important scene feature of the important scene feature similarity data 16a described with reference to FIG. It is calculated based on the similarity. In the example shown in FIG. 7, the similarity between video data takes values from 0 to 1. The recommendation score is calculated based on the similarity, and takes a value of 0 to 100 in the example shown in FIG. In the embodiment, the higher the similarity and the recommendation score, the more similar the video data is, and it is easy to recommend the similar video data.

  The similar video list data storage unit 18 is a storage area in the storage device 10 in which similar video list data 18a is stored. The similar video list data 18 a is output by the similar video determination unit 24 and is referred to by the similar video information output unit 25.

  The similar video list data 18a is information of video data recommended to the user as a video similar to the reference video data 11a. The similar video list data 18a is data in which identifiers, titles, descriptions, recommendation scores, and the like of video data to be recommended are associated.

  The video management data storage unit 19 is a storage area in the storage device 10 in which video management data 19a is stored. As shown in FIG. 8, the video management data 19a is data in which a video identifier, a video name, and a description of the video data are associated with each other. The video management data 19a is referred to in order to display information of video data recommended for the user.

  As shown in FIG. 2, the central processing control device 20 of the similar video output device 1 includes a reference video data acquisition unit 21, a structural feature similarity calculation unit 22, an important scene feature similarity calculation unit 23, and a similar video determination unit 24. And similar video information output means 25.

  The reference video data acquisition unit 21 acquires the reference video data 11 a and stores it in the reference video data storage unit 11. The similar video output device 1 searches for video data similar to the reference video data 11a. The reference video data 11 a may be an identifier of video data stored in the video data storage unit 12.

  When video data that is not stored in the video data storage unit 12 is input as the reference video data 11a, the structural feature similarity calculation means 22 described later has a structural feature amount for the input reference video data 11a. The structural feature similarity between the reference video data 11a and each video data in the video data storage unit 12 is calculated. Similarly, the important scene feature similarity calculating unit 23 described later calculates the important scene feature amount with respect to the input reference video data 11a, and the importance of the reference video data 11a and each video data in the video data storage unit 12 is calculated. A scene feature similarity is calculated. Further, the similar video determination means 24 described later includes the structural feature similarity between the reference video data 11a and each video data in the video data storage unit 12, the reference video data 11a, and each video data in the video data storage unit 12. The similarity between the reference video data 11a and each video data in the video data storage unit 12 and the recommended score are calculated based on the similarity of the important scene features of the video, and information of video data similar to the reference video data 11a is output. .

  In the embodiment of the present invention, a case where an identifier of video data stored in the video data storage unit 12 is designated as the reference video data 11a will be described. In addition, a structural feature similarity calculating unit 22, an important scene feature similarity calculating unit 23, and a similar video determining unit 24, which will be described later, calculate similarities between all video data stored in the video data storage unit 12. The case will be described.

  The structural feature similarity calculation unit 22 includes a structural feature analysis unit 221 and a similarity calculation unit 222.

  The structural feature analysis unit 221 calculates a structural feature amount for each of the plurality of video data stored in the video data storage unit 12. The structural feature analysis unit 221 divides the video data into a plurality of sections so that the length of time is uniform, for example, and calculates the time when the structural features appear in each section as a structural feature amount. Here, the structural feature analysis means 221 uses the appearance time of each structural feature of the color, movement, acoustic feature, cut division, music section, speech section, and telop section as the structural feature amount. Calculate for the interval. The structural feature analysis means 221 inserts a record in which the structural feature amount is associated with the structural feature amount data 13a using the video identifier, the identifier of the section in the video data, and the structural feature identifier as keys. .

  The similarity calculation means 222 calculates the structural feature similarity between the video data based on the structural feature quantity of the structural feature quantity data 13a, and outputs the structural feature similarity data 14a. Here, the similarity calculation unit 222 calculates, for each video identifier, a feature vector having a plurality of structural feature quantities associated with the video identifier as elements. The similarity calculation means 222 calculates the similarity of this feature vector as the similarity between video data. The similarity calculation means 222 calculates the first video identifier, the second video identifier, the structural feature similarity between the video data of the first video identifier and the video data of the second video identifier. The associated record is inserted into the structural feature similarity data 14a.

  With reference to FIG. 9, the structural feature similarity calculation processing by the structural feature similarity calculation means 22 will be described.

  First, the processing from step S101 to step S103 is repeated for each video data stored in the video data storage unit 12. In step S101, the structural feature similarity calculating unit 22 divides the video data into a plurality of sections with equal time.

  The processing in step S102 and step S103 is repeated for each of the plurality of sections. The structural feature similarity calculating unit 22 calculates a structural feature amount for each section. At this time, the structural feature similarity calculating means 22 calculates the length of time of the color, motion, acoustic feature, cut division, music section, speech section, and telop section in each section as the structural feature amount, and step S103. Are recorded in the structural feature data 13a. When the processing from step S102 to step S103 is completed for all the sections, the processing from step S101 to step S103 is continued for the next video data.

  When the processing of steps S101 to S103 is completed for all the video data, the structural feature similarity calculating unit 22 calculates the structural feature similarity between the video data in steps S104 to S107. First, in step S104, the structural feature similarity calculating unit 22 acquires the structural feature amount of each section from the structural feature amount data 13a for arbitrary first video data. In step S105, the structural feature similarity calculating unit 22 acquires the structural feature amount of each section from the structural feature amount data 13a for the second video data.

  In step S106, the structural feature similarity calculating unit 22 calculates the first feature from the structural feature amount of the first video data acquired in step S104 and the structural feature amount of the second video data acquired in step S105. The similarity between the video data and the second video data is calculated. Further, in step S107, the structural feature similarity calculating unit 22 obtains the identifier of the first video data, the identifier of the second video data, and the structural feature similarity based on the structural feature amount calculated in step S106. Correspondingly, the structural feature similarity data 14a is recorded.

  Here, the processing of the structural feature similarity calculation means 22 will be described in detail. In the embodiment of the present invention, the structural features include a cut division of video data, a music section, a speech section, a telop section, and the like. In the embodiment of the present invention, the structural feature amount of the video data is calculated based on the time when these structural features appear in the video data. For example, as shown in FIG. 10A, the structural feature amount of the music section is information on the length of the music section and the position of the music section in the video data. In the embodiment of the present invention, video data having similar structural features can be recommended.

  An existing processing method can be used as a structural feature extraction method. For example, a method described in Japanese Patent No. 2869398 is conceivable as a method for detecting a cut position from video data. As a method for extracting a music section, a method described in Japanese Patent No. 4572218 can be considered. As a method for extracting an utterance section, a method described in Japanese Patent No. 3105465 can be considered. As a telop section extraction method, a method described in Japanese Patent No. 3479592 can be considered.

  As shown in FIG. 10B, the structural feature similarity calculating unit 22 divides the video into several sections, analyzes the structural feature amount in each section, and creates a feature vector. For example, the structural feature similarity calculation unit 22 divides the video into several sections so that the times are equal. Further, the structural feature similarity calculating unit 22 expresses the ratio of the length of the section in which the structural feature appears with respect to the length of each section as a numerical value, and creates a feature vector having these as elements. Here, when the increase / decrease is important rather than the absolute value of the feature quantity, the structural feature similarity calculation unit 22 may normalize the feature vector.

Here, a feature vector whose element is a ratio of the length of the section in which the structural feature of each section of the video data A appears is v ^A, and the length of the section in which the structural feature of each section of the video data B appears. consider the case of a feature vector for the ratio between the element and v ^B. Structural features similarity calculation unit 22, by calculating the similarity of the feature vectors ^{v A} and the feature vector ^{v B,} the feature vector ^{v A} and the feature vector ^{v B} of the similarity ^{S (v} A, v ^B) and calculate.

Here, an arbitrary scale is used in calculating the similarity of the feature vectors of the structural features. For example, if cosine similarity is used as a similarity measure for feature vectors, S (v ^A , v ^B ) is given by equation (1).

Here, v _i ^A represents the i-th value of v ^A , and v _i ^B represents the i-th value of v ^B. n is the number of dimensions of the feature vector.

Here, when there are N _s (> 0) structural feature extraction methods and structural feature amounts to be used, the structural feature similarity calculating unit 22 determines that the video data A, the video data B, Similarity S (v ^A , v ^B ) is obtained. The structural feature similarity calculating unit 22 calculates the similarity S ^AB based on the structural feature based on S (v ^A , v ^B ) obtained for each, and the similarity S ^AB = {S _i ^{A, B} | i = 1 to N _s }.

The structural feature similarity calculating means 22 generates a record in which the similarity S ^AB calculated in this way is associated with the identifier of the video data A and the identifier of the video data B, and inserts it into the structural feature similarity data 14a. And remember.

  Next, the important scene feature similarity calculating unit 23 will be described. The important scene feature similarity calculation unit 23 includes an important scene extraction unit 231, a video feature analysis unit 232, and a similarity calculation unit 233.

  The important scene extraction unit 231 extracts an important scene from each of the plurality of video data stored in the video data storage unit 12. As an important scene extraction method, an existing processing method such as a digest creation method can be used. In the embodiment of the present invention, a plurality of important scenes may be extracted from one video data.

  For each important scene of the plurality of video data stored in the video data storage unit 12, the video feature analysis unit 232 calculates the feature quantity of the important scene included in the video data as the important scene feature quantity. Here, the video feature analysis means 232 calculates one or more of the color, motion, and acoustic features of the important scene as the important scene feature quantity for each important scene of each video data. The video feature analysis means 232 may calculate each important scene feature amount based on the frame image at the intermediate point of each important scene. For example, the video feature analysis unit 232 calculates the color histogram of the frame image at the intermediate point as the feature amount of the important scene color. The video feature analysis means 232 inserts a record in which the important scene feature amount is associated with the important scene feature amount data 15a using the video identifier, the important scene identifier in the video data, and the important scene feature identifier as keys. .

  The similarity calculation means 233 calculates the important scene feature similarity between the video data based on the important scene feature quantity of the important scene feature quantity data 15a, and outputs the important scene feature similarity data 16a. Here, the similarity calculation means 233 calculates, for each video identifier, a feature vector having a plurality of important scene feature quantities associated with the video identifier as elements. The similarity calculation means 233 calculates the similarity of this feature vector as the similarity between video data. The similarity calculating means 233 calculates the first video identifier, the second video identifier, the important scene feature similarity between the video data of the first video identifier and the video data of the second video identifier. The associated record is inserted into the important scene feature similarity data 16a.

  With reference to FIG. 9, the important scene feature similarity calculating process by the important scene feature similarity calculating unit 23 will be described.

  First, the processing from step S201 to step S203 is repeated for each video data stored in the video data storage unit 12. In step S201, the important scene feature similarity calculating unit 23 extracts an important scene from each video data. Here, one or more important scenes may be extracted from one video data.

  The process of step S202 and step S203 is repeated for each important scene. The important scene feature similarity calculating unit 23 calculates an important scene feature amount for each important scene. At this time, the important scene feature similarity calculating unit 23 calculates one or more of important scene colors, movements, and acoustic features in the middle frame of each important scene as an important scene feature amount. Recorded in the scene feature data 15a. When the processing from step S202 to step S203 is completed for all important scenes, the processing from step S201 to step S203 is continued for the next video data.

  When the processing from step S201 to step S203 is completed for all the video data, the important scene feature similarity calculating unit 23 calculates the important scene feature similarity between the video data in steps S204 to S207. First, in step S204, the important scene feature similarity calculating unit 23 acquires the important scene feature quantity of each important scene from the important scene feature quantity data 15a for the arbitrary first video data. In step S205, the important scene feature similarity calculating unit 23 acquires the important scene feature quantity of each important scene from the important scene feature quantity data 15a for the second video data.

  In step S206, the important scene feature similarity calculating unit 23 calculates the first scene from the important scene feature amount of the first video data acquired in step S204 and the important scene feature amount of the second video data acquired in step S205. The similarity between the video data and the second video data is calculated. Further, in step S207, the important scene feature similarity calculating unit 23 obtains the identifier of the first video data, the identifier of the second video data, and the important scene feature similarity based on the important scene feature amount calculated in step S206. Correspondingly, it is recorded in the important scene feature similarity data 16a.

  Here, the processing of the important scene feature similarity calculating means 23 will be described in detail. An existing processing method can be used as a method by which the important scene extracting unit 231 extracts an important scene from video data. For example, a digest video creation method described in Japanese Patent No. 4358723 may be considered. Since the digest of the video data can be created by the digest video creation technique, the section selected as the digest can be regarded as an important scene of the video data.

  As shown in FIG. 12A, the important scene feature similarity calculating unit 23 calculates the important scene feature of the video data for the important scene extracted by the important scene extracting unit 231 as shown in FIG. To analyze. The important scene feature similarity calculation means 23 can use color, movement, acoustic features, etc. as the important scene feature quantity.

  An arbitrary scale is used for calculating the similarity of the important scene feature quantity. For example, when a color histogram is used as the important scene feature quantity, the important scene feature similarity calculating unit 23 extracts several important scenes from the video data as shown in FIG. Extract frames at points. The important scene feature similarity calculating means 23 creates a color histogram of the frame and expresses it as a feature vector. Here, the distance in the color space to be used is used as the distance between the dimensions of the histogram.

Here, a feature amount extracted from each important scene of the video data A is denoted by v ^A , and a feature amount extracted from each important scene of the video B is denoted by v ^B. v ^A and ^{v B} of the similarity ^{I (v} A, ^{v B),} for example, is calculated by equation (2).

The important scene feature similarity calculating means 23, as shown in FIG. 10B, includes each important scene A ₁ , A ₂ ,... Included in the video data A and each video data B ₁ , B ₂ . Calculate the similarity of the scene. The value of I (v ^A , v ^B ) increases as there is an important scene with a high degree of similarity. Here, N _A is the number of important scene extracted from the video data A, N _B is the number of important scene extracted from the video data B. v _i ^A is the i-th important scene of the video data A, and v _j ^B is the j-th important scene of the video data B. D (v _i ^A , v _j ^B ) is the similarity between the important scene v _i ^A and the important scene v _j ^B.

Here, Earth Mover's Distance (Y. Rubner, C. Tomasi, and L. Guibas. "The earth mover's distance as a metric for image retrieval," International Journal of Computer Vision Vol. 40, No. 2, pp. 99- 121, 2000.). If Earth Mover's Distance between v _i ^A and v _j ^B is E (v _i ^A , v _j ^B ), D (v _i ^A , v _j ^B ) is calculated by equation (3).

Alternatively, the important scene feature similarity calculating unit 23, when the feature vector of the image data A v ^A, the feature vector of the image data B was placed with v ^B, v calculates the cosine similarity of ^A and v ^B Thus, the similarity D (v _i ^A , v _j ^B ) between the important scene A _i and the important scene B _j can be calculated. This degree of similarity D (v _i ^A , v _j ^B ) is expressed by equation (4).

Here, v _i ^A is the i-th value of the feature vector of the video data A, and v _i ^B is the i-th value of the feature vector of the video data B. n is the number of dimensions of the feature vector.

Here, when there are N _I (> 0) important scene extraction methods and important scene feature quantities to be used, the important scene feature similarity calculating unit 23 determines the important scene and the video data of the video data A for each. The similarity score I (v ^A , v ^B ) of the ^B important scene is obtained. The important scene feature similarity calculating means 23 calculates the similarity I ^AB based on the important scene feature based on I (v ^A , v ^B ) obtained for each, and the similarity I ^AB = {I _i ^{A, B} | i = 1 to N _I }.

The important scene feature similarity calculating unit 23 generates a record in which the similarity I ^AB calculated in this way is associated with the identifier of the video data A and the identifier of the video data B, and inserts it into the important scene feature similarity data 16a. And remember.

  Next, the similar video determining means 24 will be described. The similar video determination unit 24 extracts video data similar to the reference video data 11a from the video data storage unit 12 based on the structural feature similarity data 14a and the important scene feature similarity data 16a.

  The similar video determination unit 24 may extract video data similar to the reference video data 11a using only one of the structural feature and the important scene feature. For example, when extracting similar video data based only on structural features, the similar video determination means 24 sets the structural feature similarity to the similarity between the video data based on the structural feature similarity data 14a. To do. When extracting similar video data based only on the important scene feature, the similar video determination means 24 sets the important scene feature similarity to the similarity between the video data based on the important scene feature similarity data 16a.

  When video data similar to the reference video data 11a is extracted based on both the structural feature similarity data 14a and the important scene feature similarity data 16a, each is weighted and evaluated, and similar video data is extracted. Also good. This weight may be designated in advance by a user or the like, or a default weight may be designated by an administrator or the like.

  When the similarity between the video data is calculated, the similar video determination unit 24 calculates a recommendation score of each video data with respect to the reference video data 11a based on the similarity. The recommendation score is higher as the similarity is higher, and indicates that the recommendation is preferentially recommended. The similar video determination means 24 extracts video data similar to the reference video data 11a based on the recommendation score. Here, the video data extracted by the similar video determination means 24 may not be the video data itself but only the identifier of the video data. The similar video determination unit 24 may extract a predetermined number of video data having a high degree of similarity, or may extract video data having a predetermined similarity or higher.

  The similar video determination means 24 generates similar video list data 18a including identifiers of these video data as video data for recommending video data having a high similarity, and stores the similar video list data 18a in the similar video list data storage unit 18. In the similar video list data 18a, in order to indicate the degree of recommendation to the user, the recommendation score may be associated with the identifier of the video data to be recommended.

  With reference to FIG. 13, the similar video determination process by the similar video determination means 24 will be described.

  The similar video determination unit 24 repeats the processing from step S301 to step S305 for any two video data stored in the video data storage unit 12.

  First, in step S301, the similar video determination means 24 acquires the structural feature similarity between the first video data and the second video data from the structural feature similarity data 14a. Similarly, in step S302, the similar video determination unit 24 acquires the important scene feature similarity between the first video data and the second video data from the important scene feature similarity data 16a.

  In step S303, the similar video determination unit 24 determines the first based on the structural feature similarity and the important scene feature similarity between the first video data and the second video data acquired in steps S301 and S302. The similarity between the video data and the second video data is calculated. At this time, the similar video determination means 24 calculates the similarity between the first video data and the second video data by weighting the structural feature similarity and the important scene feature similarity, respectively.

  In step S304, the similar video determination unit 24 records the first video data identifier, the second video data identifier, and the similarity between the first video data and the second video data. Is inserted into the similarity data 17a.

  In step S305, the similar video determination unit 24 refers to the similarity data 17a, and acquires an identifier of video data having a high similarity to the reference video data 11a as an identifier of recommended video data. Further, the similar video determination unit 24 generates similar video list data 18a including the identifier of the acquired recommended video data and stores it in the storage device 10.

Here, the processing of the similar video determination means 24 will be described in detail. The similar video determination means 24 calculates the similarity between the video data A and the video data B from the similarity S ^AB calculated based on the structural feature and the similarity I ^AB calculated based on the important scene feature. A recommendation score R (A, B) of video data B for a user who has watched video data A is calculated.

  The recommendation score R (A, B) is a recommendation score R (A, B) when the value of the structural feature similarity based on the structural feature is large or when the value of the important scene feature similarity based on the important scene is large. ) Is also calculated to be large.

For example, when a recommendation score is calculated by a weighted linear sum of each element of S ^AB and each element of I ^AB , R (A, B) is given by Expression (5).

Here, k _p ^I is the weight of I _p ^{A, B} , and k _q ^S is the weight of S _q ^{A, B.} When it is desired to calculate the recommendation score with emphasis on the important scene feature included in the video data, the value of k _p ^I is set large. When the recommendation score is to be calculated with emphasis on the presentation of video data, the values of I _p ^{A, B} are set large.

Only one of the similarity I ^AB between the video data A and the video data B calculated based on the important scene and the similarity S ^AB between the video data A and the video data B calculated based on the structural features is obtained. The similarity R (A, B) between the video data A and the video data B may be calculated by using it. When the similarity R (A, B) is calculated using only the similarity I ^AB between the video data A and the video data B calculated based on the important scene, the similarity R (A, B) is expressed by Equation (6). Given. When the similarity R (A, B) is calculated using only the similarity S ^AB between the video data A and the video data B calculated based on the structural features, the similarity R (A, B) is expressed by Equation (7). Given in.

The weights k _p ^I and I _p ^{A, B} may be set as appropriate depending on the types of video data stored in the reference video data 11a and the video data storage unit 12. For example, in the case of video data that has not been edited much, such as a home video, it is preferable to use a similarity calculated based on an important scene. In addition, when recommending a video for which it is difficult to extract an important scene, it is preferable to use a similarity calculated based on a structural feature. In other cases, it is preferable to use both the similarity calculated based on the important scene and the similarity calculated based on the structural features.

  The similar video determination means 24 calculates a recommendation score from the similarity R (A, B). For example, a value obtained by converting the similarity R (A, B) into a 100% notation may be output as the recommendation score. In order to inform the user of the recommendation level in an easy-to-understand manner, the recommendation score may be written with a symbol or the like. For example, when the similarity is high and the recommendation score is high, the similar video determination unit 24 describes the recommendation score as “★★★” or the like, and when the similarity is relatively low and the recommendation score is low, the recommendation score is “★” or the like. May be written.

  When the user accesses the video data A, the similar video determination unit 24 selects a predetermined number of video data in order from video data having a larger value of R (A, B) among arbitrary video data X in the video data storage unit 12. Similar video list data 18a is created for.

  Next, the similar video information output means 25 will be described. The similar video information output means 25 reads the similar video list data 18a and the video management data 19a from the storage device 10, and acquires information such as the title and content of these video data based on the identifier of the recommended video data. The similar video determination means 24 associates the identifier of video data to be recommended with these titles and outputs them. In the example referred to in FIG. 2, the case where the image is displayed on the display device 30 of the similar video output device 1 will be described. The similar video information output means 25 may be displayed on a display device of another computer via a communication control device (not shown) and a communication network, for example.

  Here, each process of the structural feature similarity calculation unit 22 and the important scene feature similarity calculation unit 23 may be executed when the reference video data 11a is input by the reference video data acquisition unit 21, or may be executed in advance. It may be executed. For example, if the structural feature similarity data 14a is calculated in advance by the structural feature similarity calculator 22 and the important scene feature similarity data 16a is calculated in advance by the important scene feature similarity calculator 23, similar video determination means 24 can easily generate the similarity data 17a and the similar video list data 18a. Further, the similarity data 17a may be calculated in advance by the similar video determination means 24. Thus, by calculating similarity data in advance, the processing time required for outputting similar video information can be shortened.

  As described above, according to the similar video output device 1 according to the embodiment of the present invention, similar video data can be specified based on the similarity based on the structural feature of the video data and the similarity based on the important scene feature. it can.

  The similarity based on the structural feature is the similarity of the feature related to the presentation of the video data. Similarities based on structural features are extracted by extracting features related to the production of the video, such as cut splits and music sections, and analyzing where each feature appears in the time axis, such as the first half or the second half of the video. The degree is calculated.

  The similarity based on the important scene feature is a feature similarity related to a scene that tends to remain in the viewer's impression. The similarity based on the important scene feature is calculated by analyzing features such as color only for the important scene that tends to remain in the viewer's impression.

  In general, it is considered that a video having similar characteristics regarding the presentation of video data or a characteristic in an important scene has a similar impression on the video data of the viewer. Therefore, the similar video output device 1 according to the embodiment of the present invention calculates, for each video data, a structural feature quantity related to the rendering of the video data and an important scene feature quantity in the important scene, and these feature quantities are used as the reference video. Video data similar to the data 11a is output as recommended video data. Thereby, the similar video output device 1 according to the embodiment of the present invention can recommend video data having a similar impression to the viewer's video data to the reference video data 11a.

  The structural feature similarity calculating means 22 extracts structural features related to performance from the video data, analyzes how often each feature appears on the time axis of the video data, and analyzes the structural feature similarity. Calculate the degree. Based on such structural feature similarity, the degree of similarity of video data based on information such as whether structural features often appear in the first half, the second half, or evenly throughout. Is calculated. In video data such as movies, an effect of increasing the number of cuts is given to give a sense of urgency. For this reason, it is considered that video data having similar characteristics related to presentations also have similar impressions to viewers, and the similar video output device 1 uses videos having similar impressions to viewers based on structural feature similarity. Can recommend data.

  Further, the structural feature similarity calculating means 22 divides the video data into several sections, and calculates the similarity of the feature vectors whose elements are the appearance frequency of the structural features such as cut division and music sections in each section. calculate. This makes it possible to analyze information related to performance such as increasing or decreasing the frequency of cuts to give an impression such as a sense of urgency, and to calculate the similarity of video data based on structural features.

  The important scene feature similarity calculating unit 23 calculates the similarity of the video data based on the important scene features that are likely to remain in the viewer's impression in the video data. Features used as important scene features include color, motion, and acoustic features. Accordingly, the similar video output device 1 can recommend video data having a similar impression that the video data gives to the viewer.

  For example, in a horror movie, there are a bright scene and a dark scene, but in a horror movie, a dark scene is generally considered as an important scene that tends to remain in an impression. Therefore, the important scene feature similarity calculating unit 23 analyzes the features limited to the important scenes, so that the similar video output device 1 is similar to video data from the viewpoint of “there are many dark scenes in the important scenes”, for example. Can be found. When the features of a scene that is likely to remain in the viewer's impression are similar, it is considered that the impression given to the viewer by the video data is also similar, and the similar video output device 1 gives the viewer based on the similarity of the important scene feature. Can recommend videos with similar impressions.

  Further, the important scene feature similarity calculating means 23 analyzes the features limited to the important scenes and calculates the similarity. As a result, since the number of scenes to be analyzed is reduced as compared with the case of calculating the similarity of the entire video, the important scene feature similarity calculating unit 23 can reduce the amount of calculation processing for calculating the similarity.

  As described above, the similar video output device 1 according to the embodiment of the present invention analyzes the structural characteristics of video data, the characteristics of important scenes of video data, or both, thereby analyzing the similarity of impressions of video data. The degree can be calculated. Thereby, the similar video output device 1 according to the embodiment of the present invention can recommend video data having an impression similar to that of the reference video data 11a.

(Modification)
In the embodiment of the present invention, the case where there is one reference video data has been described. However, in the modification, a case where a plurality of reference video data is designated will be described.

  In the modification, the similar video determining unit 24a of the similar video output device 1 recommends by adding the similarities of the plurality of reference video data when the plurality of input reference video data is input. The video can be determined.

For example, given N _A video sets A = {A _i | i = 1 to N _A } as input, R ′ (A, B) shown in Equation 8 is changed to R ( Used in place of A, B). Thereby, the similar video determination means 24a can generate the similar video list data 18b according to the modification.

  As described above, in the modified example of the present invention, information on video data similar to the reference video data is output based on the structural feature similarity and the important scene feature similarity for a plurality of reference video data. can do.

(Other embodiments)
As described above, the best mode for carrying out the invention and the modifications thereof have been described. However, it should not be understood that the description and drawings constituting a part of this disclosure limit the present invention. From this disclosure, various alternative embodiments, examples, and operational techniques will be apparent to those skilled in the art.

  For example, the similar video output device described in the best embodiment of the present invention may be configured on one piece of hardware as shown in FIG. 2, or a plurality of pieces of hardware may be used depending on the functions and the number of processes. It may be configured above. Further, it may be realized on an existing information processing system.

  It goes without saying that the present invention includes various embodiments not described herein. Therefore, the technical scope of the present invention is defined only by the invention specifying matters according to the scope of claims reasonable from the above description.

DESCRIPTION OF SYMBOLS 1 Similar video output apparatus 10 Storage apparatus 11 Reference video data storage part 12 Video data storage part 13 Structural feature-value data storage part 14 Structural feature similarity data storage part 15 Important scene feature-value data storage part 16 Important scene feature similarity Data storage unit 17 Similarity data storage unit 18 Similar video list data storage unit 19 Video management data storage unit 20 Central processing controller 21 Reference video data acquisition unit 22 Structural feature similarity calculation unit 23 Important scene feature similarity calculation unit 24 Similar video determination means 25 Similar video information output means 30 Display device 221 Structural feature analysis means 222, 233 Similarity calculation means 231 Important scene extraction means 232 Video feature analysis means

Claims (11)

A similar video output method for outputting similar video data,
For each of a plurality of video data stored in the video data storage unit, a time at which a structural feature appears for each section obtained by dividing the video data is calculated as a structural feature amount. A structural feature similarity calculating step for calculating the structural feature similarity between the data and outputting the structural feature similarity data;
A similar video determination step of extracting video data similar to reference video data from the video data storage unit based on the structural feature similarity data;
A similar video output method comprising: The similar image according to claim 1, wherein the structural feature is at least one of a color, a motion, an acoustic feature, a cut division, a music segment, a speech segment, and a telop segment in the segment. output method. For each of the plurality of video data, an important scene feature amount included in the video data is calculated as an important scene feature amount, and an important scene feature similarity between the video data is calculated based on the important scene feature amount. And an important scene feature similarity calculating step for outputting important scene feature similarity data,
The similar video determining step extracts video data similar to the reference video data from the video data storage unit based on the structural feature similarity data and the important scene feature similarity data. The similar image output method according to claim 1 or 2. The similar video output method according to claim 3, wherein the important scene feature is at least one of a color, a motion, and an acoustic feature of the important scene. A similar video output method for outputting similar video data,
For each of a plurality of video data stored in the video data storage unit, a feature amount of an important scene included in the video data is calculated as an important scene feature amount, and an important scene between the video data is calculated based on the important scene feature amount. An important scene feature similarity calculating step for calculating feature similarity and outputting important scene feature similarity data;
A similar video determination step of extracting video data similar to reference video data from the video data storage unit based on the important scene feature similarity data;
A similar video output method comprising: A similar video output device that outputs similar video data,
A video data storage unit storing a plurality of video data;
For each of the plurality of video data, the appearance time of the structural feature for each section obtained by dividing the video data is calculated as the structural feature amount, and the structural feature similarity between the video data is calculated based on the structural feature amount. A structural feature similarity calculating means for calculating a degree and outputting structural feature similarity data;
Similar video determination means for extracting video data similar to reference video data from the video data storage unit based on the structural feature similarity data;
A similar video output device comprising: The similar image according to claim 6, wherein the structural feature is any one or more of a color, a motion, an acoustic feature, a cut division, a music section, a speech section, and a telop section in the section. Output device. For each of the plurality of video data, an important scene feature amount included in the video data is calculated as an important scene feature amount, and an important scene feature similarity between the video data is calculated based on the important scene feature amount. , Further comprising important scene feature similarity calculating means for outputting important scene feature similarity data,
The similar video determining means extracts video data similar to the reference video data from the video data storage unit based on the structural feature similarity data and the important scene feature similarity data. The similar image output device according to claim 6 or 7. The similar video output device according to claim 8, wherein the important scene feature is at least one of a color, a motion, and an acoustic feature of the important scene. A similar video output device that outputs similar video data,
A video data storage unit storing a plurality of video data;
For each of the plurality of video data, an important scene feature amount included in the video data is calculated as an important scene feature amount, and an important scene feature similarity between the video data is calculated based on the important scene feature amount. , Important scene feature similarity calculating means for outputting important scene feature similarity data;
Similar video determining means for extracting video data similar to reference video data from the video data storage unit based on the important scene feature similarity data;
A similar video output device comprising:   A similar video output program for causing a computer to execute the steps according to any one of claims 1 to 5.

Images