JP2007200249A - Image search method, device, program, and computer readable storage medium - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To improve similar image searching accuracy in image search aimed at an image so called a shot and to realize speedup by processing based on a label type alone. <P>SOLUTION: An image inputted when content of the inputted image is changed is divided into shots, and from the shots, composition information (a position and a dimension of the area, complexity of a pattern, a color feature, a movement amount and the like) serving as feature quantities of a main area is extracted for classifying the shots based on the composition information. According to the classification category result, similarity between an inputted example image section and the shot is calculated, and based on the similarity, an image section is outputted. <P>COPYRIGHT: (C)2007,JPO&INPIT

Description

  The present invention relates to a video search method and apparatus, a program, and a computer-readable recording medium, and in particular, a video segment feature description method focusing on a video production method, and a video segment composed of a plurality of shots using the same. The present invention relates to a video search method and apparatus, a program, and a computer-readable recording medium.

  In recent years, with the spread of digital broadcasting and cable television, it has become easier for individuals to obtain video data of various genres. In addition, with the advancement of HDDs and storage media, a large amount of video data is stored on a hard disk, DVD, etc., and a viewing style has become established at a time convenient for the individual.

  On the other hand, as the amount of recorded and accumulated video data increases, searching for a video section that has one meaning (action) in a plurality of shots, such as a specific event scene, from an enormous amount of video data. Have difficulty. Accordingly, a technique for analyzing a large amount of video content and searching for a video section desired by a user has attracted attention.

  Many conventional video segment search methods target a predetermined event and search for an event with high accuracy by learning in advance features included in the defined event (for example, non-patent literature). 1).

  As an approach that does not use learning, there is a technique based on examples used in conventional image retrieval. This method is a method in which a video similar to the video desired by the user is prepared in advance, and the video desired by the user is searched from the feature amount included in the video. Since the method based on the example does not fix the event, it has a feature that it can flexibly respond to various search requests (various events).

  One of the methods based on the examples is a video search method that focuses on time-series features of video feature values. For example, using the DC component of each Intra (intra-frame code) frame of an MPEG stream, the average color information of the frame is obtained, placed in a three-dimensional color space, the trajectory is projected on the time axis, and converted into waveform information. In some cases, a video image search is realized by enlarging / reducing and comparing waveforms (for example, see Non-Patent Document 2).

Further, as an example of the feature amount of the video, there is a video search method that focuses on the image feature amount of the video frame. For example, there is a technique in which a video is divided into shots, the video is represented by a set of representative frames of each shot, and the video is searched by comparing the representative frame sets (see Non-Patent Document 3).
Takahiro Mochizuki, Satoshi Suganuma, Nobuyuki Yagi, "Scene Patterning and Baseball Indexing Using Hidden Markov Models" IEICE Technical Report, PRMU2005-28, pp.37-42, 2005 Katsunao Takahashi, Hideyoshi Tominaga, Maki Sugiura, Mayo Yokoi, Nobuyoshi Terashima “Highly Efficient Video Retrieval Method Using Characteristic Image Prints”, Transactions of the Institute of Image Electronics Engineers of Japan, Vol.29, No.6, 2000 Seiji Hotta, Kohei Inoue, Kiichi Urahama “Similar search of video based on distance between image sets”, Journal of the Institute of Image Information and Television Engineers, Vol.54, No.11, 2000

  In the conventional learning-based method (Non-Patent Document 1), it is necessary to prepare learning data in advance. In general, it is difficult to collect learning data automatically, and in reality, human labor is required. Therefore, as the video genre and search event types increase, the amount of work and the cost of hiring workers become a problem. Further, there is a problem that it is not possible to immediately respond to the user's various search needs other than predetermined events.

  On the other hand, the example-based method (Non-Patent Document 2) uses time series features of average colors and is suitable for searching for the same video (for example, CM) as the example video, but is similar to the example video. When searching for a video (for example, a soccer PK scene), the video is not necessarily shot with the same length of time as that of the example video, so that there is a problem that the similarity of time-series features decreases. In addition, since the entire frame is represented by an average color, there is a problem of identification accuracy in that different frames are identified with human eyes.

  Further, in the method using the image feature amount of the video frame (Non-patent Document 3), the shot is represented by one representative frame, the motion information of the object in the shot is lost, and the moving object peculiar to the video is displayed. There is a problem that the search cannot be performed.

  Further, as the feature of the moving image is described in detail, there is a problem that the time required for comparing the feature amount between videos increases and it takes more time to return the search result.

  The present invention has been made in view of the above points, and improves the search accuracy of similar videos in a video search for a video of about 10 seconds called a shot, and can increase the speed by processing only the label type. It is an object of the present invention to provide a search method, apparatus, program, and computer-readable recording medium.

  FIG. 1 is a diagram for explaining the principle of the present invention.

The present invention (Claim 1) is a video search method for searching for a video section similar to the video section exemplified by the user,
A shot dividing step (step 1) in which the shot dividing means divides the video into partial video sections (hereinafter referred to as shots) at the point where the content of the inputted video has changed, and stores it in the storage means;
A composition information extracting unit that extracts composition information that is a feature quantity of a main region from a shot and stores the extracted composition information in a storage unit (step 2);
A shot classification step (step 3) in which the shot classification unit classifies the shots based on the composition information and stores the classification category result in the storage unit;
A similarity calculation step (step 4) in which the similarity calculation means calculates the similarity between the input example video section and the shot based on the classification category result of the shot obtained in the shot classification step;
The search result display means performs a search result display step (step 5) for outputting the video section based on the similarity.

In the present invention (Claim 2), in the shot classification step (Step 3),
The unclassified input shot is compared with the classified shot, and if it is less than a preset threshold, the unclassified input shot is classified into an existing shot category, and if it is equal to or greater than the threshold, it is classified into a new classification category.

  FIG. 2 is a principle configuration diagram of the present invention.

The present invention (Claim 3) is a video search device for searching for a video section similar to the video section exemplified by the user,
A shot dividing unit 102 that divides the video into partial video sections (hereinafter referred to as shots) at the point where the content of the input video has changed, and stores it in the storage unit 108;
Composition information extraction means 103 that extracts composition information that is a feature quantity of a main region from a shot and stores it in the storage means 108;
A shot classification unit 104 that classifies the shots stored in the storage unit 108 based on the composition information and stores the classification category result in the storage unit;
Similarity calculation means 106 for calculating the similarity between the input example video section and shot partial video section based on the shot classification category result obtained by the shot classification means 104;
And search result display means 107 for outputting a video section based on the degree of similarity.

Further, the present invention (Claim 4), in the shot classification means 104,
Compare the unclassified input shot with the classified shot, and if it is less than the preset threshold, classify the unclassified input shot into the existing shot category, and if it is above the threshold, classify it into a new classification category. Including.

The present invention (Claim 5) is a video search program for searching for a video section similar to the video section exemplified by the user,
A video search program for causing a computer to function as the video device according to claim 3.

The present invention (Claim 6) is a computer-readable recording medium storing a video search program for searching a video section similar to the video section exemplified by the user,
A computer-readable recording medium storing a program that causes a computer to function as the video device according to claim 3.

  Since many broadcast videos are produced based on video production technology, if the video type is limited, the program structure (scene development, subject composition, etc.) is reproducible. For example, many regular programs that are broadcast weekly or every day at the same time zone have a story that determines the scenes to be filmed by specific broadcast writers and directors and the development of the scenes. The nature is high.

  In sports programs, the video is reproducible due to the nature of sports performed based on rules determined by the sports themselves. For example, a similar pitching scene repeatedly appears in any program of a baseball program.

  In addition, the skilled cameraman has the skill to shoot appropriately according to the shooting scene and development, so in the same program structure, the position and size of the subject on the screen, camera operation etc. are also reproducible Be looked at.

  Therefore, in the same type of broadcast video produced by broadcast writers, directors, skilled cameramen, etc., similar scenes repeatedly occur in the program structure, and in those similar scenes, the composition of the subject is reproducible. I can say that.

  The present invention focuses on such video production technology, and models video based on features such as subject placement, size of the subject, and amount of movement of the subject on the screen, in units of shots with varying image contents. To do. As a result, it is not a strict similarity conscious of specific conditions (same time length, same frame) as in Non-Patent Document 2, but a rough similarity such as an image feature amount in a specific frame as in Non-Patent Document 3. There is no video segment search method that gives similarity by modeling video based on a shot composition set based on video shooting technology. In the present invention, by using a shot as a unit, robustness can be given in the time direction, and the analysis accuracy of the image content can be enhanced by the composition information. In addition, it focuses on TV program video production technology, and in addition to not requiring prior learning, it can be applied to any video section. Note that these features are not limited to a specific video such as a baseball video.

  Further, the present invention extracts the composition information composed of both the time axis and the image space axis, thereby improving the accuracy of analyzing the shot contents. In addition, shots are classified into several categories based on this composition information, labeling is performed, and the similarity of video is defined as the similarity of a set of shot labels, greatly increasing the cost of comparison between video sections. Therefore, the video search method can search a desired video section at high speed.

  The present invention is a similar video segment search technique based on a process in which a set of camera operations and compositions included in a video segment are similar if the story of the program is the same. In particular, it can be expected to be effective for images in which similarity is seen in camera operation and composition, such as in sports competitions.

  In addition, unlike conventional event detection, there is a feature that prior learning is not required.

  As described above, according to the present invention, instead of using all the features of the entire image for each shot whose image content changes, the main region is selected and extracted as a composition feature of the shot, and applied to video search. . Thereby, the time length of the video section can be made flexible, and the identification accuracy can be increased by the composition of the screen. It focuses on video production technology for TV programs and is not limited to a specific video such as a baseball video.

  In addition, in the present invention, shot category classification and labeling are performed on videos in the video database in advance, and comparison between videos having a large amount of information is replaced with calculation of similarity of a small number of shot label sets. Similar video sections can be searched at high speed. In addition, since shot classification is performed based on composition information, a searched video section is composed of similar shooting compositions, and can be expected to be closer to human subjectivity. In addition, it is possible to suppress erroneous searches by filtering search objects using bibliographic data of video.

  Hereinafter, embodiments of the present invention will be described with reference to the drawings.

  FIG. 3 shows a configuration of a video search apparatus according to an embodiment of the present invention.

  The video search apparatus shown in the figure divides video data into partial video sections called shots, classifies shots based on composition information, defines video similarity based on similarity of shot classification labels, and allows the user to The desired video section is searched.

  In the present embodiment, the video search apparatus 100 includes a video reception unit 101, a shot division unit 102, a composition information extraction unit 103, a shot classification unit 104, a search condition input unit 105, a similarity calculation unit 106, and a search result display unit 107. The video database 108 is assumed to be configured.

  In addition, an external information storage device 110 and a video playback device 111 are connected to the video search device 100.

  The video reception unit 101 acquires a video from the outside via an external information storage unit 110 such as a video distribution server or a video reception device (for example, a tuner), inputs the video to the shot division unit 102, and stores the video in the video database 108. . At this time, bibliographic information such as a title and a video genre may be acquired together with the video using metadata previously associated with the video or an EPG (electronic program guide) service, and stored in the video database 108.

  The shot division unit 102 detects a point where the configuration of the video content, which is called a cut point, changes greatly from the video input from the video reception unit 101, and divides the video for each cut point. A shot is defined as a time interval between adjacent cuts. The shot dividing unit 102 inputs the obtained shot to the composition information extracting unit 103. Further, it is stored in the shot name name, start time S_Time, and end time E_Time fields of the video management table of the video database 108 shown in FIG. The shot division unit 102 can use, for example, a shot division method disclosed in JP-A-11-018028.

  The composition information extraction unit 103 obtains composition information that characterizes a shot for each shot input from the shot division unit 102. Specifically, the main area from the shot (represents the main area constituting the screen. For example, using a general technique used in image area division, each frame is divided into a plurality of areas, and the head of the shot is The region from the frame to the final frame can be determined), and the feature amount such as the position, size, pattern complexity, color feature, and movement amount is extracted as composition information. The calculated composition information is input to the shot classification unit 104 and stored in the composition information (region 1 to region i) of the video management table of the video database 108.

  The shot classification unit 104 performs category classification of shots based on the similarity of the composition information of each shot input from the composition information extraction unit 103. Further, the shot classification unit 104 stores the classification label in the shot label [Label] of the video management table of the video database 108 based on the category classification result of the shot, and the label shown in FIG. Store in the management table (details will be described later). As described above, the video and its feature amount are accumulated in the video database 108.

  For example, as shown in FIG. 6, the video playback device 111 is used by a user to view video stored in the video database 108. When the user finds a video section to be searched, the user transitions to the search condition input unit 105 (for example, FIG. 7) by pressing the “Search” button at the lower right of FIG.

  The search condition input unit 105 determines an example video section having a role of transmitting the user's search intention to the system, and inputs the example video section to the similarity calculation unit 106. In order to show similar video sections, for example, as shown in FIG. 7, representative images of a plurality of shots (for example, the heads of shots) that move forward and backward with respect to the time when the “search” button is pressed in the video playback device 111 are used as a reference. Image) may be displayed, and an example video section may be determined by selecting radio buttons of in (start point) and out (end point) at the bottom of each representative image.

  At this time, when bibliographic information such as EPG is held together with the video in the video database 108, the range of the target video is specified using the bibliographic information of the video itself for the purpose of improving the video search accuracy. can do.

  The similarity calculation unit 106 uses the similarity between the example video segment input from the search condition input unit 105 and the video segment stored in the video database 108 as the similarity of the shot label set based on the composition information, and uses a predetermined formula And is input to the search result display unit 107. For example, an expression defined below may be used as the similarity calculation method.

検索結果表示部１０７は、類似度算出部１０６で算出される類似度に基づき、類似度の高い順に、映像データベース１０８から該当する類似映像区間を取得し、ユーザに結果を提示する。例えば、検索結果の一覧性を高めるため、１映像区間につき、各ショットの代表フレームを複数枚表示してもよい。

The search result display unit 107 acquires corresponding similar video sections from the video database 108 in descending order of similarity based on the similarity calculated by the similarity calculation unit 106 and presents the result to the user. For example, a plurality of representative frames for each shot may be displayed for one video section in order to improve the list of search results.

  Next, the shot classification unit 104 will be described.

  FIG. 8 is a flowchart of the operation of the shot classification unit according to the embodiment of the present invention.

  Step 11) The composition information of the unclassified shot input from the composition information extraction unit 103 is received and stored in a temporary memory (not shown).

  Step 12) The label management table of the video database 108 is acquired, and shots of the label management table are stored in a temporary memory (not shown).

  Step 13) The composition information of the shot received in Step 11 and the composition information of the shot acquired in Step 12 are read from a temporary memory (not shown), and distances are calculated for all shot groups. At this time, the distance between the shots is, for example, the sum of the distances between the regions having the closest feature vector distances, and the distance scale includes a sum of absolute values of the differences and a weighted square distance sum defined by the following expression. It is possible to use.

ステップ１４） ステップ１３で求めた距離が、予め定めた閾値以下のショットの組を、同一のカテゴリに分類する。

Step 14) A set of shots in which the distance obtained in Step 13 is equal to or less than a predetermined threshold is classified into the same category.

  Step 15) For each category classified in Step 14, it is determined whether or not a shot of the label management table of the video database 108 is included. If “yes”, the process proceeds to step 16, and if “no”, the process proceeds to step 18.

  Step 16) For each category, a shot that has already been given a label within the same category with respect to a shot that has not been given a label within the category (in the present embodiment, it will be described as shot Z) (this embodiment) The label D of the shot management D of the video database 108 is stored, and the classification label D is stored in the Label column of the shot Z of the video management table of the video database 108.

  Step 17) Using all shots belonging to the D label among the shots in the video management table of the video database 108, an average value of the composition information is obtained and replaced with the composition information of the shot D in the label management table.

  Step 18) A new label is given and stored in the video management table and the label management table.

  FIG. 9 is a flowchart of a procedure for generating video feature information (video management table, label management table) which is a clue to search in an embodiment of the present invention.

  Step 101) The video reception unit 101 acquires a video from the outside via an external information storage device such as a video distribution server or a video reception device, and stores the video in a video storage area of the video database 108.

  Step 102) The video data acquired in Step 101 is divided into shots by the shot division unit 102. The time and shot label of the first and last frames of the shot obtained in this way are stored in the video management table and label management table of the video database 108.

  Step 103) For each shot divided in Step 102, the composition information extraction unit 103 extracts the shot composition information as shot information and stores it in the video management table and label management table of the video database 108.

  Step 104) Using the composition information calculated in step 103, the shot classification unit 104 classifies the shots.

  An example of the video management table of the video database 108 obtained as described above is shown in FIG. 4, and an example of the label management table is shown in FIG. Further, FIG. 10 shows an image diagram in which a label is given to the video in the above flowchart.

  Next, a description will be given of processing for searching for a video based on the shot classification result (shot classification label) obtained in advance in the video search apparatus 100.

  FIG. 11 is a flowchart of processing for searching for similar videos according to an embodiment of the present invention.

  Step 201) The search condition input unit 105 designates an exemplary video section representing the user's search intention. At this time, detailed conditions of the search target video may be designated. For example, conditions such as video genre, title name, recording date and time.

  Step 202) The similarity calculation unit 106 refers to the Label column of the video management table corresponding to the example section video from the video database 108 and obtains a label set of shots. Further, a label set of candidate videos to be searched is acquired from the video database 108 in time series.

  Step 203) The similarity calculation unit 106 determines the search window width based on the label set of shots in the example video section. The search window width is for acquiring a searched video section from the search target videos in order to compare the video sections. As a method for determining the search window width, for example, the number of shot labels included in the example video section or the time width of the example video section may be used.

  Step 204) The similarity calculation unit 106 sets a search window width from the top of the label set obtained in Step 202 as shown in FIG. 12, applies the search window determined in Step 203, Get a set of shot labels.

  Step 205) The similarity calculation unit 106 calculates the similarity based on the shot label set of the example video section obtained in Step 203 and the shot label set to be searched obtained in Step 204.

  Step 206) It is determined whether or not the input search target video is the last. If “Yes”, the process proceeds to Step 207, and if “No”, the process proceeds to Step 204.

  Step 207) The search result display unit 107 displays the video sections on the display means in descending order of the calculated similarity.

  According to the present invention, the operation of the above-described video search apparatus can be constructed as a program, installed in a computer used as the video search apparatus, executed, or distributed via a network.

  Further, the constructed program can be stored in a portable storage medium such as a hard disk, a flexible disk, or a CD-ROM, and can be installed or distributed in a computer.

  The present invention is not limited to the above-described embodiment, and various modifications and applications can be made within the scope of the claims.

  The present invention is applicable to a technique for detecting a video section. For example, the present invention can be applied to a search for a video section in which similarity is seen in the operation and composition of a camera, such as sports competition, which has reproducibility in the configuration of a broadcast video.

It is a figure for demonstrating the principle of this invention. It is a principle block diagram of this invention. It is a block diagram of the video search device in one embodiment of the present invention. It is an example of the image | video management table of the image | video database in one embodiment of this invention. It is an example of the label management table of the video database in one embodiment of the present invention. It is an example of the viewing-and-listening video by the video reproduction apparatus in one embodiment of this invention. It is an example of a screen which selects the example video section in one embodiment of the present invention. It is a flowchart of operation | movement of the shot classification | category part in one embodiment of this invention. It is a flowchart of the procedure which produces | generates the image | video feature information (an image | video management table, a label management table) in one embodiment of this invention. It is an image figure which provides a label to the image | video in one embodiment of this invention. It is a flowchart of the process which searches the similar image | video in one embodiment of this invention. It is an example which calculates | requires the shot label set which calculates | requires a to-be-searched area from the search object image | video in one embodiment of this invention.

Explanation of symbols

DESCRIPTION OF SYMBOLS 100 Image | video search device 101 Image | video reception part 102 Shot division | segmentation means, Shot division | segmentation part 103 Composition information extraction means, Composition information extraction part 104 Shot classification | category means, shot classification | category part 105 Search condition input part 106 Similarity degree calculation means, Similarity degree calculation part 107 Search result display means, search result display unit 108 Video database 110 External information storage device 111 Video playback device

Claims (6)

A video search method for searching for a video segment similar to the video segment exemplified by the user,
A shot dividing step in which the shot dividing means divides the video into partial video sections (hereinafter referred to as shots) at the point where the content of the input video has changed, and stores it in the storage means;
Composition information extracting means extracts composition information that is a feature quantity of a main area from the shot, and stores it in the storage means;
A shot classification step in which the shot classification unit classifies the shots based on the composition information and stores a classification category result in the storage unit;
Similarity calculation means, based on the classification category result of the shot obtained in the shot classification step, a similarity calculation step of calculating the similarity between the input example video section and the shot;
A search result display means for outputting a video section based on the similarity, a search result display step,
The video search method characterized by performing. In the shot classification step,
Compare the unclassified input shot with the classified shot, and if it is less than a preset threshold, classify the unclassified input shot into an existing shot category, and if it is above the threshold, classify it into a new classification category.
The video search method according to claim 1. A video search device for searching for a video section similar to the video section exemplified by the user,
Shot dividing means for dividing the video into partial video sections (hereinafter referred to as shots) at a point where the content of the input video has changed, and storing the divided video in a storage means;
Composition information extraction means for extracting composition information, which is a feature amount of a main area, from the shot and storing it in a storage means;
Shot classification means for classifying the shots stored in the storage means based on the composition information, and storing a classification category result in the storage means;
Based on the classification category result of the shot obtained by the shot classification means, similarity calculation means for calculating the similarity between the input example video section and the shot;
Search result display means for outputting a video section based on the similarity;
A video search apparatus characterized by comprising: The shot classification means includes
A means for comparing an unclassified input shot with the classified shot, and classifying an unclassified input shot into an existing shot category if it is less than a preset threshold, and classifying it into a new classification category if it is greater than or equal to the threshold including,
The video search device according to claim 3. A video search program for searching for a video segment similar to the video segment exemplified by the user,
Computer
5. A video search program which functions as the video device according to claim 3 or 4. A computer-readable recording medium storing a video search program for searching for a video section similar to the video section exemplified by the user,
Computer
A computer-readable recording medium storing a program for functioning as the video apparatus according to claim 3.

Images