JP2014137637A - Image processor and image processing program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To properly acquire the feature information of each scene included in a video.SOLUTION: An image processor for extracting the feature information of each scene included in a video comprises: sampling acquisition means for sampling a predetermined frame image from a sample video; block feature information generation means for dividing each frame image acquired by the sampling acquisition means respectively for one or more scales, and for generating the feature information of each of those divided blocks; scene division means for dividing a scene from an object video from which the feature information is generated; and histogram generation means for generating a histogram based on the appearance ratio of each block by using the blocks acquired by the block feature information generation means for each scene divided by the scene division means.

Description

  The present invention relates to an image processing apparatus and an image processing program for a frame image included in a video.

  Conventionally, it has become possible to store a large amount of video on a hard disk due to advances in recording technology. In addition, with the development of the network environment, it is possible to access a wide variety of videos through a communication network such as the Internet. Therefore, a search technique for quickly searching for a desired video is useful.

  Here, as a general video search technique, keyword search related to video content can be cited (see, for example, Patent Document 1). However, when the amount of video becomes enormous, it is very expensive to assign appropriate keywords and text information to each scene. Moreover, the given information includes a sense blur due to a difference in workers, and there is a possibility that the search accuracy is lowered. Therefore, research on “visual search” based on the similarity of image features has been actively conducted as an approach different from keyword search. The conventional visual search is a search in shot units divided by camera switching, and “similarity of representative frame images” is used as a shot similarity as it is for high-speed search.

Tomoki Masuda, Daisuke Yamamoto, Shigeki Ohira, Katashi Nagao, "Video Scene Retrieval Using Online Video Annotation", New Frontiers in artificial Intelligence, Awarded Papers, LNAI 4914, Springer-Verlag, pp. 54-62 (2008)

  However, since the above-described search by shot unit may satisfy only a part of the search intention, a search mechanism using “scene” composed of a plurality of shots as a unit is required. Further, in the search by scene unit, the “look” of the frame image in the middle of the scene may be significantly different from the representative frame image. Therefore, it is difficult to position one frame image as a “scene representative”.

  For example, an approach using a plurality of images (for example, representative images of all shots) as a scene representative is also conceivable, but in this case, in order to obtain the similarity between scenes, Since the similarity calculation is required, the calculation cost becomes very high.

  The present invention has been made in view of the above-described problems, and an object thereof is to provide an image processing apparatus and an image processing program for appropriately acquiring feature information for each scene included in a video.

  An image processing apparatus according to an aspect of the present invention is obtained by a sampling acquisition unit that samples a predetermined frame image from a sample video in the image processing apparatus that extracts feature information of each scene included in the video, and the sampling acquisition unit. Block feature information generating means for generating feature information for each divided block and dividing each frame image by one or a plurality of scales, and scene division for dividing a scene from a target video for generating the feature information And a histogram generating means for generating a histogram based on the appearance ratio for each block using the block obtained by the block feature information generating means for each scene divided by the scene dividing means.

  An image processing program according to an aspect of the present invention is an image processing program for causing a computer to execute image processing for extracting feature information of each scene included in a video. Sampling acquisition means for sampling the frame image, block feature information generation means for generating feature information for each divided block for each frame image obtained by the sampling acquisition means, and generating the feature information for each divided block; Based on the appearance ratio of each block using the block obtained by the block feature information generating means for each scene divided by the scene dividing means for dividing the scene from the target video for generating information Histogram generator for generating histograms To function as.

  According to the present invention, it is possible to appropriately acquire feature information for each scene included in a video.

It is a conceptual diagram of a multiscale image piece word histogram. It is a figure which shows the example of a block image in this embodiment. It is a figure which shows an example of a function structure of an image processing apparatus. It is a flowchart which shows an example of the production | generation process of a multiscale image piece word. It is a figure which shows the flow of a production | generation of an image fragment word. It is a flowchart which shows an example of a multiscale image piece word histogram production | generation process. It is a figure which shows the flow of a production | generation of a multiscale image piece word histogram. It is a flowchart which shows an example of a search process. Distance is a diagram showing an example of calculation of D _i. It is a figure which shows 12 types of scenes used as a query, and the content of the correct video set about each. It is a figure which shows the rough calculation example of a relevance degree. It is a figure which shows an example of the search result in this embodiment. It is a figure which shows an example of the comparison method. It is a figure which shows the comparative example of an experimental result. It is a figure which shows an example of an accuracy comparison.

<About the present invention>
The present invention acquires feature information for a video (for example, for each scene) using a plurality of frame images included in the video. Specifically, a histogram of image fragment words having one or a plurality of different image sizes (hereinafter referred to as “multi-scale”) for each frame image (multi-scale image fragment word histogram, Histogram of Multi-scale Image Piece Word). Hereinafter, the scenes are classified using feature information based on “H-MIPW” as necessary.

  An image piece is, for example, each block image obtained by dividing one frame image by dividing it into a predetermined image size. The image size (scale) may be, for example, a square or other shapes. The word is, for example, predetermined feature information such as a reference vector, but is not limited thereto. H-MIPW is based on a still image classification method based on, for example, the type and appearance ratio (frequency) of a block image, and expands the block size to a multi-scale and a moving image feature.

  Here, FIG. 1 is a conceptual diagram of a multiscale image fragment word histogram. FIG. 2 is a diagram illustrating an example of a block image in the present embodiment. In the example of FIG. 1, a frame image sampled from a predetermined scene is divided into blocks for each of one or more types of image sizes, and a multiscale image fragment word histogram (H− MIPW) is generated, and how many types of block images are present in the scene (appearance ratio) is acquired.

  Here, the type of block image has a strong causal relationship with the content (subject) being shown. For example, as shown in FIG. 2, the contents of the video such as “sky”, “mountain, forest”, “sunset”, and the like can be acquired from each block obtained by dividing the frame image. Therefore, the above-described H-MIPW can be considered as one of the features that comprehensively express the contents of the scene. In the present embodiment, scene feature information including a plurality of frame images is acquired based on H-MIPW, for example.

  In the present embodiment, since the scene is represented by one histogram, the similarity can be calculated at high speed by appropriate scene classification, and based on the similarity of the video content using the acquired feature information. Allows scene search. Hereinafter, embodiments in which an image processing apparatus and an image processing program are suitably implemented will be described in detail with reference to the drawings.

<Example of functional configuration of image processing apparatus>
FIG. 3 is a diagram illustrating an example of a functional configuration of the image processing apparatus. The image processing device 10 illustrated in the example of FIG. 3 includes a feature extraction device 20 and a scene search device 30 when roughly classified. Note that the image processing apparatus 10 according to the present embodiment may be configured to include any one of the feature extraction apparatus 20 and the scene search apparatus 30.

  The feature extraction device 20 inputs a set of preparatory (sample) frame images, and generates an image fragment word. In addition, the feature extraction apparatus 20 inputs, for example, a scene (a plurality of frame images) obtained by dividing a video at a predetermined interval (for example, a fixed interval or a video separator), and the above-described image fragment word histogram (for the scene). H-MIPW) is calculated. Based on the H-MIPW similarity obtained by the feature extraction device 20, the scene retrieval device 30 retrieves a scene corresponding to a requested scene such as a user from video information stored in advance. Hereinafter, the feature extraction device 20 and the scene search device 30 will be specifically described.

  The feature extraction device 20 includes a sampling acquisition unit 21, a divided block setting unit 22, an image fragment word generation unit (block feature information generation unit) 23, a scene division unit 24, and a histogram generation unit 25. In addition, the scene search device 30 includes a histogram generation unit 31 and a search unit 32.

The sampling acquisition means 21 samples a frame image at a predetermined interval (for example, T ₁ frame) from a preliminarily accumulated (sample) video set 41 and prepares a preparatory frame image set 42 (P ₁ ,...・, P _{N_P} ) is output. The predetermined interval (T ₁ ) is, for example, a predetermined constant frame interval, but is not limited thereto, and may be, for example, a constant time interval, and each shot ( For example, it may be the first image of video switching).

  The divided block setting unit 22 sets at least one of the size (scale, image size), type, number, and the like of one or a plurality of image pieces (image blocks) generated by the image piece word generation unit 23. To do. For example, the divided block setting means 22 can set the size of the image piece as three types of 4 × 4 pixels, 8 × 8 pixels, and 16 × 32 pixels, but the size and number of the image pieces are limited to this. Is not to be done. The image block setting may be set by the user in advance, and is automatically set according to the resolution of the input video, the genre of the video (for example, news program, sports, drama), etc. May be. Further, the divided block setting means 22 has at least one of the size, type, number, and the like of the image pieces based on the saliency map (Saliency Map) representing the “conspicuous area” in the entire image with respect to the video. May be set. In the saliency map, an area having a property different from that of the surrounding area is extracted as an “area having high saliency (attracting attention)”.

  The image fragment word generation unit 23 generates an image fragment word 43 (W) from the preparation frame image set 42 based on, for example, the conditions set by the divided block setting unit 22. A specific method for generating an image fragment word will be described later.

The scene dividing means 24 is a scene for every predetermined interval (for example, T ₂ frame) with respect to the search target video 44 (V ₁ ,..., V _{N_T} ) designated by the user or the like using the input means. _Are automatically divided to generate a search target scene 45 (S ₁ ,..., S _{N —} S).

Note that the search target video 44 means a target video from which feature information for each scene is extracted, and in the present embodiment, as an example, a search target video in the scene search device 30 described later is shown. Further, the predetermined interval (T ₂ ) is, for example, a preset constant frame interval, but is not limited to this, and may be, for example, a fixed time interval, or may be the first of video segmentation. It may be an interval of frames. Further, the predetermined interval (T ₂ ) may be the same interval as the predetermined interval (T ₁ ) described above, or may be a different interval.

The histogram generation means 25 inputs search target scenes 45 (S ₁ ,..., S _{N —} S), which are scenes obtained by dividing the video at regular intervals, and the image fragment word histogram 46 (H _{1 ,.}・, H _{N_S} ) is output. A specific generation example of the image fragment word histogram 46 in the histogram generation means 25 will be described later.

  As described above, by using the feature of the image piece (block region) unit using the feature extraction device 20, it is possible to extract the feature information of the image with high accuracy that leads to the improvement of the search accuracy, for example.

  The preparation video set 41, the preparation frame image set 42, the image fragment word 43, the search target video 44, the search target scene 45, and the image fragment word histogram 46 are stored in the image processing apparatus 10 or the like. Or may be managed by an external device (for example, a database server) or the like. When managed by an external device, the image processing apparatus 10 is connected in a state where data can be transmitted to and received from the external device via a communication network represented by the Internet or a LAN (Local Area Network), for example. Reading stored data and writing to an external device can be performed.

In the scene search device 30, the histogram generation unit 31 generates a histogram for a requested scene input from a user or the like, similar to the histogram generation unit 25 in the feature extraction device 20 described above. In the example of FIG. 3, a histogram is generated for the search target scene 51 (V _Q ) requested by the user or the like, and an image fragment histogram 52 (H _Q ) for the requested scene is output.

  Based on the image fragment word histogram 52 of the requested scene, the retrieval unit 32 retrieves the same scene by referring to the image fragment word histogram 46 of each scene acquired by the feature extraction device 20 described above, and the retrieval result 53. Is output. For example, the search result 53 may be a scene having a similarity between image pieces equal to or greater than a preset threshold, but is not limited thereto. For example, a predetermined number of scenes may be output in descending order of similarity. The requested scene 51, the image fragment word histogram 52 of the requested scene, and the search result 53 described above may be stored in, for example, a preset storage unit, or may be managed in an external database or the like.

  As described above, in the present embodiment, since the image piece is considered to have a strong correlation with the content in the image, H-MIPW can be an effective moving image feature for scene search based on the similarity of the video content. Therefore, a highly accurate search can be performed on the requested scene, and a scene with high similarity can be acquired.

  Next, a calculation example of the above-described multi-scale image fragment word (hereinafter referred to as “MIPWORD” as necessary) representing the type of block image and a predetermined scene unit H-MIPW will be specifically described.

<Example of Multiscale Image Single Word (MIPWord)>
A method for generating a multi-scale image fragment word (MIPWord) in the image fragment word generation means 23 described above will be described. The MIPWord is generated by using, for example, a preparation video randomly selected from the search target video. FIG. 4 is a flowchart illustrating an example of a multiscale image fragment word generation process. FIG. 5 is a diagram showing a flow of generation of an image fragment word.

In the example of FIG. 4, the image fragment word generation process samples a predetermined frame image from the preparation video set (S01). For sampling, for example, frame images at regular intervals may be acquired, or frame images may be acquired based on video segmentation or the like. Next, in the image fragment word generation process, each sampled frame image is divided into blocks of one or a plurality of scales (S02). In the process of S02, for example, each frame image is divided into blocks at a plurality of scales of scale 1 (n _W1 × n _H1 ),..., Scale N _d (n _WNd × n _HNd ).

  Next, the image fragment word generation process calculates a predetermined feature vector (feature information) for each divided block image (S03). Examples of the predetermined feature vector include a color feature and a texture feature, but are not limited thereto. Other features may be used, and a plurality of feature information may be combined. Examples of the color feature include an RGB average value vector and a hue histogram. The texture features include, for example, a fractal sequence, an edge direction histogram, a CS-LBP (Center Symmetric-Local Binary Pattern) feature, and the like.

Next, in the image fragment word generation process, the block image sets are clustered (classified) based on the similarity of the feature vectors at each scale i (i = 1,..., N _d ) (S04). In the process of S04, the clustering method can be a division optimization method such as the K-Means method, but is not limited to this. Let K _i clusters in each scale i generated by the process of S04 be C [i, 1],..., C [i, K _i ].

Next, in the image fragment word generation processing, for example, an image fragment word W = {w [1, 1],..., W having the center vector w [i, k] of each cluster C [i, k] as an element. [I, k],..., W [N _d , K _Nd ]} are generated as multi-scale image fragment words (MIPWord) (S05). Then, the generated multiscale image fragment word (MIPWord) is stored in storage means (for example, image fragment word 43) or the like (S06).

The example of FIG. 5 shows the flow of MIPWord generation when the block division scale N _d = 2 in the process shown in FIG. As shown in FIG. 5, the same preparation (sample) video is divided into blocks at a plurality of scales (image sizes), and each piece of image divided at each scale is clustered based on a feature vector, Generate an image fragment word.

  For example, when a genre (for example, news, various sports (soccer, baseball), etc.) such as a search target video or a search request scene is determined in advance, the preparation video is a preparation video of the same genre. However, the present invention is not limited to this. Further, the scale is set to an arbitrary scale, type, and number, for example, by the divided block setting means 22 described above. The scale may be arbitrarily set according to the resolution of the input video.

<Example of Calculation of Multi-Scale Image Single Word Histogram (H-MIPW) in Scene Unit>
Next, a calculation example of H-MIPW for a predetermined scene unit will be described with reference to the drawings. In the present embodiment, the H-MIPW of each scene of the search target video is calculated based on the multiscale image piece word (MIPWord).

  Here, FIG. 6 is a flowchart showing an example of the multiscale image fragment word histogram generation processing. FIG. 7 is a diagram showing a flow of generating a multiscale image fragment word histogram.

In FIG. 6, the multi-scale image fragment word histogram generation processing is performed by, for example, a histogram H = {h [1] having the same number as the number of vectors w {i, k} constituting MIPWord (W) generated from each scale. 1],..., H [i, k],..., H [N _d , K _Nd ]} are prepared (S11), and the initial value of each element is set to 0 (S12).

Next, in the multiscale image fragment word histogram generation process, frame images are sampled from each shot of the scene S at predetermined intervals (for example, T frames) (S13). Next, in the multiscale image piece word histogram generation processing, each sampled frame image is divided into blocks of one or a plurality of scales (S14). The scale at this time may be, for example, the same scale as in S02 described above (scale 1 (n _W1 × n _W1 ),..., Scale N _d (n _WNd × N _HNd ). A predetermined number (for example, three types) of scales included in a plurality of scales (for example, five types) obtained by processing may be used.

Next, in the multiscale image piece word histogram generation process, a feature vector is calculated for each block image obtained in the process of S14 as in the process of S03 described above (S15). Next, in each scale i (i = 1,..., N _d ), each element of the histogram H is added to all block images (S16). In the process of S16, specifically, among W [i, k] (k = 1,..., K _i ) of MIPWord (W), the one having the highest similarity with the feature vector of the block image is represented by w. [I, k ′]. In the process of S16, 1 is added to the element h [i, k ′] of the histogram H corresponding to the feature vector w [i, k ′] having the highest similarity.

In the multiscale image fragment word histogram generation process, each element of the histogram H is divided by the total number of sampled frame images (S18), and the calculated histogram H = {h [1,1],. [I, k],..., H [N _d , K _Nd ]} are set as the H-MIPW of the scene S and stored in the storage means (for example, the image fragment word histogram 46) or the like (S19).

The example of FIG. 7 shows the flow of processing when the block division scale N _d = 2 with respect to the multiscale image fragment word histogram generation processing shown in FIG. In the example of FIG. 7, a frame image is sampled at a predetermined interval (T) for each scene (multiple shots) S included in the search target video, and is divided into a plurality of scales.

  In the example of FIG. 7, w [j, k] closest to the feature vector of each block is obtained for the generated MIPWORD (W) based on the feature vector of each divided block, and the corresponding h Add [j, k]. As a result, the H-MIPW of the scene S can be acquired as shown in the example of FIG. Therefore, in this embodiment, feature information for each scene can be extracted, and image classification can be performed quickly and appropriately.

<Scene Search Using Multiscale Image Single Word Histogram (H-MIPW)>
Next, an example of scene search using a multiscale image fragment word histogram (H-MIPW) in the scene search device 30 will be described.

FIG. 8 is a flowchart illustrating an example of the search process. In the example of FIG. 8, the search process, the initial value i = 1 of the variable that identifies each element (S21), the search target scene S and the image piece word histogram H _i of _i, the image piece word histogram H _Q requests scenes A distance D _i is calculated (S22). Here, FIG. 9 is a diagram illustrating a calculation example of the distance D _i . In the present embodiment, as shown in FIG. 9, the distance D _i between the image fragment word histograms H _Q and H _i of each of the request scene and the search target scene S _{i is obtained} for each element, and thus based on similarity. Perform a search.

That is, in the search process, i = i + 1 is set (S23), and the vector distance D _i is calculated for the next element in order. Here, for example, it is determined whether or not i is greater than N_S (the last of the elements) (S24). If i is not greater than N_S (NO in S24), the process returns to S22. Furthermore, (in S24, YES) when the value of i is greater than n_s distance towards _{D i} is small, due to high degree of similarity, the distance _D Top _{N HIT} pieces of scene search result set in advance from the smaller _i Is output (S25). That is, the processing of S25 is the same as outputting the top N _HIT scene search results from the one with the higher similarity.

  As a result, the image processing apparatus 10 can set appropriate feature information for a scene included in a video, and can realize a highly accurate scene search using the set feature information.

<Experimental result>
Next, in order to clarify the effect of this embodiment, a scene search experiment based on the similarity of H-MIPW of each scene targeting an actual program video will be described as an example. The performance of H-MIPW in terms of “search” is verified.

<Experimental conditions>
As an experimental condition, 254 nature-related broadcast program videos are used as an example of a use video, and 100 preparation videos for MIPWord generation are used. Also, 254 videos to be searched are used. The scene delimiter was fixed at the number of shots in one scene, and one scene for every five shots. The total number of scenes is about 7300, the frame image normalization size is 320 × 180, the block division scale is Nd = 2, scale 1 (16 × 16 pixels), and scale 2 (8 × 8 pixels). The number of MIPWords is 750 for both scale 1 and scale 2.

  Here, FIG. 10 is a diagram showing twelve types of scenes to be queried and the correct video content set for each. Each image is a head image of a shot constituting the scene. The query scenes (Q1 to Q12) shown in the example of FIG. 10 are randomly selected from the search target videos in consideration of easy setting of correct answers. The correct video content is set from the viewpoint of “a subject that appears in any one of the shots and is considered to be important to some extent in terms of content”, but is not limited to this. For example, the correct video content of the query scene Q1 in FIG. 10 is {mountain, sky and mountain (sky + mountain), flower, branch, bird}. The correct video content of the query scene Q2 includes {building distant view, building close view, city distant view} and the like.

  In the present embodiment, the similarity by the histogram intersection between the H-MIPW of the query scene and the H-MIPW of all search target scenes is calculated, and the search target scenes are rearranged in descending order of the similarity, thereby obtaining a search result. Can be obtained.

<Measure for accuracy evaluation>
Here, a scale (relevance) for evaluating the accuracy of the search result will be described. The degree of relevance can be determined in consideration of both aspects such as whether each shot of the scene is related to the correct video content and how much the correct video content is covered by the scene, but is limited to this. Instead, for example, any of the above may be used. Here, the degree of association R between a certain scene and the correct video content can be set as “R = (2RsRc / (Rs + Rc)) (1)”. Here, Rs is “a shot ratio including any correct video content in a scene”. Rc represents “the ratio of correct video content included in any shot”.

  FIG. 11 is a diagram illustrating a schematic calculation example of the degree of association. Of the five shots that make up a scene shown in FIG. 11, the shot that shows one of the correct video contents of this scene, {moon, mountain, sea, fish}, is marked 3 in FIG. (Frame images 1, 2, 4). Therefore, the above-described shot ratio Rs is 3/5 = 60%.

  On the other hand, among the four correct video contents {moon, mountain, sea, fish}, three (moon, mountain, sea) marked with ○ are shown in any shot of the scene. Therefore, the ratio Rc of the correct video content described above is 3/4 = 75%. Then, according to the above-described equation (1), the degree of association R between this scene and the correct video content is 66.7%.

<Scene search result example>
Next, an example of a scene search result using H-MIPW will be described. Here, FIG. 12 is a diagram illustrating an example of a search result in the present embodiment. 12 (a) to 12 (c) show search result examples 1 to 3. FIG. Specifically, FIG. 12A shows the top 20 search results for the query scene Q1 shown in FIG. Similarly, FIG. 12B shows the top 20 search results for the query scene Q3, and FIG. 12C shows the top 20 search results for the query scene Q9.

  Each image is a head image of each shot constituting the scene. The ◯ mark attached to the upper left of the image indicates “a shot including any of the correct video contents” in the relevance calculation. In the table on the right side of the scene of the search result, the ◯ mark indicates “correct video content included in any shot”. When the relevance is calculated for each of FIGS. 12A to 12C, the average value of the relevance with the correct video content of the top 20 scenes of the search results is 52.1% in FIG. (B) was 68.9% and FIG. 12 (c) was 68.9%. In the above example, the genre of the program video is narrowed down to the natural program. However, it can be said that the result is highly accurate considering that a variety of meaning “miscellaneous” scene sets are targeted for retrieval.

<Accuracy comparison results with related methods>
Next, in order to objectively demonstrate the effectiveness of the present embodiment, the accuracy comparison result with the related method will be described. First, in order to verify the effect of block division using multiple scales, two methods are compared. FIG. 13 is a diagram illustrating an example of the comparison method. FIG. 13A shows a conventional method of Bag of Visual Words using local features in order to demonstrate the superiority of generating a word with an image fragment (block image) as a comparison method 1. . As representative local features, a luminance gradient-based SIFT (Scale Invariant Feature Transform) feature, a SURF (Speeded Up Robust Features) feature, a Color-SIFT feature obtained by extending the SIFT feature for color images, and the like can be used. . In this comparison, the Color-SIFT feature is a comparison target, and the number of words of Visual Words is 1000.

  FIG. 13B shows a method based on the image fragment word histogram in the block division with only one type of scale to be divided (scale 1 (16 × 16 pixels)) as the comparison method 2 in the present embodiment. ing.

  The local feature Bag of Words method, which is the comparison method 1, shows a somewhat good performance for identifying a copy of an image. However, unlike the block image shown in the comparison method 2, the relevance between each word and the subject is weak. Therefore, for example, when a plurality of frames are integrated and processed as in a scene search, a case where two scenes having different video contents and Bag of Visual Words are similar is likely to occur.

  Moreover, FIG. 14 is a figure which shows the comparative example of an experimental result. FIG. 14 shows the top 20 scenes of the search results obtained by the respective methods for the query scene Q12. 14A shows a search result by the above-described comparison method 1, and FIG. 14B shows a search result by the above-described comparison method 2 (one kind of division scale). FIG. 14C shows a method using an image fragment word histogram as a comparison method 3 in block division with multiple types of scales in this embodiment. Further, each image in FIG. 14 and a circle in the figure indicate the content of the correct video included in any shot.

  Comparing FIG. 14A to FIG. 14C, FIG. 14C has the largest number of circles. Here, in the scene search in the present embodiment, the maximum number of search results displayed on the first page is about 20. Therefore, the accuracy can be evaluated by the average value of the degree of association R with the correct video content of the top 20 search results.

  The accuracy calculated for each of FIGS. 14A to 14C is 43.2% in FIG. 14A, 63.5% in FIG. 14B, and 73.2 in FIG. 14C. It can be seen that the comparison method 3 which is an example of this embodiment has the highest search accuracy.

  Here, FIG. 15 is a diagram illustrating an example of accuracy comparison. FIG. 15A shows the accuracy by the respective methods of FIGS. 14A to 14C described above as evaluation results for each query. FIG. 15B shows the accuracy of the total (Total) obtained by averaging the accuracy of all the queries shown in FIG.

  Referring to FIG. 15A, the comparison between the comparison method 3 and the comparison method 1 is highly accurate in the eight query scenes (Q1 to Q4, Q7, Q8, Q10, and Q12). Further, as shown in FIG. 15B, the accuracy was improved by 13% as a whole. As for comparison between the comparison method 3 and the comparison method 2, the accuracy of the present embodiment is high in nine query scenes (Q2 to Q5, Q7, Q9 to Q12), and the accuracy is improved by 4% as a whole.

  Thereby, it is possible to show the superiority of the present embodiment using words based on image pieces (block images) instead of local features (Color-SIFT). Further, it can be seen that the image pieces preferably generate multiple scales. According to the present embodiment, by using H-MIPW, it is possible to realize a highly accurate scene search based on the similarity of video contents.

<Execution program>
Here, the above-described image processing apparatus 10 includes, for example, a CPU (Central Processing Unit), a volatile storage device such as a RAM (Random Access Memory), a nonvolatile storage device such as a ROM (Read Only Memory), a mouse and a keyboard. It can be configured by a computer having an input device such as a pointing device, a display device for displaying images and data, and an interface device for communicating with the outside.

  Therefore, the above-described functions of the image processing apparatus 10 can be realized by causing the CPU to execute a program describing these functions. These programs can also be stored and distributed in a recording medium such as a magnetic disk (floppy (registered trademark) disk, hard disk, etc.), an optical disk (CD-ROM, DVD, etc.), or a semiconductor memory.

  That is, an image processing can be realized by generating an execution program (image processing program) for causing a computer to execute the processing in each configuration described above and installing the program in, for example, a general-purpose personal computer or server. it can. In addition, about the process by the execution program in this embodiment, each process mentioned above is realizable, for example.

  As described above, according to the present embodiment, the feature information for each scene included in the video can be appropriately acquired. Further, the similarity can be quickly acquired by appropriate scene classification based on the feature information. Therefore, it is possible to realize a highly accurate scene search based on the similarity of the image features of the entire scene.

  For example, high-precision image classification can be performed by using a multi-scale image fragment word histogram based on types and appearance ratios of multi-scale block images as moving image features for scene search including a plurality of cuts. Further, by applying this embodiment, for example, a scene search for a broadcast program video or a scene search based on the similarity of video content can be performed. Therefore, for example, it is possible to realize a highly accurate scene search based on the similarity of the image features of the entire scene, not the representative thumbnail image of the scene as in the prior art.

  The preferred embodiment has been described in detail above, but the disclosed technique is not limited to the specific embodiment, and various modifications, within the scope of the disclosed technique described in the claims, It can be changed.

DESCRIPTION OF SYMBOLS 10 Image processing apparatus 20 Feature extraction apparatus 21 Sampling acquisition means 22 Division | segmentation block setting means 23 Image piece word generation means (block characteristic information generation means)
24 Scene division means 25, 31 Histogram generation means 30 Scene search device 32 Search means 41 Preparation video set 42 Preparation frame image set 43 Image fragment word 44 Search object video 45 Search object scene 46, 52 Image fragment word histogram 51 Required scene 53 Results

Claims (5)

In an image processing apparatus that extracts feature information of each scene included in a video,
Sampling acquisition means for sampling a predetermined frame image from a sample video;
Block feature information generating means for dividing each frame image obtained by the sampling acquisition means for each of one or a plurality of scales and generating feature information for each divided block;
Scene dividing means for dividing a scene from the target video for generating the feature information;
Histogram generation means for generating a histogram based on the appearance ratio for each block using the block obtained by the block feature information generation means for each scene divided by the scene division means. apparatus. The histogram generation means generates a block unit histogram for a search request scene from a user,
The image processing apparatus according to claim 1, further comprising: search means for searching for a corresponding scene by referring to the histogram generated by the histogram generation means using the generated search request scene.   3. The divided block setting unit that sets at least one of the size, type, and number of one or a plurality of blocks generated by the block feature information generation unit. Image processing apparatus.   The image processing apparatus according to claim 1, wherein the feature information is a color feature or a texture feature. In an image processing program for causing a computer to execute image processing for extracting feature information of each scene included in a video,
The computer,
Sampling acquisition means for sampling a predetermined frame image from a sample video;
Block feature information generating means for dividing each frame image obtained by the sampling acquisition means into one or a plurality of scales and generating feature information for each divided block;
Scene dividing means for dividing a scene from the target video for generating the feature information; and
An image processing program for functioning as a histogram generation unit that generates a histogram based on an appearance ratio for each block using a block obtained by the block feature information generation unit for each scene divided by the scene division unit.