JP2017021606A - Method, device, and program for searching for dynamic images - Google Patents

Abstract

PROBLEM TO BE SOLVED: To conduct searching or grouping using a dynamic image itself as a query more precisely and more easily with lower cost compared to a conventional case.SOLUTION: A feature quantity extraction part 22 extracts the respective feature quantities of a query dynamic image and a target dynamic image. A dynamic image representative frame selection part 23 acquires representative frames in chronological order of the query dynamic image and the target dynamic image on the basis of the relation of the feature quantities, using one of a time-interval k average method, a time-division k average method, a time-binding pair-wise nearest neighbor method, and a time-division pair-wise nearest neighbor method. Further, a similarity calculation part 24 obtains the similarity between the query dynamic image and the target dynamic image from the similarity between the representative frames of the query dynamic image and the target dynamic image and searches for a dynamic image which is similar to the query dynamic image on the basis of the obtained similarity.SELECTED DRAWING: Figure 1

Description

  The present invention relates to a moving image search method, a moving image search device, and a program thereof.

A huge amount of moving image data is accumulated on the Internet. A text-based search method is known as a frequently used method for searching these moving images. This text-based search method searches for a desired moving image by associating appropriate text information with each moving image and performing a text search. However, this method has the following problems.
(A) The association (labeling) of the moving image and the text information needs to be performed manually, which is expensive.
(B) Since an inappropriate label with poor versatility is attached, the accuracy and efficiency of search may be reduced.

  As a method for solving the above problem, a content-based search using the image content (content) itself as a query is expected.

  As a content-based search method, the query image and the search target image can be obtained by combining information about the entire image such as brightness change and complexity of the entire image, complexity of the edge amount, simple composition information using the luminance distribution, or a combination thereof. There has been proposed a method for calculating the degree of similarity (see, for example, Patent Document 1). Further, by dividing a plurality of search target images into element images of a predetermined size, selecting a representative element image from the divided element images, and using a search reference image drawn using the representative element image A technique for searching for an image has also been proposed (for example, see Patent Document 2).

  However, these proposed methods search still images using still images as queries, and do not consider a method for searching for moving images as queries.

JP 2001-56820 A JP 2002-140331 A

  As described above, in the prior art, there is only a disclosure about a method using a text base or a still image as a query, and what is disclosed about a moving image search based on a moving image content using a moving image as a query is disclosed. There was no consideration. An object of the present invention is to provide a moving image search method, a moving image search apparatus, and a program thereof that can perform a search using a moving image itself as a query with high accuracy while keeping calculation cost low.

  The present invention is a moving image search method by a moving image search apparatus for calculating a similarity of a target moving image with respect to a query moving image and searching for a moving image similar to the query moving image. A first feature value is extracted for each frame from the frame, a second feature value is extracted for each frame from the frame of the target moving image, and the query moving image is extracted with respect to the extracted first feature value. The first representative frame is acquired in time series using either the time-segment k-average method, the time-division k-average method, the time-constrained pair-wise nearest neighbor method, or the time-segment pair-wise nearest neighbor method, For the second feature amount extracted for the target moving image, a time-segment k-average method, a time-division k-average method, a time-constrained pairwise nearest neighbor method, or a time-segmented pairwise A second representative frame is acquired in a time series using any of the nearest neighbor method, and between the first representative frame in the query moving image and the second representative frame in the target moving image The similarity between the query moving image and the target moving image is obtained from the similarity, and the moving image similar to the query moving image is searched.

  The present invention is also a moving image search apparatus for calculating a similarity of a target moving image to be searched with respect to a query moving image and searching for a moving image similar to the query moving image, wherein the query moving image And a feature amount extraction unit that extracts a first feature amount for each frame related to the query moving image and a second feature amount for each frame related to the target moving image from the frame of the target moving image, and the query For the moving image and the target moving image, the time-segment k-average method, the time-division k-average method, the time-constrained pairwise nearest neighbor method, or the extracted first feature value and second feature value, A representative frame selection unit that acquires a first representative frame related to the first feature quantity and a second representative frame related to the second feature quantity in time series using any of the time-wise pairwise, nearest, and neighbor methods When, The similarity between the query moving image and the target moving image is obtained from the similarity between the first representative frame in the query moving image and the second representative frame in the target moving image. A moving image search unit that searches for the moving image similar to the query moving image.

  With the above-described moving image search method, moving image search apparatus, and program thereof according to the present invention, a search using a moving image as a query can be performed with high accuracy while keeping the calculation cost low.

It is a block diagram which shows the system configuration | structure of the moving image search device by embodiment of this invention. It is explanatory drawing of the operation | movement flow of the moving image search device by embodiment of this invention. It is explanatory drawing which shows the difference between the clustering which considered the time series information applied in embodiment of this invention, and the clustering which does not consider the time series information of a prior art. In the embodiment of the present invention, it is a flowchart showing a representative frame selection algorithm by an affinity propagation method (TB-AP, Time-Bound Affinity Propagation) using a time window. It is explanatory drawing explaining the difference of message exchange in embodiment of this invention and a prior art. In embodiment of this invention, it is explanatory drawing which shows the link matrix of message exchange. In the embodiment of the present invention, the time window in TB-AP and the time block in the improved k-means method and harmonic competition method are each an explanatory diagram showing on the time axis. It is explanatory drawing of the time division k average method (TP k-means) in embodiment of this invention. In the time-split k-means (TS k-means) in the embodiment of the present invention, all frames are time-divided into k equal parts, and the center of gravity is obtained for each cluster. It is explanatory drawing. Regarding the time-split k-means (TS k-means) in the embodiment of the present invention, the centroid of the cluster that currently belongs in each frame and the centroid of the clusters before and after adjacent in time series It is explanatory drawing which shows the state which calculated the distance between. It is explanatory drawing which shows the state which performed the reclustering about the time division | segmentation k average method (Time-Split k-means: TS k-means) in embodiment of this invention. It is explanatory drawing which shows the state which divided | segmented the cluster between the frames in which a time series is discontinuous about the time-division k average method (Time-Split k-means: TS k-means) in embodiment of this invention. It is explanatory drawing which shows the state which calculated the gravity center in each cluster, after dividing | segmenting a cluster about the time division | segmentation k-means (Time-Split k-means: TS k-means) in embodiment of this invention. It is explanatory drawing which shows the merge of the data point in the pairwise nearest neighbor method (PNN) in embodiment of this invention. It is explanatory drawing of the time division | segmentation pairwise nearest neighbor method (TP-PNN) in embodiment of this invention. It is explanatory drawing of the time constraint pairwise nearest neighbor method (TB-PNN) in embodiment of this invention. In the embodiment of the present invention, it is a flowchart showing a similarity calculation algorithm by M distance. It is explanatory drawing showing the production | generation process of the original moving image data used in experiment. It is explanatory drawing which shows the structure of the moving image data used in experiment. This is a plot of each of the five Gaussian clusters divided into two in the time direction and made into 10 groups. This figure was used to examine the appearance and calculation speed of representative points (typical still images) reflecting temporal characteristics. is there. Is a diagram showing the distance of each representative still image of the moving image v _A and moving image v _B as matrix. It is a figure which shows an example of the table for global alignment obtained by experiment. It is a figure which shows an example of the table for local alignment obtained by experiment. It is the figure which blacked out the part which abbreviate | omitted calculation corresponding to the table for global alignment. It is a graph which shows the precision-recall rate curve in the moving image search obtained by experiment. As another experimental example, it is a graph showing a precision-recall rate curve in a moving image search obtained by an experiment using a different representative frame selection algorithm, without performing post-processing of redundant representative frame deletion at the time of clustering The relevance rate-recall rate curves in the five clustering methods when the similarity is calculated as it is are shown. As another experimental example, it is a graph showing a precision-recall rate curve in a moving image search obtained by an experiment using a different representative frame selection algorithm, and when post-processing of redundant representative frame deletion is performed during clustering The relevance rate-recall rate curves in the five clustering methods are expressed. It is a graph which shows the precision-recall rate curve in the moving image search obtained by the experiment using CSD as a feature-value as another experiment example. As another experimental example, it is a graph which shows the relevance rate-reproducibility curve in the moving image search obtained by experiment using Frame Signature as a feature-value. It is an example of the hardware block diagram at the time of using a computer as a processing apparatus of the moving image search apparatus according to an embodiment of the present invention.

  Hereinafter, a moving image search method and a moving image search apparatus according to the present invention will be described with reference to the accompanying drawings.

  FIG. 1 schematically shows an example of a system configuration of a moving image search apparatus according to the present embodiment. In the figure, 1 is a storage device such as a hard disk drive used as a database for storing and storing a set of moving images, 2 is a processing device such as a computer that searches for similar images, and 3 is a moving image used for searching and a moving image to be searched A display 4 displays an image or a search result, and 4 is an input device such as a mouse or a keyboard. The storage device 1 is connected to the processing device 2 in a readable state. The processing device 2 can display a moving image stored in the storage device 1 on a display 3 as a display device as necessary. Note that the storage device 1 may be built in the processing device 2 or may be constructed on a computer such as a server connected to the processing device 2 via communication means. In addition, the query image and the search target image can be stored in separate storage devices, and the form is not particularly limited. The processing device 2 is connected to the input device 4 and can appropriately accept input from the user.

  The processing device 2 includes a moving image capturing unit 21, a feature amount extracting unit 22, and a moving image representative frame selecting unit 23 in order to process the query moving image and the target moving image. The moving image capturing unit 21 captures a query moving image selected by the user via the input device 4 or a target moving image to be searched from the moving images stored in the storage device 1. The feature amount extraction unit 22 extracts the feature amount of the query moving image or the target moving image captured by the moving image capturing unit 21. The moving image representative frame selection unit 23 selects one or more representative frames in the query moving image or the target moving image based on the feature amount of the query moving image or the target moving image extracted by the feature amount extraction unit 22. . The moving image capturing unit 21, the feature amount extracting unit 22, and the moving image representative frame selecting unit 23 may be configured separately for the query moving image and the target moving image.

  In addition, the processing device 2 includes a similarity calculation unit 24 and a similar moving image display control unit 25 in order to search for a target moving image similar to the query moving image. The similarity calculation unit 24 compares the feature amount of each representative frame in the query moving image with the feature amount of each representative frame in the target moving image, and calculates the similarity. The similar moving image display control unit 25 can display one or more target moving images on the display 3 in descending order of the similarity calculated by the similarity calculating unit 24.

  When a computer is used as the processing device 2, a part or all of each function for processing can be realized by installing application software for performing similar moving image search. It is possible to execute processing according to the application software that has been set.

  FIG. 2 shows an outline of the processing of the processing device 2 in the present embodiment. In the figure, the left route represents the processing flow for the query moving image, and the right route represents the processing flow for the target moving image. With the processing in FIG. 2, finally, the similarity of the target moving image to the query moving image is calculated.

  A processing flow of the query moving image of the left route in FIG. 2 will be described. In S110, the query moving image is taken from the database of the storage device 1. In step S120, the feature amount of the query moving image is extracted. As feature quantities, one of the image feature quantities proposed in MPEG-7, Color Structure Descriptor (CSD), and Video Signature (Multimedia content description interface, ISO / IEC) International Standard of ISO / IEC 15938-3, Amendment 4 (2010)) can be used. In step S130, a representative frame of the query moving image is determined as a typical still image. At this time, the representative frame is determined using a method that takes into account the factors of the elapsed time of the moving image. Although depending on the number of frames of the query moving image, a plurality of representative frames are usually selected, and the feature amount set of each typical still image corresponding to the selected representative frame is set as the feature amount of the query moving image. .

  The processing flow of the target moving image on the right path in FIG. 2 is performed in the same procedure as the processing flow of the query moving image. That is, in S140, the target moving image is captured from the database of the storage device 1. In step S150, the feature amount of the target moving image is extracted. The CSD or video signature described above can be used as the feature amount. In step S160, a representative frame that is a typical still image of the target moving image is determined. At this time, as described above, the representative frame is determined using a method that takes into account the element of the elapsed time of the moving image. Although depending on the number of frames of the target moving image, a plurality of representative frames are usually selected, and a set of feature amounts of each typical still image corresponding to the selected representative frame is set as a feature amount of the target moving image. . Either the left route processing or the right route processing in FIG. 2 may be performed first or in parallel.

  Note that the moving image representative frame selection unit 23 includes a time-bound affinity propagation method (TB-AP), a time-part k-average method (Time-Partition) as a technique that takes into account the elements of the elapsed time of the moving image. k-means (TP-k-means), Time-Split k-means (TS-k-means), Time-bound pairwise nearest neighbor (TB-PNN) ), Or the time-partition pairwise nearest neighbor (TP-PNN) method. These will be described in detail later.

  Also, instead of the TP-k-means method, it is also possible to apply a harmonic competition that introduces a time block that is a generalization of the method. The harmonic competition method is described in “Matsuyama, Yasuo,“ Harmonic Competition: A Self-Organizing Multiple Criteria Optimization, ”IEEE Transactions on Neural Networks, Vol.7, No.3, pp.652-668, MAY 1996. Has been.

  Thus, when the representative frames (typical still images) are selected for the query moving image and the target moving image, the similarity calculation process of S170 is performed. Since the number of typical still images here depends on each moving image, the similarity between the query moving image and the target moving image is calculated by a weighted M distance that allows gap insertion. The M distance will be described in detail later. In S180, based on the similarity score calculated in S170, a target moving image with a high similarity is retrieved and displayed on the display 3.

  In the processing flow described above, the following describes the three processes of feature amount extraction in S120 and S150, representative frame selection in S130 and S160, and similarity calculation in S170.

-Extraction of feature amount for frame In the present embodiment, extraction of the feature amount of each frame included in the moving image is first executed. The feature amount extracted here is expressed as a vector {x _t } ⁿ _{t = 1} in consideration of time series. For example, so that the processing by the application software used in the similar image search can handle a moving image having color information, the vector is expressed by the above-described MPEG-7 CSD, and all the vectors are normalized with the CSD histogram. can do.

However, the feature amount of each frame of the moving image is not limited to that using CSD, and any feature amount may be used as long as the feature amount is a vector amount. The above-mentioned CSD histogram feature values are normalized by applying the following conditions.
-Normalization conditions (i) Each vector component is non-negative.
(Ii) The sum of all vector components is set to 1.0.
This is obtained by the following procedure.
(A) Each frame of the moving image is changed to a vector amount representing the feature (feature amount extraction).
(B) A method that can normalize each vector is employed so as to ensure a similarity comparison method and versatility when it is realized as software.

  The procedure (b) is always possible by using substantially the minimum value and sum of each vector. CSD is merely an example of the feature vector in the procedure (a), and a simpler one that covers only the entire image frame is a histogram.

  CSD is a simple feature amount, but it is preferable because good results can be obtained. Moreover, although the amount of calculation increases instead of CSD, it is also possible to use a feature amount by another method. For example, SIFT (Scale-Invariant Feature Transform), SURF (Speeded Up Robust Features), FREAK (Fast Retina Keypoint), bag of visual words, etc. can be used for the feature amount of each frame of the moving image. In addition, the aforementioned video signature can be used. It is also possible to combine these feature quantities other than CSD and feature quantities including CSD.

  For example, when {CSD, FREAK, bag of visual words, SIFT, Video Signature ...} is described as a group of individual normalized features, and the individual normalized features are expressed as NormalizedFeatureVector (i), As a feature quantity of the image frame, a composite feature quantity vector CompositeFeatureVector defined in the following Equation 1 can be used. However, when using a feature quantity that does not need to be normalized, NormalizedFeatureVector outputs the input feature quantity as it is.

However, the combined feature vector CompositeFeatureVector satisfies the normalization conditions: w _i ≧ 0 and Σ ^N _{i = 0} w _i = 1 in the above equation. The above formula 1 is merely an example of obtaining a combined feature vector CompositeFeatureVector. For example, a product obtained by multiplying feature amounts can be used as the combined feature vector CompositeFeatureVector. Therefore, the composite feature quantity vector CompositeFeatureVector is obtained by the following general formula 2. This is a mathematical expression of the expression that what is finally normalized by combining several individual feature quantities FeatureVector (i) becomes a composite feature quantity vector CompositeFeatureVector.

-Selection of representative frames to keep time series After extracting the feature values for each frame, the representative frames are selected. Here, the processing device 2 performs clustering based on the extracted feature amount of each frame in the moving image, and selects a frame representing each cluster. A frame representing each cluster is referred to as a typical still image. In this embodiment, time-series information is added to the selection of a typical still image.

FIG. 3 is a diagram for explaining a difference in clustering between the time-constrained affinity propagation method (TB-AP) considering the time series information used in the present embodiment and the normal affinity propagation method (AP). The time series data in the upper part of the figure represents n = 10 time series feature vectors {x _t } ¹⁰ _{t = 1} from time t = 1 to time t = 10 extracted as feature quantities. Each box here represents a frame at each time t. For example, a number 1 is assigned to a frame at time t = 1, and a number 5 is assigned to a frame at time t = 5. Also, the frames drawn with the same gradation in the figure represent similar frames (for example, one scene in a moving image) and belong to the same class. In the illustrated example, there are three types of sets of similar frames: {1,2,10} class, {3,4,5} class, {6,7,8,9} class.

When these frames are clustered by feature amount, when the normal affinity propagation method (AP) is used, for example, frames {2, 4, 7} Elected. However, this does not take time series information into consideration, and is not appropriate as a label for searching for moving images. By correctly considering the time series, the typical still image is selected as {2, 4, 7, 10} as shown in the lower right part of the figure. In the time-constrained affinity propagation method (TB-AP) that considers the elapsed time of a moving image, clustering considering such time series becomes possible. That is, in the example of FIG. 3, when the time-constrained affinity propagation method is applied, the typical still image is correctly {(1, 2 , 0) (1, 4 , 1), (1, 7 , 2), (0, 10 , 0)}.

  Note that the number of frames of these typical still images varies according to the length of the moving image and the content. Thus, each typical still image includes a responsible frame on its left and right in time order.

  Hereinafter, the TB-AP algorithm, which is one of the algorithms for selecting a typical still image that keeps time series, will be described with reference to the flowchart of FIG.

S210: Calculation of Similarity Here, for a moving image frame from time t = 1 to time t = n, n time-series feature vectors {x _t } ⁿ _{t = 1} extracted as feature values are given. As described above, each feature vector {x _t } ⁿ _{t = 1} is a normalized CSD histogram with a total of one. From the feature vectors _{x i} of the time t = i, define the time t = j Similarity measure _{s (x} i, _{x j)} representing the relationship between the feature vector _{x j} of the as s (i, j). Then, when and only when the feature vector x _i is more similar to the feature vector x _k at time t = k than the feature vector x _j , s (k, i)> s (k, j) is satisfied. It shall have properties.

  In the present embodiment, the similarity measure s (i, j) calculated by the following equation (3) is used.

  However, bar (D) (hereinafter, the symbol D with an overline as in the first term on the right side of Equation 3 is expressed as bar (D) in the text) is input by the input device 4, for example. Is a constant parameter that can be set.

  As long as the similarity is set according to the rule that “the larger the value is, the greater the relationship is, and the smaller the value is, the smaller the relationship is”, the similarity is not limited to the equation 3 but may be any type. The selection of bar (D) does not affect the result of TB-AP, and is set to 0 here. However, this parameter is an important factor in comparing moving images, and can take a value other than 0.

S220: Initialize matrix A Here, a matrix A = [a _ij ] having an initial value of zero is prepared (a _ij ← 0). This matrix A is a matrix representing availability. The availability is a message sent from the data point j of the typical still image candidate to the data point i, and is a value representing how appropriate it is to select the data point i as the typical still image of the data point j.

S230: Matrix R is calculated (or updated) using matrix A
Here, first, the length of the sliding time window is set to W = 2ω−1 so that its center is located at the diagonal component, and then the feature vector x is considered in consideration of the time series in the time window. _{i, x} j, to select the _{x k.}

Next, the component r _{ij of the} matrix R representing the responsibility is calculated and updated using the availability matrix A described above. Responsibility is a message sent from data point i to data point j of a typical still image candidate, and is a value representing how appropriate data point i is as a typical still image of data point j. In the calculation of the responsibility matrix R and the calculation of the availability matrix A, which will be described later, the data points i and j are set within the range of the following equation (4). In other words, in TB-AP, the limit shown in Equation 4 is set in the range for message exchange.

  The responsibility matrix R is usually symmetric and is updated using the matrix A as shown in Equation 5.

In Equation 5, ρ _ij is a response propagation value between data point i and data point j. Further, λε (0,1) is a damping coefficient as a design parameter, and by using this, the convergence in S250 described later can be delayed and a stable result can be obtained. The initial value of the matrix R is also a zero matrix, but if the damping coefficient λ is 0, for example, the response propagation value ρ _ij is updated as it is as the value of the matrix R.

S240: Calculate Matrix A Here, the data points i and j are limited to the range indicated by Equation 4, and the component a _ij of the availability matrix A is updated using Equation 6 below.

In Equation 6, α _ij is a propagation value of availability between data point i and data point j.

S250: Determine whether or not the sum of the matrix A and the matrix R has converged. The component a _{ij of the} matrix A and the component r _{ij of the} matrix R calculated as described above are added, and the value a _ij + r _ij The steps S230 and S240 are repeated until the standard is satisfied. On the other hand, when the value of a _ij + r _ij satisfies the convergence criterion, the process proceeds to the next step S260.

S260: Representative Frame Selection Here, the index defined in Equation 7 below is adopted as the index of the typical still image.

That is, the still image at the data point where the value of a _ij + r _ij is the maximum is adopted as the typical still image. For example, the data point i belongs to a cluster centered on the data point j having the maximum value of a _ij + r _ij . A set of typical still images finally selected as representative frames is found by collecting such indexes.

  In the above-described flow of S230 to S260, ω necessary for setting the time window is a parameter for limiting the range of message exchange when calculating the matrix A and the matrix R. FIG. 5 shows how messages are exchanged under this restriction. In the figure, “AP” in the upper part indicates a range of message exchange by the affinity propagation method as a conventional technique. The lower “TB-AP” indicates the range of message exchange by the time-constrained affinity propagation method in the present embodiment. Here, it is assumed that the same frames as described in FIG. 3 are arranged in time series.

  In normal “AP”, messages are exchanged for all data points, but in this embodiment, messages are exchanged for neighboring data points limited by a time window. Specifically, in the example of “AP” shown in FIG. 5, data exchange is performed over the entire range from data point j = 1 to j = 10 with data point i = 5 as the center. On the other hand, in the example of “TB-AP” in the figure, the parameter ω = 4 is set, and the message exchange is performed within the range of the time window 2ω-1 = 7. Therefore, the range of message exchange is limited to the range of data points j = 2 to j = 8 with data point i = 5 as the center.

  FIG. 6 shows a link matrix for message exchange to which the TB-AP of this embodiment is applied. Again, parameter ω = 4 is set, and when data point i is selected from any one of 1 to 10, it is shown with which data point j message exchange is performed.

  As a characteristic of a normal AP, only data points for which messages have been exchanged are finally selected as typical still image candidates. The same can be said for the TB-AP of the present embodiment. In the TB-AP of the present embodiment, the range for exchanging messages is limited in consideration of time series. As a result, data points that are distant in time series not exchanging messages (for example, the time shown in FIG. 3). A typical still image at t = 10) can also be selected as one of the representative frames by clustering.

Further, regarding the adoption of a typical still image in TB-AP, the same typical still image can be taken even if the time frame is exceeded for consecutive similar frames. Specifically, it is assumed that the typical still image e _i for the i-th frame is determined. At this time, it is determined that the feature vectors from the k-th frame to the (i−1) -th frame before the i-th frame are similar using a threshold or the like, and the typical still image e _k for the k-th frame and the If it is determined that the typical still image e _i for the i frame is also similar, the typical still image e _k can be used as it is without generating a new typical still image e _i . As a result, the performance of the present invention can be improved.

As described above, in the present embodiment, the similar moving image search processing device 2 executes the processing based on the TB-AP, so that a set of typical still images taking time series into consideration is obtained as a query moving image and a target moving image. Respectively. Then, the extracted feature amount of the representative frame is determined as the feature amount of the moving image. In particular, the TB-AP employed in this embodiment has a time window that slides with the selected data point, so the number of message exchanges is limited, and the complexity of message transmission is O (ωn). On the other hand, the complexity of a normal AP is O (n ² ). In other words, since the calculation cost of the AP is proportional to the square of the number of data points, there is a problem that a calculation time is required when the number of data points increases. On the other hand, in the present embodiment, generally, ω <n, so that TB-AP can reduce the calculation cost lower than AP.

  Regarding the normalization of the data related to the similarity and the distance measure, the Similarity measure s (i, j) may be any as long as the algorithm converges. If the parameter λ is set appropriately, the form of the Similarity measure s (i, j) in Equation 3 results in convergence of the algorithm.

The standard choices for bar (D) are as follows:
(A) Average value of all data distances: This is allowed only when the data size is small.
(B) if each of the feature vector x _t is the sum of the components is normalized to have only non-negative components become 1, bar (D) is √2, √ ((d-2 ) / (2d )), Or √ (1- (1 / d)):
Here, d represents the dimension of the feature vector x _t existing in the simplex. √2 is the edge of the simplex, √ ((d−2) / (2d)) ≈1 / √2 is the radius of the inner sphere, and √ (1− (1 / d)) ≈1 is the outer sphere Represents the radius. Note that ≈ means an approximate symbol, and the approximation is effective when d is sufficiently large.
In addition to the above criteria, it is possible to experimentally obtain an optimum value according to the use of the system.

  For example, the above-mentioned CSD uses HSV (Hue-Saturation-Lightness) color space representation, and the ratio of each value of H, S, V is quantized to H: S: V = 12: 8: 8. Is preferred. In this case, each image is expressed by 12 × 8 × 8 = 768 colors, and the feature amount by CSD is a 768-dimensional histogram, so d = 768, and a value representing the radius of the outer sphere is used as bar (D). .

  Next, as a TB-AP alternative method, a clustering method that considers the elements of the elapsed time of moving images will be described. The improved k-means method (k-average method) and harmonic competition method will be described with reference to FIG. The left side of FIG. 7 shows a time window W that slides on the time axis in TB-AP, and the right side of FIG. 7 shows a time window that slides on the time axis in the improved k-means method and harmonic competition method. W is shown.

  TB-AP uses a series of image frames in a time window W that slides so as to overlap along a time axis as a calculation target. On the other hand, in the improved k-means and harmonic competition, which are alternative techniques, image frames in the time window W without overlap are targeted for calculation. That is, in TB-AP, the shift width of the time window W is smaller than the frame length of the time window W, and the time window W slides by one or more frames, whereas in the improved k-means and harmonic competition, the time window The shift width of W matches the frame length of the time window W, and the time window W is sliding in units of the time window W. Such a method is described as time-partition with respect to time-bound, and the improved k-means is referred to as time-partition k-means (TP k-means). ) As appropriate.

  When improved k-means (TP k-means) or harmonic competition is used, an image frame is divided into time windows W along the time axis, and clustering is performed in the divided time windows W. The representative point vector obtained here does not correspond to the original image itself, but is a feature vector of a similar image (artificial image) generated by calculation based on k-means or harmonic competition. Therefore, a two-step calculation procedure is required in which a frame having features most similar to the representative point vector is selected as a typical still image. The typical still image obtained by this two-stage calculation is close to the typical still image of TB-AP.

  Hereinafter, the processing flow of TP k-means will be described with reference to FIG. First, the moving image is divided by the time window W along the time axis and divided into a plurality of frames divided by the time window W. The frame is divided into k equal parts within the divided time window W, and the frame is allocated to each cluster. Subsequently, the center of gravity of the cluster is obtained based on the feature vector of the frame belonging to each cluster. The center of gravity can be obtained by the arithmetic average of feature vectors. For each frame, the distance between the feature vector of the frame and all the centroids in the time window is calculated, and re-clustering is performed based on the result. After the reclustering, the center of gravity of the cluster is obtained again, and the recalculation and reclustering of the distances between all the centroids in the time window and each frame are performed until the processing converges. In each cluster, a frame closest to the center of gravity after convergence is selected as a typical still image. This process is performed while moving the time window.

  Next, time-split k-means (TS k-means), which is another clustering method using k-means, will be described with reference to FIGS. 9A to 9E. Note that, in each of these drawings, consecutive numbers 1 to 25 correspond to time t of time-series moving image frames. In the TP k-means, a moving image is divided by a time window W of a predetermined size, and clustering based on k-means is performed in the time window W. In contrast, TS k-means is a technique for performing clustering based on k-means for all frames of a moving image. However, it differs from general k-means in that frame allocation and cluster subdivision are performed in consideration of time series.

  In TS k-means, the following processing is performed. First, all frames of a moving image are divided into k equal parts based on a time series, and a frame is allocated to each cluster. Subsequently, the center of gravity of the cluster is obtained based on the feature vector of the frame belonging to each cluster. The center of gravity can be obtained by the arithmetic average of feature vectors. As an example, in FIG. 9A, a total of 25 moving image frames are divided into five in time series (k = 5), and centroid 1 to centroid 5 are calculated for each cluster surrounded by a curve. Next, for each frame, the distance between the feature vector of the frame, the centroid of the cluster to which the frame currently belongs, and the centroids of the clusters before and after the time series are calculated, and reclustering is performed based on the result. As shown in FIG. 9B, for example, for the frame at time t = 18, the distance between the centroid 4 of the current cluster and the centroid 3 and centroid 5 of the previous and subsequent clusters adjacent in time series is calculated. . As a result, since the distance between this frame and the center of gravity 3 is the shortest, in the reclustering, as shown in FIG.

  Next, the discontinuity of the time series is determined in each cluster that has been reclustered, and if there is a discontinuity, the cluster is divided. As shown in FIG. 9D, for example, for clusters of centroid 3 and centroid 4, these clusters are divided between frames that are discontinuous in time series. When a cluster is divided, the number of clusters increases from k. However, when a cluster disappears due to reclustering, the number may decrease. After re-clustering and dividing, the center of gravity of the cluster is obtained again, and recalculation and re-clustering of the distance between the cluster to which the center of gravity belongs and each frame belonging to the cluster in which the time series belongs are performed until the process converges. In the example shown in FIG. 9E, after dividing the cluster of the centroid 3 and the centroid 4, the centroid 1 to the centroid 7 are calculated for each of the seven clusters, and the distance calculation described above is performed for each frame at time t = 1 to 25. Repeat reclustering until convergence. Then, a frame closest to the center of gravity after convergence in each cluster is selected as a typical still image.

  Next, a clustering method using an improved pairwise nearest neighbor (PNN) method will be described. PNN first calculates the distance between all given data points. Next, data points are merged in order from the closest data points. This is repeated until all the data points after merging are more than a certain distance. This is an algorithm for obtaining a cluster group in which data points having high similarity are collected as one set. The normal PNN is described in “W. Equitz,“ A new vector quantization clustering algorithm, ”IEEE Trans ASSP, vol. 37, pp. 1568-1575, 1989”. In an ordinary PNN, the average value of feature values of a pair is used to represent the pair. However, in the improved PNN described in the present embodiment, one of the pairs is left and the other is removed as described later. Is different. Further, there are a Time-Bound PNN (TB-PNN) and a Time-Partition PNN (TP-PNN) depending on how the time window W is set in consideration of the time series.

FIG. 10 is a diagram for explaining improved PNN merging. Figure, ○ and ● represent the data points x _i which corresponds to the feature amount of each frame, ■ represents the center of gravity of all data points. In PNN, after calculating the distance between all given data points, the closest pair of those data points is found. In the example shown in FIG. 10, the data points a and b correspond to the pair with the closest distance. The pair is then merged. “Merging” in the improved PNN according to the present embodiment means that the data near the center of gravity of all the data points is left as it is, and the far data is deleted. In the example shown in FIG. 10, the data point b is deleted and the data point a is left. In the normal PNN, the center of gravity (midpoint) of the data point a and the data point b is calculated and used as a representative value of a and b. Thereafter, the process is repeated until the distance between the remaining data points reaches a certain value (δ) or more, and the frame corresponding to the remaining data points becomes a typical still image. Each typical still image treats a frame merged with the typical still image as a member.

  FIG. 11 is a diagram for explaining the above-described TP-PNN. Similar to TP k-means, in the improved TP-PNN, a moving image is divided by a time window W having a predetermined size, and clustering based on the PNN is performed in the time window W. At this time, the difference from normal PNN is that the center of gravity (arithmetic average of feature values of each frame) is first calculated for each time window W, and the distance to the center of gravity when two data points (frame feature values) are merged. The data points that are close to each other are left as they are, and the data points that are far are deleted. This process is repeated until the distance between adjacent pairs becomes greater than a predetermined value δ, and the frame corresponding to the remaining data point is set as a typical still image. Each typical still picture belongs to a group of frames deleted by merging so far. In TP-PNN, this process is performed for each time window (not overlapping).

  On the other hand, as shown in FIG. 12, TB-PNN limits the sliding time window in the same manner as TB-AP, and calculates the distance from only the time window range of each data point. Then, the data points are merged in order from the shortest distance among the combinations of data points limited by the time window. However, when merging, unlike TP-PNN, the one close to the center of gravity of the entire video frame is left as it is, and the one far away is deleted. This process is repeated until the distance between adjacent pairs becomes a predetermined value δ or more. Similarly to TP-PNN, a frame corresponding to the remaining data point is set as a typical still image. Each typical still picture belongs to a group of frames deleted by merging so far.

  As described above, the search accuracy can be improved as compared with a normal AP by introducing one or a plurality of representative frames by introducing a time limit in consideration of an element of the elapsed time of a moving image. Similar results can be expected for the usual k-means and PNN methods. TB-AP makes it possible to select a representative frame in one step from a series of image frames in the time window, which can simplify the calculation procedure. The other four methods of TP-k-means, TS k-means, TB-PNN, and TP-PNN have the advantage that the amount of calculation is smaller than that of TB-AP. Regarding accuracy, it may be slightly inferior to TB-AP, but it is insignificant, and both can maintain high accuracy.

  When selecting a representative frame that keeps the time series described above, if the time window width is too small or a scene with almost no change in the moving image continues, representative frames are selected more than necessary by successive similar frames. There is a possibility that. In such a case, a difference between representative frames having consecutive time sequences is obtained. If the difference is equal to or less than the threshold value ε, redundant representative frames can be obtained by deleting subsequent representative frames and grouping them. The frame can be deleted.

Calculation of similarity by analysis between moving images Next, a method of calculating similarity by analysis between moving images used when comparing different moving images having different sets of typical still images will be described. In order to search between moving images based on content, it is necessary to compare the similarity between the query moving image and the target moving image. Here, one of the two methods described below is performed. In the present embodiment, the comparison target is a moving image, but it is not limited to this as long as it is time-series data having an exemplar such as a representative frame.

(1) Comparison of Similarity Applying M Distance for Finding Global Alignment In this embodiment, a distance called M distance is used as a scale for comparing similarity. The M distance for global alignment is a generalization of the Levenshtein distance (L-distance) for measuring the similarity of character strings and the Needleman-Wunsch algorithm for obtaining global alignment in bioinformatics. Therefore, the M distance for global alignment includes the Levenshtein distance and the Needleman-Wunsch algorithm as special cases.

  Hereinafter, an algorithm using the M distance for global alignment will be described in detail with reference to the flowchart of FIG.

S310: Prepare typical still images of the query moving image and the target moving image For the moving image v _A corresponding to the query moving image, a set of typical still images {e ^A _i } and a frame owned by the typical still image ^A set of numbers {E ^A _i } (i = 1, 2,...) Is given. _A similar set {e ^B _j }, {E ^B _j } (j = 1, 2,...) Is also given to the moving image v _B corresponding to the target moving image to be compared with the moving image v _A. It is done. Here, the Similarity measure s (i, j) in Equation 3 is selected as the similarity between representative frames.

S320: Create a table and follow the path according to the following dynamic programming procedure.
_A matrix of (m + 1) × (n + 1) is created as a global alignment table, where m is the number of typical still images of the moving image v _A and n is the number of typical still images of the moving image v _B. Each component f (i, j) of the matrix is calculated by the following procedure by dynamic programming.

  First, in S321, a gap penalty g set as a design parameter is selected. By introducing this gap penalty g, even if the number m of the typical still images of the query moving image and the number n of the typical still images of the target moving image are different from each other, Similarity can be determined correctly.

  In the next S322, for the row of {i = 0} and the column of {j = 0}, the matrix g of the table is configured so that the value decreases stepwise by multiplying the gap g by the total number of frames. Calculate each component. Specifically, the row of {i = 0} calculates the matrix component f (0, j) based on the following formula 8.

  For the column of {j = 0}, the matrix component f (i, 0) is calculated based on the following equation (9).

  Furthermore, in S323, starting from the position of (i, j) = (1, 1), the remaining matrix components f (i, j) other than the row of {i = 0} and the column of {j = 0} are It calculates based on following Formula 10.

  Then, an arrow is drawn toward the component that gives the maximum value. This calculation is continued until all matrix components are calculated. Here, r (i, j) represents a weight reflecting the ownership of the typical still image, and the value is calculated based on the following equation 11, for example.

  As shown in Equation 10, the matrix component f (i, j) here is sequentially generated in the order in which the number of rows or columns increases. Then, the matrix component f (i, j) of i row and j column passes through (i-1) row j column and reaches i row j column, i row j through i row (j-1) column. The route with the highest cost is selected as the optimum route from the route to the column and the route to the i row and j column via the (i-1) row (j-1) column, and the value at that time is the value of the matrix It is determined sequentially as a component.

S330: Determination of global alignment After calculation of all matrix components constituting the table is completed, a path of global alignment is obtained by tracing from the last row and last column components of the matrix f (i, j) based on the arrows. be able to.

S340: Calculation of similarity The similarity u (A, B) between the moving image v _A and the moving image v _B is calculated based on the following equation (12).

However, h (x) is a positive increase function, and f _last is a numerical value of the last row and last column of the matrix. W is an averaging function. The averaging function can be, for example, an arithmetic average of Σ _i E _i ^A and Σ _i E _i ^B.

(2) Comparison of Similarity Applying M Distance for Obtaining Local Alignment When the moving images to be compared are greatly different from each other, the above-described global alignment is deviated from human senses. In such a case, local alignment can be used. Local alignment is useful when the sequences are not similar overall and you want to find partial similarities. The algorithm described below is an adaptation of the above global alignment method to local alignment, and uses M distance for local alignment as a measure of similarity. This M distance for local alignment is a generalization of the Smith-Waterman algorithm for obtaining local alignment in bioinformatics. Therefore, the M distance for local alignment applied in the present embodiment includes the Smith-Waterman algorithm as a special case.

  Hereinafter, an algorithm using the M distance for local alignment will be described in detail with reference to the flowchart of FIG. However, since there are many procedures that overlap with the already described M distance for global alignment, only the differences are described.

S310: Prepare typical still images of the query moving image and the target moving image. This is the same processing procedure as the M distance for global alignment.

S320: Create a table and follow the path according to the following dynamic programming procedure.
_A matrix of (m + 1) × (n + 1) is created as a local alignment table where m is the number of typical still images of the moving image v _A and n is the number of typical still images of the moving image v _B. Each component of the matrix is calculated by the following procedure by dynamic programming.

  First, in S321, a gap penalty g set as a design parameter is selected. This is the same processing procedure as the M distance for global alignment.

  In next step S322, all components are set to 0 for the row of {i = 0} and the column of {j = 0}.

  Furthermore, in S323, starting from the position of (i, j) = (1, 1), the remaining matrix components f (i, j) other than the row of {i = 0} and the column of {j = 0} are It calculates based on following Formula 13.

  Then, an arrow is drawn toward the component that gives the maximum value that is not zero. On the right side of Equation 13, unlike the Equation 10, the leftmost component in the max symbol is 0. Thereby, when the score value becomes negative, the value can be reset to 0 and the alignment can be newly started.

S330: Determination of local alignment After calculation of all matrix components constituting the table is completed, the path of local alignment is traced from the component of the maximum value f _{max in} the matrix f (i, j) based on the arrow. Can be obtained.

S340: Calculation of Similarity Using the same processing procedure as the M distance for global alignment, the similarity u (A, B) between moving image v _A and moving image v _B is calculated based on Equation 12.

As described above, the similar moving image search processing apparatus 2 executes the procedure of extracting feature amounts for frames, selecting representative frames that maintain time series, and calculating similarity by analyzing between moving images. calculates a similarity u (a, B) of each of the moving image _{v B} for video images _{v a.} Thereby, a similar moving image similar to the query moving image can be searched from the target moving image.

  Next, the details and results of the experiment using the moving image retrieval apparatus will be described.

  First, as for the experimental data, since the end user of the search is a human, the final determination of the similarity is strongly dependent on the sensitivity of the human. Therefore, it is desirable that the determination of similarity is not dependent on the subject as much as possible. However, with a simple set, it would be too easy to judge even with a plain device, so a data set that satisfies the following conditions was prepared in the experiment.

・ The original video data is not labeled.
・ The precision and recall in the search can be automatically judged.
It is assumed that each moving image has information that captures the time change of the scene so that the time-dependent property as shown in the lower right of FIG. 3 can be determined.

  FIG. 14 shows the generation process of the original moving image data. The group A moving image represents a “bird's eye view”, and the group B moving image represents a “bird”. In the experiment, 20 animation classes are created from these groups A and B (see FIG. 15). Since each class 1 to 20 constitutes 21 moving images, a total of 420 different test videos were used.

・ Preliminary experiment
Since the convergence of the AP algorithm depends on the data, it is necessary to confirm the range of the parameter λ in Equations 5 and 6. As a set of trial data, five groups of Gaussian clusters were generated, and further divided into two groups for each time direction to form ten groups. FIG. 16 illustrates these data.

  With respect to the collection of trial data shown in FIG. 16, experiments were performed on the TB-AP proposed in the present embodiment and a normal AP. The main purpose is to confirm the range of the parameter λ for convergence and the effect of the time window parameter ω. Through repeated experiments, the selection of the parameter λε [0.3,1.0−ε] depends on the characteristics of the data. However, ε is a small positive amount, and ε = 0.02 in the experiment of FIG.

Table 1 shows the tendency of convergence and the calculation time. Convergence was checked by the sum of a (i, k) and r (i, k) normalized by the initial value to see if it was less than ^10-5 . Table 1 shows the number of typical still images “exemplars”, the number of iterations “iterations”, and the number of message passing “message passing” when the parameter ω of the time window W is changed. From Table 1, for example, when ω = 10, it can be seen that the TB-AP of the present embodiment is 6 times faster than the message transmission of a normal AP.

  When ω = 1, it means that all frames are typical still images. ω = n = 100 corresponds to the case of a normal AP that does not include time-dependent typical still image extraction. This was obtained as a comparative example in order to see the convergence tendency. In the case of ω = 5, 10, 20, 50, a monotonic increase is observed in the number of message transmissions, which is directly related to the CPU time because of the size of the time window. However, since the ratio of the number of message transmissions of TB-AP to the number of message transmissions of normal AP is kept small, it can be said that these are in the desirable range.

  TB-AP was applied to the prepared moving image set (classes 1 to 20) of FIG. 15 to find a typical still image with a reliable frame. This is an important procedure for assigning “numerical annotations” to each moving image. The unorganized set of moving images shown in FIG. 15 is structured in terms of numerical tags. Is equivalent. Such tags can be calculated either online or offline.

Thus, for each typical still image of the two classes of moving images (moving image v _A and moving image v _B ) obtained by TB-AP, they are obtained from their Similarity measures s (i, j). An example of the obtained distance matrix is shown in FIG. In the figure, since the moving image v _A has four typical still images and the moving image v _B has three typical still images, the matrix is a 3 × 4 matrix. The numbers in the table represent the components of the distance matrix between the typical still images (representative frames) of the moving image v _A and the moving image v _B. For example, the representative frame 2 of the moving image v _A and the moving image v _B The distance from the representative frame 3 is 0.26. The distance is quantified corresponding to the degree of similarity between the representative frames. Here, the smaller the distance, the more similar the representative frames, and the larger the distance, the more different the representative frames are from each other. Become.

FIG. 18 is an example of a table created by applying the M distance for global alignment described above. In creating this global alignment table, the algorithm parameters were set to bar (D) = 1 / √ (1− (1 / d)), d = 3 × 256 = 768, g = 0.05, respectively. . In the figure, numeral 26.60 described at the lower right of the table is the numerical value f _last of the final component, and the optimal path traced from there is shown by shading as global alignment. Thereafter, in order to obtain the similarity u (A, B) between the moving image v _A and the moving image v _B , the calculated numerical value f _last is normalized by an averaging function. For example, when normalizing by the sum of products of the gap penalty g and the set of the number of frames, in FIG. 18, the sum is 0.05 × (8 + 12 + 10 + 7) + 0.05 × (12 + 7 + 10) = 1.85 + 1.45 = 3.30. Therefore, the normalized similarity u (A, B) is calculated as 26.60 / (1.85 + 1.45) = 0.06. The matching pattern can be found by tracing the arrow from the last row and last column of the matrix. In this way, it is possible to calculate an appropriate similarity u (A, B) between the moving image v _A and the moving image v _B based on the weighted M distance for global alignment.

FIG. 19 is an example of a table created by applying the M distance for local alignment described above. The algorithm parameters were set as bar (D) = √ (1− (1 / d)), d = 3 × 256 = 768, g = 0.05. In the local alignment table, it can be confirmed that the components whose scores are negative in the global alignment table, that is, the components calculated in the equations 8 and 9, are all reset to zero. Here, the maximum value f _max is 26.95, and the optimum path traced from the component of the maximum value f _max is indicated by shading as local alignment. Thus, even if the moving images to be compared are greatly different, the appropriate similarity u (A, B) is calculated between the moving image v _A and the moving image v _B by the weighted M distance for local alignment. Is possible.

  As a preferred modification in the global alignment, the above-described calculation of the similarity based on the M distance has an arbitrary width r with respect to the ratio of the entire sequence as shown in FIG. 20 in order to reduce the amount of calculation. It can also be used for calculations. This is a calculation method that is speeded up by specializing the M distance in similarity calculation of content-based moving image search. In FIG. 20, the blacked-out portion indicates a portion where the calculation is omitted. This figure also corresponds to the table of FIG. 18 with global alignment. In this case, the range of calculation is determined by the following equation (14).

In Equation 14, m is the number of representative frames of the moving image v _A , and n is the number of representative frames of the moving image v _B. The width r is a parameter that represents the degree of narrowing of the calculation range in the similarity comparison, and is 100 × (1 compared to the calculation for calculating all the matrix components by narrowing the calculation range as shown in Equation 14. -R) The calculation amount can be reduced by ² %. In the example of FIG. 20, although the calculation range is narrowed, it can be confirmed that the same result as the calculation result of FIG. 18 in which the calculation range is not narrowed is calculated.

  Next, the relationship between the above-described time window parameter ω and the recall and relevance ratio after calculating the similarity u (A, B) will be described. Here, after all the moving images are classified in descending order of similarity, the recall rate and the matching rate are defined as in the following equation (15).

Recall rate: = Number of correct moving images found by search / Number of moving images belonging to the same class Precision: = Number of correct moving images found by searching / Number of upper moving images searched

  The reproduction rate means the ratio of search results that satisfy the search request to all moving images that satisfy the search request, and is an index that represents the completeness of the search. On the other hand, the relevance rate represents the ratio of search results that satisfy the search request to all search results, and is an index that represents the accuracy of the search. Generally, the higher the recall and relevance rate, the higher the search performance.

  FIG. 21 shows the result of the precision-recall rate curve obtained in the experiment. In the figure, the vertical axis represents the precision, and the horizontal axis represents the recall. The higher the curve goes to the upper right of the graph, the higher the search performance. Here, each experiment with time window parameters “ω = 5” and “ω = 10” to which TB-AP is applied is shown. As described in the frame in the figure, the function h (x) expressed by Equation 12 was defined, and the coefficient a used therein was set to a = 1. For comparison, the experimental results of “Plain AP” using ordinary AP are also shown.

  From these experimental results, in the region of interest that requires a recall rate of up to 20% (the region surrounded by “ROI” in the figure), the precision of TB-AP in this embodiment is very high, and the processing capability is satisfactory. It can be read that On the other hand, since the normal AP does not consider the time axis, it can be seen that the performance is inferior to that of the TB-AP of the present invention.

  Next, differences in search results due to differences in clustering methods used when selecting representative frames will be described. FIG. 22A and FIG. 22B show the experimental results of the precision-recall rate curve in the five clustering methods described above.

  As experiment data, 2100 different moving images were prepared. Next, divide these into 21 pieces to make a total of 100 movie sets. In each set, one of the 21 moving images was used as a query, and 10% or more frames of this query were inserted into the remaining 20 moving images at positions obtained by uniform random numbers. As a result, 2000 test videos excluding the query were set as the search target group.

  With respect to the above experimental data, feature amounts for frames were extracted, representative frames for keeping time series were selected, and similarity was calculated by analysis between moving images, and a moving image related to the query was searched. The feature quantity was CSD, and the similarity was obtained based on local alignment.

  FIG. 22A shows relevance ratio-recall ratio curves in five clustering methods when similarity is calculated without performing post-processing of redundant representative frame deletion during clustering. All of the five methods yielded good results with almost no difference.

  FIG. 22B shows relevance ratio-recall ratio curves in five clustering methods when post-processing of redundant representative frame deletion is performed during clustering. At this time, the threshold ε = 0.05. By reducing the number of representative frames to almost half, the matching rate is reduced compared to those without post processing, but the difference is small.

  In the above five clustering methods, other TP k-means, TS k-means, TP-PNN, and TB-PNN have a smaller calculation amount than TB-AP, so that the processing speed can be significantly increased. Moreover, in order to avoid local optimization, it is necessary to devise by setting the initial value, but for TB-AP, TP-PNN, and TB-PNN, inappropriate local optimization does not occur from the algorithm. .

  Next, as another experimental example, an experimental example in which the feature amount related to the video signature is applied to the present invention instead of the CSD used in the experimental example will be described below. Video Signature is a specification published in the ISO / IEC standard (Multimedia content description interface, International Standard of ISO / IEC 15938-3, Amendment 4 (2010)), to determine the identity of content. The feature quantity that can be uniquely identified is defined. Since it is possible to detect content duplication (copying) with high accuracy, it is expected that it can be used to detect unauthorized copying and unauthorized distribution on the Internet.

  In the Video Signature standard, the feature amount data is composed of Frame Signature, Frame Confidence, and Word that describe the feature of each frame, and BagOfWords that describes the feature of each section composed of a plurality of continuous frames. In this experimental example, calculation is performed by paying attention to the frame signature. Note that this does not mean that it is limited to Frame Signature, but other feature quantities in the Video Signature standard can be used, or multiple feature quantities can be used as composite feature quantity vectors as described above. it can.

  The frame signature is a feature quantity that divides each frame of the video into partial areas, ternary the luminance difference between the partial areas, and represents it as a 380-dimensional vector. At this time, the central region in the frame is sampled as a partial region more densely than the peripheral region. In this experimental example, each frame was divided into 32 × 32 partial areas, and the average value of Y luminance in the color space of YCbCr was used as luminance information of each partial area.

Similar to the previous clustering experiment, 2100 different moving images were prepared as experimental data. Next, divide these into 21 pieces to make a total of 100 movie sets. In each set, one of the 21 moving images was used as a query, and 10% or more frames of this query were inserted into the remaining 20 moving images at positions obtained by uniform random numbers. As a result, 2000 test videos excluding the query were set as the search target group. Further, instead of simply inserting a part of the query, the following five processes are performed on the moving image to be inserted to make it difficult to search.
1) Increase frame rate (delete frame) (frame)
The frame rate is increased by deleting one frame every 2 or 3 or 6 frames (determined by a uniform random number) of the moving image to be inserted.
2) Gray scale (gray)
For each frame image of the moving image to be inserted,
Y = 0.11448 x R + 0.58661 x G + 0.29891 x B
The gray scale conversion is given by the following formula. Here, R, G, and B (red, green, and blue) represent a color space that represents each frame image, and each value takes an integer value of 0 to 255.
3) Decrease brightness (br)
The R, G, and B values of each frame image of the moving image to be inserted are multiplied by a variable P that represents the rate of decrease in luminance. The value of P is a uniform random number in the range of 0.6 to 0.9.
4) Size change (reduction and enlargement) (resize)
Each frame image size of the moving image to be inserted is changed by taking a uniform random number from a range of 0.5 to 2.0 times.
5) JPEG (Joint Photographic Experts Group) conversion (jpeg)
In JPEG conversion, compression is performed by reducing a part of information among frequency components of an image.
In the experiment, the quality of the moving image to be inserted was lowered by increasing the compression rate. In this experiment, OpenCV of the image processing library is used, and compression is performed by taking a uniform random number from 20 to 80 for the variable CV_IMWRITE_JPEG_QUALITY that determines JPEG quality. It was. This variable CV_IMWRITE_JPEG_QUALITY takes a value from 0 to 100. When the value is 100, the image quality is the highest.

  For the above experimental data, we extracted feature quantities for frames, selected representative frames to keep time series, and calculated similarity by analyzing between moving images, and searched for moving images related to the query. . As a comparative example, an experiment using CSD as a feature amount was also performed.

  FIG. 23A and FIG. 23B show the result of the precision-recall rate curve obtained in the experiment. FIG. 23A shows a case where CSD is used, and FIG. 23B shows a case where Frame Signature is used. TP-PNN is used as a typical still image acquisition method. Gap penalty g = 0.2 for each parameter, δ = 0.1 when using CSD, bar (D) = 0.05, distance calculation uses Euclidean distance, Frame Signature In this case, δ = 320, bar (D) = 15, and distance calculation uses an absolute distance. At this time, δ is a threshold value of the condition for merging with TP-PNN. For each parameter, for proper comparison, δ is set so that the compression rate (number of representative frames acquired) is equal, and the value of bar (D) is set so that the result is the best in each method. Is selected.

  As shown in FIG. 23A, when the brightness is low, the search by CSD becomes difficult. When it is made grayscale, it can hardly be searched. Also, JPEG conversion reduced accuracy. However, high accuracy was maintained for frame rate conversion and size conversion. On the other hand, as shown in FIG. 23B, when Video Frame Signature is used, it was confirmed that the difference in accuracy reduction was reduced for all the conversions that were experimented compared to CSD.

  As described above, adopting Frame Signature has an advantage that it is more resistant to changes such as luminance changes than the case of using CSD. In addition, since the data amount of the feature vector is smaller than that of CSD, there is an advantage that distance calculation at the time of comparison can be performed at high speed.

  As mentioned above, the system configuration and algorithm of the moving image search apparatus in the present embodiment have been presented. However, the moving image has a large capacity size, and each typical still image and a set of frames for which they are responsible are usually calculated offline. The Therefore, such a set can be used as a numerical label for the structural information of a large data set. In the present embodiment, considering that each typical still image includes a responsible frame, a novel similarity calculation method including the Levenshtein distance is proposed as the M distance. This M distance can be used for both the global alignment and the local alignment for the selected typical still image.

  Further, in the experimental example in which the difference between the search results by the five clustering methods described above is used, the representative frame is selected using the same clustering method for the query moving image and the target moving image. By doing so, it is possible to select representative frames with the same standard, and it is possible to prevent the occurrence of inconsistency due to selection of representative frames using different standards in the similarity comparison. However, it is not always necessary to use the same clustering method. Considering the data size of the moving image to be searched, the processing speed of clustering, the search accuracy, etc., different clustering methods are used for the query moving image and the target moving image. It may be used to select a representative frame.

  The moving image search processing by the processing device 2 described above can be executed by an electronic circuit using an FPGA (field-programmable gate array) or the like, but all or part of the functions can be executed by a computer. it can. FIG. 24 is an example of a hardware configuration when the computer 100 is used as the processing device 2. A central processing unit (CPU) 102 that performs various controls, a main storage device 103, an auxiliary storage device 104, a communication interface 105, a drive device 106, and the like are connected to the computer 100 via a bus 101.

  The CPU 102 can read out a program stored in the auxiliary storage device 104 and execute the moving image search process according to the present invention in accordance with the instruction. The main storage device 103 is a storage device such as a random access memory, and temporarily stores a part of programs, data, and the like stored in the auxiliary storage device 104, and temporarily stores the processing results by the CPU 102. To do. The auxiliary storage device 104 is a storage device such as a hard disk drive or SSD (Solid State Drive), for example, and stores a program for executing the moving image search processing according to the present invention and data used for the processing. The communication interface 105 is for communicating with an external server or an external storage device via a network. The computer 100 can download a program from an external server or read / write data from / to an external storage device via the communication interface 105. The drive device 106 is used to read data from and write data to the recording medium 107. The recording medium 107 records programs and data, and includes a ROM (Read Only Memory) using a CD (Compact Disk), a DVD (Digital Versatile Disk), or the like. Further, as the recording medium 107, other electronic device memory such as a USB (Universal Serial Bus) memory may be used. In addition, these hardware is shown as an example, can be selected as needed, and other elements may be added.

  In summary, in this embodiment, a system of a moving image search apparatus that calculates a similarity of a target moving image to be searched for a query moving image and searches for a moving image similar to the query moving image. We propose a video retrieval method.

  In particular, in this method, the feature amount extraction unit 22 incorporated in the processing device 2 extracts the first feature amount for each frame from the query moving image frame, and the second amount for each frame from the target moving image frame. Extract features. Next, the moving image representative frame selection unit 23 incorporated in the processing device 2 performs TP k-means, TS k-means on the query feature image with respect to the first feature amount extracted by the feature amount extraction unit 22. , TB-PNN, or TP-PNN is used to acquire the first representative frame in time series, and for the second feature amount extracted by the feature amount extraction unit 22 for the target moving image, A second representative frame is acquired in time series using any one of TP k-means, TS k-means, TB-PNN, or TP-PNN. Then, the similarity calculation unit 24 or the similar moving image display control unit 25 as a moving image search unit incorporated in the processing device 2 performs the first representative frame in the query moving image and the second representative frame in the target moving image. The similarity between the query moving image and the target moving image is obtained from the similarity between the representative frames between and the moving image similar to the query moving image.

  When such a method is adopted, a time restriction that considers an element of the elapsed time of the moving image is introduced to the query moving image and the target moving image, and one or more first representative frames and the second representative frame Each representative frame can be acquired. Then, using the first representative frame and the second representative frame in time series, the similarity between the query moving image and the target moving image is determined, so that the query moving image can be selected from the target moving images. Can be searched with higher accuracy than a normal AP. Similar results can be expected with the k-means and PNN methods. The TP k-means, TS k-means, TB-PNN, and TP-PNN methods require less computation than TB-AP, and as a result, search and grouping that uses the moving image itself as a query, It is possible to easily carry out with high accuracy while keeping the calculation cost low.

  In the above method, it is preferable that the feature quantity extraction unit 22 uses a frame signature defined as a feature quantity related to the video signature as the first feature quantity and the second feature quantity. Thereby, compared with the case where CSD is used, high search accuracy can be maintained even if there is a change such as luminance change. In addition, since the amount of data is small compared to CSD, distance calculation at the time of comparison can be performed at high speed.

In searching for a moving image similar to the query moving image, the number of frames E ^A _i represented by the i-th first representative frame in the query image and the number of frames E represented by the j-th second representative frame in the target moving image. ^B _j is obtained, and the similarity measure s (i, j) between the i-th first representative frame in the query image and the j-th second representative frame in the target moving image is used as the similarity between the representative frames. Get as.

Then, after setting the gap penalty g, when the number of first representative frames in the query moving image is m and the number of second representative frames in the target moving image is n, (m + 1) × ( For each component f (i, j) of the matrix of (n + 1), each component f (0, j) in the row with i = 0 is calculated based on Equation 8, and each component f (i, j) in the column with j = 0 is calculated. 0) is calculated based on Equation (9). Further, when the weight calculated according to the number of frames E ^A _i and E ^B _j is r (i, j), the remaining matrix components f (i) other than the row of i = 0 and the column of j = 0. , J) is calculated based on Equation (10). When the numerical value f _last of the last row and the last column of the matrix is calculated in this way, the similarity u (A, B) between the query moving image and the target moving image is calculated by using the numerical value f _last , for example, using Equation 11. Then, a moving image similar to the query moving image is searched.

  If such a method is adopted, even if the number of the first representative frame and the second representative frame is different between the query moving image and the target moving image, by introducing the gap penalty g, Search can be performed by correctly determining the similarity between the representative frames and the similarity in time series. Also, a global alignment path can be obtained from each component of the matrix f (i, j) thus obtained. In addition, the calculation of similarity suitable for the moving image data obtained here includes the Levenshtein distance and the Needleman-Wunsch algorithm as mere special cases, and can be applied to a technique for measuring the similarity of life information sequences. .

As another method, in the search for a moving image similar to the above-described query moving image, the number of frames E ^A _i and E ^B _j and the similarity measure s (i, j) are acquired, and the gap penalty g is set. Then, for each component f (i, j) of the matrix of (m + 1) × (n + 1), each component f (0, j) in the row where i = 0 and each component f (i, j) in the column where j = 0. 0) are all set to 0. Then, assuming that the weight calculated according to the number of frames E ^A _i and E ^B _j is r (i, j), the remaining matrix components f (i) other than the row of i = 0 and the column of j = 0. , J) is calculated based on Equation (13). When the maximum value f _max is calculated in each element of the matrix in this way, the similarity u (A, B) between the query moving image and the target moving image is calculated by using the numerical value f _max , for example, Equation 12. It is also possible to calculate f _last instead of f _max and search for a moving image similar to the query moving image.

  Also in this case, even if the number of the first representative frame and the second representative frame is different between the query moving image and the target moving image, the similarity between both the representative frames can be obtained by introducing the gap penalty g. It is possible to perform a search by correctly determining the similarity between the sex and the time series. Further, a path of local alignment can be obtained from each component of the matrix f (i, j) thus obtained. In addition, the calculation of the similarity suitable for the moving image data obtained here includes the Smith-Waterman algorithm as a special case, and can be applied to a technique for measuring the similarity of life information sequences.

  The above-described operational effects are similarly realized in a moving image search apparatus capable of executing the above processing and a moving image search program for causing a computer to execute the above processing.

  As mentioned above, although embodiment of this invention was described, this invention is not limited to such embodiment at all, Of course, in the range which does not deviate from the summary of this invention, it can implement with a various form. It is. For example, the setting of each parameter value described above can be appropriately changed via the input device 4.

22 feature amount extraction unit 23 moving image representative frame selection unit 24 similarity calculation unit (moving image search unit)
25 Similar video display control unit (video search unit)

Claims (10)

A moving image search method by a moving image search apparatus for calculating a similarity of a target moving image with respect to a query moving image and searching for a moving image similar to the query moving image,
Extract the first feature value for each frame from the frame of the query video,
Extract the second feature value for each frame from the frame of the target moving image,
For the query moving image, any one of the time-segment k-average method, the time-division k-average method, the time-constrained pair-wise nearest neighbor method, or the time-segment pair-wise nearest neighbor method is applied to the extracted first feature amount. To obtain the first representative frame in time series,
For the extracted second feature value, any one of the time-segment k-average method, the time-division k-average method, the time-constrained pair-wise nearest neighbor method, or the time-segment pair-wise nearest neighbor method is applied to the target moving image. To obtain the second representative frame in time series,
A similarity between the query moving image and the target moving image is obtained from a similarity between the first representative frame in the query moving image and the second representative frame in the target moving image. , Search for the moving image similar to the query moving image,
A moving image search method characterized by the above.   The moving image search method according to claim 1, wherein a frame signature is used as the first feature amount and the second feature amount.   3. The moving image search method according to claim 1, wherein the acquisition of the first representative frame and the acquisition of the second representative frame are performed using the same method. In the search for similar moving images,
The number of frames E ^A _i represented by the i th first representative frame in the query video, the number of frames E ^B _j represented by the j th second representative frame in the target video, and the query video Obtaining the inter-representative frame similarity s (i, j) between the i-th first representative frame and the j-th second representative frame in the target moving image,
Set gap penalty g,
When the number of the first representative frames in the query moving image is m and the number of the second representative frames in the target moving image is n, each component of the matrix of (m + 1) × (n + 1) For f (i, j)
Calculate each component f (0, j) of the row of i = 0 based on the following equation 1;
Calculate each component f (i, 0) in the column of j = 0 based on the following formula 2;
When the weight calculated according to the number of frames E ^A _i and E ^B _j is r (i, j), the remaining matrix components f (i, j) other than i = 0 rows and j = 0 columns are used. j) is calculated based on the following equation 3,
The said similarity between the said query moving image and the said target moving image is calculated using the numerical value of the last row | line | column last column of the said matrix, The any one of Claims 1-3 characterized by the above-mentioned. Video search method. In the search for similar moving images,
The number of frames E ^A _i represented by the i th first representative frame in the query video, the number of frames E ^B _j represented by the j th second representative frame in the target video, and the query video Obtaining the inter-representative frame similarity s (i, j) between the i-th first representative frame and the j-th second representative frame in the target moving image,
Set gap penalty g,
When the number of the first representative frames in the query moving image is m and the number of the second representative frames in the target moving image is n, each component of the matrix of (m + 1) × (n + 1) For f (i, j)
Each component f (0, j) in the row with i = 0 and each component f (i, 0) in the column with j = 0 are all set to 0,
When the weight calculated according to the number of frames E ^A _i and E ^B _j is r (i, j), the remaining matrix components f (i, j) other than i = 0 rows and j = 0 columns are used. j) is calculated based on the following equation 4,
4. The similarity between the query moving image and the target moving image is calculated using a maximum value among the components of the matrix, according to any one of claims 1 to 3. The moving image search method described. A moving image search apparatus for calculating a similarity of a target moving image to be searched with respect to a query moving image and searching for a moving image similar to the query moving image,
A feature amount extraction unit that extracts a first feature amount for each frame related to the query moving image and a second feature amount for each frame related to the target moving image from the frame of the query moving image and the target moving image. When,
For the query moving image and the target moving image, with respect to the extracted first feature quantity and second feature quantity, a time-division k-average method, a time-division k-average method, a time-constrained pairwise nearest neighbor method Or a representative frame for acquiring a first representative frame related to the first feature value and a second representative frame related to the second feature value in time series using either the time-wise pairwise nearest neighbor method. A selection department;
A similarity between the query moving image and the target moving image is obtained from a similarity between the first representative frame in the query moving image and the second representative frame in the target moving image. And a moving image search unit for searching for the moving image similar to the query moving image.   The moving image search apparatus according to claim 6, wherein the feature amount extraction unit uses a frame signature as the first feature amount and the second feature amount. The moving image search unit
The number of frames E ^A _i represented by the i th first representative frame in the query video, the number of frames E ^B _j represented by the j th second representative frame in the target video, and the query video Obtaining the inter-representative frame similarity s (i, j) between the i-th first representative frame and the j-th second representative frame in the target moving image,
Set gap penalty g,
When the number of the first representative frames in the query moving image is m and the number of the second representative frames in the target moving image is n, each component of the matrix of (m + 1) × (n + 1) For f (i, j)
Calculate each component f (0, j) in the row of i = 0 based on the following equation 5:
Calculate each component f (i, 0) in the column of j = 0 based on the following equation 6:
When the weight calculated according to the number of frames E ^A _i and E ^B _j is r (i, j), the remaining matrix components f (i, j) other than i = 0 rows and j = 0 columns are used. j) is calculated based on the following equation 7,
The moving image according to claim 6 or 7, wherein the similarity between the query moving image and the target moving image is calculated using a numerical value of a last row and a last column of the matrix. Image retrieval device. The moving image search unit
The number of frames E ^A _i represented by the i th first representative frame in the query video, the number of frames E ^B _j represented by the j th second representative frame in the target video, and the query video Obtaining the inter-representative frame similarity s (i, j) between the i-th first representative frame and the j-th second representative frame in the target moving image,
Set gap penalty g,
When the number of the first representative frames in the query moving image is m and the number of the second representative frames in the target moving image is n, each component of the matrix of (m + 1) × (n + 1) For f (i, j)
Each component f (0, j) in the row with i = 0 and each component f (i, 0) in the column with j = 0 are all set to 0,
When the weight calculated according to the number of frames E ^A _i and E ^B _j is r (i, j), the remaining matrix components f (i, j) other than i = 0 rows and j = 0 columns are used. j) is calculated based on the following equation 8,
8. The configuration according to claim 6, wherein the similarity between the query moving image and the target moving image is calculated using a maximum value among the components of the matrix. Video search device.   A moving image search program for causing a computer to execute the moving image search process according to any one of claims 1 to 5.