JP4177689B2 - Video feature information generator - Google Patents

Description

[0001]
BACKGROUND OF THE INVENTION
  The present invention generates video feature information that generates simple video feature information that characterizes the video data from the video data.apparatusAbout.
[0002]
[Prior art]
In general, in order to search for desired video data from a large amount of video data, a video search technique is known in which feature quantities are extracted from video data in advance and video data is searched based on the feature quantities. Yes.
[0003]
Conventionally, in order to extract feature amounts from video data for video search, the exact shape of the subject and background in the video data is cut out, and feature amounts of each shape are extracted, or video data is configured. For example, a vector indicating the accurate movement of a large number of feature points arranged on each frame image is extracted as a feature amount.
[0004]
For example, an object (subject) is detected from video data, and a feature quantity including the position, shape, and movement of the object and a feature quantity including background movement are generated as feature quantity data of the video data. There is a technique for performing video search based on the feature amount data (see Patent Document 1).
[0005]
[Patent Document 1]
JP 2000-222584 A (paragraph number 25-30, FIG. 1)
[0006]
[Problems to be solved by the invention]
However, the conventional technique has a problem in that since a detailed feature amount is extracted from video data, a calculation time for calculating the feature amount becomes long. Also, as in the past, when video data is searched based on detailed feature amounts, it is complicated to narrow down search information reflecting the searcher's search intention and to calculate the similarity of search information in video data. Therefore, there is a problem that it takes a long time to search the video data in accordance with the search intention of the searcher.
[0007]
Further, in the conventional technique, when it is desired to express a searcher's search intention with a simple expression such as a sketch representing an object (subject) region, a background region, and a vector indicating their motion in the video data, The more detailed the amount, the more complicated the sketch representation. For this reason, the search information envisioned by the searcher must be expressed in a more complicated manner, and as a result, a search suitable for the searcher's sense (similar sense) cannot be performed.
[0008]
  The present invention has been made in view of the above problems, and by extracting simple information from video data as a feature amount, the calculation time for extracting the feature amount is shortened, and the search is performed. Video feature information generation that enables users to search for desired video data by video search suitable for the user's similar senseapparatusThe purpose is to provide.
[0018]
[Means for Solving the Problems]
  The present invention has been made to achieve the above-mentioned object, and first, claim 1The video feature information generating device described in the above is a video feature information generating device that generates video feature information that characterizes the scene for each scene of the video data, and includes a frame image sampling unit, a node initial setting unit, and a node tracking Means, node classification means, class image feature value generation means, and feature information generation means.
[0019]
According to such a configuration, the video feature information generation apparatus extracts the frame image from the video data at a specific sampling interval by the frame image sampling unit, and extracts the frame image extracted by the frame image sampling unit by the node initial setting unit. A node (reference point) serving as a reference for extracting image features is set in the first frame. Note that the nodes may be set at specific intervals, or the positions may be set based on templates that are determined in advance.
[0020]
  Then, the video feature information generating apparatus searches for nodes having similar image feature amounts in the neighboring image regions by the node tracking means, thereby corresponding to frame images after the first frame.NodeIn addition to tracking, each node image tracked by the node tracking means for each frame image by the node classification meansKnotNodes that are similar in the image feature quantity of the neighboring image area are grouped (clustered) into one class (cluster)..
[0021]
  In addition, the video feature information generation device uses the class image feature amount generation means toFrom the first frame to the last frame of the scene,Classified into the same class (cluster) by node classification meansKnotImage feature of each frame image corresponding to the neighborhood image areaImage feature amount having the smallest distance from the average image feature amount in the class,The classRepresentClass image featureAsGenerate. This class image feature amount is an image feature amount that best represents the features of images of the same class (cluster)..
[0022]
  Then, the video feature information generation device uses the feature information generation unit to generate the class image feature amount generated by the class image feature amount generation unit, the coordinate information of the rectangular area including the node in the class, and the first frame of the scene. Motion information that is a motion vector of the position centroid of the rectangular area from to the last frame is generated as video feature information.
  In the video feature information generation device, the feature information generation unit extracts a representative texture (pattern) from the frame image most similar to the class image feature amount. Then, by representing each of the rectangular areas for each neighboring image area classified by the node classification means with this texture, the characteristics of the rectangular area itself can be expressed by representative image data. Thereby, the video feature information of the video data can be expressed by visual image data.
[0023]
  Claims2The video feature information generating device according to claim 11The video feature information generating apparatus according to claim 1, further comprising a scene dividing unit that divides the video data for each scene, wherein the frame image sampling unit is configured to generate a frame image at a specific sampling interval for each scene divided by the scene dividing unit. Is extracted.
[0024]
According to this configuration, the video feature information generation apparatus detects scene switching (scene change) of the input video data by the scene dividing unit, and divides the video data for each scene. As a result, the video feature information generation apparatus can automatically divide video data for each scene that becomes a break of node features, thereby improving the efficiency for generating video feature information.
[0032]
DETAILED DESCRIPTION OF THE INVENTION
Hereinafter, embodiments of the present invention will be described with reference to the drawings.
[Configuration of Video Feature Information Generation Device]
FIG. 1 is a block diagram showing a configuration of a video feature information generating apparatus according to the present invention. The video feature information generation apparatus 1 divides video data into partial videos (hereinafter referred to as scenes), and these scenes are roughly positioned and sizes of subjects and backgrounds, representative images / patterns (hereinafter referred to as textures). ) And video feature information (simple scene information) expressed by rough motion (hereinafter referred to as motion vector). Here, the video feature information generating apparatus 1 includes a scene dividing unit 2, a frame image sampling unit 3, a node initial setting unit 4, a node tracking unit 5, a node clustering unit 6, a texture clustering unit 7, a cluster An information storage unit 8 and a feature information generation unit 9 are provided. In addition, the video feature information generation apparatus 1 is assumed to input video data from video storage means 20 such as a hard disk that stores video data.
[0033]
The scene dividing unit 2 divides the video data input from the video storage unit 20 into scenes. The scene division in the scene dividing means 2 detects a point (scene change) where the configuration of the video data is largely switched, and divides the video data at each switching point. This scene change can be detected by an existing method. For example, by calculating the difference between time-series consecutive frame images constituting video data, and assuming that the sum of absolute values of the differences is greater than a predetermined threshold, it is assumed that there is no continuity between the frame images Detect scene changes.
[0034]
The scene dividing unit 2 sequentially outputs frame images to the frame image sampling unit 3 for each scene obtained by dividing the video data. The scene dividing unit 2 detects (divides) the next scene when the feature information generating unit 9 described later generates video feature information (scene simple information) of one scene.
[0035]
The frame image sampling means 3 extracts the frame image input from the scene dividing means 2 at a preset sampling interval. In this frame image sampling means 3, when the extracted frame image is the first frame of the scene, the frame image (first frame) is output to the node initial setting means 4, and the extracted frame image is other than the first frame of the scene. In some cases, the frame image (after the first frame) is output to the node tracking means 5.
[0036]
The node initial setting means 4 sets the nodes in a grid pattern at predetermined intervals on the frame image (first frame) when the frame image of the first frame is input from the frame image sampling means 3. Here, the nodal point initial setting means 4 is configured by adding the neighborhood image feature amount calculation unit 4a. Note that a node refers to a point serving as a reference for extracting a feature of a frame image. In addition, the nodes may be set at specific intervals, or the positions may be set based on templates that have been arranged in advance.
[0037]
The neighborhood image feature amount calculation unit 4a calculates an image feature amount from the vicinity of a node set on the frame image (first frame) (neighboring image region: a region having a predetermined size (such as a square region)). . As this image feature amount, a feature amount common in the field of image processing may be used. For example, the average value of each color component of RGB, the distribution of the edge amount when the image is edged, the fractal dimension indicating the complexity of the image, and the like can be used.
[0038]
Here, the nodes will be described with reference to FIG. 7 (refer to FIG. 1 as appropriate). FIG. 7 is a diagram showing a state in which the node initial setting means 4 has arranged the nodes PT in a grid pattern at a certain interval on the frame image of the first frame. Here, for convenience of explanation, a node PT is illustrated on the frame image, but this node PT only indicates a position corresponding to a grid-like point on the frame image. In this way, the node initial setting means 4 sets the nodes PT in a grid pattern with Npx horizontal and Npy vertical (Npx and Npy are preset).
[0039]
  In addition, node initial settingmeansThe neighborhood image feature amount calculation unit 4a of 4 calculates image feature amounts from neighborhood regions (neighboring image regions: square regions of Rfv pixels × Rfv pixels) centered on each node PT, and associates them with each node PT.
[0040]
The node initial setting means 4 generates information (node information PTI) as shown in FIG. 6A for each node, and stores the node information PTI in the cluster information storage means 8. As shown in FIG. 6A, the node information PTI includes a frame image number (frame number) PIfn in which the node is set, and the current position of the node on the frame image (coordinate information: x coordinate and y coordinate). PTIps and an image feature amount (neighboring image feature) PIfv calculated from the neighborhood region of the node at the current position.
Returning to FIG. 1, the description will be continued.
[0041]
The node tracking unit 5 includes an image feature amount (neighboring of the neighborhood image region of the node associated with the previous frame image stored in the cluster information storage unit 8 in the frame image input from the frame image sampling unit 3. Image feature amount) is referred to, and a neighborhood image region similar to the neighborhood image feature amount is tracked. Here, the node tracking unit 5 is configured by adding the neighborhood image feature amount calculation unit 5a. Here, the tracking of the neighborhood image region is performed by tracking the position of the corresponding node.
[0042]
The neighborhood image feature quantity calculation unit 5a calculates an image feature quantity from a neighborhood area centered on a node (neighboring image area: a circular area having a specific radius). As a result, the node tracking unit 5 compares the adjacent image feature quantity of the node associated with the previous frame image with the image feature quantity of the current frame image in the vicinity area of the node position, and the most difference is obtained. A pixel having a smaller image feature amount is set as a node of the current frame image.
[0043]
Note that the image feature amount calculated by the neighborhood image feature amount calculation unit 5a is a feature amount that is general in the field of image processing, as in the neighborhood image feature amount calculation unit 4a. Here, the neighborhood image feature quantity calculation unit 4a and the neighborhood image feature quantity calculation unit 5a are configured as separate units, but are configured as one unit, and the node initial setting unit 4 and the node tracking unit 5 are common. It is good also as making it operate.
[0044]
Here, with reference to FIG. 8 (refer to FIG. 1 as appropriate), tracking of nodes on a frame image will be described. FIG. 8 is a diagram illustrating a state in which the node tracking unit 5 tracks the node on the frame image. In this case, the node PT (marked with ● in the figure) in the previous frame image (here, the first frame in FIG. 7) is the node PT in the current frame image._B(X mark in the figure) indicates that tracking was performed. As described above, the node tracking unit 5 moves the node to a position where the difference in the image feature amount is minimized based on the content of the frame image that changes every moment.
Returning to FIG. 1, the description will be continued.
[0045]
The node clustering means (node classification means) 6 classifies (clusters) the nodes in one frame image in which the position of the node is determined by the node initial setting means 4 or the node tracking means 5 based on the neighborhood image feature amount and the coordinate information. The classification information (cluster information) is stored in the cluster information storage means 8.
[0046]
The node clustering means 6 generates information (node cluster information PC) as shown in FIG. 6B and stores (stores) the node cluster information PC in the cluster information storage means 8. As shown in FIG. 6B, the node cluster information PC includes, for each frame image, all node PCpt belonging to the node cluster (to be a pointer indicating the node information PTI) and neighboring image features of all nodes belonging to the node cluster. The average image feature amount PCfv of the amount PTFvv (FIG. 6A) and the cluster number PCcn assigned to the node cluster.
[0047]
The texture clustering means (class image feature quantity generating means) 7 sequentially inputs the frame images in which the positions of the nodes are determined by the node initial setting means 4 or the node tracking means 5, and each of the frames from the first frame to the last frame in one scene An image feature amount that represents a cluster is generated.
[0048]
The texture clustering means 7 creates information (texture cluster information TC) as shown in FIG. 6C in the cluster information storage means 8 and updates (stores) it every time a frame image is input. As shown in FIG. 6C, the texture cluster information TC includes all nodes TCpt belonging to the texture cluster (to be a pointer indicating the node information PTI) and neighboring image feature values PIfv belonging to the texture cluster (FIG. 6). (See (a)) and the average image feature amount TCfv. The average image feature amount TCfv is an N-dimensional vector composed of an average value of each element when the image feature amount is an N-dimensional vector.
[0049]
Further, the texture clustering means 7 generates a representative texture structure representing each texture cluster at the stage when the final frame in the scene is input. Here, among all the nodes TCpt belonging to the texture cluster (see FIG. 6C), the neighboring image feature amount PIfv (see FIG. 6A) of all the nodes belonging to the cluster and all the nodes belonging to the texture cluster. The image feature amount with the smallest distance from the average feature amount TCfv (see FIG. 6C) is set as a representative image feature amount representing a texture cluster. In addition, a rectangular area having a predetermined size is extracted from the nodes having the representative image feature amount to obtain a representative texture image.
[0050]
The representative texture structure has a structure as shown in FIG. That is, the representative texture structure PTX includes a preset representative texture image PTXimg of Tw pixels × Th pixels, and a neighborhood image feature amount (representative image feature amount) PTXfv of the center coordinates of the representative texture image PTXimg.
[0051]
The cluster information storage means (storage means) 8 stores cluster information and is a general storage medium such as a hard disk. The cluster information storage unit 8 stores cluster information CI of each frame image and a representative texture structure PTX generated for each scene.
The cluster information includes the node information PTI (FIG. 6A), the node cluster information PC (FIG. 6B), and the texture cluster information TC (see FIG. 6C) of the frame image. included.
[0052]
The feature information generation means 9 refers to the cluster information storage means 8 and generates video feature information (scene simple information) that characterizes the scene for each scene. Here, the feature information generation means 9 is composed of a scene feature information generation unit 9a and an initialization unit 9b.
[0053]
The scene feature information generation unit 9a includes the coordinate information of the rectangular area including the nodes classified by the node clustering means (node classification means) 6, and the class image generated by the texture clustering means (class image feature value generation means) 7. A feature amount (representative texture image and representative image feature amount) and motion information of a rectangular area are generated as video feature information. The scene feature information generation unit 9a generates video feature information as image data and text data.
[0054]
When generating video feature information with image data, the interior of the rectangular area including the nodes of the cluster is filled with a representative texture image. Then, the motion of the position centroid of the cluster from the first frame to the last frame is drawn in a straight line as the motion vector of the cluster.
[0055]
FIG. 10 shows an example of image data generated by the scene feature information generation unit 9a. As shown in FIG. 10, the rectangular area of the cluster is filled with the representative texture image, and the movement of the cluster is indicated by a straight line. Thereby, the simple information of the scene of the video data can be visualized.
[0056]
Also, when generating video feature information with text data, the frame number of the first and last frames, the coordinate information of the rectangular area including the nodes of the cluster, the representative image feature amount, and the cluster from the first frame to the last frame The motion vector of the position centroid is written into a text file as text data.
[0057]
FIG. 11 shows an example of text data generated by the scene feature information generation unit 9a. As shown in FIG. 11, for each scene, the first frame number Ns, the last frame number Ne, and the coordinate information {(x0, y0), (x1, y1), (x2, y2), (x3, y3)}, the image feature amount (representative image feature amount) of the texture structure {(f (0), f (1), f (2),..., f (N−1)}, and x of the motion vector And y component {vx, vy}, information for one cluster is generated.
[0058]
Here, the representative image feature amount {(f (0), f (1),...) Represents the average value of each RGB color component, the distribution of the edge amount when the image is edged, and the like. Here, the representative image feature quantity is represented as an N-dimensional image feature vector.
[0059]
The initialization unit 9b determines whether or not the frame image is the final frame of the scene, and initializes the cluster information in the cluster information storage unit 8 when the frame image is the final frame. As a result, cluster information of a new scene is generated in the cluster information storage means 8 when a scene change (scene change) occurs.
[0060]
The configuration of the video feature information generation device 1 according to the present invention has been described above, but the present invention is not limited to this. Here, the scene dividing means 2 for dividing the video data into scenes is provided. However, when the video data is input for each scene, the scene dividing means 2 is deleted from the video feature information generating device 1 and configured. Also good. Here, in the scene feature information generation unit 9a, the video feature information is generated as image data and text data, but either one may be generated.
[0061]
In addition, the video feature information generation apparatus 1 can also realize each unit as a function program in a computer, and can also function as a video feature information generation program by combining the function programs.
[0062]
[Operation of video feature information generation device]
Next, the operation of the video feature information generation apparatus will be described with reference to FIG. 2 (refer to FIG. 1 as appropriate). FIG. 2 is a flowchart showing the operation of the video feature information generation apparatus.
[0063]
First, the video feature information generation apparatus 1 inputs video data from the video storage unit 20 by the scene dividing unit 2 and detects a scene change (scene change), thereby dividing the video data into scenes (step S1). ). Then, the frame image sampling means 3 extracts frame images from the scene at preset sampling intervals (step S2).
[0064]
Here, the frame image sampling means 3 determines whether or not the extracted frame image is the first frame of the scene (step S3). If it is the first frame (Yes), the frame image is sent to the node initial setting means 4. Output. Then, the node initial setting means 4 sets the nodes in a grid pattern with respect to the frame image of the first frame (step S4), and the neighborhood image feature quantity calculation unit 4a calculates the image feature quantity from the neighborhood of each node ( Step S5).
[0065]
On the other hand, if the extracted frame image is other than the first frame (No in step S3), the frame image is output to the node tracking means 5. Then, the node tracking means 5 calculates the image feature amount around the node (circular region having a specific radius) in the current frame image (step S6), and refers to the image feature amount near the node in the previous frame image. The position of the node in the current frame image is moved (tracked) (step S7).
Thus, in step S5 or step S7, the position of the node and the image feature amount in each frame image are generated as the node information PTI (see FIG. 6A).
[0066]
Then, the video feature information generation device 1 classifies (clusters) the nodes based on the node information of the frame image by the node clustering means (node classification means) 6 and assigns a cluster number to the cluster information (nodes). Cluster information PC: see FIG. 6B) is generated (step S8). Also, the video feature information generation apparatus 1 uses the texture clustering means 7 to obtain the texture cluster information TC (FIG. 6C) from the same cluster information (node cluster information) from the first frame image to the currently input frame image. Reference) is generated (step S9).
[0067]
Here, the video feature information generation device 1 determines whether or not the input frame image is the final frame of the scene (step S10), and if it is not the final frame (No), the process returns to step S2 to return to the next Classification (clustering) of frame images is performed.
[0068]
On the other hand, when the input frame image is the final frame (Yes in step S10), the texture clustering unit 7 generates a representative texture structure PTX (see FIG. 6D) that is representative in each texture cluster (step (d) in FIG. 6). S11).
Then, the video feature information generation device 1 generates and outputs video feature information (scene simple information) that characterizes the scene by the scene feature information generation unit 9a of the feature information generation unit 9 (step S12).
[0069]
Through the above operation, video feature information in a scene with video data is generated. If the input of video data continues, the initialization unit 9b of the feature information generation unit 9 initializes the cluster information and the representative texture structure in the cluster information storage unit 8, and repeats the operation from step S1. As a result, the video feature information (simple scene information) for each scene is sequentially generated.
[0070]
(Detailed operation of node clustering means)
Next, the operation of the node clustering means 6 will be described in more detail with reference to FIG. FIG. 3 is a flowchart showing the operation of the node clustering means 6. This operation corresponds to the operation in step S8 in FIG. Further, in the following description, the symbols used in FIG.
[0071]
First, the node clustering means 6 calculates the distances of the neighboring image feature amounts PIfv of all the node pairs on one input frame image (step S21). Then, nodes whose distances are not more than a certain threshold are classified (clustered) so as to be included in the same cluster (step S22). Then, an average image feature value PCfv in the vicinity of all nodes PCpt included in the same cluster is calculated (step S23). Then, the node clustering means 6 separates the node sets from which the positions PTIps of all the nodes belonging to the node cluster are separated as different node clusters (step S24), and attaches the cluster number PCcn to each node cluster.
[0072]
Here, the separation of the node clusters will be described with reference to FIG. 9 (refer to FIG. 1 as appropriate). FIG. 9 is an example of a screen on which the node clustering means 6 performs node cluster separation processing, where (a) is a screen clustered based on image feature values, and (b) is a different node set for distant node sets. This is a cluster screen. As shown in FIG. 9A, when clustering is performed based on the image feature amount, the left and right node clusters are separated from each other, but both become the same node cluster. Therefore, by separating the node sets that are separated from each other, different node clusters are generated as shown in FIG. 9 (b) (in FIG. 9 (b), different node clusters are represented by the difference in the diagonal direction in the node cluster). is doing).
Returning to the flowchart of FIG.
[0073]
The node clustering means 6 determines whether or not the currently processed frame image is the first frame (step S25). If it is the first frame, the node clustering means 6 proceeds to step S27 and displays the clustering result as cluster information of each frame image. Write to the cluster information storage means 8 as (node cluster information PC).
[0074]
On the other hand, if the frame image currently being processed is not the first frame (No in step S25), the following cluster number renumbering process is performed (step S26).
[0075]
That is, if more than half of the nodes belonging to the node cluster (processing target node cluster) separated and generated in step S24 are included in a node cluster in the previous frame image, the nodes are attached to the node cluster in the previous frame image. The assigned cluster number PCcn is set as the cluster number PCcn of the node cluster to be processed. If more than half of the nodes belonging to the node cluster to be processed are not included in the node cluster of the previous frame image, “0” is assigned as the cluster number PCcn. Then, by assigning an unused cluster number to all the node clusters having “0” added to the cluster number PCcn, a unique cluster number is assigned to all the node clusters.
[0076]
Then, this clustering result is written in the cluster information storage means 8 as cluster information (node cluster information PC) of each frame image (step S27). With the above operation, the node clustering means 6 classifies (clusters) the nodes for each frame image.
[0077]
(Detailed operation of texture clustering means)
Next, the operation of the texture clustering means 7 will be described in more detail with reference to FIG. 4 (refer to FIG. 1 as appropriate). FIG. 4 is a flowchart showing the operation of the texture clustering means 7. This operation corresponds to the operation in step S9 in FIG. Further, in the following description, the symbols used in FIG.
[0078]
First, the texture clustering means 7 inputs one node on the frame image (step S31). Then, it is determined whether or not the current number of texture clusters is “0” (step S32). If “0” (Yes), a texture cluster including only the input nodes is generated (step S33). Proceed to S38. Note that the average image feature amount of the texture cluster is the image feature amount in the vicinity of the input node.
[0079]
On the other hand, when the number of current texture clusters is not “0” (No in step S32), the image feature amount (neighboring image feature amount) PIfv in the vicinity of the input node and the average image feature of all the nodes of each texture cluster. The distance from the quantity TCfv is calculated (step S34). Here, a texture cluster having a minimum distance and not more than a predetermined threshold is searched (step S35). Then, the search result is determined (step S36), and if the texture cluster is not found (Ng), the process proceeds to step S33. On the other hand, when a texture cluster is found (Ok), the input node is added to the node TCpt belonging to the texture cluster, and the average image feature value TCfv of all nodes in the texture cluster is updated (step S37).
[0080]
Here, it is determined whether or not processing has been performed for all nodes on the frame image (step S38). If processing has not been performed for all nodes (No), the process returns to step S31 to continue the operation. On the other hand, when processing has been performed for all nodes on the frame image (Yes in step S38), the following representative texture structure generation processing is performed (step S39).
[0081]
That is, in each texture cluster, a node having the closest distance between the neighboring image feature quantity PTFvv of the node and the average image feature quantity TCfv of the texture cluster is selected, and the neighboring image feature quantity PIfv of the node is represented as a representative of the texture cluster. The image feature amount is PTXfv. Then, from the frame image corresponding to the frame number of the node, a rectangular area having a predetermined size with the coordinates of the node as a center is extracted and used as a representative texture image.
[0082]
Then, the representative image feature amount PTXfv and the representative texture image PTXimg are written in the cluster information storage unit 8 as the representative texture structure PTX (step S40).
Through the above operation, the texture clustering unit 7 generates the same number of representative texture structures as the number of texture clusters.
[0083]
(Detailed operation of feature information generation means)
Next, the operation of the feature information generation unit 9 will be described in more detail with reference to FIG. 5 (refer to FIG. 1 as appropriate). FIG. 5 is a flowchart showing the operation of the feature information generating unit 9. This operation corresponds to the operation in step S12 in FIG. Further, in the following description, the symbols used in FIG.
[0084]
First, the feature information generation unit 9 initializes image data to be output (sets all pixels to “0”) and opens a text file for writing text data to be output (step S41). Then, the feature information generation unit 9 extracts a cluster that can be traced to the last frame (clusters having the same cluster number exist in the last frame) from the clusters of the first frame (step S42).
[0085]
Here, the feature information generation means 9 analyzes the positions PTIps of all the nodes of the extracted cluster, and sets the coordinates of the four corners of the rectangle that surrounds all the nodes most efficiently (the area is small) (step S43).
Further, the feature information generating unit 9 selects a representative texture structure PTX having a representative image feature PTXfv closest to the average image feature PCfv of all nodes belonging to the cluster (Step S44).
[0086]
Further, the feature information generation means 9 calculates the position centroid (sx, sy) of the cluster of the first frame from the positions PTIps of all the nodes of the cluster (step S45), and the feature information generating means 9 of the last frame having the same cluster number as that cluster The cluster position centroid (ex, ey) is calculated from the cluster (step S46). Then, the difference (ex-sx, ey-sy) in the position centroid is calculated as the motion vector of the cluster (step S47).
[0087]
Then, the feature information generation unit 9 fills the inside of the rectangle obtained in step S43 with the representative texture image PTXimg of the representative texture structure PTX selected in step S44, and starts the position gravity center (sx, sy) calculated in step S45. As described above, image data in which a motion vector (ex-sx, ey-sy) is drawn as a straight line is generated (step S48).
[0088]
  Further, the feature information generating unit 9 includes the first frame number, the last frame number, the coordinates of the four corners enclosed by the rectangle (rectangular coordinate information), and the representative texture structure PTX selected in step S44.representativeEach information of the image feature amount and the motion vector (ex-sx, ey-sy) is written into the text file (step S49).
[0089]
Through the above operation, the feature information generation unit 9 generates and outputs video feature information (scene simple information) characterizing a scene as image data and text data (text file).
[0090]
【The invention's effect】
As described above, the video feature information generation method, the video feature information generation device, and the video feature information generation program according to the present invention have the following excellent effects.
[0091]
  Claim1According to the described invention, rough features such as the position, size, and motion vector of an object (subject) and background can be generated from video data as video feature information. It can be used as search information. At this time, since the video feature information expresses a rough feature, it is easy to express the search intention of the searcher, and video data suitable for the search intention of the searcher can be searched.
[0092]
  In addition, according to the present invention, when the searcher searches from the video data, the number of search conditions (parameters) can be reduced, so that the searcher can easily perform the search operation. Furthermore, since the search conditions are simplified, there is an effect that the search speed can be increased.
  In addition, according to the present invention, image data in which a moving area is filled with a texture that characterizes a scene can be generated as video feature information, so that a scene feature of video data can be visually captured. . Thus, the searcher can visually recognize the change for each scene from the video feature information that is the image data, so that the searcher can quickly search for the scene of the video data desired by the searcher. .
[0093]
  Claim2According to the invention described in the above, since the scene of the video data is detected and video feature information is generated for each scene, simple and simple video feature information is automatically generated even in video data in which scenes such as movies change. can do.
[Brief description of the drawings]
FIG. 1 is a block diagram illustrating a configuration of a video feature information generation apparatus according to the present invention.
FIG. 2 is a flowchart showing the operation of the video feature information generation apparatus according to the present invention.
FIG. 3 is a flowchart showing the operation of the node clustering means.
FIG. 4 is a flowchart showing the operation of the texture clustering means.
FIG. 5 is a flowchart showing an operation of feature information generating means.
FIG. 6A is a data structure diagram showing a data structure of node information.
(B) It is a data structure figure which shows the data structure of node cluster information.
(C) It is a data structure figure which shows the structure of the data of texture cluster information.
(D) It is a data structure figure which shows the structure of the data of a representative texture structure.
FIG. 7 is a diagram illustrating an example of a frame image in which nodes are set (arranged) in a grid pattern.
FIG. 8 is an explanatory diagram for explaining a state in which nodes are tracked on a frame image.
FIG. 9 is an explanatory diagram for explaining node cluster separation processing of the node clustering means;
FIG. 10 is a diagram illustrating an example of an image of video feature information generated as image data.
FIG. 11 is a diagram illustrating an example of data content of video feature information generated as text data.
[Explanation of symbols]
1 Video feature information generator
2 Scene division means
3 Frame image sampling means
4 Node initial setting means
4a Neighborhood image feature amount calculation unit
5 Node tracking means
5a Neighborhood image feature amount calculation unit
6 Node clustering means (node classification means)
7 Texture clustering means (class image feature generation means)
8 Cluster information storage means (storage means)
9 Feature information generation means
9a Scene feature information generator
9b Initialization part
20 Video storage means

Claims (2)

A video feature information generating device that generates video feature information characterizing each scene of the video data,
Frame image sampling means for extracting frame images from the video data at specific sampling intervals;
A node initial setting means for setting a reference node for extracting the feature of the image in the first frame of the frame image extracted by the frame image sampling means;
A node tracking unit that tracks the node corresponding to the frame image after the first frame based on the image feature amount of the neighborhood image region of the node set by the node initial setting unit;
Node classification means for classifying the nodes on the basis of the image feature amount of the neighborhood image area of each node tracked by the node tracking means for each frame image;
From the first frame to the last frame of the scene, among the image feature values of the respective frame images corresponding to the neighboring image areas of the nodes classified into the same class by the node classification means, the average image features in the class A class image feature amount generating unit that generates an image feature amount having the smallest distance from the amount as a class image feature amount representing the class;
The class image feature amount generated by the class image feature amount generation means, the coordinate information of the rectangular region including the node in the class, and the position centroid of the rectangular region from the first frame to the last frame of the scene the motion information is a motion vector, and generates as Film image feature information of the text data, the rectangular area is divided on the basis of the neighboring image regions classified by the node classifying unit, most to the class image feature amount Feature information generating means for generating an image in which texture extracted from a similar frame image is spread as video feature information as image data ;
A video feature information generating apparatus comprising: A scene dividing unit that divides the video data for each scene, wherein the frame image sampling unit extracts a frame image at a specific sampling interval for each scene divided by the scene dividing unit. image feature information generating device according to one.

Images