JP6151908B2 - Learning device, identification device, and program thereof - Google Patents

Description

  The present invention relates to an image feature amount calculation device, a learning device, an identification device, and a program thereof. In particular, the present invention relates to an image feature amount calculation device, a learning device, an identification device, and a program for calculating an image feature amount in order to detect an object included in a video or an image.

  As a method for analyzing the contents of a video, there is a method of extracting a feature amount from a video frame and determining whether or not a specific subject is reflected based on the feature amount. There is also a method for performing machine learning for making such a determination. In this learning, the parameters of the determiner are adjusted using learning data (images) to which positive or negative example labels are assigned. That is, learning is performed using an image to which a correct answer indicating whether or not a specific subject has moved is given. When this method is used, it is a feature that a determination device for detecting various subjects can be realized only by changing the learning data without changing the framework of the learning method itself.

  In order to classify frame images into two classes depending on whether or not a specific subject is reflected, first, image data is converted into some feature vector. An example of the simplest method for obtaining a feature vector is a statistic (for example, an average value or variance) regarding pixel values of R (red), G (green), and B (blue) of each pixel of the entire image. This is a method of calculating feature vectors of several dimensions by arranging them as elements. Another example of a technique for obtaining a feature vector is to calculate frequency components of the entire image and use the intensity distribution as a feature vector (to obtain a feature vector in which intensity values for each frequency component are arranged as elements). Is the method. As yet another technique, Non-Patent Document 1 discloses a feature of a frame image by detecting feature points from the frame image, calculating gradient features from the surrounding area, and then determining their appearance frequency histogram. A method for calculating a vector is shown. This method is called the Bag of Visual Words (BoVW) method.

  Non-Patent Document 2 discloses a method for dividing a frame image into a plurality of regions, calculating a feature vector for each region, and calculating the feature vector of the entire frame image by connecting the calculated vectors. It is shown. The frame image dividing method specifically shown is, for example, vertical and horizontal 2 × 2 division or vertical and horizontal 1 × 3 division. The technique described in Non-Patent Document 2 solves the problem that the position of the subject in the frame image cannot be reflected in the feature vector and the problem that the subject and other background region features are mixed. I am trying.

G. Csurka, C. Bray, C. Dance and L. Fan, “Visual categorization with bags of keypoints,”, In Proc. ECCV Workshop on Statistical Learning in Computer Vision, pp. 59-74, 2004 S.-F. Chang, J. He, Y.-G. Jiang, EE Khoury, C.-W.Ngo, A. Yanagawa and E. Zavesky, `` Columbia University / VIREO-City / IRIT TRECVID2008 high-level feature extraction and interactive video search, '' In Proc. TRECVID 2008 Workshop, 2008

  However, in the technique described in Non-Patent Document 2, when a frame image is divided, division at a fixed size and a fixed position is performed, for example, vertical and horizontal 2 × 2 division or vertical and horizontal 1 × 3 division. . If the size and method of division are fixed in this way, there arises a problem that the robustness against the size variation of the subject is insufficient. For example, even in the same car, the entire frame image may appear as a subject up, or the frame image may appear smaller in the corners of the frame image. If the image size of the divided area is fixed, there is a possibility that a car that deviates from the size cannot be detected.

  As another problem, when a frame image is divided, the target subject may straddle the boundary of the region. When the subject crosses the boundary of the area, the information on the entire subject is not accurately reflected in the feature vector obtained from the divided image.

  These problems lead to a decrease in accuracy when detecting a specific subject from the frame image. The present invention has been made in view of such circumstances, and provides an image feature amount calculation device, a learning device, an identification device, and a program thereof for performing highly accurate subject detection.

  [1] In order to solve the above-described problem, an image feature amount calculation apparatus according to an aspect of the present invention includes a region image extraction unit that specifies a range of region images of a plurality of sizes included in an input image, and the input image Based on the above, the feature amount of each of the region images specified by the region image extraction unit is calculated, and the feature amount of the input image is generated by connecting the feature amounts calculated from the plurality of region images. A quantity calculation unit.

Here, the “region image” is an image of a partial region of the input image. Note that an image in the same region as the input image is also a region image. The fact that the region image has a plurality of sizes means that region images having various vertical and horizontal sizes (units such as the number of pixels) are used. When the multiple sizes are the number of pixels that change stepwise with a predetermined difference (that is, when the stepwise region image is used so that the lengths of the vertical or horizontal sides of the rectangular image form an even number sequence) There is also a possibility. In addition, there may be a case where the number of pixels changes stepwise at a predetermined ratio (that is, a case where a stepwise region image is used so that the lengths of the vertical or horizontal sides of the rectangular image form a geometric progression). . In addition, there may be a case where the size of the region image becomes more irregular and stepwise.
The “feature amount of each region image” is a feature amount (scalar or vector) of an image obtained from one of the region images.
“Connecting feature amounts calculated from a plurality of region images” means, for example, an operation for obtaining a feature vector by simply arranging (connecting) the feature amounts obtained from the respective region images as elements. It is.

  “Specify the range of area images of multiple sizes” and “Calculate the feature quantities of each of the specified area images, and connect the feature quantities calculated from the multiple area images to combine the features of the input image The combination with “generate quantity” is one of the configurations having the technical features of the present embodiment. The area image has a plurality of sizes, so that the subject included in the input image protrudes from a certain area image, is relatively small in a certain area image, or is suitable for an intermediate area image. When it fits. When the subject protrudes from the region image, the feature of the subject in the image may not be extracted well from the region image. When the subject appears small in the area image, the feature information extracted from the area image may have insufficient information on the characteristics of the subject. When the subject fits within the area image reasonably, the feature amount extracted from the area image favorably represents the feature of the subject as information. Then, by connecting the feature values calculated from each of the plurality of region images, it is relatively possible that the feature as an image of a certain subject is well included in any place of the connected feature values. Become expensive. Therefore, according to such a technical configuration, even if the size of the subject is changed, it is possible to extract a feature amount that captures the feature of the subject satisfactorily.

[2] A learning device according to an aspect of the present invention includes an image feature amount calculation device according to [1], information indicating whether the input image is a positive example or a negative example, and the feature amount. A discriminator for obtaining a parameter of a discriminator for discriminating whether an unknown input image is a positive example or a negative example based on a combination of feature amounts of the input image generated by the calculation unit And a learning unit.
Here, the process for obtaining the parameters of the discriminator is a machine learning process based on the learning data. The discriminator receives a feature amount extracted from an unknown input image according to a predetermined model as an input, and the input image is a positive example or a negative example as a result of calculation using the feature amount and a parameter. Outputs information that represents. The parameter is usually a plurality of variables, and an optimal set of parameter values can be obtained by performing the process of the classifier learning unit. “A positive example or a negative example” indicates whether an input image belongs to a predetermined cluster. As a specific example, it represents whether or not a predetermined subject (a person, a car, a mountain, a dog, a cat, etc.) is reflected in the input image. Thereby, learning using a good feature amount is possible.

[3] Further, according to one aspect of the present invention, in the learning device, the region image extraction unit specifies a range of the region image so that at least a part of the plurality of region images having the same size overlap each other. It is characterized by doing.
This is realized by setting the value α or β described in the embodiment to less than 1 (0 <α <1 or 0 <β <1). This increases the possibility that the feature amount extraction unit can extract a feature amount that well represents the feature of the subject.
Furthermore, when 0 <α ≦ 0.5, or when 0 <α ≦ 0.5, any pixel in the original key frame image falls within the range of at least two region images of the same size. Will be included. That is, in this case, the possibility that the subject can be captured in a region image of an appropriate size is further increased. That is, a better feature amount can be extracted.

[4] In addition, an identification device according to an aspect of the present invention provides an image feature amount calculation device according to [1], a parameter learned in advance, and a feature amount of the input image generated by the feature amount calculation unit. And an identification unit for identifying whether the input image is a positive example or a negative example.
Accordingly, it is possible to identify whether the input image is a positive example or a negative example based on the image feature amount obtained by the image feature amount calculation device and the learned parameter.

[5] In addition, according to one aspect of the present invention, in the above-described identification device, information indicating whether the input image input as learning data is a positive example or a negative example, and the feature amount calculation unit A discriminator learning unit for obtaining a parameter of a discriminator for discriminating whether the unknown input image is a positive example or a negative example based on the generated combination of feature amounts of the input image; Further, the identifying unit identifies whether the unknown input image is a positive example or a negative example by using the parameter obtained by the classifier learning unit as the previously learned parameter. It is characterized by.
Thereby, this identification device performs a learning process and an identification process.

[6] Further, according to one aspect of the present invention, in the identification device, the region image extraction unit specifies a range of the region image so that at least a part of the plurality of region images having the same size overlap each other. It is characterized by doing.
When at least a part of the plurality of region images having the same size overlap each other, there is a high possibility that a feature amount that favorably represents the feature of the subject can be calculated.

  [7] Further, according to one embodiment of the present invention, a computer is designated by the region image extraction unit that designates a range of region images of a plurality of sizes included in the input image, and based on the input image. A program for functioning as a feature amount calculation unit that calculates a feature amount of each of the region images and generates a feature amount of the input image by connecting the feature amounts calculated from a plurality of the region images. It is.

According to the present invention, it is possible to determine the appearance of a subject with high accuracy without being affected by changes in the position and size of the subject in an image.
In particular, by calculating a feature amount that maintains information obtained from each of a plurality of size area images as information, it is possible to obtain and use a feature amount that is robust against changes in the size of the subject.
In particular, it is possible to obtain and use feature amounts that are robust against changes in the position of the subject by using region images having the same size so that they overlap at least partially. That is, a good result can be obtained even for a subject existing at the grid boundary.

It is the block diagram which showed schematic function structure of the identification device by 1st Embodiment of this invention. It is a block diagram which shows the detailed functional structure of the feature-value calculation part by the embodiment. It is the schematic which shows the range of the area image extracted based on a frame image by the embodiment. It is a flowchart which shows the procedure of the process which the area | region image extraction part by the same embodiment extracts an area | region image. It is the schematic which shows the relationship between the various feature vectors which the feature-value calculation part by the embodiment calculates, and the area | region image extracted by an area | region image extraction part. It is the schematic which shows the structural example of the data for learning in the embodiment.

Next, embodiments of the present invention will be described with reference to the drawings.
[First Embodiment]
FIG. 1 is a block diagram showing a schematic functional configuration of the identification device 2 according to the first embodiment. As illustrated, the identification device 2 includes a learning device 1 therein. The learning device 1 includes a learning video input unit 11, a key frame image extraction unit 13, a region image extraction unit 15, a feature amount calculation unit 17, and a discriminator learning unit 19. The identification device 2 further includes a video input unit 12, a key frame image extraction unit 14, a region image extraction unit 16, a feature amount calculation unit 18, and an identification unit 20. Although not shown, the combination of the region image extraction unit 15 and the feature amount calculation unit 17 functions as an image feature amount calculation device. Similarly, the combination of the region image extraction unit 16 and the feature amount calculation unit 18 functions as an image feature amount calculation device.

The learning device 1 performs machine learning of the identification unit 20 based on the read learning data.
The identification device 2 determines whether or not a specific subject appears in the input video by the identification unit that has been learned by the learning device 1.

The learning video input unit 11 obtains learning video data from the outside.
The key frame image extraction unit 13 extracts a key frame image from the learning video acquired by the learning video input unit 11. As a specific method, the key frame image extraction unit 13 detects a shot boundary from a video, divides the video into shots, and then acquires a frame image from the beginning or an intermediate position of each shot. Note that the shot boundary is detected, for example, by detecting a point where the sum of the differential values of the pixel values in the time direction exceeds a predetermined threshold and shows a peak. In addition, in a video where there is no shot boundary or a video in which the time length of one shot is very long, the key frame image extraction unit 13 extracts a key frame image at a predetermined time interval or a motion vector between frames. A key frame image is extracted at a timing when the size of the frame becomes greater than or equal to a threshold value.

The region image extraction unit 15 extracts region images of a plurality of sizes included in the key frame image extracted by the key frame image extraction unit 13, and designates the range of these region images. The key frame image extraction unit 13 outputs information regarding the range of the extracted region image.
The feature amount calculation unit 17 calculates a feature vector from the frame image extracted by the key frame image extraction unit 13. The feature vector calculation method will be described in detail later.
The discriminator learning unit 19 learns a discriminator for determining whether or not a subject is shown from learning data to which a positive or negative example label is assigned. The input data to the discriminator learning unit 19 is a feature amount (feature vector) calculated by the feature amount calculation unit 17 based on the key frame image, and “positive example” or “ A label indicating which of the “negative examples” is attached. The discriminator learning unit 19 uses this label as a correct answer and performs machine learning processing. As a learning method by the classifier learning unit 19, a general machine learning method such as a support vector machine, a neural network, or a Bayesian network can be used. A configuration example of the learning data will be described in detail later with reference to FIG.

The video input unit 12 acquires video data from the outside. This video data is video data that is a target for determining whether or not a specific subject is shown.
The key frame image extraction unit 14 extracts a key frame image by the same method as the key frame image extraction unit 13. However, the key frame image extraction unit 14 targets not video data for learning but video data acquired by the video input unit 12.
The region image extraction unit 16 extracts a region image from the key frame image extracted by the key frame image extraction unit 14 by the same method as the region image extraction unit 15.
The feature amount calculation unit 18 extracts the feature amount of the key frame image by the same method as the feature amount calculation unit 17.
The identifying unit 20 identifies whether the input image (unknown image) is a positive example or a negative example based on the feature amount calculated by the feature amount calculating unit 18. Note that the discriminator 20 has been learned in advance by the discriminator learning unit 19. In other words, the parameters used by the identification unit 20 for identification are optimized in advance by the learning process performed by the classifier learning unit 19.

  As a result, the identification device 2 performs a process of determining whether or not a specific subject appears in the input video, and outputs a determination result.

  As already described, the key frame image extraction units 13 and 14 have the same function. The area image extraction units 15 and 16 have the same function. Also, the feature quantity calculation units 17 and 18 have the same function. Therefore, these functional blocks having the same function may be configured by sharing these units.

  FIG. 2 is a block diagram illustrating a detailed functional configuration of the feature amount calculation unit 17. As shown in the figure, the feature amount calculation unit 17 includes a feature point detection unit 171, a local feature quantization unit 174, a local feature vector generation unit 177, a color statistical feature calculation unit 172, and a color feature vector generation unit 178. A texture feature calculation unit 173, a texture feature vector generation unit 179, and a feature vector generation unit 170.

  Further, as shown in FIG. 2, the frame image data is input to the region image extraction unit 15 and the feature amount calculation unit 17. The region image extraction unit 15 sequentially extracts images of a grid region (this is referred to as “region image”) obtained by cutting out the portion from the input frame image. Then, the region image extraction unit 15 supplies information indicating the range of each region image to the local feature vector generation unit 177, the color feature vector generation unit 178, and the texture feature vector generation unit 179. The shape of the region image is typically a rectangle, and in this case, the information indicating the range of the region image includes the coordinate values of the pixels at the upper left corner and the lower right corner of the region image, and the pixels at the upper left corner of the region image. Coordinate values and vertical and horizontal sizes.

The feature point detection unit 171 extracts feature points from the entire input frame image using a feature point detection method.
The local feature quantization unit 174 quantizes the feature of the local region around the feature point detected by the feature point detection unit 171.
The local feature vector generation unit 177 generates a local feature vector by concatenating local feature amounts for each region image.
The color statistical feature calculation unit 172 performs color space conversion based on the input frame image data, and calculates a feature amount in the color space after conversion.
The color feature vector generation unit 178 generates a color feature vector by concatenating the color feature amounts for each area image.
The texture feature calculation unit 173 calculates the texture feature of the input frame image data by performing processing such as wavelet transform.
The texture feature vector generation unit 179 calculates a texture feature vector based on the statistical feature value for each region image of the pixel value as a result of the wavelet transform.
The feature vector generation unit 170 generates a vector obtained by connecting the local feature vector, the color feature vector, and the texture feature vector.
Details of the processing of these units will be described later.

  The feature amount calculation unit 18 also has the same configuration as the feature amount calculation unit 17. The area image extraction unit 16 supplies information related to the extracted region image to the feature amount calculation unit 18 in the same manner as the information related to the region image extracted by the region image extraction unit 15 is supplied to the feature amount calculation unit 17.

Next, details of each feature amount extraction will be described.
(A) Extraction of local feature vectors The above-described bag of visual words method is used to extract local feature vectors.
The feature point detection unit 171 extracts feature points from the entire input frame image using a feature point detection method such as SIFT (Scale-invariant feature transform) or SURF (Supeeded-Up. Robust Features). SIFT and SURF are methods for detecting local features in an image. Reference documents [David G. Lowe, “Object recognition from local scale-invariant features,” In Proc. IEEE International Conference on Computer Vision, respectively. , vol. 2, pp. 1150-1157, 1999.] and [Herbert Bay, Tinne Tuytelaars, and L Van Gool, `` SURF: Speeded Up Robust Features, '' In Proc. 9th European Conference on Computer Vision, vol. 3951, pp. 404--417, 2006.] for details.

  Then, the local feature quantization unit 174 quantizes the features in the local region around the feature points detected by the feature point detection unit 171. Specifically, the local feature quantization unit 174 performs quantization by clustering gradient feature amounts calculated from local regions around the feature points. For this purpose, the local feature quantization unit 174 obtains a representative value for each cluster by clustering gradient feature amounts obtained from learning data in advance using, for example, k-means. Then, the local feature quantization unit 174 assigns the feature amount calculated from the input data to the cluster corresponding to the closest representative value.

Then, the local feature vector generation unit 177 obtains information on the range of each region image from the region image extraction unit 15, and the frequency of appearance of the quantized gradient feature amount with respect to the feature points included in a certain region image. A histogram is obtained, and a sequence of frequency values constituting the histogram is obtained. The local feature vector generation unit 177 performs the above processing for all region images. Then, the local feature vector generation unit 177 generates a local feature vector by concatenating the sequence of frequency values obtained from each region image with respect to all the region images. Let this local feature vector be _Vl . It should be noted that “connection with respect to all region images” will be described in detail later with reference to FIG.

(B) Extraction of Color Feature Vector The color statistical feature calculation unit 172 converts input frame image data into an HSV color space and a Lab color space. The HSV color space is a color space including three components of hue (Hue), saturation (Saturation), and lightness (Value). The Lab color space is a color space composed of components of lightness (L) and complementary color dimensions (a and b). For example, conversion from RGB pixel values to the HSV color space and the Lab color space is performed using an existing technique. As a result of the color space conversion, the color statistical feature calculation unit 172 outputs the pixel value of each component c for each pixel included in the frame image. Note that cε {h, s, v, l, a, b}, and each of these h, s, v, l, a, b is a component of the HSV color space and the Lab color space.

The color feature vector generation unit 178 obtains information on the range of each region image from the region image extraction unit 15, and for each region image, for each component c, the average μ _c , standard deviation σ _c , The cubic root ω _c of the skewness is calculated. Specifically, the color feature vector generation unit 178 calculates these values according to the following expressions (1), (2), and (3).

In Expressions (1) to (3), x is an abscissa value, y is an ordinate value, and f _c (x, y) is a pixel value of the component c at coordinates (x, y). In each of x and y, the range in which the sum is calculated using the symbol Σ is the range of the region image. H _S and W _S are the vertical size (height) and horizontal size (width) of the area image, respectively. The unit of H _S and W _S is a pixel [pixels]. H _S and W _S will be further described later.

The color feature vector generation unit 178 performs the above processing for all region images. The color feature vector generation unit 178 then generates a sequence of values (μ _h , σ _h , ω _h , μ _s , σ _s , ω _s , μ _v , σ _v , ω _v , μ _l calculated from each region image. , [Sigma] _l , [omega] _l , [mu] _a , [sigma] _a , [omega] _a , [mu] _b , [sigma] _b , [omega] _b ) are concatenated with respect to all region images to generate a color feature vector. The color feature vector and V _c. It should be noted that “connection with respect to all region images” will be described in detail later with reference to FIG.

(C) Extraction of Texture Feature Vector Here, a feature amount reflecting the texture of the image is calculated based on the Haar wavelet. First, the texture feature calculation unit 173 applies Haar wavelet transform in three stages based on input frame image data. Next, the texture feature vector generation unit 179 obtains information on the range of each region image from the region image extraction unit 15, calculates the variance of the pixel values of each subband region for each region image, A row of variance values is used as a feature amount in the region image. Then, the texture feature vectors are generated by concatenating these numerical sequences for all the region images. The texture feature vector and V _t. It should be noted that “connection with respect to all region images” will be described in detail later with reference to FIG.

As described above, the local feature vector generating unit 177 generates the local feature vector V _l , the color feature vector generating unit 178 generates the color feature vector V _c , and the texture feature vector generating unit 179 is the texture feature vector V _t. Is generated. Then, the feature vector generation unit 170 obtains a feature vector V by connecting these three vectors. This V is as shown in the following formula (4). The vector V connected by the feature vector generation unit 170 is a feature amount output from the feature amount calculation unit 17.

As described above, the feature amount calculation unit 17 calculates the feature amount of each of the region images specified by the region image extraction unit 15 based on the input image, and the feature amount calculated from the plurality of region images. To generate feature quantities (feature vectors V _l , V _c , V _t , V) of the input image. The feature amount calculated by the feature amount calculation unit 17 holds each feature of a plurality of area images as information.

FIG. 3 is a schematic diagram showing the range of the area image of the grid area extracted by the area image extraction units 15 and 16. Hereinafter, the processing by the region image extraction unit 15 will be described as a representative, but the processing by the region image extraction unit 16 is the same.
The region image extraction unit 15 changes the size of the region image stepwise. In the example shown in the figure, the size of the region image to be extracted is gradually reduced in the order of (a), (b), and (c). When the size of the input original frame image is vertical (height) H and horizontal (width) W, the size of the area image on the Sth scale (S = 1, 2, 3,...) Is , Vertical H _S , and horizontal W _S, which are represented by the following equation (5).

Here, δ is a constant representing the degree of scale change, and 0 <δ <1. The value of δ can be set as appropriate within the range of this inequality. As an example, in the case shown in the figure, δ = 0.5. In the case of FIG. 5A, S = 1, H ₁ = H, and W ₁ = W. In the case of FIG. 5B, S = 2, H ₂ = δH, and W ₂ = δW. In the case of FIG. 3C, S = 3, H ₃ = δ ² H, and W ₃ = δ ² W. In addition, as shown in the figure, the region image extraction unit 15 extracts the range of the region image while sequentially moving in increments of the vertical direction H _S × α and the horizontal direction W _S × β. Here, α and β are constants that can be set as appropriate, and 0 <α ≦ 1 and 0 <β ≦ 1. As an example, in the figure, α = β = 0.5.

In each of FIGS. 9A to 9C, only the upper left corner of the frame of the area image is indicated by a black circle and vertical and horizontal thick lines. Note that the coordinates of the pixel in the upper left corner of the entire frame image are (x, y) = (0, 0). In FIG. 5A, S = 1, and the entire frame image corresponds to a region image. That is, the number N ₁ of area images in the case of S = 1 is 1. In FIG. 5B, S = 2, and the x-coordinate (abscissa) values are 0, βδW, 2βδW, and y-coordinate (ordinate) of the pixel in the upper left corner of each area image. The values are 0, αδH, and 2αδH. The broken line frame shown as an example in FIG. 5B shows a region image in which the coordinate position of the pixel at the upper left corner is (x, y) = (βδW, αδH). The number N ₂ of area images in the case of S = 2 is 9. In FIG. 3C, S = 3, and the values of the x coordinate (abscissa) in the upper left corner pixel of each area image are 0, βδ ² W, 2βδ ² W, 3βδ ² W, 4βδ. ² W, 5βδ ² W, 6βδ ² W. The values of the y coordinate (ordinate) are 0, αδ ² H, 2αδ ² H, 3αδ ² H, 4αδ ² H, 5αδ ² H, and 6αδ ² H. A broken-line frame shown as an example in FIG. 4C shows a region image in which the coordinate position of the pixel at the upper left corner is (x, y) = (5βδ ² W, 4αδ ² H). The number N ₃ of area images in the case of S = 3 is 49.

That is, as described above, the region image extraction unit 15 specifies the region image range so that at least some of the plurality of region images having the same size overlap each other. Part of the plurality of region images having the same size overlaps each other when α <1 in the vertical direction. In the lateral direction, β <1. As a result, even when the subject exists at a position that crosses the frame (boundary line) of the region image (that is, when it does not fit in the one region image), the subject is converted into another region image of the same size. There is a possibility to fit. This makes it possible to more appropriately extract the feature amount representing the image feature of the subject.
In particular, when 0 <α ≦ 0.5 or 0 <β ≦ 0.5, any pixel in the original key frame image is within the range of at least two region images of the same size. Will be included. That is, in this case, the possibility that the subject can be captured in a region image of an appropriate size is further increased. That is, a better feature amount can be extracted.

FIG. 4 is a flowchart showing a region image extraction processing procedure performed by the region image extraction unit 15. Hereinafter, it demonstrates along this flowchart. The processing by the region image extraction unit 16 is the same as this.
First, in step S1, the region image extraction unit 15 initializes the value of the variable S to 1. As described above, S is a value for indicating the scale of the area image.
Next, in step S2, the area image extraction unit 15 determines whether or not the value of the variable S is less than a preset upper limit (set scale). If it is less than the upper limit (step S2: YES), the process proceeds to the next step S3. In other cases (step S2: NO), the process of the entire flowchart ends.
Next, in step S3, the area image extraction unit 15 initializes the value of the variable y to 0. This y represents the value of the ordinate. By the processing of this step, the ordinate of the area image is initialized.

Next, in step S4, the region image extraction unit 15 determines whether or not the condition represented by the inequality y + H _S <H is satisfied with respect to the variable y. If this condition is satisfied (step S4: YES, that is, if the area image can still be taken in the vertical direction), the process proceeds to the next step S5. If not satisfied (step S4: NO, that is, the lower end of the frame image is reached and the vertical image is reached). If no area image can be taken in the direction), the process branches to step S10.
Next, when the process proceeds to step S5, the area image extraction unit 15 initializes the value of the variable x to 0. This x represents the value of the abscissa. By the processing in this step, the abscissa of the area image is initialized.

Next, in step _S <b> 6, the region image extraction unit 15 determines whether or not the condition represented by the inequality of x + W _S <W is satisfied with respect to the variable x. If this condition is satisfied (step S6: YES, that is, if the region image can still be taken in the horizontal direction), the process proceeds to the next step S7, and if not satisfied (step S6: NO, that is, the right end of the frame image is reached and If no more area images can be taken in the direction), the process branches to step S9.
Next, when the process proceeds to step S7, the area image extraction unit 15 sets the height H _S with the coordinates (x, y) as the base point (the pixel at the upper left corner) according to the values of the variables x and y at that time. It extracts an area image by grid width W _S. Then, the region image extraction unit 15 passes information indicating the range of the extracted region image to the local feature vector generation unit 177, the color feature vector generation unit 178, and the texture feature vector generation unit 179. In response to this, each of the local feature vector generation unit 177, the color feature vector generation unit 178, and the texture feature vector generation unit 179 calculates a feature amount related to the region image by the method described above.

In step S8, the area image extracting unit 15, the value of the variable x is increased in increments of β · _{W S.} This is a process for advancing the abscissa value of a region image to the coordinates of the next region image. After the process in this step, the process returns to step S6.
When the process proceeds from step S6 to step S9, the region image extraction unit 15 increases the value of the variable y by an increment of α · H _S. This is a process for advancing the value of the ordinate of the area image to the coordinates of the next area image. After the process of this step, the process returns to step S4.
When the process proceeds from step S4 to step S10, the area image extraction unit 15 updates the value of the variable S to the next value. That is, the value (S + 1) is stored in the storage area of the variable S. This is a process for advancing the scale of the area image to the next stage. After this step, the process returns to step S2.

  Through the series of processes described above, the region image extraction unit 15 extracts all the region images illustrated in FIG. 3 and passes information indicating the range of each region image to the feature amount calculation unit 17. After the region image extraction unit 15 finishes extracting all the region images, each of the local feature vector generation unit 177, the color feature vector generation unit 178, and the texture feature vector generation unit 179 includes each region image as described above. Outputs a feature vector in which all corresponding feature columns are arranged. Then, the feature vector generation unit 170 outputs a feature vector obtained by connecting these feature vectors. The relationship between the region image extraction unit 16 and the feature amount calculation unit 18 is the same as this.

  In this way, by changing the size of the region image in stages, extracting feature amounts from each region image, and using feature amounts (feature vectors) including the feature amounts for each region image as information, Robustness can be obtained against changes in the size of the subject included in the video.

FIG. 5 is a schematic diagram illustrating a relationship between a plurality of region images extracted by the above-described method and feature vectors. In the figure, (a) to (d) correspond to the scale stage of the region image, and S = 1, 2, 3, 4 in order in each case. As described above, the size of the entire frame image is vertical (height) H and horizontal (width) W. The size of the region image is vertical (height) δ ^S-1 · H and horizontal (width) δ ^S-1 · W according to the value of ^S. In each of (a) to (d), one of the region images is indicated by a broken line. In the figure, the connected feature vectors are shown in the form of a two-dimensional graph. In this graph, the horizontal axis represents the arrangement order of feature amounts (scalar), and the vertical axis represents the size of a value common to each feature amount. The feature amount column included in the range indicated by “a1” is a feature amount obtained from the region image included in FIG. The column of feature amounts included in the ranges indicated by “b1”, “b2”, “b3”,... Is a feature amount obtained from a plurality of region images included in FIG. In the figure, only “b4” is shown and the rest is omitted, but in actuality, the feature quantity columns follow the number of area images. The same applies to (c) and (d) in the figure, and a sequence of feature amounts for each area image follows. In the present embodiment, a feature vector is generated in such a manner by connecting feature value columns for all region images. That is, for each of the local feature, the color feature, and the texture feature, a feature value (or a sequence of values) for each region image connected with respect to all the region images as described in FIG. A feature vector, a color feature vector, and a texture feature vector.

Next, an example of a method for configuring learning data will be described.
FIG. 6 is a schematic diagram illustrating a configuration example of learning data. The learning data is stored in a storage device inside the learning device 1. As already described, positive or negative labels are assigned to the learning data. The learning data is configured using an object-oriented database, for example, and has a table structure as shown in the figure. The data includes data items of video number, video data location, frame identification information, and subject type (1 to 40). This data stores information about a plurality of video data. One video data is associated with one or more key frames. The video number is a number assigned to identify video data. The video data location is information representing the actual location of the video data, and for example, information on a path name in the file system is used. The frame identification information is information for identifying each of a plurality of key frames included in one video data. As the frame identification information, a simple key frame serial number may be used, or the frame position in the video is specified in the format of “hh: mm: ss.nnn” (hour: minute: second.frame number) or the like. Information may be used. The column corresponding to each subject type stores a “positive” or “negative” label (information indicating whether the input image is a positive example or a negative example). These labels are label information indicating correct answers as to whether or not each of the subject types (1 to 40) is included in each key frame extracted by the key frame image extraction unit 13 as a subject. Note that the sixth to 39th data of the subject type are not shown in the figure. The “positive” label indicates that the subject is included in the key frame image. The “negative” label indicates that the subject is not included in the key frame image. The value of this label is used as teacher data at the time of learning. The number of types of subjects is not limited to 40, but may be more or less.

  Note that the value of the “positive” or “negative” label is given by hand after the key frame image extracting unit 13 extracts the key frame image, and is written in the learning data.

  As described above, in this embodiment, a video is specified by machine learning using learning data with a positive example (a certain object / event is shown) and a negative example (not shown). It is determined whether or not the subject has appeared. Therefore, even when the appearance position or size of the subject in the frame image changes, an image feature amount that can robustly determine a specific subject is calculated. Specifically, the video frame image is divided into grid regions (region images) of various sizes, feature amounts are calculated for each grid region, and the robustness against the size variation is ensured by connecting them. The size of the grid area is changed in stages. In addition, by making the grid areas overlap, it is possible to deal with an object existing at the boundary of the grid areas.

[Evaluation experiment]
The results of an evaluation experiment performed using actual video data for this embodiment are as follows. In this experiment, 40 types of subjects were detected for about 600 hours of video, and the detection accuracy was evaluated. The detection accuracy was calculated by applying determination processing to all the frame images in the test video, rearranging them in descending order from the highest score, and calculating the estimated average precision for the top 2000 cases. The set values were δ = 0.5, α = 0.5, and β = 0.5. The range of the scale of the area image is 1 ≦ S ≦ 4.

  As a comparison target for evaluation (a technique according to the prior art), a method of dividing a frame image into a fixed grid size was used. Specifically, the feature amount in each divided region was obtained by using a division method that divides the frame image into 2 × 2 vertical and horizontal divisions and a division method that divides the vertical and horizontal 3 × 1 divisions.

  As a result, it was confirmed that the detection accuracy was improved as compared with the conventional method. When the accuracy was compared according to the type of subject, there was an accuracy improvement of 4% at the maximum. Table 1 shows a comparison result of detection accuracy between the method according to the present embodiment and the conventional method. As shown here, in the method according to the present embodiment, a better result than the conventional method was obtained in the estimated average precision (average of 40 types of subjects).

[Second Embodiment]
Next, a second embodiment of the present invention will be described. The first embodiment performs both learning processing and identification processing, whereas the second embodiment performs only learning processing. The functional configuration of the present embodiment has only the functions of the learning device 1 and the identification unit 20 among the functions included in the functional block diagram of FIG. The learning apparatus 1 includes a learning video input unit 11, a key frame image extraction unit 13, a region image extraction unit 15, a feature amount calculation unit 17, and a discriminator learning unit 19 in the first embodiment. It is the same as the form. Further, the processing functions of the respective parts listed here and their functions and effects are also the same as those described in the first embodiment, so that the description thereof is omitted. With this configuration, the learning device of the present embodiment can perform machine learning using a good feature amount and generate the identification unit 20 (optimizing parameter values through learning).

[Third Embodiment]
Next, a third embodiment of the present invention will be described. While the first embodiment performs both learning processing and identification processing, the third embodiment performs only identification processing. The functional configuration of the present embodiment includes only the video input unit 12, the key frame image extraction unit 14, the region image extraction unit 16, the feature amount calculation unit 18, and the identification unit 20 among the functions included in the functional block diagram of FIG. The learning apparatus 1 is not included. The processing functions of the respective parts listed here and their functions and effects are also the same as those described in the first embodiment, so that the description thereof is omitted. The identification unit 20 has been learned in advance. With this configuration, the identification device of the present embodiment can perform identification processing using a good feature amount.

[Fourth Embodiment]
Next, a fourth embodiment of the present invention will be described. In the fourth embodiment, only the function of the image feature amount calculation apparatus described in the first embodiment is implemented as a single apparatus. As described above, the image feature amount calculation device is realized as a device in which the region image extraction unit 15 and the feature amount calculation unit 17 are combined. Since the functions, operations, and effects of the region image extraction unit 15 and the feature amount calculation unit 17 in this image feature amount calculation apparatus are as described above, description thereof is omitted here. With the configuration of the present embodiment, the image feature amount calculation apparatus can calculate an image feature amount that is favorable, that is, robust against changes in the size of the subject, based on the input image.

[Fifth Embodiment]
In the first to fourth embodiments, the range is moved at equal intervals when extracting the region image. The region image extraction units 15 and 16 in the present embodiment extract region images by a method different from that in the first to fourth embodiments. The region image extraction method described below can be applied to the first to fourth embodiments. At that time, technical matters other than the method of extracting the region image are as described in the respective embodiments, and thus description thereof is omitted here. The region image extraction units 15 and 16 in the present embodiment extract region images by one of the following methods.

  In the first method, the density for extracting the region image is changed according to the position in the input image. Specifically, in the flowchart described with reference to FIG. 4, the set values α and β are not always constant. For example, in the region close to the center of the frame image, the values α and β are decreased, and the frame image In the region near the periphery of, the values of α and β are relatively increased. This leads to higher detection accuracy when the subject is present in a region near the center of the frame image. On the contrary, when it is desired to relatively increase the detection accuracy of the subject in the peripheral portion of the frame image, the values of α and β are relatively increased in the peripheral portion. In this case, 0 <α ≦ 1 and 0 <β ≦ 1. In this way, by making a difference in the density at which the region image is extracted, while suppressing the total amount of calculation for calculating the feature amount and identifying the subject, only by the priority region in the image Detailed calculations can be performed.

  In the second method, the density of extracting the region image is relatively increased in the region where the target subject is highly likely to exist. Data representing the possibility (probability value) of the presence of a subject according to the location in the image is supplied from the outside. Thereby, the effect similar to the 1st method is acquired. That is, it is possible to perform fine calculation only with the important region in the image while suppressing the total calculation amount for calculating the feature amount and identifying the subject.

  In the third method, a plurality of region images having the same size are extracted at random locations in the frame image.

[Implementation by the computer program of the first to fifth embodiments]
In addition, you may make it implement | achieve the function of each process part in each embodiment mentioned above with a computer. In that case, the program for realizing these functions may be recorded on a computer-readable recording medium, and the program recorded on the recording medium may be read into a computer system and executed. Here, the “computer system” includes an OS and hardware such as peripheral devices. The “computer-readable recording medium” refers to a storage device such as a flexible disk, a magneto-optical disk, a portable medium such as a ROM and a CD-ROM, and a hard disk incorporated in a computer system. Furthermore, a “computer-readable recording medium” dynamically holds a program for a short time, like a communication line when transmitting a program via a network such as the Internet or a communication line such as a telephone line. In this case, a volatile memory inside a computer system serving as a server or a client in that case may be included, and a program that holds a program for a certain period of time. The program may be a program for realizing a part of the above-described functions, or may be a program that can realize the above-described functions in combination with a program already recorded in a computer system.

Although a plurality of embodiments have been described above, the present invention can also be implemented in the following modifications.
(Modification 1) In the above-described embodiment, α = 0.5 and β = 0.5 are set as an example. In addition, an example has been described in which the extraction density of the region image is increased by setting α ≦ 0.5 or β ≦ 0.5. However, α> 0.5 or β> 0.5 may be set.
(Modification 2) In the above-described embodiment, a local feature vector, a color feature vector, or a texture feature vector is used as the feature amount of an image. In the modification, other feature amounts may be used.
(Modification 3) In the embodiment described above, a label value of “positive example” or “negative example” is given to the key frame extracted by the key frame image extraction unit 13 in the learning device 1. In the modified example, instead, a key frame image corresponding to the video is extracted in advance, and the learning device 1 captures the extracted key frame image and label value data as a set from the outside. The learning device 1 does not particularly use the video data itself, and performs a learning process based on the key frame image and the label value.
(Modification 4) When the region image extraction unit extracts a region image by the processing of the flowchart of FIG. 4, if a surplus occurs at the lower end or right end of the original frame image, the lower end or right end of the region image Adjust the increment of the coordinates of the region image so that is exactly aligned with the lower or right edge of the frame image. Alternatively, the coordinates of the region image may be determined by protruding the lower end or the right end of the frame image. When a part of the area image protrudes outside the frame image, it is assumed that there is a uniform pixel value for the protruding part (that is, there is no image information in that part), and the subsequent feature value calculation And so on.

  As mentioned above, although embodiment of this invention and its modification were explained in full detail, the concrete structure is not restricted to this embodiment, The design etc. of the range which does not deviate from the summary of this invention are included.

  The present invention can be used for video content management and the like.

DESCRIPTION OF SYMBOLS 1 Learning apparatus 2 Identification apparatus 11 Video input part 12 for learning Video input part 13, 14 Key frame image extraction part 15, 16 Area image extraction part 17, 18 Feature amount calculation part 19 Classifier learning part 20 Identification part 170 Feature vector generation Unit 171 feature point detection unit 172 color statistical feature calculation unit 173 texture feature calculation unit 174 local feature quantization unit 177 local feature vector generation unit 178 color feature vector generation unit 179 texture feature vector generation unit

Claims (5)

An area image extraction unit for designating a range of area images of a plurality of sizes included in the input image;
Based on the input image, the feature amount of each of the region images specified by the region image extraction unit is calculated, and the feature amount of the input image is connected by connecting the feature amounts calculated from the plurality of region images. A feature amount calculation unit for generating
An unknown input image is a positive example based on information indicating whether the input image is a positive example or a negative example and a combination of feature amounts of the input image generated by the feature amount calculation unit. A discriminator learning unit for obtaining a parameter of a discriminator for discriminating either a negative example or a negative example;
Comprising
The area image extracting unit, the vertical H _S at least a portion of the pixel and horizontal W _S plurality of the area images of the same size of pixels so as to overlap each other in the vertical direction (H _S × α) pixels and lateral (W _S Xβ) The range of the region image is designated while sequentially moving in increments of pixels , and 0 <α ≦ 0.5 or 0 <β ≦ 0.5.
A learning apparatus characterized by that. An area image extraction unit for designating a range of area images of a plurality of sizes included in the input image;
Based on the input image, the feature amount of each of the region images specified by the region image extraction unit is calculated, and the feature amount of the input image is connected by connecting the feature amounts calculated from the plurality of region images. A feature amount calculation unit for generating
An identification unit for identifying whether the input image is a positive example or a negative example based on a parameter learned in advance and a feature amount of the input image generated by the feature amount calculation unit;
Comprising
The area image extracting unit, the vertical H _S at least a portion of the pixel and horizontal W _S plurality of the area images of the same size of pixels so as to overlap each other in the vertical direction (H _S × α) pixels and lateral (W _S Xβ) The range of the region image is designated while sequentially moving in increments of pixels , and 0 <α ≦ 0.5 or 0 <β ≦ 0.5.
An identification device characterized by that. Based on information indicating whether the input image input as learning data is a positive example or a negative example, and a combination of feature amounts of the input image generated by the feature amount calculation unit, an unknown A discriminator learning unit for obtaining a discriminator parameter for discriminating whether the input image is a positive example or a negative example;
The identifying unit identifies whether the unknown input image is a positive example or a negative example by using the parameter obtained by the classifier learning unit as the previously learned parameter.
The identification device according to claim 2. A program for causing a computer to function as the learning device according to claim 1. A program for causing a computer to function as the identification device according to claim 2 or 3.

Images