JP4953211B2 - Feature extraction apparatus and feature extraction method - Google Patents

Description

  The present invention relates to a feature extraction apparatus and a feature extraction method, and more particularly to a feature extraction apparatus and a feature extraction method for extracting new feature data that can be used for recognition of the shape and motion of an object from three-dimensional data.

  Conventionally, a moving image photographed by a video camera is regarded as three-dimensional data, and the object is recognized from three-dimensional data obtained from a technology for recognizing the movement of an object or a CT scanner device, a range finder device (distance image photographing device), or the like. There was no technology for recognizing and identifying the shape of the image at high speed and with high accuracy.

In view of this, the present inventors have developed a technique for extracting a three-dimensional higher-order local autocorrelation feature from a moving image and recognizing the motion. In the following Patent Document 1 filed by the present inventors, feature data is extracted from three-dimensional data of a moving image by a three-dimensional higher-order local autocorrelation feature extraction method, and the extracted feature data is statistically analyzed such as multivariate analysis. A technique has been disclosed in which new feature data is generated by conversion by a conventional method, and determination is made by comparing with registered data.
JP 2005-92346 A

  Since the above-described conventional three-dimensional higher-order local autocorrelation feature is an integral feature, it has a feature (position invariance) that it is resistant to positional displacement of an object. However, if the image has multi-valued pixel values (pixel values), the target image information is lost from the simple sum of products of the pixel values, and in this case, the recognition accuracy decreases. There was a problem of end.

  An object of the present invention is to solve the above-described problems of the conventional example, and to extract new feature data that can be widely used for recognition of the shape and motion of an object from three-dimensional data. And a feature extraction method.

  In the case of three-dimensional image (luminance) data, the feature extraction device of the present invention extracts new feature data based on the curvature of the isoluminance curved surface. First, a gradient vector is obtained for each pixel of the three-dimensional image, and a luminance inclination angle (horizontal angle from the reference direction: θ and vertical angle: φ) is calculated from the direction of the gradient vector. Next, considering the autocorrelation limited to a local region in the three-dimensional image data, the correlation of the angle of the gradient vector is further obtained for the pixel set corresponding to the position autocorrelation. Here, two correlations are obtained by combining a spatial correlation (relative positional relationship between pixel groups) and a gradient vector correlation (angle correlation). The feature amount is an integral value of the entire region of these correlation values.

  The feature extraction apparatus according to the present invention includes a gradient vector calculation unit that calculates at least a gradient vector representing a gradient direction of a data value from each data value of three-dimensional data, and a local autocorrelation unit that calculates a local autocorrelation value for each gradient vector. And an adding means for adding the local autocorrelation values calculated for the respective gradient vectors.

  Further, in the above feature extraction apparatus, the local autocorrelation means uses a plurality of correlation patterns indicating the data position of interest and its neighboring data positions that do not overlap with each other in parallel movement, and the data represented by each correlation pattern. Another feature is that the correlation of gradient vector angles is obtained for a set.

  In the above feature extraction apparatus, the gradient vector is expressed as a quantization vector by a plurality of quantization elements representing different angles in the angle information indicating the inclination direction of the data value, and the correlation between the angles of the gradient vectors is Another feature is that a correlation value is obtained by multiplying each combination of quantization elements of a quantization vector by the value of the quantization element.

  In the feature extraction apparatus, the gradient vector is expressed as a three-dimensional vector including information corresponding to the amount of inclination in addition to the angle information indicating the direction of inclination of the data value, and the adding means is based on the amount of inclination of the data value. Another feature is that the correlation values are added with weights.

  In the feature extraction apparatus described above, the gradient vector calculation unit is characterized in that it extracts information on the inclination direction and the inclination amount by applying a filter having a predetermined coefficient to a local region including the data of interest.

  In the feature extraction method of the present invention, a step of calculating a gradient vector representing at least a gradient direction of the data value from each data value of the three-dimensional data, a step of calculating a local autocorrelation value for each gradient vector, and a calculation for each gradient vector are performed. The main feature is to include a step of adding the local autocorrelation values.

The present invention has the following effects.
(1) The correlation information between the inclination direction and the position of the data value is obtained, which means that the inclination and curvature information of the target contour curved surface in the three-dimensional space are extracted. Effectively extracted, the ability to identify the object is enhanced.

(2) Since a plurality of objects can be recognized simultaneously (additiveness) without cutting out the objects (position invariance), it is not necessary to recognize in advance where and how many objects are present.

(3) Although the overall feature dimension is large, the number of feature elements to be calculated in each pixel (pixel) is very small, so the amount of calculation for feature extraction is small. Further, the calculation amount is constant regardless of the number of objects. Accordingly, high-speed (real time) processing is possible.

  In the following embodiment, an example in which moving image data captured by a video camera is read and processed offline will be described. However, the feature extraction and recognition processing of the present invention is performed in real time on a moving image captured by a video camera, for example. It is also possible to execute it.

  As an example of data to be processed, an example in which a moving image is regarded as three-dimensional data in the embodiment is disclosed. However, the present invention relates to 3 obtained from, for example, a CT scanner device or a range finder device (distance image capturing device). It can be applied to arbitrary data arranged three-dimensionally, including dimensional data.

  FIG. 1 is a block diagram showing a configuration of a feature extraction apparatus according to the present invention. The video camera 10 captures a moving image of the object and transfers the image file to the computer 11 via a cable or a memory card. The video camera 10 may be a monochrome camera or a color camera. The computer 11 may be, for example, a known personal computer (PC) provided with a general-purpose interface circuit for capturing moving images or a memory card reader. The present invention is realized by creating and installing a program for executing processing to be described later on any known computer 11 such as a personal computer.

  The monitor device 12 is a well-known output device of the computer 11 and is used for displaying the recognition result, for example, the type and number of detected objects, images, etc. to the operator. The keyboard 13 and the mouse 14 are well-known input devices used for input by the operator. As an input device for three-dimensional image data, in addition to the video camera 10, a CT (Computer Tomography) scanner device 15, a range finder device 16 that captures a distance image using a laser beam or the like may be used.

  In the embodiment, for example, the image data input from the video camera 10 may be processed in real time, or may be temporarily stored and read and processed. Further, the video camera 10 may be connected to the computer 11 via an arbitrary communication network.

  FIG. 2 is an explanatory diagram showing the structure of three-dimensional (image) data. In the uncompressed moving image data photographed by the video camera 10, data (pixels) having at least luminance values as data values (pixel values) are arranged on a two-dimensional (xy) plane, and the two-dimensional (image) data. Are arranged in order in the direction of the time axis t. By regarding the time axis t as the z-axis, the moving image data is regarded as three-dimensional (image) data. If the data value (pixel value) has a luminance value for each RGB, it may be processed after being converted to a monochrome luminance value, or may be processed separately for each RGB luminance value, Furthermore, the separately processed results may be integrated. In the case of a CT (Computer Tomography) scanner device 15, three-dimensional image data is output from the device. In the case of the range finder device 16 that captures a distance image using laser light or the like, the distance image data is converted into three-dimensional data.

  FIG. 3 is a flowchart showing the contents of processing in the feature extraction apparatus of the present invention. In S10, image data is read from the video camera 10 or the hard disk device. In S11, a gradient vector field is calculated from the image data by a method described later. The gradient vector is a three-dimensional vector made up of data representing the luminance gradient direction and gradient amount.

  In S12, a local autocorrelation value of the gradient vector field is calculated by a method described later. Considering the autocorrelation limited to the local region in the image, the angle correlation of the gradient vector is further obtained for the pixel set corresponding to the position autocorrelation. Here, two correlations, a spatial correlation (relative positional relationship between pixel groups) and a gradient vector correlation (angle correlation) are combined. This correlation value includes data corresponding to the inclination and curvature of the target contour curved surface (isoluminous curved surface).

  In S13, the feature data is obtained by adding the local autocorrelation values over the entire image. That is, a local autocorrelation (pattern) histogram is calculated from the gradient vector field. The feature data of the present invention extracted by the above processing is analyzed using a multivariate analysis method such as a known multiple regression analysis or factor analysis according to the recognition target and purpose, and identification of movement, recognition of a three-dimensional shape, etc. Is performed.

FIG. 4 is a flowchart showing the contents of the gradient vector calculation process (S11) of the present invention. In S20, one unprocessed data (target pixel) is selected from the three-dimensional data. In S21, data values of local regions centered on (including) the data of interest are multiplied by a gradient operator and added to obtain gradient (luminance gradient) data in the X, Y, and Z directions. This gradient data corresponds to partial differential values in the X, Y, and Z directions of the data values.
Assuming that the data (pixel luminance value) is I (x, y, z), the three-dimensional gradient vector is defined as Equation 1. FIG. 6 is an explanatory diagram showing a gradient vector.

  FIG. 5 is an explanatory diagram illustrating an example of a gradient operator. The gradient operator is a filter that can determine the gradient of luminance from the three-dimensional image data. The Sobel filter shown in FIG. 5 includes coefficients for 3 × 3 × 3 pixels as shown in the figure for calculating luminance gradients in the X, Y, and Z directions. The luminance gradient value in the X direction at the target pixel is set by aligning the center of the X direction filter with the target pixel and multiplying by the filter coefficient (at the same position) corresponding to each of the 3 × 3 × 3 pixels centered on the target pixel. And the multiplication results are added.

  For example, if there is no change in luminance around the pixel of interest, the output of the filter calculation is 0, but if the luminance increases toward the right (X-axis direction) in FIG. 5, the filter calculation output in the X direction is a positive value. If the brightness decreases as going to the right, the value becomes negative. The magnitude of the calculated value increases as the luminance gradient increases. The same calculation is performed for the Y and Z directions.

  In S22, the luminance horizontal tilt direction θ, vertical tilt direction φ, and tilt amount N are obtained. The horizontal inclination direction θ, the vertical inclination direction φ, and the inclination amount N of the luminance are obtained as in the following Expression 2. arctan is an arc tangent function.

  The horizontal tilt direction θ and the vertical tilt direction φ correspond to the vector direction, and the tilt amount N corresponds to the vector norm (size). The obtained horizontal tilt direction θ, vertical tilt direction φ, and tilt amount N are stored in correspondence with pixels. Here, arctan is a function that takes a value of θ = −π to π depending on the signs of x, y, and z. In other words, the inclination direction (angle) has directions in all directions of the three-dimensional spherical surface. As a processing procedure, the gradient vector (Equation 1) can be directly quantized using a quantization vector described later.

  In S23, it is determined whether or not the processing has been completed for all the pixels. If the determination result is negative, the process proceeds to S20. If the determination is affirmative, the process ends and the process proceeds to the next step.

  FIG. 8 is a flowchart showing the contents of the local autocorrelation value calculation process (S12). In S30, the angles of all gradient vectors are converted into quantized angle vectors. FIG. 7 is an explanatory diagram illustrating an example of quantization of a three-dimensional gradient vector. The three-dimensional gradient vector quantizes the angles θ and Φ of the gradient vector using a plurality of reference vectors (black dots) distributed almost uniformly on the sphere. In the example of FIG. 7, for example, the angle vectors θ and Φ of the gradient vector are quantized using reference vectors A0 to G0 in 42 directions distributed almost uniformly on the spherical surface.

  Numbers 0 to 7 of the reference vector represent angles from the X axis, which is the reference direction on the XY plane. For example, 0 is 0 degrees, 1 is 45 degrees, and 2 is 90 degrees. Alphabets A to G represent elevation angles from the XY plane. For example, A is 90 degrees, B is 60 degrees, and C is 30 degrees.

The representation shown in FIG. 7A is the nearest neighbor quantum in which only the element corresponding to the reference vector closest to the angles θ and Φ of the vector f (θ, φ) is 1 (B1 in FIG. 7) and the other is 0. This is an example. The expression shown in FIG. 7 (b) shows the four reference vectors surrounding the angles [theta] and [phi] of the vector f ([theta], [phi]) and the corresponding element values as the angular difference (t and 1-t) between [theta] and the reference vector. ), A linear interpolation expression example in which the value is proportional to the angular difference (u and 1-u) between Φ and the reference vector. Note that the number and arrangement of element vectors can be arbitrarily changed.
Further, as described above, the distance between the gradient vector and these quantization reference vectors can be directly calculated to determine the quantization element value. In this case, for example, the distance between the gradient vector and the quantization vector is calculated, and the top few quantization reference vectors are searched in ascending order of the distance. The weight for the selected quantization reference vector is determined so as to increase as the distance decreases (for example, exp (−distance) or 1 / distance), and is normalized so that the sum of the weights becomes 1.

  In S31, an unprocessed correlation pattern is selected. FIG. 9 is an explanatory diagram showing a correlation pattern. The correlation pattern for obtaining autocorrelation (up to the first order correlation) is one correlation pattern representing the zeroth order correlation of only the center target pixel (the center of the B layer), and the target pixel and one surrounding pixel. There are a total of 14 correlation patterns representing the first-order correlation.

  There are 26 types of primary correlation patterns in total. However, for example, the combination of the center pixel of interest and the left pixel is the same combination as the combination pattern of the center pixel of interest and the right pixel when the pixel of interest is moved one place to the left. Accordingly, the overlapping correlation pattern is eliminated by leaving one pixel by moving the target pixel in an arbitrary direction. As a result, as shown in FIG. 9, for example, the correlation value can be calculated using only the data of the A layer and the B layer without using the data of the C layer. Although the first-order correlation is considered here, the second-order or higher-order correlation (three-point relationship or higher) can be defined in the same manner.

  In S32, unprocessed data (pixels) is selected. In S33, as a correlation value, products for all combinations between vector elements of the quantization angle vector are obtained based on the correlation pattern. That is, for example, if two n-dimensional quantization angle vectors are a = [a_1... A_n] and b = [b_1... B_n], the correlation value element is C (i, j) = a_i × b_j. There are n × n combinations. However, since there are several non-zero elements among the quantized angle vector elements, the amount of calculation is greatly reduced by performing the correlation calculation only for these non-zero elements.

  In the case of a three-dimensional gradient vector, the correlation value is the quantization vector itself in the case of a zero-order correlation pattern. In the embodiment, since the dimension of the quantization vector is 42 as shown in FIG. 7, the dimension of the zero-order correlation is 42. In the case of the primary correlation pattern, the dimension is 42 × 42 = 1766. When expressed by an equation, the following Equation 3 is obtained. Note that f is a quantization gradient vector, and W is a correlation value. In addition, as described above, the operator “XX” is an operator that generates a correlation value vector whose element is a product between arbitrary elements of each quantization gradient vector.

  In S34, the calculated correlation value is stored for each pixel. In S35, it is determined whether or not processing has been completed for all pixels. If the determination result is negative, the process proceeds to S32, but if the determination is affirmative, the process proceeds to S36. In S36, it is determined whether or not the processing has been completed for all correlation patterns. If the determination result is negative, the process proceeds to S31, but if the determination is affirmative, the process ends. When the processing is completed for all the correlation patterns, the dimension of the feature data obtained by combining all the correlation values is 42 + 42 × 42 × 13 = 22974 in the case of the quantization shown in FIG.

  FIG. 10 is a flowchart showing the content A of the correlation value integration process (S13). In S40, all correlation values are added with a weight of 1 for each element of 22974 dimensions. When expressed by an expression, the following expression 4 is obtained.

  FIG. 11 is a flowchart showing the content B of the correlation value integration process (S13). In S41, all correlation values are weighted and added by the norm (gradient amount) N of the gradient vector for each element of 22974 dimensions. As a weighting method, for example, a method may be used in which a minimum value of norms corresponding to two pixels that have been correlated by a correlation pattern is multiplied as a weight. When expressed by an equation, Equation 5 shown below is obtained.

  Further, a method of multiplying the maximum value of the norm, a method of multiplying the product of the two norms, a method of multiplying by taking the logarithm of any one of the above weights, etc. can be considered. Multidimensional feature data is obtained by the processing as described above.

  Although the embodiments have been described above, the following modifications may be considered in the present invention. In the embodiment, the rotation of the image (target object) is not considered, but the rotation invariant (target) is obtained by adding all the duplicated data when the image is rotated among the extracted high-dimensional feature data. The same feature data can be obtained in any direction.

  In the embodiment, an example in which the gradient vector is distributed in an arbitrary direction on the entire sphere has been disclosed. However, when extracting the feature of “motion” from a moving image, the motion is expressed in three-dimensional coordinates. The shape of the generated curved surface is important, and the curved surface shape is characterized by the normal vector (group), that is, the gradient vector. This normal vector can be taken in two ways for a point object, but both are equivalent for a curved surface. This orientation is determined by color information such as clothes color, and this information is often irrelevant to movement information. Therefore, if a point-symmetric vector in the normal vector is identified, it is possible to quantize with a reference vector only in the hemispherical direction as the quantization direction. In this way, the dimension of the feature data can be greatly reduced.

  As a correlation pattern for obtaining autocorrelation, there are one correlation pattern representing the zeroth order correlation of only the target pixel and thirteen correlation patterns representing the first order correlation composed of the target pixel and one surrounding pixel. In the case of motion recognition, a configuration in which the first-order correlation is limited to the same frame, that is, the displacement in the time (t) direction = 0 is also conceivable. This limited correlation pattern corresponds to four correlation patterns (10) to (13) in FIG. In this way, the number of dimensions of the feature data decreases, and if it is a moving image, the target information at that time can be sufficiently extracted.

  In the embodiment, the example in which the angles θ and Φ of the gradient vectors are quantized using the reference vectors A0 to G0 in 42 directions distributed almost uniformly on the spherical surface has been disclosed. However, the dimension of the correlation vector becomes as large as 42 × 42 = 1766 dimensions. Therefore, in the case of motion recognition, only the motion information is important. Therefore, as reference vectors, for example, reference vectors D0 to D7 whose components in the time (t) direction are zero in FIG. It may be used. In this way, the number of dimensions of feature data can be reduced.

  The reference vectors D0 to D7 have zero component in the time (t) direction, and as a result of quantization, the pixels determined to have gradient vectors in the vicinity of these reference vectors are very small in luminance in the time direction. This is a pixel of a still image portion such as a background. For this reason, it is expected that information that is irrelevant to these movements will be deleted in advance, so that it becomes a more effective feature quantity when the recognition target is an action. As a result of experiments, it was confirmed that the above method is effective in recognition of motion.

  For applications such as scene understanding in moving images, information on the image (shape) of a stationary part such as a background is also necessary, so it is better to extract features using all reference vectors. In this case, it is possible to take advantage of the present invention that feature extraction of motion + stationary shape, which cannot be extracted by the conventional background difference + stereoscopic higher-order local autocorrelation feature method described in Patent Document 1, can be performed.

  As an embodiment, an example of performing software processing using a computer has been disclosed. However, the hardware for implementing the present invention is not limited to a computer, and can be realized by a combination of logic circuits. High-speed processing is possible.

It is a block diagram which shows the structure of the feature extraction apparatus by this invention. It is explanatory drawing which shows the structure of three-dimensional image data. It is a flowchart which shows the content of the process in the feature extraction apparatus of this invention. It is a flowchart which shows the content of the gradient vector calculation process (S11) of this invention. It is explanatory drawing which shows the example of a gradient operator. It is explanatory drawing which shows a gradient vector. It is explanatory drawing which shows the example of quantization of a three-dimensional gradient vector. It is a flowchart which shows the content of a local autocorrelation value calculation process (S12). It is explanatory drawing which shows a correlation pattern. It is a flowchart which shows the content A of a correlation value integration process (S13). It is a flowchart which shows the content B of a correlation value integration process (S13).

Explanation of symbols

DESCRIPTION OF SYMBOLS 10 ... Video camera 11 ... Computer 12 ... Monitor apparatus 13 ... Keyboard 14 ... Mouse 15 ... CT scanner apparatus

Claims (4)

Gradient vector calculation means for calculating, from each data value of the three-dimensional data, a gradient vector expressed as a quantization vector by a plurality of quantization elements each representing an angle in which the angle information indicating the inclination direction of the data value is different from each other ;
A plurality of correlation patterns indicating the data position of interest and its neighboring data positions that do not overlap in translation are used, and for each combination of quantization elements of the gradient vector represented by each correlation pattern, Local autocorrelation means for calculating a local autocorrelation value by calculating a correlation value by multiplying the values ;
A feature extraction apparatus comprising: addition means for adding the local autocorrelation values calculated for each gradient vector. The gradient vector is expressed as a three-dimensional vector including information corresponding to a tilt amount in addition to angle information indicating a tilt direction of a data value,
The feature extraction apparatus according to claim 1 , wherein the adding means adds a correlation value by adding a weight according to an inclination amount of a data value.   2. The feature extraction apparatus according to claim 1, wherein the gradient vector calculating means extracts information on a tilt direction and a tilt amount by applying a filter having a predetermined coefficient to a local region including attention data. Calculating, from each data value of the three-dimensional data, a gradient vector expressed as a quantization vector by a plurality of quantization elements each representing an angle in which the angle information indicating the inclination direction of the data value is different ;
A plurality of correlation patterns indicating the data position of interest and its neighboring data positions that do not overlap in translation are used, and for each combination of quantization elements of the gradient vector represented by each correlation pattern, Calculating a local autocorrelation value by determining a correlation value by multiplying the values ;
A feature extraction method comprising the step of adding the local autocorrelation values calculated for each gradient vector.

Images