JP5166335B2 - Apparatus, method and program for extracting features - Google Patents

Description

  The present invention relates to an apparatus, a method, and a program for extracting features.

  A technique for detecting a specific pattern included in an input image or a technique for classifying a plurality of patterns into a known class is called a pattern recognition (pattern identification) technique. Pattern recognition consists of two stages: a feature extraction process that extracts only features that are effective for identification from an image, and a discrimination process that uses the extracted features. The feature extraction process in the previous stage is extremely important for realizing high discrimination performance with a small amount of calculation. Non-Patent Document 1 proposes a feature amount extracted by an analysis method called CoHOG (Co-occurrence Histograms of Oriented Gradients).

T. Watanabe et al., “Co-occurrence Histograms of Oriented Gradients for Pedestrian Detection”, T. Wada, F. Huang, and S. Lin (Eds.): PSIVT 2009, LNCS 5414, Springer-Verlag Berlin Heidelberg, pp 37-47, 2009.

  However, in the conventional method, since the feature amount is generally calculated based on a single image, there is a case where high identification performance cannot be realized by the identification process. That is, when detecting or recognizing a specific target object, it is desirable to use information between different images. However, in the method of Non-Patent Document 1, for example, a feature is obtained from a plurality of different images. It was not considered to calculate the amount.

  The present invention has been made in view of the above, and an object of the present invention is to provide an apparatus, a method, and a program capable of extracting a feature quantity having high discrimination performance.

In order to solve the above-described problems and achieve the object, one embodiment of the present invention includes a detection unit that detects partial images corresponding to each other between a plurality of images, each of the plurality of partial images, and the portion. A pixel feature calculation unit that calculates a pixel feature amount for each pixel included in the image, a pixel feature amount calculated for each pixel included in the partial image for each partial image, and the pixel predetermined An intra-image feature that calculates a co-occurrence frequency with a pixel feature amount calculated for other pixels separated by a distance and calculates an intra-image feature amount that includes each of the co-occurrence frequencies calculated for each pixel A calculation unit and a feature amount including an element obtained by integrating the co-occurrence frequencies included in each of the plurality of image feature amounts calculated for each of the plurality of partial images into one corresponding pixel. Image features Characterized in that it comprises a generation unit for generating a.

  According to the present invention, it is possible to extract a feature amount having high identification performance.

It is a block diagram which shows an example of a structure of the feature extraction apparatus which concerns on this Embodiment. It is a figure which shows the relationship between a feature extraction apparatus and the object to detect. It is a figure which shows an example of the stereo camera coordinate system used by this Embodiment. It is a figure which shows an example of the area | region which an area | region setting part detects and sets. It is a schematic diagram for demonstrating an example of the calculation method of parallax. It is a figure which shows an example of the rectangle produced | generated in a stereo process area | region. It is a schematic diagram for demonstrating the evaluation method of a rectangle. It is a figure explaining an example of the method of quantizing a luminance gradient direction. It is a figure which shows an example of the division | segmentation method of a candidate area | region. It is the figure which expanded and displayed the mesh of FIG. It is a figure which shows an example of the matrix showing a co-occurrence histogram. It is a figure which shows an example of the some displacement vector whose Chebyshev distance is 1 from an attention pixel. It is a figure which shows an example of the attention pixel in the case of calculating the feature vector between images. It is a figure which shows an example of the reference pixel in the case of calculating the feature vector between images. It is a figure which shows an example of the matrix showing the co-occurrence histogram in the case of calculating the feature vector between images. It is a figure which shows the relationship of the calculated feature vector. It is a flowchart which shows the whole flow of the feature extraction process in this Embodiment. It is a figure explaining illustratively the composition of the device which realizes each function of the feature extraction device concerning this embodiment.

  Exemplary embodiments of a feature extraction apparatus, a feature extraction method, and a feature extraction program according to the present invention will be explained below in detail with reference to the accompanying drawings.

  The feature extraction device according to the present embodiment is a combination of pixel feature amounts of pixels in the same partial image for a plurality of corresponding partial images detected from a plurality of pieces of image data as candidate regions for detecting an object. In-image feature amounts representing the co-occurrence frequencies are calculated. Then, an image feature amount is generated by integrating a plurality of in-image feature amounts calculated for each of the corresponding partial images. Thereby, the feature-value which has the high identification capability in consideration of the some image can be obtained.

  Furthermore, the feature extraction apparatus according to the present embodiment shares the combination of the pixel feature amount of a pixel in one partial image and the pixel feature amount of a pixel in another partial image among the corresponding partial images. An inter-image feature amount representing the occurrence frequency is calculated. And the image feature-value which combined this inter-image feature-value is produced | generated. Accordingly, it is possible to obtain a feature amount having higher discrimination ability in consideration of a plurality of images.

  FIG. 1 is a block diagram showing an example of the configuration of the feature extraction apparatus 100 according to the present embodiment. As shown in FIG. 1, the feature extraction apparatus 100 includes image input units 101a and 101b, image storage units 102a and 102b, an area setting unit 110, a normalization unit 103, a pixel feature calculation unit 104, an in-image A feature calculation unit 105, an inter-image feature calculation unit 106, a generation unit 107, and an output unit 108 are provided. Note that the image input units 101 a and 101 b and the image storage units 102 a and 102 b may be arranged outside the feature extraction device 100 and connected to the feature extraction device 100.

  In the present embodiment, as shown in FIG. 2, the traveling direction of the automobile using two cameras 11 a and 11 b functioning as image input units 101 a and 101 b mounted in front of the moving body 201 (automobile). It is assumed that the feature amount used when detecting the pedestrian 211 existing on the road surface 210 is extracted.

  Returning to FIG. 1, the image input units 101 a and 101 b input a plurality of images having different shooting viewpoints using at least two cameras. If there is an overlap in the field of view, the positions and directions of the cameras can be arbitrarily set. In the present embodiment, it is assumed that the same two cameras 11a and 11b are arranged in parallel on the left and right to shoot a stereo image.

  FIG. 3 is a diagram illustrating an example of a stereo camera coordinate system used in the present embodiment. In the present embodiment, the origin O of the stereo camera coordinate system is taken as the viewpoint (lens center) of the right camera (camera 11b). Further, a straight line connecting the viewpoints of the left and right cameras is set as the X axis, the Y axis is set vertically downward, and the Z axis is set in the optical axis direction of the camera.

If the distance between cameras (baseline length) is B, the position of the left camera can be expressed as (−B, 0, 0). For the sake of simplicity, if the road surface is modeled as a plane and is ignored because the slope in the horizontal direction is very small, the equation for the road surface (road plane) is expressed by the following equation (1).
Y = αZ + β (1)

  α and β represent the inclination of the road plane observed from the stereo camera and the height of the stereo camera from the road surface, respectively. Hereinafter, α and β are collectively referred to as surface parameters. In general, the slope of the road surface varies from place to place, and the camera vibrates when traveling on a moving body, so that the surface parameters change from moment to moment as the moving body moves.

  Here, a coordinate system for each image of the stereo image will be described. An image coordinate system is set for each image constituting the stereo image. An x-axis and a y-axis are set in the horizontal direction and the vertical direction, respectively, in an image captured by the right camera (camera 11b) (hereinafter referred to as “right image”). Similarly, an x ′ axis and a y ′ axis are set in the horizontal direction and the vertical direction, respectively, in an image captured by the left camera (camera 11a) (hereinafter referred to as “left image”). It is assumed that the horizontal direction (x-axis and x′-axis) of each image matches the X-axis direction of the stereo camera coordinate system.

  In such a case, if the corresponding point on the left image of the point (x, y) on the right image is (x ′, y ′), y = y ′. Therefore, only the difference in position in the horizontal direction needs to be considered. Hereinafter, the difference in the horizontal position of the corresponding point between the right image and the left image is referred to as parallax. In the following, the parallax is expressed as d = x−x ′ with the right image as the reference image.

  Returning to FIG. 1, the image storage units 102a and 102b store the image data acquired by the image input units 101a and 101b, respectively. The image storage units 102a and 102b can be configured by any commonly used storage medium such as an HDD (Hard Disk Drive), an optical disk, a memory card, and a RAM (Random Access Memory).

The region setting unit 110 sets a region (candidate region) that is a candidate for detecting a three-dimensional object existing on the road. FIG. 4 is a diagram illustrating an example of an area detected and set by the area setting unit 110. As shown in the figure, the region setting unit 110 extracts corresponding regions between stereo images (left image and right image) as a pair. In the figure, an example is shown in which rectangles R _{1 to} R ₃ of the right image indicated by dotted lines and rectangles R ′ _{1 to} R ′ _{3 of} the corresponding left image are set as candidate areas.

  As shown in FIG. 1, the region setting unit 110 includes a parallax calculation unit 111, a parameter calculation unit 112, and a detection unit 113 as detailed configurations.

  The parallax calculation unit 111 calculates parallax between stereo images acquired by the image input units 101a and 101b and stored in the image storage units 102a and 102b. Specifically, the parallax calculation unit 111 obtains corresponding points (corresponding points) between a plurality of images constituting a stereo image, and each of the obtained corresponding points represents a difference in position on the plurality of images. Is calculated.

  Next, a parallax calculation method by the parallax calculation unit 111 will be described with reference to FIG. FIG. 5 is a schematic diagram for explaining an example of a parallax calculation method. The parallax calculation unit 111 searches the same scanning line on the left image for an arbitrary point (x, y) of the right image that is the reference image, and finds a corresponding point (x ′, y) = (x + d, y). Ask.

  Since d ≧ 0, the parallax calculation unit 111 needs to examine only the point on the right side of the same coordinate on the left image as shown in FIG. More specifically, the parallax calculation unit 111 sets a window around a point (x, y) on the right image as shown in FIG. The surrounding points are obtained from the same scanning line of the left image.

For example, normalized cross-correlation C can be used as an evaluation measure of similarity of luminance patterns. When the size of the search window is (2w + 1) × (2w + 1) pixels and the luminance in the window set for the left and right images is expressed as f (ξ, η) and g (ξ, η), the normalized cross-correlation C is (2).

  If a corresponding point is searched for any point on the reference image using the normalized cross-correlation C, a parallax map including the parallax for all the points can be obtained.

Returning to FIG. 1, the parameter calculation unit 112 calculates the surface parameters α and β using the parallax map calculated by the parallax calculation unit 111. First, a method for obtaining the position (X, Y, Z) of the point in the three-dimensional space from the parallax d of the point (x, y) on the reference image will be described. The following relational expression (5) is established between the point (X, Y, Z) in the space and (x ′, y) and (x, y) which are the projected images on the left and right images.

When the equation (5) is solved for X, Y, and Z, the following equation (6) is obtained.

  The parameter calculation unit 112 obtains the position (three-dimensional position) in the three-dimensional space of the point on the reference image from which the parallax is obtained by the parallax calculation unit 111 using each of the above equations. Then, the parameter calculation unit 112 selects a point that is close to the road surface from among the obtained three-dimensional positions, and substitutes it into the above equation (1) (Y = αZ + β) that is the road surface equation. Then, α and β which are surface parameters are calculated.

The parameter calculation unit 112 extracts a point that is close to the road surface as a point that satisfies the following expression (7).

Here, ΔY is a threshold value, and an appropriate value is set in advance. Y _p represents the point (X, Y, Z) Y coordinate of the intersection of the parallel straight lines and the reference road surface in Y axis through. The reference road surface refers to a road surface that is a predetermined reference such as a flat road. The parameters of the reference road surface are measured on a flat road when the moving body is stationary, for example. If the parameters of the reference road surface are α ′ and β ′, Y _p is given by the following equation (8).
Y _p = α′Z + β ′ (8)

  Using the calculated parameters, the detection unit 113 detects partial images (candidate regions) that correspond to each other among a plurality of images as a region for detecting an object. The detection unit 113 detects a candidate area according to the following procedure.

  First, the detection unit 113 sets a rectangle having an arbitrary point (x, y) in the stereo processing region with the lower side as a midpoint. FIG. 6 is a diagram illustrating an example of a rectangle generated in the stereo processing area. By using the surface parameters α and β, the detection unit 113 sets a rectangle whose lower side is in contact with the road surface and whose size is determined in consideration of the vertical and horizontal sizes of a person.

For example, when the representative values of the human height and the horizontal width are H and W, respectively, the height h and the width w on the rectangular image can be obtained as follows. That is, since the following formula (9) is established from the formula (1) (Y = αZ + β) and the formula (5), which are road surface equations, the following formula (10) is obtained.

  As described above, the size of the rectangular image generated by the detection unit 113 varies depending on the vertical position on the image, that is, the y coordinate. The detection unit 113 generates a plurality of types of rectangles for each point (x, y) on the image as shown in FIG. In the figure, an example of setting three types of rectangles is shown.

  Next, the detection unit 113 evaluates the possibility that a person (pedestrian) is included in the rectangle from the parallax in the rectangle set to each point in this way, and detects a rectangle having a high possibility as a candidate area. To do. Specifically, the detection unit 113 evaluates a rectangle having a uniformity degree that represents a degree of uniformity of parallax included in the rectangle that is greater than a predetermined threshold as a rectangle that is likely to include a person. FIG. 7 is a schematic diagram for explaining a rectangular evaluation method.

As shown in FIG. 7A, when a person is included in a rectangle 901, the depth in the rectangle is almost uniform, and the corresponding parallax is also uniform. Further, since the point (x, y) is in contact with the road surface, the parallax at this point is given by the following equation (11).

Therefore, if the parallax at an arbitrary point in the rectangle is d _i , the detection unit 113 can evaluate the uniformity of the parallax with the number N of points that satisfy the following expression (12). Here, Δd represents a predetermined parallax difference threshold (difference threshold).

In order to consider the influence of the size of the rectangle, the detection unit 113 normalizes the number N with the rectangular area S = w × h, and uses the rectangle that satisfies the following expression (13) as a candidate region. sign up.

Here, N _min is a threshold value, and an appropriate value is set in advance. Note that, as shown in FIG. 7B, even in a rectangle 902 smaller than an object to be detected (a person such as a pedestrian), the parallax within the rectangle becomes uniform, and the above equation (13) may be satisfied. In order to prevent such a rectangle from becoming a candidate, the detection unit 113 selects only a rectangle having the largest area as a candidate area when a plurality of rectangles satisfy the above expression (13) for a certain point. You may comprise as follows. For example, when a rectangle 901 and a rectangle 902 are candidate regions as shown in FIG. 7B, only the rectangle 901 may be selected as a candidate region.

  Moreover, the evaluation method of the said uniformity is an example, and any evaluation method can be applied if it can evaluate the uniformity of the parallax within a rectangle. Moreover, you may comprise so that the uniformity which represents that a degree of uniformity is so large that a value is small may be used.

The detection unit 113 detects N (N ≧ 0) candidate regions R _{1 to} R _N by the above processing. FIG. 4 described above shows an example in which _three candidate regions R ₁ , R ₂ , and R ₃ are detected. The detection unit 113 further extracts candidate regions R ′ _{1 to} R ′ _N in the left image corresponding to these candidate regions detected in the right image. Since the lower side of each rectangle is in contact with the road surface and the parallax is given by equation (11), candidate regions R ′ _{1 to} R ′ _N are detected from the y coordinate of the lower side of each rectangle. In the following, N (≧ 0) corresponding regions in which the candidate regions R _{1 to} R _N detected in the right image and the corresponding candidate regions R ′ _{1 to} R ′ _{N of} the left image are associated with ( R ₁ , R ′ ₁ ) to (R _N , R ′ _N ).

  The normalization unit 103 normalizes each candidate area included in the N sets of corresponding areas set by the area setting unit 110 to a predetermined size. The normalization size is arbitrary, but in the present embodiment, it is assumed that it is aligned with a vertically long rectangle of 48 × 24 pixels.

  The pixel feature calculation unit 104 calculates the feature amount of the image data of the candidate area normalized by the normalization unit 103 for each pixel. For example, the pixel feature calculation unit 104 calculates the luminance gradient direction as the pixel feature amount. The gradient direction of the luminance is a feature amount that is robust against illumination variation and the like, and is an effective feature amount even in an environment where the change in brightness is large. When the change in brightness is relatively small, the luminance value itself may be used as the feature amount. Hereinafter, a case where the luminance gradient direction is used as the pixel feature amount will be described. Further, the pixel feature calculation unit 104 quantizes the calculated luminance gradient direction into a discrete value in an appropriate range. FIG. 8 is a diagram for explaining an example of a method of quantizing the luminance gradient direction. In the figure, a case where quantization is performed in 8 directions of 0 to 7 is illustrated.

  The in-image feature calculation unit 105 calculates an in-image feature vector for each normalized area. The in-image feature vector is a feature amount (intra-image feature amount) represented in a vector format having the co-occurrence frequency of pixel feature amounts calculated for a plurality of pixels in an area as an element.

  The in-image feature calculation unit 105 calculates one feature vector for one candidate region. Since 2N normalization candidate regions are extracted, the in-image feature calculation unit 105 calculates 2N feature vectors. The in-image feature calculation unit 105 first divides each candidate region normalized by the normalization unit 103 into a plurality of partial regions. FIG. 9 is a diagram illustrating an example of a candidate region dividing method. FIG. 9 shows an example in which the candidate area is divided into a plurality of square partial areas (square areas) by dividing the candidate area into a mesh shape.

  In this example, the in-image feature calculation unit 105 calculates a co-occurrence histogram for each mesh (square area) divided into meshes next. Although the number of meshes is arbitrary, for example, as shown in FIG. Since the entire region is 48 × 24 pixels, one section (one partial region) is a square region of 6 × 6 pixels.

  The co-occurrence histogram will be described below. FIG. 10 is an enlarged view of the mesh of FIG. In the figure, one rectangle represents one pixel in the mesh. An arrow indicates the luminance gradient direction calculated for the pixel. First, consider a pixel of interest r = (x, y) and a pixel r + δ that is separated from the pixel of interest by a displacement (displacement vector) δ = (δx, δy). FIG. 10 shows the case where δ = (1, 0).

ここで「＃」は括弧内の集合の要素数を示す。また、Ｉ（ｘ）等は画像上の点ｘでの画素特徴量を示す。行列ｈｉｊは、ある変位ベクトルδによって定義される２つの画素における画素特徴量の組み合わせのメッシュ内での分布を示す２次元のヒストグラム（共起ヒストグラム）である。行列ｈｉｊは１つの変位ベクトルに対して定義されるため、Ｄ種類の変位ベクトルを用いた場合にはＤ個の２次元ヒストグラム（共起ヒストグラム）が生成される。 Next, a matrix as shown in FIG. 11 is defined with the pixel feature values of r and r + δ as i and j, respectively. In the case of FIG. 10, since the combination of the pixel feature value i of r and the pixel feature value j of r + δ is (i, j) = (0, 1), it corresponds to the element h01 of the matrix. In general, the matrix hij is defined as the following equation (14).

Here, “#” indicates the number of elements in the set in parentheses. I (x) and the like indicate pixel feature amounts at a point x on the image. The matrix hij is a two-dimensional histogram (co-occurrence histogram) showing the distribution in the mesh of combinations of pixel feature values in two pixels defined by a certain displacement vector δ. Since the matrix hij is defined for one displacement vector, D two-dimensional histograms (co-occurrence histograms) are generated when D types of displacement vectors are used.

FIG. 12 is a diagram illustrating an example of a plurality of displacement vectors whose Chebyshev distance is 1 from the target pixel. For example, δ = (− 1, 0) can be substituted with δ = (1, 0) if the target pixel and the reference pixel are interchanged. Note that the reference pixel represents a pixel separated from the target pixel by a displacement vector. For this reason, there are four types of displacement vectors of δ ₁ to δ _{4 with} a Chebyshev distance of 1, as shown in FIG.

  The in-image feature calculation unit 105 calculates, for example, four co-occurrence histograms using each of four types of displacement vectors as shown in FIG. The generation method and the number of displacement vectors are not limited to this. For example, the displacement vector may be generated using the Manhattan distance or the Euclidean distance.

  The brightness gradient direction has 8 steps (see FIG. 8). When four types of displacement vectors are used, the in-image feature calculation unit 105 has 8 × 8 × 4 = 256 dimensions for each mesh in FIG. A feature vector is calculated. In this example, since the total number of meshes is 8 × 4 = 32, a 256 × 32 = 8192-dimensional feature vector is generated from one candidate region. This 8192-dimensional feature vector corresponds to the in-image feature vector.

  Since a pair of candidate areas is extracted from the left and right images as described above, the in-image feature calculation unit 105 calculates one in-image feature vector for each candidate area. In the following, the in-image feature vector calculated from the candidate area of the left image is denoted as λ, and the in-image feature vector calculated from the candidate area of the right image is denoted as μ.

  The inter-image feature calculation unit 106 calculates an inter-image feature vector for each corresponding region. The inter-image feature vector is a feature amount (inter-image feature amount) expressed in a vector format having the co-occurrence frequency of pixel feature amounts calculated for a plurality of pixels respectively included in different images. Specifically, the inter-image feature calculation unit 106 extracts the target pixel r = (x, y) and the reference pixel r + δ from different images, and uses the same method as the in-image feature calculation unit 105 to perform the co-occurrence histogram. Are used as elements of the feature vector. Similarly to the above-described example, when the luminance gradient direction has 8 steps, the displacement vector has 4 types, and the image is divided into 32 meshes, the inter-image feature calculation unit 106 converts the 8192-dimensional feature vector between the images. Calculate as a feature vector. Hereinafter, the inter-image feature vector is denoted as ψ.

  FIG. 13 and FIG. 14 are diagrams illustrating an example of a target pixel and a reference pixel when calculating an inter-image feature vector, respectively. FIG. 13 shows a pixel of interest in the candidate area R detected from the right image. FIG. 14 shows reference pixels in the candidate area R ′ corresponding to the candidate area R in FIG. 13. 13 and 14 show a case where the displacement vector δ = (1, 0).

FIG. 15 is a diagram illustrating an example of a matrix representing a co-occurrence histogram when an inter-image feature vector is calculated. In the example of FIGS. 13 and 14, since the combination of the pixel feature value of r and the pixel feature value of r + δ is (i, j) = (0, 1), the element g01 of the matrix shown in FIG. 15 corresponds. . In general, a matrix gij representing a co-occurrence histogram between images is defined as the following equation (15).

  FIG. 16 is a diagram illustrating the relationship between the calculated feature vectors. As shown in FIG. 16, in-image feature vectors λ and μ are calculated from the candidate region R ′ for the left image and the candidate region R for the right image, respectively. Further, an inter-image feature vector ψ is calculated from the candidate region R ′ and the candidate region R.

The generation unit 107 generates a new feature vector Ψ from three feature vectors as shown in FIG. 16 as an image feature amount for each candidate pattern. For example, the generation unit 107 combines the three vectors as shown in the following equation (16) to generate the feature vector Ψ. Here, | λ |, | μ |, and | ψ | represent the number of dimensions of the feature vectors λ, μ, and ψ, respectively.

Note that the generation unit 107 may combine the two intra-image feature vectors λ and μ and then combine it with the inter-image feature vector ψ. When a vector obtained by integrating λ and μ is Λ, the vector Λ is expressed by the following equation (17). The feature vector ψ is a | Λ | + | ψ | -dimensional feature vector as shown in the following equation (18).

  For example, the generation unit 107 can integrate the in-image feature vectors λ and μ by calculating an arithmetic mean or a geometric mean for each element. Note that the method of integrating the feature vectors in the image is not limited to these. By integrating the in-image feature vectors in this way, the size of the finally calculated feature vector Ψ can be reduced.

  The output unit 108 outputs the feature vector Ψ generated by the generation unit 107 for each candidate pattern.

  Next, feature extraction processing by the feature extraction apparatus 100 according to the present embodiment configured as described above will be described with reference to FIG. FIG. 17 is a flowchart showing the overall flow of feature extraction processing in the present embodiment.

  First, the image input units 101a and 101b input left and right images captured by the cameras 11a and 11b constituting the stereo camera (step S1701). The input left and right images are stored in the image storage units 102a and 102b.

  Next, the parallax calculation unit 111 obtains corresponding points between the input left image and right image, calculates the parallax for each point, and creates a parallax map (step S1702). Next, the parameter calculation unit 112 calculates the surface parameters α and β of the road plane using the parallax map (step S1703).

Next, the detection unit 113 generates a candidate area for the stereo processing area, and detects a candidate area having a parallax uniformity greater than a threshold among the generated candidate areas (step S1704). As a result, N sets of corresponding regions (R ₁ , R ′ ₁ ) in which the candidate regions R _{1 to} R _N detected in the right image are associated with the corresponding candidate regions R ′ _{1 to} R ′ _{N of} the left image. ~ (R _N , R ' _N ) is obtained.

  Next, the normalization unit 103 normalizes each candidate area included in each corresponding area (step S1705).

  Next, the pixel feature calculation unit 104 calculates the pixel feature amount of the image data of the normalized candidate area for each pixel (step S1706). Also, the in-image feature calculation unit 105 calculates an in-image feature amount (intra-image feature vector) for each normalized candidate region using the calculated pixel feature amount (step S1707). Further, the inter-image feature calculation unit 106 calculates an inter-image feature amount (inter-image feature vector) for each normalized candidate area using the calculated pixel feature amount (step S1708).

  Next, the generation unit 107 integrates the in-image feature amount calculated from the candidate region of the right image and the in-image feature amount calculated from the corresponding candidate region of the left image. By combining the feature quantity, an image feature quantity for each candidate pattern is generated (step S1709). Then, the output unit 108 outputs the generated image feature amount (step S1710) and ends the feature extraction process.

  As described above, the in-image feature quantity extracted in the same image and the inter-image feature quantity extracted between different images can be calculated, and an image feature quantity describing both can be calculated. Thus, it is possible to calculate a feature amount that can realize extremely high discrimination performance when used for pattern discrimination.

  In this embodiment, the case where a pedestrian existing in the traveling direction is detected using a camera installed in front of the moving body has been described. However, a camera may be installed on the side or rear of the moving body. . Further, the present invention can also be applied when a camera is mounted on a moving body other than a moving body such as a mobile robot. Further, the present invention is not limited to the case where a camera is installed on a moving body, and can also be applied to a case where an object is detected from an image captured by a monitoring camera installed at a fixed position.

  Moreover, although the case where a person (pedestrian) was detected was demonstrated, a detection target object is not limited to this, Even if it is another detection target, it is applicable. Further, the present invention can also be applied when detecting multi-class objects such as simultaneous detection of a person and a moving object.

  In addition, the stereo view when two cameras are arranged in parallel on the left and right has been described. However, the number of cameras is arbitrary as long as there are two or more cameras. You may arrange as follows.

  Moreover, although the case where the region setting unit 110 generates a candidate region based on stereo parallax has been described, the motion of each point on a time-series image obtained using one fixed camera is calculated, and the motion information is used. You may comprise so that a candidate area | region may be extracted. In the above example, a three-dimensional object (an object having a height) existing on the road is extracted as a candidate area. In this case, a moving object is extracted as a candidate.

  In addition, although the case where the luminance gradient direction is quantized in eight steps has been described as the pixel feature amount calculated by the pixel feature calculation unit 104, the number of quantization is not limited to this. For example, quantization may be performed in four directions in which the top and bottom (2 and 6 in FIG. 8), left and right (0 and 4 in FIG. 8), etc. are identified. Also, the magnitude of the luminance gradient may be used instead of the direction of the luminance gradient. Further, output values of filters such as a Gaussian filter, a Sobel filter, and a Laplacian filter may be used as the pixel feature amount. Furthermore, some of these may be combined to calculate a plurality of types of pixel feature amounts for each pixel.

  As described above, in the feature extraction device according to the present embodiment, the in-image feature amount representing the co-occurrence frequency of the combination of pixel feature amounts of the pixels in the same partial image with respect to the partial images detected from each of the plurality of image data. And an image feature quantity obtained by integrating a plurality of feature quantities in the image can be generated. Furthermore, in the feature extraction device according to the present embodiment, an inter-image feature amount that represents a co-occurrence frequency of a combination of a pixel feature amount of a pixel in a partial image and a pixel feature amount of a pixel in another partial image is calculated. It is possible to generate an image feature amount that is calculated and combined with this inter-image feature amount. Accordingly, it is possible to obtain a feature amount having higher discrimination ability in consideration of a plurality of images.

  In addition, since the feature extraction apparatus according to the present embodiment can extract feature quantities effective for identification with a relatively small amount of computation, for example, when applied to object recognition or object detection, the identification performance thereof Can be greatly improved.

  FIG. 18 is a diagram for exemplarily explaining the configuration of a device such as a computer that implements each function of the feature extraction device 100 according to the present embodiment. 18 includes, for example, a main processing unit 400, an input unit 410, a display unit 420, a storage unit 490, an image input unit 101, an input I / F 419, a display I / F 429, an image input I / F 489, and a memory I. / F499.

  The main processing unit 400 implements each function of the feature extraction apparatus 100 by executing a program that executes each procedure in FIG. The main processing unit 400 includes, for example, a CPU 401, a ROM 408, and a RAM 409. The CPU 401 controls each device included in the computer by executing a program. The ROM 408 stores programs, parameters, and the like, for example, and provides them to the CPU 401. The RAM 409 is provided as a work memory when the CPU 401 executes a program, for example, and can also function as the image storage unit 102 in FIG.

  The input unit 410 is input means such as a keyboard and a mouse, for example, and inputs instructions to the computer. The display unit 420 displays the processing result of the CPU 401 and the like.

  An input I / F 419, a display I / F 429, a memory I / F 499, and an image input I / F 489 are an input unit 410, a display unit 420, a storage unit 490, and an image input unit 101 via a bus, respectively. 400 is an interface when connected to 400.

  The image data processed by the feature extraction device 100 according to the present embodiment is acquired by the image input unit 101 or input from the outside via a network, for example. Therefore, the feature extraction apparatus 100 according to the present embodiment may include the image input unit 101 inside, or may be connected to an external image input unit so as to be communicable. The image data processed by the feature extraction device 100 according to the present embodiment may also be read from, for example, a recording medium inserted in a driving device (not shown).

  The feature amount of the image data extracted by the feature extraction apparatus 100 according to the present embodiment is output from the display unit 420 or the network, for example. The feature amount of the image data extracted by the feature extraction device 100 according to the present embodiment may also be recorded in, for example, a recording medium inserted in the drive unit or the storage unit 490.

  The program of the feature extraction apparatus 100 in FIG. 1 is stored in a non-volatile storage device such as the ROM 408 and the storage unit 490, or is recorded on a recording medium such as a CD or DVD, and is inserted into the drive unit. Thus, the computer may read and execute the program.

  It should be noted that the present invention is not limited to the above-described embodiment as it is, and can be embodied by modifying the constituent elements without departing from the scope of the invention in the implementation stage. In addition, various inventions can be formed by appropriately combining a plurality of constituent elements disclosed in the above embodiments. For example, some components may be deleted from all the components shown in the embodiment. Furthermore, constituent elements over different embodiments may be appropriately combined.

104 pixel feature calculation unit 105 in-image feature calculation unit 107 generation unit 113 detection unit

Claims (5)

A detection unit for detecting partial images corresponding to each other between a plurality of images;
A pixel feature calculation unit that calculates a pixel feature amount for each of the plurality of partial images and for each pixel included in the partial image;
For each partial image, for each pixel included in the partial image, the calculated pixel feature amount and the pixel feature amount calculated for other pixels that are separated from the pixel by a predetermined distance are shared. An in-image feature calculating unit that calculates an occurrence frequency and calculates an in-image feature amount including each of the co-occurrence frequencies calculated for each pixel as an element ;
An image feature amount representing a feature amount including an element obtained by integrating the co-occurrence frequencies included in each of the plurality of in-image feature amounts calculated for each of the plurality of partial images into one for each corresponding pixel. A generating unit to generate;
A feature extraction apparatus comprising: Pixel feature values calculated for pixels included in the first partial image among the plurality of partial images, and pixel feature values calculated for pixels included in the second partial image among the plurality of partial images And an inter-image feature calculation unit that calculates an inter-image feature amount that represents a feature amount including the calculated co-occurrence frequency,
The generation unit generates the image feature amount including a plurality of the image feature amounts calculated for each of the plurality of partial images and the inter-image feature amount;
The feature extraction apparatus according to claim 1. A parallax calculator that calculates parallax between the plurality of images;
The detection unit is a partial image included in the plurality of images, and detects the partial image in which the parallax uniformity of each point included in the partial image is greater than a predetermined threshold;
The feature extraction apparatus according to claim 1. Computer
A detection unit for detecting partial images corresponding to each other between a plurality of images;
A pixel feature calculation unit that calculates a pixel feature amount for each of the plurality of partial images and for each pixel included in the partial image;
For each partial image, for each pixel included in the partial image, the calculated pixel feature amount and the pixel feature amount calculated for other pixels that are separated from the pixel by a predetermined distance are shared. An in-image feature calculating unit that calculates an occurrence frequency and calculates an in-image feature amount including each of the co-occurrence frequencies calculated for each pixel as an element ;
An image feature amount representing a feature amount including an element obtained by integrating the co-occurrence frequencies included in each of the plurality of in-image feature amounts calculated for each of the plurality of partial images into one for each corresponding pixel. A feature extraction program for functioning as a generation unit. A detection step in which the detection unit detects partial images corresponding to each other between the plurality of images;
A pixel feature calculation step in which a pixel feature calculation unit calculates a pixel feature amount for each of the plurality of partial images and for each pixel included in the partial image;
An in-image feature calculation unit calculates, for each partial image, for each pixel included in the partial image, the calculated pixel feature amount and other pixels that are separated from the pixel by a predetermined distance. An intra-image feature calculation step of calculating a co-occurrence frequency with the pixel feature amount and calculating an in-image feature amount including each of the co-occurrence frequencies calculated for each pixel ;
The generation unit represents a feature amount including an element obtained by integrating the co-occurrence frequencies included in each of the plurality of in-image feature amounts calculated for each of the plurality of partial images into one for each corresponding pixel. A generation step for generating image features;
A feature extraction method characterized by comprising :

Images